== sanity test 398c: run fio to test AIO ================= 19:48:47 (1755042527)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.211362 s, 198 MB/s
osc.lustre-OST0000-osc-ffff898b200a3800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=189872: Tue Aug 12 19:49:07 2025
  write: IOPS=174, BW=699KiB/s (716kB/s)(10.0MiB/14640msec); 0 zone resets
    slat (usec): min=26, max=7171, avg=144.87, stdev=269.38
    clat (msec): min=8, max=189, avg=90.88, stdev=26.09
     lat (msec): min=8, max=189, avg=91.02, stdev=26.10
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   56], 10.00th=[   61], 20.00th=[   68],
     | 30.00th=[   73], 40.00th=[   81], 50.00th=[   87], 60.00th=[   95],
     | 70.00th=[  105], 80.00th=[  116], 90.00th=[  129], 95.00th=[  138],
     | 99.00th=[  148], 99.50th=[  163], 99.90th=[  190], 99.95th=[  190],
     | 99.99th=[  190]
   bw (  KiB/s): min=  462, max=  908, per=24.63%, avg=689.00, stdev=141.02, samples=28
   iops        : min=  115, max=  227, avg=171.96, stdev=35.35, samples=28
  lat (msec)   : 10=0.04%, 20=0.08%, 50=2.03%, 100=62.93%, 250=34.92%
  cpu          : usr=0.40%, sys=2.22%, ctx=1464, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189873: Tue Aug 12 19:49:07 2025
  write: IOPS=174, BW=700KiB/s (717kB/s)(10.0MiB/14631msec); 0 zone resets
    slat (usec): min=23, max=25183, avg=164.62, stdev=549.57
    clat (msec): min=6, max=189, avg=90.84, stdev=26.23
     lat (msec): min=6, max=189, avg=91.01, stdev=26.21
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   57], 10.00th=[   61], 20.00th=[   67],
     | 30.00th=[   73], 40.00th=[   81], 50.00th=[   87], 60.00th=[   95],
     | 70.00th=[  106], 80.00th=[  116], 90.00th=[  128], 95.00th=[  138],
     | 99.00th=[  150], 99.50th=[  163], 99.90th=[  190], 99.95th=[  190],
     | 99.99th=[  190]
   bw (  KiB/s): min=  446, max=  908, per=24.66%, avg=689.71, stdev=144.80, samples=28
   iops        : min=  111, max=  227, avg=172.14, stdev=36.27, samples=28
  lat (msec)   : 10=0.08%, 20=0.20%, 50=1.95%, 100=62.11%, 250=35.66%
  cpu          : usr=0.40%, sys=2.26%, ctx=1597, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189874: Tue Aug 12 19:49:07 2025
  write: IOPS=176, BW=704KiB/s (721kB/s)(10.0MiB/14544msec); 0 zone resets
    slat (usec): min=22, max=4189, avg=140.46, stdev=233.98
    clat (msec): min=17, max=190, avg=90.64, stdev=25.75
     lat (msec): min=17, max=191, avg=90.78, stdev=25.76
    clat percentiles (msec):
     |  1.00th=[   48],  5.00th=[   56], 10.00th=[   61], 20.00th=[   67],
     | 30.00th=[   73], 40.00th=[   82], 50.00th=[   87], 60.00th=[   94],
     | 70.00th=[  105], 80.00th=[  115], 90.00th=[  128], 95.00th=[  136],
     | 99.00th=[  150], 99.50th=[  167], 99.90th=[  190], 99.95th=[  190],
     | 99.99th=[  190]
   bw (  KiB/s): min=  430, max=  956, per=24.83%, avg=694.39, stdev=146.93, samples=28
   iops        : min=  107, max=  239, avg=173.32, stdev=36.82, samples=28
  lat (msec)   : 20=0.12%, 50=1.80%, 100=63.63%, 250=34.45%
  cpu          : usr=0.33%, sys=2.01%, ctx=1324, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189875: Tue Aug 12 19:49:07 2025
  write: IOPS=174, BW=699KiB/s (716kB/s)(10.0MiB/14640msec); 0 zone resets
    slat (usec): min=23, max=5345, avg=135.34, stdev=222.81
    clat (msec): min=5, max=189, avg=90.86, stdev=26.04
     lat (msec): min=5, max=189, avg=91.00, stdev=26.05
    clat percentiles (msec):
     |  1.00th=[   45],  5.00th=[   55], 10.00th=[   60], 20.00th=[   67],
     | 30.00th=[   74], 40.00th=[   81], 50.00th=[   87], 60.00th=[   95],
     | 70.00th=[  105], 80.00th=[  116], 90.00th=[  128], 95.00th=[  138],
     | 99.00th=[  153], 99.50th=[  163], 99.90th=[  190], 99.95th=[  190],
     | 99.99th=[  190]
   bw (  KiB/s): min=  454, max=  932, per=24.54%, avg=686.39, stdev=140.71, samples=28
   iops        : min=  113, max=  233, avg=171.29, stdev=35.27, samples=28
  lat (msec)   : 10=0.08%, 50=2.19%, 100=63.87%, 250=33.87%
  cpu          : usr=0.30%, sys=1.96%, ctx=1310, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=2798KiB/s (2865kB/s), 699KiB/s-704KiB/s (716kB/s-721kB/s), io=40.0MiB (41.9MB), run=14544-14640msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=189887: Tue Aug 12 19:49:22 2025
  read: IOPS=104, BW=418KiB/s (428kB/s)(5048KiB/12074msec)
    slat (usec): min=1535, max=19371, avg=5098.17, stdev=2223.83
    clat (msec): min=18, max=164, avg=71.51, stdev=20.41
     lat (msec): min=28, max=174, avg=76.61, stdev=21.32
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   45], 10.00th=[   50], 20.00th=[   55],
     | 30.00th=[   60], 40.00th=[   64], 50.00th=[   68], 60.00th=[   73],
     | 70.00th=[   80], 80.00th=[   87], 90.00th=[   97], 95.00th=[  110],
     | 99.00th=[  142], 99.50th=[  148], 99.90th=[  161], 99.95th=[  165],
     | 99.99th=[  165]
   bw (  KiB/s): min=  247, max=  584, per=25.30%, avg=420.52, stdev=84.16, samples=23
   iops        : min=   61, max=  146, avg=104.91, stdev=21.20, samples=23
  write: IOPS=107, BW=430KiB/s (440kB/s)(5192KiB/12074msec); 0 zone resets
    slat (usec): min=27, max=3505, avg=145.28, stdev=187.18
    clat (msec): min=33, max=174, avg=73.39, stdev=19.62
     lat (msec): min=33, max=174, avg=73.54, stdev=19.63
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   50], 10.00th=[   53], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   66], 50.00th=[   70], 60.00th=[   75],
     | 70.00th=[   81], 80.00th=[   87], 90.00th=[   99], 95.00th=[  109],
     | 99.00th=[  142], 99.50th=[  150], 99.90th=[  157], 99.95th=[  176],
     | 99.99th=[  176]
   bw (  KiB/s): min=  263, max=  624, per=24.87%, avg=430.17, stdev=98.29, samples=23
   iops        : min=   65, max=  156, avg=107.35, stdev=24.63, samples=23
  lat (msec)   : 20=0.04%, 50=8.36%, 100=82.97%, 250=8.63%
  cpu          : usr=0.59%, sys=3.65%, ctx=3006, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189888: Tue Aug 12 19:49:22 2025
  read: IOPS=103, BW=414KiB/s (424kB/s)(4976KiB/12015msec)
    slat (usec): min=47, max=18591, avg=5042.37, stdev=2258.13
    clat (msec): min=7, max=170, avg=70.88, stdev=20.16
     lat (msec): min=7, max=174, avg=75.92, stdev=21.02
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   44], 10.00th=[   49], 20.00th=[   55],
     | 30.00th=[   60], 40.00th=[   65], 50.00th=[   69], 60.00th=[   73],
     | 70.00th=[   79], 80.00th=[   85], 90.00th=[   95], 95.00th=[  107],
     | 99.00th=[  138], 99.50th=[  155], 99.90th=[  167], 99.95th=[  171],
     | 99.99th=[  171]
   bw (  KiB/s): min=  271, max=  560, per=24.99%, avg=415.30, stdev=86.79, samples=23
   iops        : min=   67, max=  140, avg=103.61, stdev=21.80, samples=23
  write: IOPS=109, BW=438KiB/s (449kB/s)(5264KiB/12015msec); 0 zone resets
    slat (usec): min=27, max=6277, avg=146.64, stdev=248.77
    clat (msec): min=14, max=170, avg=73.92, stdev=20.00
     lat (msec): min=14, max=170, avg=74.07, stdev=20.01
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   48], 10.00th=[   53], 20.00th=[   58],
     | 30.00th=[   63], 40.00th=[   67], 50.00th=[   71], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   88], 90.00th=[  100], 95.00th=[  111],
     | 99.00th=[  136], 99.50th=[  146], 99.90th=[  163], 99.95th=[  171],
     | 99.99th=[  171]
   bw (  KiB/s): min=  247, max=  632, per=25.38%, avg=439.00, stdev=96.75, samples=23
   iops        : min=   61, max=  158, avg=109.57, stdev=24.28, samples=23
  lat (msec)   : 10=0.04%, 20=0.08%, 50=9.61%, 100=81.52%, 250=8.75%
  cpu          : usr=0.54%, sys=3.55%, ctx=2916, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189889: Tue Aug 12 19:49:22 2025
  read: IOPS=104, BW=416KiB/s (426kB/s)(4984KiB/11969msec)
    slat (usec): min=1406, max=18479, avg=5018.97, stdev=2174.08
    clat (msec): min=5, max=150, avg=70.17, stdev=18.92
     lat (msec): min=9, max=158, avg=75.19, stdev=19.85
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   45], 10.00th=[   50], 20.00th=[   54],
     | 30.00th=[   59], 40.00th=[   64], 50.00th=[   68], 60.00th=[   72],
     | 70.00th=[   79], 80.00th=[   85], 90.00th=[   95], 95.00th=[  105],
     | 99.00th=[  125], 99.50th=[  132], 99.90th=[  144], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  KiB/s): min=  223, max=  600, per=25.23%, avg=419.26, stdev=101.32, samples=23
   iops        : min=   55, max=  150, avg=104.61, stdev=25.41, samples=23
  write: IOPS=109, BW=439KiB/s (450kB/s)(5256KiB/11969msec); 0 zone resets
    slat (usec): min=30, max=1939, avg=140.29, stdev=169.09
    clat (msec): min=22, max=161, avg=73.50, stdev=19.79
     lat (msec): min=22, max=161, avg=73.64, stdev=19.81
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   50], 10.00th=[   53], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   66], 50.00th=[   70], 60.00th=[   75],
     | 70.00th=[   82], 80.00th=[   88], 90.00th=[   99], 95.00th=[  113],
     | 99.00th=[  138], 99.50th=[  146], 99.90th=[  159], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  255, max=  624, per=25.34%, avg=438.43, stdev=84.57, samples=23
   iops        : min=   63, max=  156, avg=109.39, stdev=21.27, samples=23
  lat (msec)   : 10=0.12%, 50=8.83%, 100=83.09%, 250=7.97%
  cpu          : usr=0.57%, sys=3.62%, ctx=2997, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189890: Tue Aug 12 19:49:22 2025
  read: IOPS=104, BW=419KiB/s (429kB/s)(5060KiB/12073msec)
    slat (usec): min=1603, max=23847, avg=5057.76, stdev=2224.13
    clat (msec): min=30, max=157, avg=71.43, stdev=19.60
     lat (msec): min=34, max=161, avg=76.49, stdev=20.45
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   47], 10.00th=[   50], 20.00th=[   55],
     | 30.00th=[   60], 40.00th=[   64], 50.00th=[   68], 60.00th=[   73],
     | 70.00th=[   80], 80.00th=[   87], 90.00th=[   97], 95.00th=[  108],
     | 99.00th=[  136], 99.50th=[  144], 99.90th=[  155], 99.95th=[  157],
     | 99.99th=[  157]
   bw (  KiB/s): min=  239, max=  576, per=25.28%, avg=420.22, stdev=82.73, samples=23
   iops        : min=   59, max=  144, avg=104.83, stdev=20.83, samples=23
  write: IOPS=107, BW=429KiB/s (439kB/s)(5180KiB/12073msec); 0 zone resets
    slat (usec): min=29, max=4369, avg=141.80, stdev=206.73
    clat (msec): min=14, max=148, avg=73.45, stdev=19.44
     lat (msec): min=14, max=148, avg=73.59, stdev=19.46
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   49], 10.00th=[   53], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   66], 50.00th=[   71], 60.00th=[   75],
     | 70.00th=[   81], 80.00th=[   88], 90.00th=[   99], 95.00th=[  111],
     | 99.00th=[  136], 99.50th=[  144], 99.90th=[  148], 99.95th=[  148],
     | 99.99th=[  148]
   bw (  KiB/s): min=  288, max=  640, per=24.77%, avg=428.48, stdev=97.96, samples=23
   iops        : min=   72, max=  160, avg=106.91, stdev=24.58, samples=23
  lat (msec)   : 20=0.04%, 50=8.44%, 100=82.85%, 250=8.67%
  cpu          : usr=0.59%, sys=3.45%, ctx=3025, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1662KiB/s (1702kB/s), 414KiB/s-419KiB/s (424kB/s-429kB/s), io=19.6MiB (20.5MB), run=11969-12074msec
  WRITE: bw=1730KiB/s (1772kB/s), 429KiB/s-439KiB/s (439kB/s-450kB/s), io=20.4MiB (21.4MB), run=11969-12074msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=189895: Tue Aug 12 19:49:27 2025
  read: IOPS=5, BW=231MiB/s (242MB/s)(40.0MiB/173msec)
    slat (nsec): min=17347k, max=17347k, avg=17346636.00, stdev= 0.00
    clat (nsec): min=146575k, max=146575k, avg=146574973.00, stdev= 0.00
     lat (nsec): min=163929k, max=163929k, avg=163929208.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  146],  5.00th=[  146], 10.00th=[  146], 20.00th=[  146],
     | 30.00th=[  146], 40.00th=[  146], 50.00th=[  146], 60.00th=[  146],
     | 70.00th=[  146], 80.00th=[  146], 90.00th=[  146], 95.00th=[  146],
     | 99.00th=[  146], 99.50th=[  146], 99.90th=[  146], 99.95th=[  146],
     | 99.99th=[  146]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.58%, sys=8.72%, ctx=4, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=40.0MiB (41.9MB), run=173-173msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout