== sanity test 398c: run fio to test AIO ================= 20:43:16 (1786840996)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.125641 s, 334 MB/s
osc.lustre-OST0000-osc-ffff934d05b9c800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=189675: Sat Aug 15 20:43:26 2026
  write: IOPS=424, BW=1699KiB/s (1740kB/s)(10.0MiB/6026msec); 0 zone resets
    slat (usec): min=50, max=28684, avg=224.87, stdev=578.12
    clat (usec): min=6484, max=68161, avg=37118.56, stdev=9893.50
     lat (usec): min=6615, max=68512, avg=37344.58, stdev=9929.03
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[25297], 10.00th=[28443], 20.00th=[31327],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[40109], 90.00th=[55313], 95.00th=[61080],
     | 99.00th=[66323], 99.50th=[66847], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1048, max= 2057, per=24.72%, avg=1680.09, stdev=344.80, samples=11
   iops        : min=  262, max=  514, avg=420.00, stdev=86.17, samples=11
  lat (msec)   : 10=0.12%, 20=0.74%, 50=85.94%, 100=13.20%
  cpu          : usr=0.48%, sys=7.17%, ctx=2835, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189676: Sat Aug 15 20:43:26 2026
  write: IOPS=426, BW=1706KiB/s (1747kB/s)(10.0MiB/6003msec); 0 zone resets
    slat (usec): min=41, max=1221, avg=211.47, stdev=128.90
    clat (usec): min=8854, max=70806, avg=37248.22, stdev=9980.32
     lat (usec): min=9035, max=70894, avg=37461.11, stdev=10026.93
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[25035], 10.00th=[28181], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[40633], 90.00th=[55837], 95.00th=[61604],
     | 99.00th=[65799], 99.50th=[66323], 99.90th=[68682], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min= 1040, max= 2159, per=24.81%, avg=1686.45, stdev=360.61, samples=11
   iops        : min=  260, max=  539, avg=421.55, stdev=90.05, samples=11
  lat (msec)   : 10=0.04%, 20=0.86%, 50=85.86%, 100=13.24%
  cpu          : usr=0.53%, sys=6.51%, ctx=2775, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189677: Sat Aug 15 20:43:26 2026
  write: IOPS=424, BW=1699KiB/s (1740kB/s)(10.0MiB/6026msec); 0 zone resets
    slat (usec): min=48, max=1088, avg=212.48, stdev=132.27
    clat (usec): min=4610, max=68497, avg=37102.73, stdev=9823.89
     lat (usec): min=4765, max=68704, avg=37316.38, stdev=9875.89
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[25297], 10.00th=[28443], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[40109], 90.00th=[55837], 95.00th=[61080],
     | 99.00th=[65799], 99.50th=[65799], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1048, max= 2019, per=24.66%, avg=1675.91, stdev=343.30, samples=11
   iops        : min=  262, max=  504, avg=418.91, stdev=85.75, samples=11
  lat (msec)   : 10=0.04%, 20=0.39%, 50=86.52%, 100=13.05%
  cpu          : usr=0.43%, sys=7.04%, ctx=2882, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189678: Sat Aug 15 20:43:26 2026
  write: IOPS=424, BW=1699KiB/s (1740kB/s)(10.0MiB/6026msec); 0 zone resets
    slat (usec): min=46, max=31348, avg=229.75, stdev=631.31
    clat (usec): min=11599, max=68914, avg=37116.77, stdev=9816.25
     lat (usec): min=15669, max=69121, avg=37347.87, stdev=9866.38
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[25297], 10.00th=[28443], 20.00th=[31327],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[40109], 90.00th=[55313], 95.00th=[61080],
     | 99.00th=[65799], 99.50th=[66847], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1032, max= 2044, per=24.64%, avg=1674.55, stdev=343.70, samples=11
   iops        : min=  258, max=  511, avg=418.64, stdev=85.93, samples=11
  lat (msec)   : 20=0.86%, 50=86.02%, 100=13.12%
  cpu          : usr=0.53%, sys=7.17%, ctx=2798, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6797KiB/s (6960kB/s), 1699KiB/s-1706KiB/s (1740kB/s-1747kB/s), io=40.0MiB (41.9MB), run=6003-6026msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=189688: Sat Aug 15 20:43:35 2026
  read: IOPS=193, BW=773KiB/s (791kB/s)(5048KiB/6532msec)
    slat (usec): min=1137, max=8257, avg=2803.64, stdev=936.89
    clat (usec): min=7282, max=59807, avg=38505.67, stdev=6977.60
     lat (usec): min=10200, max=63570, avg=41310.71, stdev=7315.65
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28443], 10.00th=[30278], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[58459], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  572, max=  928, per=24.84%, avg=763.00, stdev=127.37, samples=12
   iops        : min=  143, max=  232, avg=190.75, stdev=31.84, samples=12
  write: IOPS=198, BW=795KiB/s (814kB/s)(5192KiB/6532msec); 0 zone resets
    slat (usec): min=65, max=2073, avg=170.45, stdev=98.51
    clat (usec): min=15075, max=58064, avg=39187.74, stdev=6663.10
     lat (usec): min=15224, max=58211, avg=39359.26, stdev=6664.50
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30278], 10.00th=[31851], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[40109],
     | 70.00th=[42730], 80.00th=[45876], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[54264], 99.50th=[55313], 99.90th=[57934], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  585, max=  960, per=24.77%, avg=792.08, stdev=116.57, samples=12
   iops        : min=  146, max=  240, avg=198.00, stdev=29.18, samples=12
  lat (msec)   : 10=0.04%, 20=0.20%, 50=92.93%, 100=6.84%
  cpu          : usr=0.40%, sys=5.86%, ctx=4854, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189689: Sat Aug 15 20:43:35 2026
  read: IOPS=191, BW=766KiB/s (784kB/s)(4976KiB/6498msec)
    slat (usec): min=44, max=8514, avg=2811.53, stdev=975.47
    clat (usec): min=7576, max=61926, avg=38518.01, stdev=7080.74
     lat (usec): min=7621, max=65024, avg=41330.93, stdev=7435.31
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28705], 10.00th=[30278], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[41681], 80.00th=[44827], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[57410], 99.90th=[61604], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  576, max=  952, per=24.66%, avg=757.42, stdev=129.41, samples=12
   iops        : min=  144, max=  238, avg=189.33, stdev=32.38, samples=12
  write: IOPS=202, BW=810KiB/s (830kB/s)(5264KiB/6498msec); 0 zone resets
    slat (usec): min=42, max=3994, avg=166.78, stdev=132.82
    clat (usec): min=21181, max=60256, avg=39622.71, stdev=6591.39
     lat (usec): min=21303, max=60397, avg=39790.62, stdev=6593.27
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[30802], 10.00th=[32113], 20.00th=[33817],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[38536], 60.00th=[40633],
     | 70.00th=[43254], 80.00th=[46400], 90.00th=[49546], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  616, max= 1040, per=25.46%, avg=814.08, stdev=110.41, samples=12
   iops        : min=  154, max=  260, avg=203.50, stdev=27.62, samples=12
  lat (msec)   : 10=0.08%, 20=0.12%, 50=91.91%, 100=7.89%
  cpu          : usr=0.45%, sys=5.83%, ctx=4747, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189690: Sat Aug 15 20:43:35 2026
  read: IOPS=191, BW=765KiB/s (783kB/s)(4984KiB/6518msec)
    slat (usec): min=1039, max=7854, avg=2851.54, stdev=965.75
    clat (usec): min=11461, max=60014, avg=38795.50, stdev=7165.77
     lat (usec): min=13798, max=63482, avg=41648.53, stdev=7485.97
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28705], 10.00th=[30278], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[45876], 90.00th=[49546], 95.00th=[51119],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  553, max=  912, per=24.85%, avg=763.42, stdev=118.91, samples=12
   iops        : min=  138, max=  228, avg=190.83, stdev=29.77, samples=12
  write: IOPS=201, BW=806KiB/s (826kB/s)(5256KiB/6518msec); 0 zone resets
    slat (usec): min=62, max=1842, avg=170.02, stdev=97.51
    clat (usec): min=21178, max=57608, avg=38985.53, stdev=6386.66
     lat (usec): min=21309, max=57702, avg=39156.65, stdev=6387.80
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30540], 10.00th=[31851], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[43254], 80.00th=[45351], 90.00th=[47973], 95.00th=[50594],
     | 99.00th=[54264], 99.50th=[55313], 99.90th=[56886], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  624, max= 1016, per=24.97%, avg=798.58, stdev=144.14, samples=12
   iops        : min=  156, max=  254, avg=199.58, stdev=36.11, samples=12
  lat (msec)   : 20=0.04%, 50=93.09%, 100=6.88%
  cpu          : usr=0.32%, sys=5.98%, ctx=4726, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189691: Sat Aug 15 20:43:35 2026
  read: IOPS=193, BW=775KiB/s (794kB/s)(5060KiB/6528msec)
    slat (usec): min=1123, max=8342, avg=2849.05, stdev=997.78
    clat (usec): min=19693, max=60453, avg=38618.02, stdev=6959.76
     lat (usec): min=21786, max=64480, avg=41468.59, stdev=7285.32
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28705], 10.00th=[30540], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[41681], 80.00th=[44827], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[57410], 99.90th=[59507], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  592, max=  912, per=25.07%, avg=770.00, stdev=115.28, samples=12
   iops        : min=  148, max=  228, avg=192.50, stdev=28.82, samples=12
  write: IOPS=198, BW=794KiB/s (813kB/s)(5180KiB/6528msec); 0 zone resets
    slat (usec): min=56, max=1738, avg=165.17, stdev=85.18
    clat (usec): min=21791, max=57882, avg=39421.90, stdev=6464.80
     lat (usec): min=21858, max=57964, avg=39588.15, stdev=6462.56
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30540], 10.00th=[31851], 20.00th=[33817],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[38011], 60.00th=[40109],
     | 70.00th=[43254], 80.00th=[45876], 90.00th=[49021], 95.00th=[50594],
     | 99.00th=[54789], 99.50th=[55837], 99.90th=[57410], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  618, max=  912, per=24.69%, avg=789.50, stdev=110.74, samples=12
   iops        : min=  154, max=  228, avg=197.33, stdev=27.76, samples=12
  lat (msec)   : 20=0.04%, 50=93.32%, 100=6.64%
  cpu          : usr=0.43%, sys=5.91%, ctx=4689, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3072KiB/s (3146kB/s), 765KiB/s-775KiB/s (783kB/s-794kB/s), io=19.6MiB (20.5MB), run=6498-6532msec
  WRITE: bw=3198KiB/s (3275kB/s), 794KiB/s-810KiB/s (813kB/s-830kB/s), io=20.4MiB (21.4MB), run=6498-6532msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=189696: Sat Aug 15 20:43:39 2026
  read: IOPS=10, BW=435MiB/s (456MB/s)(40.0MiB/92msec)
    slat (nsec): min=10606k, max=10606k, avg=10605849.00, stdev= 0.00
    clat (nsec): min=76783k, max=76783k, avg=76782549.00, stdev= 0.00
     lat (nsec): min=87391k, max=87391k, avg=87391003.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[77071],  5.00th=[77071], 10.00th=[77071], 20.00th=[77071],
     | 30.00th=[77071], 40.00th=[77071], 50.00th=[77071], 60.00th=[77071],
     | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071],
     | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[77071]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.99%, ctx=11, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=435MiB/s (456MB/s), 435MiB/s-435MiB/s (456MB/s-456MB/s), io=40.0MiB (41.9MB), run=92-92msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout