== sanity test 398c: run fio to test AIO ================= 13:32:55 (1772476375)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.119461 s, 351 MB/s
osc.lustre-OST0000-osc-ffff99e412446800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183277: Mon Mar  2 13:33:03 2026
  write: IOPS=558, BW=2233KiB/s (2286kB/s)(10.0MiB/4586msec); 0 zone resets
    slat (usec): min=35, max=22856, avg=164.66, stdev=457.68
    clat (usec): min=7674, max=48914, avg=28377.72, stdev=2529.83
     lat (usec): min=15208, max=49059, avg=28543.39, stdev=2499.95
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[24511], 10.00th=[25560], 20.00th=[26608],
     | 30.00th=[27395], 40.00th=[27919], 50.00th=[28443], 60.00th=[28967],
     | 70.00th=[29230], 80.00th=[30016], 90.00th=[31065], 95.00th=[32375],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[45876], 99.95th=[46400],
     | 99.99th=[49021]
   bw (  KiB/s): min= 2160, max= 2304, per=25.05%, avg=2230.62, stdev=53.53, samples=8
   iops        : min=  540, max=  576, avg=557.62, stdev=13.37, samples=8
  lat (msec)   : 10=0.04%, 20=0.08%, 50=99.88%
  cpu          : usr=0.52%, sys=7.02%, ctx=2751, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183278: Mon Mar  2 13:33:03 2026
  write: IOPS=556, BW=2227KiB/s (2280kB/s)(10.0MiB/4599msec); 0 zone resets
    slat (usec): min=34, max=613, avg=149.44, stdev=83.85
    clat (usec): min=6241, max=35541, avg=28296.44, stdev=2371.69
     lat (usec): min=6344, max=35714, avg=28446.98, stdev=2374.73
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[24511], 10.00th=[25297], 20.00th=[26608],
     | 30.00th=[27132], 40.00th=[27919], 50.00th=[28443], 60.00th=[28705],
     | 70.00th=[29230], 80.00th=[30016], 90.00th=[31065], 95.00th=[32113],
     | 99.00th=[34341], 99.50th=[35390], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 2136, max= 2312, per=24.90%, avg=2217.88, stdev=64.84, samples=8
   iops        : min=  534, max=  578, avg=554.38, stdev=16.34, samples=8
  lat (msec)   : 10=0.04%, 20=0.23%, 50=99.73%
  cpu          : usr=0.61%, sys=6.29%, ctx=2509, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183279: Mon Mar  2 13:33:03 2026
  write: IOPS=559, BW=2240KiB/s (2293kB/s)(10.0MiB/4572msec); 0 zone resets
    slat (usec): min=34, max=620, avg=152.77, stdev=85.74
    clat (usec): min=5820, max=52462, avg=28373.18, stdev=2717.80
     lat (usec): min=5916, max=52649, avg=28527.33, stdev=2720.16
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[24511], 10.00th=[25560], 20.00th=[26608],
     | 30.00th=[27395], 40.00th=[27919], 50.00th=[28443], 60.00th=[28967],
     | 70.00th=[29492], 80.00th=[30016], 90.00th=[31327], 95.00th=[32113],
     | 99.00th=[34341], 99.50th=[35914], 99.90th=[49021], 99.95th=[49546],
     | 99.99th=[52691]
   bw (  KiB/s): min= 2120, max= 2296, per=25.16%, avg=2240.87, stdev=63.13, samples=8
   iops        : min=  530, max=  574, avg=560.12, stdev=15.73, samples=8
  lat (msec)   : 10=0.12%, 20=0.08%, 50=99.77%, 100=0.04%
  cpu          : usr=0.39%, sys=6.58%, ctx=2670, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183280: Mon Mar  2 13:33:03 2026
  write: IOPS=557, BW=2228KiB/s (2281kB/s)(10.0MiB/4596msec); 0 zone resets
    slat (usec): min=35, max=1430, avg=148.67, stdev=85.68
    clat (usec): min=5943, max=41599, avg=28312.51, stdev=2347.37
     lat (usec): min=6321, max=41733, avg=28462.24, stdev=2352.31
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[24249], 10.00th=[25560], 20.00th=[26608],
     | 30.00th=[27395], 40.00th=[27919], 50.00th=[28443], 60.00th=[28705],
     | 70.00th=[29230], 80.00th=[30016], 90.00th=[31065], 95.00th=[32113],
     | 99.00th=[33817], 99.50th=[34341], 99.90th=[41157], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 2104, max= 2304, per=24.96%, avg=2223.12, stdev=71.24, samples=8
   iops        : min=  526, max=  576, avg=555.75, stdev=17.84, samples=8
  lat (msec)   : 10=0.04%, 20=0.16%, 50=99.80%
  cpu          : usr=0.57%, sys=6.64%, ctx=2645, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8906KiB/s (9120kB/s), 2227KiB/s-2240KiB/s (2280kB/s-2293kB/s), io=40.0MiB (41.9MB), run=4572-4599msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183291: Mon Mar  2 13:33:11 2026
  read: IOPS=225, BW=902KiB/s (924kB/s)(5048KiB/5597msec)
    slat (usec): min=950, max=14245, avg=2365.15, stdev=780.92
    clat (usec): min=8332, max=53319, avg=33260.45, stdev=5312.60
     lat (usec): min=10659, max=57126, avg=35627.00, stdev=5477.42
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[25560], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30540], 40.00th=[31851], 50.00th=[32900], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[37487], 90.00th=[39584], 95.00th=[43254],
     | 99.00th=[47973], 99.50th=[49546], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  720, max= 1008, per=25.02%, avg=897.00, stdev=102.46, samples=10
   iops        : min=  180, max=  252, avg=224.20, stdev=25.64, samples=10
  write: IOPS=231, BW=928KiB/s (950kB/s)(5192KiB/5597msec); 0 zone resets
    slat (usec): min=38, max=672, avg=149.18, stdev=74.53
    clat (usec): min=13877, max=58460, avg=33578.02, stdev=5068.88
     lat (usec): min=14022, max=58564, avg=33728.10, stdev=5066.40
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27132], 10.00th=[28705], 20.00th=[29754],
     | 30.00th=[30802], 40.00th=[31589], 50.00th=[32637], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[40109], 95.00th=[44303],
     | 99.00th=[50594], 99.50th=[53216], 99.90th=[55837], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  776, max= 1080, per=24.98%, avg=932.20, stdev=104.37, samples=10
   iops        : min=  194, max=  270, avg=233.00, stdev=26.17, samples=10
  lat (msec)   : 10=0.04%, 20=0.20%, 50=98.98%, 100=0.78%
  cpu          : usr=0.38%, sys=6.22%, ctx=4814, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183292: Mon Mar  2 13:33:11 2026
  read: IOPS=222, BW=891KiB/s (913kB/s)(4976KiB/5583msec)
    slat (usec): min=932, max=16759, avg=2346.09, stdev=829.17
    clat (usec): min=11681, max=53862, avg=33521.44, stdev=5473.37
     lat (usec): min=13837, max=57374, avg=35869.06, stdev=5680.95
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25297], 10.00th=[26870], 20.00th=[29230],
     | 30.00th=[30802], 40.00th=[31851], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[40633], 95.00th=[43779],
     | 99.00th=[50070], 99.50th=[51643], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  736, max=  968, per=24.79%, avg=888.70, stdev=75.16, samples=10
   iops        : min=  184, max=  242, avg=222.10, stdev=18.92, samples=10
  write: IOPS=235, BW=943KiB/s (965kB/s)(5264KiB/5583msec); 0 zone resets
    slat (usec): min=46, max=801, avg=149.35, stdev=72.77
    clat (usec): min=16209, max=55234, avg=33379.69, stdev=4718.38
     lat (usec): min=16298, max=55391, avg=33530.13, stdev=4718.40
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[27395], 10.00th=[28443], 20.00th=[29492],
     | 30.00th=[30802], 40.00th=[31851], 50.00th=[32637], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[42206],
     | 99.00th=[49546], 99.50th=[52167], 99.90th=[54789], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  784, max= 1048, per=25.43%, avg=949.10, stdev=96.33, samples=10
   iops        : min=  196, max=  262, avg=237.20, stdev=24.08, samples=10
  lat (msec)   : 20=0.12%, 50=98.95%, 100=0.94%
  cpu          : usr=0.30%, sys=6.25%, ctx=4750, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183293: Mon Mar  2 13:33:11 2026
  read: IOPS=222, BW=891KiB/s (913kB/s)(4984KiB/5591msec)
    slat (usec): min=1019, max=13614, avg=2332.26, stdev=773.96
    clat (usec): min=17262, max=59128, avg=33382.57, stdev=5283.96
     lat (usec): min=19758, max=61151, avg=35716.16, stdev=5454.88
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25560], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30540], 40.00th=[31851], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39584], 95.00th=[42206],
     | 99.00th=[49021], 99.50th=[51119], 99.90th=[55837], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  688, max= 1032, per=24.93%, avg=893.80, stdev=97.54, samples=10
   iops        : min=  172, max=  258, avg=223.40, stdev=24.40, samples=10
  write: IOPS=235, BW=940KiB/s (963kB/s)(5256KiB/5591msec); 0 zone resets
    slat (usec): min=53, max=671, avg=153.59, stdev=79.47
    clat (usec): min=16798, max=57494, avg=33545.66, stdev=5072.38
     lat (usec): min=16904, max=57601, avg=33700.21, stdev=5071.52
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27132], 10.00th=[28443], 20.00th=[29754],
     | 30.00th=[31065], 40.00th=[31851], 50.00th=[32637], 60.00th=[33424],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39584], 95.00th=[44303],
     | 99.00th=[51119], 99.50th=[54264], 99.90th=[55837], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  816, max= 1088, per=25.23%, avg=941.70, stdev=102.34, samples=10
   iops        : min=  204, max=  272, avg=235.40, stdev=25.61, samples=10
  lat (msec)   : 20=0.16%, 50=98.91%, 100=0.94%
  cpu          : usr=0.47%, sys=6.12%, ctx=4782, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183294: Mon Mar  2 13:33:11 2026
  read: IOPS=227, BW=910KiB/s (931kB/s)(5060KiB/5563msec)
    slat (usec): min=54, max=13395, avg=2348.75, stdev=857.18
    clat (usec): min=3659, max=55602, avg=33409.93, stdev=5777.08
     lat (usec): min=4709, max=57505, avg=35760.21, stdev=5984.78
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[25297], 10.00th=[26870], 20.00th=[29230],
     | 30.00th=[30540], 40.00th=[31851], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[35390], 80.00th=[37487], 90.00th=[40109], 95.00th=[43779],
     | 99.00th=[50070], 99.50th=[52167], 99.90th=[54789], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min=  760, max=  976, per=25.40%, avg=910.70, stdev=67.71, samples=10
   iops        : min=  190, max=  244, avg=227.60, stdev=16.91, samples=10
  write: IOPS=232, BW=931KiB/s (953kB/s)(5180KiB/5563msec); 0 zone resets
    slat (usec): min=49, max=1353, avg=147.85, stdev=77.61
    clat (usec): min=8525, max=63786, avg=33516.21, stdev=5573.04
     lat (usec): min=8644, max=63911, avg=33665.16, stdev=5575.31
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[26608], 10.00th=[27919], 20.00th=[29492],
     | 30.00th=[30540], 40.00th=[31589], 50.00th=[32375], 60.00th=[33424],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[40633], 95.00th=[44827],
     | 99.00th=[54264], 99.50th=[55837], 99.90th=[63701], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  736, max= 1104, per=25.00%, avg=933.10, stdev=106.81, samples=10
   iops        : min=  184, max=  276, avg=233.20, stdev=26.74, samples=10
  lat (msec)   : 4=0.04%, 10=0.20%, 20=0.16%, 50=98.05%, 100=1.56%
  cpu          : usr=0.38%, sys=6.18%, ctx=4796, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3585KiB/s (3672kB/s), 891KiB/s-910KiB/s (913kB/s-931kB/s), io=19.6MiB (20.5MB), run=5563-5597msec
  WRITE: bw=3733KiB/s (3822kB/s), 928KiB/s-943KiB/s (950kB/s-965kB/s), io=20.4MiB (21.4MB), run=5563-5597msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183299: Mon Mar  2 13:33:14 2026
  read: IOPS=8, BW=357MiB/s (374MB/s)(40.0MiB/112msec)
    slat (nsec): min=9770.9k, max=9770.9k, avg=9770868.00, stdev= 0.00
    clat (nsec): min=97126k, max=97126k, avg=97126337.00, stdev= 0.00
     lat (nsec): min=106918k, max=106918k, avg=106918014.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[96994],  5.00th=[96994], 10.00th=[96994], 20.00th=[96994],
     | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994],
     | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994],
     | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994],
     | 99.99th=[96994]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.90%, sys=6.31%, ctx=12, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=357MiB/s (374MB/s), 357MiB/s-357MiB/s (374MB/s-374MB/s), io=40.0MiB (41.9MB), run=112-112msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout