== sanity test 398c: run fio to test AIO ================= 07:35:01 (1781609701)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.119153 s, 352 MB/s
osc.lustre-OST0000-osc-ffff8cc106657000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=185026: Tue Jun 16 07:35:09 2026
  write: IOPS=524, BW=2098KiB/s (2148kB/s)(10.0MiB/4881msec); 0 zone resets
    slat (usec): min=41, max=1011, avg=194.92, stdev=112.95
    clat (usec): min=7520, max=56996, avg=30259.31, stdev=3399.54
     lat (usec): min=7637, max=57159, avg=30455.27, stdev=3401.14
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[24773], 10.00th=[27132], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[30016], 50.00th=[30540], 60.00th=[31065],
     | 70.00th=[31589], 80.00th=[32113], 90.00th=[33162], 95.00th=[34341],
     | 99.00th=[40633], 99.50th=[41681], 99.90th=[51119], 99.95th=[53740],
     | 99.99th=[56886]
   bw (  KiB/s): min= 2024, max= 2339, per=25.14%, avg=2094.56, stdev=96.51, samples=9
   iops        : min=  506, max=  584, avg=523.56, stdev=23.89, samples=9
  lat (msec)   : 10=0.04%, 20=1.25%, 50=98.52%, 100=0.20%
  cpu          : usr=0.37%, sys=7.68%, ctx=2808, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=185027: Tue Jun 16 07:35:09 2026
  write: IOPS=521, BW=2085KiB/s (2135kB/s)(10.0MiB/4911msec); 0 zone resets
    slat (usec): min=59, max=27502, avg=215.04, stdev=550.90
    clat (usec): min=11954, max=54386, avg=30180.69, stdev=3119.90
     lat (usec): min=12043, max=54728, avg=30396.88, stdev=3128.41
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[24773], 10.00th=[27132], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[30016], 50.00th=[30540], 60.00th=[31065],
     | 70.00th=[31327], 80.00th=[31851], 90.00th=[32900], 95.00th=[33817],
     | 99.00th=[37487], 99.50th=[40633], 99.90th=[47973], 99.95th=[48497],
     | 99.99th=[54264]
   bw (  KiB/s): min= 2000, max= 2205, per=25.01%, avg=2083.22, stdev=63.16, samples=9
   iops        : min=  500, max=  551, avg=520.78, stdev=15.73, samples=9
  lat (msec)   : 20=0.94%, 50=99.02%, 100=0.04%
  cpu          : usr=0.49%, sys=8.51%, ctx=2855, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=185028: Tue Jun 16 07:35:09 2026
  write: IOPS=520, BW=2083KiB/s (2133kB/s)(10.0MiB/4915msec); 0 zone resets
    slat (usec): min=40, max=876, avg=201.34, stdev=110.56
    clat (usec): min=3853, max=45442, avg=30150.51, stdev=2831.00
     lat (usec): min=3971, max=45568, avg=30352.91, stdev=2845.04
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[25297], 10.00th=[27132], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[29754], 50.00th=[30540], 60.00th=[30802],
     | 70.00th=[31327], 80.00th=[31851], 90.00th=[32900], 95.00th=[33817],
     | 99.00th=[35914], 99.50th=[39584], 99.90th=[42730], 99.95th=[45351],
     | 99.99th=[45351]
   bw (  KiB/s): min= 2008, max= 2153, per=24.94%, avg=2077.44, stdev=44.39, samples=9
   iops        : min=  502, max=  538, avg=519.33, stdev=11.05, samples=9
  lat (msec)   : 4=0.04%, 20=0.66%, 50=99.30%
  cpu          : usr=0.51%, sys=7.86%, ctx=2760, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=185029: Tue Jun 16 07:35:09 2026
  write: IOPS=520, BW=2083KiB/s (2133kB/s)(10.0MiB/4917msec); 0 zone resets
    slat (usec): min=49, max=1030, avg=196.63, stdev=113.26
    clat (usec): min=9787, max=44001, avg=30198.38, stdev=2923.33
     lat (usec): min=10024, max=44149, avg=30396.20, stdev=2923.45
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[24773], 10.00th=[27395], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[30016], 50.00th=[30540], 60.00th=[31065],
     | 70.00th=[31589], 80.00th=[32113], 90.00th=[32900], 95.00th=[33817],
     | 99.00th=[35914], 99.50th=[40109], 99.90th=[43779], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 1992, max= 2185, per=24.96%, avg=2079.22, stdev=59.34, samples=9
   iops        : min=  498, max=  546, avg=519.78, stdev=14.78, samples=9
  lat (msec)   : 10=0.04%, 20=0.94%, 50=99.02%
  cpu          : usr=0.39%, sys=8.22%, ctx=2934, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8330KiB/s (8530kB/s), 2083KiB/s-2098KiB/s (2133kB/s-2148kB/s), io=40.0MiB (41.9MB), run=4881-4917msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=185039: Tue Jun 16 07:35:19 2026
  read: IOPS=194, BW=777KiB/s (795kB/s)(5048KiB/6500msec)
    slat (usec): min=61, max=7726, avg=2779.98, stdev=918.73
    clat (usec): min=5317, max=70326, avg=38789.34, stdev=7310.00
     lat (usec): min=5452, max=73368, avg=41570.69, stdev=7646.95
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[29230], 10.00th=[31065], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[41157], 80.00th=[43779], 90.00th=[48497], 95.00th=[51643],
     | 99.00th=[59507], 99.50th=[61604], 99.90th=[67634], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  600, max=  888, per=25.29%, avg=775.00, stdev=96.40, samples=12
   iops        : min=  150, max=  222, avg=193.75, stdev=24.10, samples=12
  write: IOPS=199, BW=799KiB/s (818kB/s)(5192KiB/6500msec); 0 zone resets
    slat (usec): min=61, max=635, avg=166.97, stdev=81.78
    clat (usec): min=17777, max=67257, avg=39395.49, stdev=6446.29
     lat (usec): min=18072, max=67425, avg=39563.53, stdev=6448.35
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31327], 10.00th=[32637], 20.00th=[34341],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[44303], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[55837], 99.50th=[56886], 99.90th=[64750], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  568, max= 1008, per=25.05%, avg=799.50, stdev=111.75, samples=12
   iops        : min=  142, max=  252, avg=199.83, stdev=27.93, samples=12
  lat (msec)   : 10=0.16%, 20=0.39%, 50=91.25%, 100=8.20%
  cpu          : usr=0.43%, sys=6.00%, ctx=4805, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=185040: Tue Jun 16 07:35:19 2026
  read: IOPS=190, BW=760KiB/s (778kB/s)(4976KiB/6546msec)
    slat (usec): min=1114, max=7410, avg=2802.40, stdev=880.69
    clat (usec): min=13441, max=64131, avg=38907.56, stdev=6531.26
     lat (usec): min=15229, max=68702, avg=41711.34, stdev=6806.17
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29492], 10.00th=[31327], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[47973], 95.00th=[51119],
     | 99.00th=[55837], 99.50th=[59507], 99.90th=[62129], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  608, max=  888, per=24.58%, avg=753.50, stdev=87.84, samples=12
   iops        : min=  152, max=  222, avg=188.33, stdev=21.99, samples=12
  write: IOPS=201, BW=804KiB/s (823kB/s)(5264KiB/6546msec); 0 zone resets
    slat (usec): min=53, max=640, avg=175.96, stdev=82.23
    clat (usec): min=24346, max=60998, avg=39439.20, stdev=6160.20
     lat (usec): min=24578, max=61386, avg=39616.34, stdev=6164.48
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[31327], 10.00th=[32637], 20.00th=[34341],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[39584],
     | 70.00th=[41681], 80.00th=[44303], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[55837], 99.50th=[58459], 99.90th=[60556], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  664, max=  928, per=25.40%, avg=810.58, stdev=92.19, samples=12
   iops        : min=  166, max=  232, avg=202.58, stdev=23.05, samples=12
  lat (msec)   : 20=0.08%, 50=93.20%, 100=6.72%
  cpu          : usr=0.37%, sys=6.05%, ctx=4753, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=185041: Tue Jun 16 07:35:19 2026
  read: IOPS=190, BW=762KiB/s (780kB/s)(4984KiB/6541msec)
    slat (usec): min=1159, max=8098, avg=2785.02, stdev=902.96
    clat (usec): min=23482, max=63522, avg=39077.37, stdev=6760.48
     lat (usec): min=25195, max=66046, avg=41863.81, stdev=7006.38
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29754], 10.00th=[31327], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[43779], 90.00th=[49546], 95.00th=[52691],
     | 99.00th=[56886], 99.50th=[58983], 99.90th=[62129], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  520, max=  920, per=24.95%, avg=764.83, stdev=123.10, samples=12
   iops        : min=  130, max=  230, avg=191.17, stdev=30.78, samples=12
  write: IOPS=200, BW=804KiB/s (823kB/s)(5256KiB/6541msec); 0 zone resets
    slat (usec): min=54, max=3044, avg=178.91, stdev=138.21
    clat (usec): min=24571, max=63536, avg=39379.44, stdev=6171.54
     lat (usec): min=24742, max=63654, avg=39559.45, stdev=6173.64
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[31327], 10.00th=[32900], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[36963], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[44303], 90.00th=[48497], 95.00th=[51643],
     | 99.00th=[55837], 99.50th=[61080], 99.90th=[63701], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  696, max=  936, per=24.99%, avg=797.50, stdev=72.86, samples=12
   iops        : min=  174, max=  234, avg=199.33, stdev=18.22, samples=12
  lat (msec)   : 50=91.88%, 100=8.12%
  cpu          : usr=0.47%, sys=5.89%, ctx=4867, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=185042: Tue Jun 16 07:35:19 2026
  read: IOPS=193, BW=773KiB/s (792kB/s)(5060KiB/6546msec)
    slat (usec): min=1194, max=6703, avg=2811.19, stdev=849.98
    clat (usec): min=3947, max=59844, avg=39028.13, stdev=6858.30
     lat (usec): min=7043, max=63758, avg=41840.81, stdev=7077.29
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[30016], 10.00th=[31589], 20.00th=[33817],
     | 30.00th=[34866], 40.00th=[36963], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[44303], 90.00th=[49546], 95.00th=[52691],
     | 99.00th=[57410], 99.50th=[58459], 99.90th=[59507], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  616, max=  864, per=25.14%, avg=770.50, stdev=81.01, samples=12
   iops        : min=  154, max=  216, avg=192.58, stdev=20.26, samples=12
  write: IOPS=197, BW=791KiB/s (810kB/s)(5180KiB/6546msec); 0 zone resets
    slat (usec): min=64, max=695, avg=175.78, stdev=81.57
    clat (usec): min=12314, max=65792, avg=39372.60, stdev=6208.92
     lat (usec): min=12500, max=65960, avg=39549.46, stdev=6208.19
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[31327], 10.00th=[32900], 20.00th=[34866],
     | 30.00th=[35914], 40.00th=[36963], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[43779], 90.00th=[48497], 95.00th=[52167],
     | 99.00th=[56886], 99.50th=[60031], 99.90th=[65274], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  648, max=  920, per=24.71%, avg=788.50, stdev=77.54, samples=12
   iops        : min=  162, max=  230, avg=197.08, stdev=19.39, samples=12
  lat (msec)   : 4=0.04%, 20=0.12%, 50=91.33%, 100=8.52%
  cpu          : usr=0.41%, sys=6.11%, ctx=4808, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3066KiB/s (3139kB/s), 760KiB/s-777KiB/s (778kB/s-795kB/s), io=19.6MiB (20.5MB), run=6500-6546msec
  WRITE: bw=3192KiB/s (3268kB/s), 791KiB/s-804KiB/s (810kB/s-823kB/s), io=20.4MiB (21.4MB), run=6500-6546msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=185047: Tue Jun 16 07:35:22 2026
  read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec)
    slat (nsec): min=9895.8k, max=9895.8k, avg=9895803.00, stdev= 0.00
    clat (nsec): min=94547k, max=94547k, avg=94546819.00, stdev= 0.00
     lat (nsec): min=104499k, max=104499k, avg=104498795.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[94897],  5.00th=[94897], 10.00th=[94897], 20.00th=[94897],
     | 30.00th=[94897], 40.00th=[94897], 50.00th=[94897], 60.00th=[94897],
     | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897],
     | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897],
     | 99.99th=[94897]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.17%, ctx=6, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout