== sanity test 398c: run fio to test AIO ================= 10:15:36 (1773238536)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.129811 s, 323 MB/s
osc.lustre-OST0000-osc-ffff9d9108794000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=184587: Wed Mar 11 10:15:44 2026
  write: IOPS=495, BW=1983KiB/s (2030kB/s)(10.0MiB/5165msec); 0 zone resets
    slat (usec): min=23, max=25437, avg=112.30, stdev=505.14
    clat (usec): min=11634, max=62862, avg=31910.62, stdev=2933.59
     lat (usec): min=11718, max=62944, avg=32024.04, stdev=2923.46
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[27657], 10.00th=[28705], 20.00th=[30016],
     | 30.00th=[30540], 40.00th=[31327], 50.00th=[31851], 60.00th=[32375],
     | 70.00th=[33162], 80.00th=[33817], 90.00th=[34866], 95.00th=[35914],
     | 99.00th=[40633], 99.50th=[41681], 99.90th=[48497], 99.95th=[52691],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1826, max= 2096, per=25.17%, avg=1988.20, stdev=78.58, samples=10
   iops        : min=  456, max=  524, avg=497.00, stdev=19.76, samples=10
  lat (msec)   : 20=0.20%, 50=99.73%, 100=0.08%
  cpu          : usr=0.64%, sys=4.53%, ctx=2719, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184588: Wed Mar 11 10:15:44 2026
  write: IOPS=497, BW=1988KiB/s (2036kB/s)(10.0MiB/5150msec); 0 zone resets
    slat (usec): min=24, max=503, avg=100.66, stdev=50.56
    clat (usec): min=11474, max=61216, avg=32031.40, stdev=3029.48
     lat (usec): min=11632, max=61311, avg=32133.27, stdev=3027.15
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[27919], 10.00th=[28705], 20.00th=[29754],
     | 30.00th=[30540], 40.00th=[31327], 50.00th=[31851], 60.00th=[32375],
     | 70.00th=[33162], 80.00th=[33817], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[40633], 99.50th=[42206], 99.90th=[57934], 99.95th=[57934],
     | 99.99th=[61080]
   bw (  KiB/s): min= 1871, max= 2104, per=25.20%, avg=1990.30, stdev=71.42, samples=10
   iops        : min=  467, max=  526, avg=497.50, stdev=18.00, samples=10
  lat (msec)   : 20=0.04%, 50=99.65%, 100=0.31%
  cpu          : usr=0.60%, sys=4.06%, ctx=2637, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184589: Wed Mar 11 10:15:44 2026
  write: IOPS=493, BW=1976KiB/s (2023kB/s)(10.0MiB/5183msec); 0 zone resets
    slat (usec): min=24, max=376, avg=97.70, stdev=48.15
    clat (usec): min=7210, max=54092, avg=31916.64, stdev=2954.75
     lat (usec): min=7278, max=54267, avg=32015.41, stdev=2954.21
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[27919], 10.00th=[28705], 20.00th=[30016],
     | 30.00th=[30802], 40.00th=[31327], 50.00th=[31851], 60.00th=[32375],
     | 70.00th=[32900], 80.00th=[33817], 90.00th=[35390], 95.00th=[35914],
     | 99.00th=[40633], 99.50th=[41157], 99.90th=[50594], 99.95th=[51119],
     | 99.99th=[54264]
   bw (  KiB/s): min= 1749, max= 2120, per=25.01%, avg=1974.90, stdev=99.20, samples=10
   iops        : min=  437, max=  530, avg=493.70, stdev=24.86, samples=10
  lat (msec)   : 10=0.23%, 50=99.65%, 100=0.12%
  cpu          : usr=0.44%, sys=3.98%, ctx=2628, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184590: Wed Mar 11 10:15:44 2026
  write: IOPS=493, BW=1975KiB/s (2022kB/s)(10.0MiB/5186msec); 0 zone resets
    slat (usec): min=30, max=516, avg=99.08, stdev=50.72
    clat (usec): min=23848, max=53123, avg=31952.23, stdev=2731.22
     lat (usec): min=23945, max=53223, avg=32052.51, stdev=2729.59
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[27919], 10.00th=[28705], 20.00th=[29754],
     | 30.00th=[30802], 40.00th=[31327], 50.00th=[31851], 60.00th=[32375],
     | 70.00th=[33162], 80.00th=[33817], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[40109], 99.50th=[41681], 99.90th=[46924], 99.95th=[50070],
     | 99.99th=[53216]
   bw (  KiB/s): min= 1749, max= 2104, per=25.07%, avg=1979.70, stdev=99.35, samples=10
   iops        : min=  437, max=  526, avg=494.90, stdev=24.90, samples=10
  lat (msec)   : 50=99.92%, 100=0.08%
  cpu          : usr=0.56%, sys=3.95%, ctx=2674, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7898KiB/s (8088kB/s), 1975KiB/s-1988KiB/s (2022kB/s-2036kB/s), io=40.0MiB (41.9MB), run=5150-5186msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=184601: Wed Mar 11 10:15:53 2026
  read: IOPS=220, BW=882KiB/s (904kB/s)(5048KiB/5721msec)
    slat (usec): min=23, max=5581, avg=2569.73, stdev=732.04
    clat (usec): min=10525, max=67708, avg=34388.01, stdev=4895.08
     lat (usec): min=14537, max=69427, avg=36959.18, stdev=4969.16
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26870], 10.00th=[28443], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[40109], 95.00th=[42730],
     | 99.00th=[46924], 99.50th=[49021], 99.90th=[62129], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  776, max=  936, per=25.21%, avg=880.27, stdev=43.88, samples=11
   iops        : min=  194, max=  234, avg=220.00, stdev=10.95, samples=11
  write: IOPS=226, BW=908KiB/s (929kB/s)(5192KiB/5721msec); 0 zone resets
    slat (usec): min=25, max=499, avg=94.37, stdev=49.95
    clat (usec): min=10122, max=54258, avg=34363.11, stdev=3980.54
     lat (usec): min=10180, max=54325, avg=34458.54, stdev=3983.91
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28967], 10.00th=[30016], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33424], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[39060], 95.00th=[41157],
     | 99.00th=[43779], 99.50th=[45876], 99.90th=[47973], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  832, max= 1072, per=25.06%, avg=910.91, stdev=70.11, samples=11
   iops        : min=  208, max=  268, avg=227.73, stdev=17.53, samples=11
  lat (msec)   : 20=0.47%, 50=99.30%, 100=0.23%
  cpu          : usr=0.65%, sys=4.49%, ctx=4643, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184602: Wed Mar 11 10:15:53 2026
  read: IOPS=217, BW=868KiB/s (889kB/s)(4976KiB/5730msec)
    slat (usec): min=25, max=9621, avg=2570.75, stdev=778.70
    clat (usec): min=21073, max=60176, avg=34346.56, stdev=4487.44
     lat (usec): min=23657, max=63019, avg=36918.92, stdev=4543.44
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27657], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[37487], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[46924], 99.50th=[49021], 99.90th=[57410], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  756, max=  936, per=24.90%, avg=869.45, stdev=51.73, samples=11
   iops        : min=  189, max=  234, avg=217.36, stdev=12.93, samples=11
  write: IOPS=229, BW=919KiB/s (941kB/s)(5264KiB/5730msec); 0 zone resets
    slat (usec): min=25, max=492, avg=89.69, stdev=44.87
    clat (usec): min=17791, max=49324, avg=34416.44, stdev=3762.43
     lat (usec): min=17832, max=49498, avg=34507.11, stdev=3761.92
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[28705], 10.00th=[30016], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39060], 95.00th=[41157],
     | 99.00th=[44827], 99.50th=[46400], 99.90th=[47449], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min=  864, max=  976, per=25.26%, avg=918.36, stdev=37.36, samples=11
   iops        : min=  216, max=  244, avg=229.55, stdev= 9.31, samples=11
  lat (msec)   : 20=0.08%, 50=99.77%, 100=0.16%
  cpu          : usr=0.47%, sys=4.59%, ctx=4663, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184603: Wed Mar 11 10:15:53 2026
  read: IOPS=216, BW=867KiB/s (888kB/s)(4984KiB/5746msec)
    slat (usec): min=1085, max=8998, avg=2590.40, stdev=754.18
    clat (usec): min=6316, max=50129, avg=34386.78, stdev=4527.67
     lat (usec): min=8571, max=52812, avg=36978.49, stdev=4585.09
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27657], 10.00th=[28967], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[39584], 95.00th=[42206],
     | 99.00th=[46924], 99.50th=[48497], 99.90th=[49546], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min=  768, max=  952, per=24.87%, avg=868.45, stdev=60.75, samples=11
   iops        : min=  192, max=  238, avg=217.09, stdev=15.21, samples=11
  write: IOPS=228, BW=915KiB/s (937kB/s)(5256KiB/5746msec); 0 zone resets
    slat (usec): min=39, max=541, avg=92.76, stdev=50.65
    clat (usec): min=8508, max=47762, avg=34312.98, stdev=3885.22
     lat (usec): min=8637, max=47887, avg=34406.76, stdev=3884.31
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28705], 10.00th=[29754], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33424], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39060], 95.00th=[41157],
     | 99.00th=[44827], 99.50th=[46400], 99.90th=[47449], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min=  816, max= 1040, per=25.00%, avg=908.73, stdev=69.99, samples=11
   iops        : min=  204, max=  260, avg=227.18, stdev=17.50, samples=11
  lat (msec)   : 10=0.16%, 20=0.08%, 50=99.73%, 100=0.04%
  cpu          : usr=0.42%, sys=4.66%, ctx=4702, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184604: Wed Mar 11 10:15:53 2026
  read: IOPS=220, BW=881KiB/s (902kB/s)(5060KiB/5744msec)
    slat (usec): min=1062, max=5618, avg=2601.44, stdev=723.57
    clat (usec): min=22617, max=50359, avg=34242.38, stdev=4342.51
     lat (usec): min=25046, max=52831, avg=36845.26, stdev=4411.81
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27395], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[31851], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[37487], 90.00th=[39584], 95.00th=[41157],
     | 99.00th=[45876], 99.50th=[48497], 99.90th=[50070], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min=  794, max=  952, per=25.29%, avg=883.09, stdev=44.18, samples=11
   iops        : min=  198, max=  238, avg=220.73, stdev=11.15, samples=11
  write: IOPS=225, BW=902KiB/s (923kB/s)(5180KiB/5744msec); 0 zone resets
    slat (usec): min=28, max=477, avg=95.44, stdev=51.62
    clat (usec): min=21477, max=57898, avg=34387.83, stdev=3662.20
     lat (usec): min=21799, max=58052, avg=34484.24, stdev=3659.53
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[28967], 10.00th=[30016], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39060], 95.00th=[40633],
     | 99.00th=[43779], 99.50th=[45876], 99.90th=[50594], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  776, max=  968, per=24.78%, avg=900.73, stdev=61.16, samples=11
   iops        : min=  194, max=  242, avg=225.18, stdev=15.29, samples=11
  lat (msec)   : 50=99.80%, 100=0.20%
  cpu          : usr=0.37%, sys=4.75%, ctx=4745, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3493KiB/s (3576kB/s), 867KiB/s-882KiB/s (888kB/s-904kB/s), io=19.6MiB (20.5MB), run=5721-5746msec
  WRITE: bw=3636KiB/s (3723kB/s), 902KiB/s-919KiB/s (923kB/s-941kB/s), io=20.4MiB (21.4MB), run=5721-5746msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=184609: Wed Mar 11 10:15:57 2026
  read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec)
    slat (nsec): min=9478.2k, max=9478.2k, avg=9478170.00, stdev= 0.00
    clat (nsec): min=94244k, max=94244k, avg=94244173.00, stdev= 0.00
     lat (nsec): min=103727k, max=103727k, avg=103727305.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[93848],  5.00th=[93848], 10.00th=[93848], 20.00th=[93848],
     | 30.00th=[93848], 40.00th=[93848], 50.00th=[93848], 60.00th=[93848],
     | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848],
     | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848],
     | 99.99th=[93848]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.92%, sys=7.34%, ctx=6, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout