== sanity test 398c: run fio to test AIO ================= 14:50:17 (1767729017)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.140683 s, 298 MB/s
osc.lustre-OST0000-osc-ffff945e4542e000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=189834: Tue Jan  6 14:50:25 2026
  write: IOPS=509, BW=2037KiB/s (2086kB/s)(10.0MiB/5026msec); 0 zone resets
    slat (usec): min=27, max=736, avg=95.65, stdev=50.58
    clat (usec): min=7687, max=59262, avg=31011.18, stdev=5301.96
     lat (usec): min=7841, max=59354, avg=31107.89, stdev=5304.53
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[23987], 10.00th=[25822], 20.00th=[27395],
     | 30.00th=[28705], 40.00th=[29492], 50.00th=[30278], 60.00th=[31327],
     | 70.00th=[32637], 80.00th=[34341], 90.00th=[37487], 95.00th=[40633],
     | 99.00th=[50070], 99.50th=[52691], 99.90th=[54789], 99.95th=[58983],
     | 99.99th=[59507]
   bw (  KiB/s): min= 1760, max= 2288, per=24.89%, avg=2028.44, stdev=190.58, samples=9
   iops        : min=  440, max=  572, avg=507.11, stdev=47.65, samples=9
  lat (msec)   : 10=0.04%, 20=1.29%, 50=97.66%, 100=1.02%
  cpu          : usr=0.56%, sys=4.36%, ctx=2855, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189835: Tue Jan  6 14:50:25 2026
  write: IOPS=514, BW=2057KiB/s (2106kB/s)(10.0MiB/4978msec); 0 zone resets
    slat (usec): min=21, max=1514, avg=89.17, stdev=60.33
    clat (usec): min=14840, max=59218, avg=30969.13, stdev=5568.54
     lat (usec): min=14900, max=59302, avg=31059.20, stdev=5574.99
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[22938], 10.00th=[25560], 20.00th=[27395],
     | 30.00th=[28443], 40.00th=[29492], 50.00th=[30278], 60.00th=[31589],
     | 70.00th=[32637], 80.00th=[34341], 90.00th=[37487], 95.00th=[41681],
     | 99.00th=[50070], 99.50th=[53216], 99.90th=[54789], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1720, max= 2432, per=25.16%, avg=2050.67, stdev=227.19, samples=9
   iops        : min=  430, max=  608, avg=512.67, stdev=56.80, samples=9
  lat (msec)   : 20=3.12%, 50=95.82%, 100=1.05%
  cpu          : usr=0.46%, sys=3.82%, ctx=2416, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189836: Tue Jan  6 14:50:25 2026
  write: IOPS=510, BW=2042KiB/s (2091kB/s)(10.0MiB/5015msec); 0 zone resets
    slat (usec): min=26, max=24720, avg=104.45, stdev=489.54
    clat (usec): min=8224, max=55131, avg=30957.59, stdev=5477.04
     lat (usec): min=8451, max=55205, avg=31063.18, stdev=5460.30
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[23462], 10.00th=[25560], 20.00th=[27395],
     | 30.00th=[28705], 40.00th=[29492], 50.00th=[30016], 60.00th=[31327],
     | 70.00th=[32375], 80.00th=[34341], 90.00th=[37487], 95.00th=[41157],
     | 99.00th=[50070], 99.50th=[52167], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1744, max= 2304, per=24.99%, avg=2036.44, stdev=206.07, samples=9
   iops        : min=  436, max=  576, avg=509.11, stdev=51.52, samples=9
  lat (msec)   : 10=0.12%, 20=2.54%, 50=96.37%, 100=0.98%
  cpu          : usr=0.48%, sys=4.45%, ctx=2666, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189837: Tue Jan  6 14:50:25 2026
  write: IOPS=509, BW=2037KiB/s (2086kB/s)(10.0MiB/5026msec); 0 zone resets
    slat (usec): min=22, max=1182, avg=90.98, stdev=49.04
    clat (usec): min=15922, max=59339, avg=30984.61, stdev=5308.48
     lat (usec): min=16021, max=59423, avg=31076.65, stdev=5315.48
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[23725], 10.00th=[25560], 20.00th=[27395],
     | 30.00th=[28443], 40.00th=[29492], 50.00th=[30278], 60.00th=[31327],
     | 70.00th=[32375], 80.00th=[34341], 90.00th=[37487], 95.00th=[40109],
     | 99.00th=[50070], 99.50th=[52167], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min= 1728, max= 2296, per=24.86%, avg=2026.11, stdev=197.19, samples=9
   iops        : min=  432, max=  574, avg=506.44, stdev=49.20, samples=9
  lat (msec)   : 20=2.07%, 50=96.91%, 100=1.02%
  cpu          : usr=0.54%, sys=3.78%, ctx=2517, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8150KiB/s (8345kB/s), 2037KiB/s-2057KiB/s (2086kB/s-2106kB/s), io=40.0MiB (41.9MB), run=4978-5026msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=189847: Tue Jan  6 14:50:34 2026
  read: IOPS=233, BW=934KiB/s (956kB/s)(5048KiB/5405msec)
    slat (usec): min=913, max=8819, avg=2504.17, stdev=876.64
    clat (usec): min=6270, max=53738, avg=32258.72, stdev=5377.17
     lat (usec): min=8368, max=56566, avg=34764.26, stdev=5480.68
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[23987], 10.00th=[25822], 20.00th=[27919],
     | 30.00th=[29492], 40.00th=[30540], 50.00th=[31851], 60.00th=[33424],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[47973], 99.50th=[50594], 99.90th=[52691], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  779, max= 1064, per=25.25%, avg=937.10, stdev=90.19, samples=10
   iops        : min=  194, max=  266, avg=234.20, stdev=22.70, samples=10
  write: IOPS=240, BW=961KiB/s (984kB/s)(5192KiB/5405msec); 0 zone resets
    slat (usec): min=26, max=1969, avg=84.49, stdev=89.34
    clat (usec): min=8870, max=53347, avg=32141.91, stdev=4686.36
     lat (usec): min=8941, max=53437, avg=32227.35, stdev=4686.17
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25297], 10.00th=[26608], 20.00th=[28443],
     | 30.00th=[29492], 40.00th=[30540], 50.00th=[31589], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[35914], 90.00th=[38011], 95.00th=[40633],
     | 99.00th=[45351], 99.50th=[47973], 99.90th=[51643], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  734, max= 1080, per=25.00%, avg=966.20, stdev=92.84, samples=10
   iops        : min=  183, max=  270, avg=241.50, stdev=23.35, samples=10
  lat (msec)   : 10=0.08%, 20=0.20%, 50=99.30%, 100=0.43%
  cpu          : usr=0.65%, sys=4.13%, ctx=4553, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189848: Tue Jan  6 14:50:34 2026
  read: IOPS=231, BW=925KiB/s (947kB/s)(4976KiB/5380msec)
    slat (usec): min=36, max=14745, avg=2508.28, stdev=937.45
    clat (usec): min=7204, max=60848, avg=32284.62, stdev=5537.66
     lat (usec): min=9620, max=63074, avg=34794.28, stdev=5615.93
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[24249], 10.00th=[25560], 20.00th=[27657],
     | 30.00th=[29230], 40.00th=[30540], 50.00th=[32113], 60.00th=[33424],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[46924], 99.50th=[49021], 99.90th=[56361], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  741, max= 1000, per=24.99%, avg=927.70, stdev=75.11, samples=10
   iops        : min=  185, max=  250, avg=231.90, stdev=18.85, samples=10
  write: IOPS=244, BW=978KiB/s (1002kB/s)(5264KiB/5380msec); 0 zone resets
    slat (usec): min=32, max=1037, avg=81.29, stdev=54.40
    clat (usec): min=7131, max=58532, avg=32312.12, stdev=5028.58
     lat (usec): min=7218, max=58601, avg=32394.34, stdev=5026.24
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[25297], 10.00th=[26870], 20.00th=[28181],
     | 30.00th=[29492], 40.00th=[30802], 50.00th=[31851], 60.00th=[32900],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[39060], 95.00th=[41157],
     | 99.00th=[46924], 99.50th=[48497], 99.90th=[54264], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  894, max= 1072, per=25.47%, avg=984.60, stdev=58.70, samples=10
   iops        : min=  223, max=  268, avg=246.10, stdev=14.76, samples=10
  lat (msec)   : 10=0.12%, 20=0.55%, 50=98.95%, 100=0.39%
  cpu          : usr=0.30%, sys=4.46%, ctx=4517, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189849: Tue Jan  6 14:50:34 2026
  read: IOPS=230, BW=922KiB/s (944kB/s)(4984KiB/5405msec)
    slat (usec): min=846, max=8588, avg=2527.08, stdev=865.20
    clat (usec): min=16182, max=55141, avg=32380.66, stdev=5292.37
     lat (usec): min=17542, max=57681, avg=34909.40, stdev=5370.04
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[24249], 10.00th=[25822], 20.00th=[27919],
     | 30.00th=[29492], 40.00th=[30802], 50.00th=[31851], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[36963], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[45876], 99.50th=[49546], 99.90th=[54264], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  773, max= 1024, per=24.99%, avg=927.70, stdev=83.84, samples=10
   iops        : min=  193, max=  256, avg=231.90, stdev=21.01, samples=10
  write: IOPS=243, BW=972KiB/s (996kB/s)(5256KiB/5405msec); 0 zone resets
    slat (usec): min=33, max=2600, avg=83.27, stdev=98.58
    clat (usec): min=4777, max=51417, avg=32255.70, stdev=4982.94
     lat (usec): min=4900, max=51470, avg=32339.92, stdev=4982.03
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25297], 10.00th=[26870], 20.00th=[28443],
     | 30.00th=[29492], 40.00th=[30540], 50.00th=[31589], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[35914], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[45876], 99.50th=[46924], 99.90th=[49546], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min=  792, max= 1176, per=25.11%, avg=970.40, stdev=107.07, samples=10
   iops        : min=  198, max=  294, avg=242.60, stdev=26.77, samples=10
  lat (msec)   : 10=0.04%, 20=0.43%, 50=99.26%, 100=0.27%
  cpu          : usr=0.43%, sys=4.40%, ctx=4568, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189850: Tue Jan  6 14:50:34 2026
  read: IOPS=234, BW=937KiB/s (960kB/s)(5060KiB/5400msec)
    slat (usec): min=926, max=9826, avg=2487.76, stdev=879.21
    clat (usec): min=2648, max=53941, avg=32162.29, stdev=5340.41
     lat (usec): min=6237, max=56012, avg=34651.60, stdev=5449.14
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[23987], 10.00th=[25822], 20.00th=[27919],
     | 30.00th=[29492], 40.00th=[30540], 50.00th=[31851], 60.00th=[32900],
     | 70.00th=[34341], 80.00th=[36439], 90.00th=[39060], 95.00th=[41157],
     | 99.00th=[44827], 99.50th=[47449], 99.90th=[49546], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  760, max= 1040, per=25.24%, avg=936.80, stdev=75.28, samples=10
   iops        : min=  190, max=  260, avg=234.20, stdev=18.82, samples=10
  write: IOPS=239, BW=959KiB/s (982kB/s)(5180KiB/5400msec); 0 zone resets
    slat (usec): min=29, max=3026, avg=84.94, stdev=93.91
    clat (usec): min=20222, max=53302, avg=32315.68, stdev=4857.63
     lat (usec): min=20292, max=53356, avg=32401.54, stdev=4861.60
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[25297], 10.00th=[26608], 20.00th=[28443],
     | 30.00th=[29492], 40.00th=[30540], 50.00th=[31851], 60.00th=[32900],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[47449], 99.50th=[50070], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  779, max= 1048, per=25.01%, avg=966.70, stdev=76.26, samples=10
   iops        : min=  194, max=  262, avg=241.60, stdev=19.27, samples=10
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.35%, 50=99.30%, 100=0.27%
  cpu          : usr=0.41%, sys=4.46%, ctx=4535, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3713KiB/s (3802kB/s), 922KiB/s-937KiB/s (944kB/s-960kB/s), io=19.6MiB (20.5MB), run=5380-5405msec
  WRITE: bw=3865KiB/s (3958kB/s), 959KiB/s-978KiB/s (982kB/s-1002kB/s), io=20.4MiB (21.4MB), run=5380-5405msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=189855: Tue Jan  6 14:50:38 2026
  read: IOPS=9, BW=385MiB/s (403MB/s)(40.0MiB/104msec)
    slat (nsec): min=10889k, max=10889k, avg=10889170.00, stdev= 0.00
    clat (nsec): min=88409k, max=88409k, avg=88409042.00, stdev= 0.00
     lat (nsec): min=99322k, max=99322k, avg=99322436.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[88605],  5.00th=[88605], 10.00th=[88605], 20.00th=[88605],
     | 30.00th=[88605], 40.00th=[88605], 50.00th=[88605], 60.00th=[88605],
     | 70.00th=[88605], 80.00th=[88605], 90.00th=[88605], 95.00th=[88605],
     | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605],
     | 99.99th=[88605]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.74%, ctx=11, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=385MiB/s (403MB/s), 385MiB/s-385MiB/s (403MB/s-403MB/s), io=40.0MiB (41.9MB), run=104-104msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout