== sanity test 398c: run fio to test AIO ================= 23:00:16 (1789354816)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.130475 s, 321 MB/s
osc.lustre-OST0000-osc-ffff9343894fc800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182070: Sun Sep 13 23:00:25 2026
  write: IOPS=508, BW=2034KiB/s (2083kB/s)(10.0MiB/5034msec); 0 zone resets
    slat (usec): min=38, max=1201, avg=195.12, stdev=107.69
    clat (usec): min=4965, max=53280, avg=31213.75, stdev=3339.54
     lat (usec): min=5156, max=53525, avg=31410.11, stdev=3340.42
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[26608], 10.00th=[27657], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30540], 50.00th=[31327], 60.00th=[32113],
     | 70.00th=[32900], 80.00th=[33424], 90.00th=[34341], 95.00th=[35390],
     | 99.00th=[39060], 99.50th=[42730], 99.90th=[50070], 99.95th=[50594],
     | 99.99th=[53216]
   bw (  KiB/s): min= 1944, max= 2116, per=25.26%, avg=2031.56, stdev=68.07, samples=9
   iops        : min=  486, max=  529, avg=507.89, stdev=17.02, samples=9
  lat (msec)   : 10=0.23%, 20=0.51%, 50=99.14%, 100=0.12%
  cpu          : usr=0.60%, sys=7.47%, ctx=2750, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182071: Sun Sep 13 23:00:25 2026
  write: IOPS=502, BW=2011KiB/s (2060kB/s)(10.0MiB/5091msec); 0 zone resets
    slat (usec): min=44, max=1058, avg=183.81, stdev=102.68
    clat (usec): min=16297, max=47285, avg=31300.29, stdev=2786.52
     lat (usec): min=16687, max=47532, avg=31485.25, stdev=2784.94
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27132], 10.00th=[27919], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30802], 50.00th=[31327], 60.00th=[32113],
     | 70.00th=[32637], 80.00th=[33424], 90.00th=[34866], 95.00th=[35914],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[43254], 99.95th=[43779],
     | 99.99th=[47449]
   bw (  KiB/s): min= 1936, max= 2072, per=24.92%, avg=2004.56, stdev=52.98, samples=9
   iops        : min=  484, max=  518, avg=501.11, stdev=13.27, samples=9
  lat (msec)   : 20=0.04%, 50=99.96%
  cpu          : usr=0.43%, sys=7.54%, ctx=2651, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182072: Sun Sep 13 23:00:25 2026
  write: IOPS=502, BW=2011KiB/s (2059kB/s)(10.0MiB/5092msec); 0 zone resets
    slat (usec): min=39, max=802, avg=169.67, stdev=106.58
    clat (usec): min=19214, max=40559, avg=31267.99, stdev=2712.75
     lat (usec): min=19287, max=40667, avg=31438.82, stdev=2721.67
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26870], 10.00th=[28181], 20.00th=[29230],
     | 30.00th=[30016], 40.00th=[30802], 50.00th=[31327], 60.00th=[32113],
     | 70.00th=[32637], 80.00th=[33424], 90.00th=[34341], 95.00th=[35390],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1890, max= 2128, per=24.88%, avg=2001.11, stdev=77.04, samples=9
   iops        : min=  472, max=  532, avg=500.22, stdev=19.35, samples=9
  lat (msec)   : 20=0.27%, 50=99.73%
  cpu          : usr=0.55%, sys=6.52%, ctx=2229, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182073: Sun Sep 13 23:00:25 2026
  write: IOPS=503, BW=2015KiB/s (2064kB/s)(10.0MiB/5081msec); 0 zone resets
    slat (usec): min=44, max=29612, avg=206.39, stdev=590.34
    clat (usec): min=7411, max=55043, avg=31242.20, stdev=2946.31
     lat (usec): min=16648, max=55281, avg=31449.76, stdev=2908.04
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[26870], 10.00th=[27657], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30540], 50.00th=[31327], 60.00th=[32113],
     | 70.00th=[32637], 80.00th=[33817], 90.00th=[34341], 95.00th=[35390],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1936, max= 2088, per=25.03%, avg=2013.11, stdev=61.86, samples=9
   iops        : min=  484, max=  522, avg=503.22, stdev=15.51, samples=9
  lat (msec)   : 10=0.04%, 20=0.23%, 50=99.57%, 100=0.16%
  cpu          : usr=0.43%, sys=8.13%, ctx=2624, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8044KiB/s (8237kB/s), 2011KiB/s-2034KiB/s (2059kB/s-2083kB/s), io=40.0MiB (41.9MB), run=5034-5092msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182083: Sun Sep 13 23:00:34 2026
  read: IOPS=211, BW=845KiB/s (865kB/s)(5048KiB/5977msec)
    slat (usec): min=44, max=8747, avg=2548.38, stdev=738.44
    clat (usec): min=12207, max=66873, avg=35834.45, stdev=4981.46
     lat (usec): min=17621, max=69954, avg=38384.22, stdev=5032.03
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28181], 10.00th=[29754], 20.00th=[31851],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[43254],
     | 99.00th=[46924], 99.50th=[47973], 99.90th=[64226], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  752, max=  896, per=25.34%, avg=846.27, stdev=43.17, samples=11
   iops        : min=  188, max=  224, avg=211.55, stdev=10.76, samples=11
  write: IOPS=217, BW=869KiB/s (890kB/s)(5192KiB/5977msec); 0 zone resets
    slat (usec): min=61, max=2524, avg=171.11, stdev=105.51
    clat (usec): min=11367, max=64120, avg=36041.25, stdev=4142.00
     lat (usec): min=11442, max=64246, avg=36213.72, stdev=4141.87
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[30016], 10.00th=[31327], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[44827], 99.50th=[46400], 99.90th=[62653], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  744, max= 1056, per=25.07%, avg=871.82, stdev=90.69, samples=11
   iops        : min=  186, max=  264, avg=217.91, stdev=22.70, samples=11
  lat (msec)   : 20=0.47%, 50=99.14%, 100=0.39%
  cpu          : usr=0.37%, sys=6.44%, ctx=4812, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182084: Sun Sep 13 23:00:34 2026
  read: IOPS=207, BW=829KiB/s (848kB/s)(4976KiB/6006msec)
    slat (usec): min=1106, max=6913, avg=2583.79, stdev=677.36
    clat (usec): min=8313, max=57373, avg=35783.75, stdev=4830.97
     lat (usec): min=11257, max=59401, avg=38369.02, stdev=4827.44
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28181], 10.00th=[29754], 20.00th=[31589],
     | 30.00th=[33162], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[39584], 90.00th=[41681], 95.00th=[43254],
     | 99.00th=[46400], 99.50th=[49021], 99.90th=[54264], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  724, max=  888, per=24.62%, avg=822.18, stdev=55.41, samples=11
   iops        : min=  181, max=  222, avg=205.55, stdev=13.85, samples=11
  write: IOPS=219, BW=876KiB/s (897kB/s)(5264KiB/6006msec); 0 zone resets
    slat (usec): min=46, max=598, avg=167.48, stdev=82.49
    clat (usec): min=18129, max=48967, avg=36092.48, stdev=3668.23
     lat (usec): min=18489, max=49122, avg=36261.13, stdev=3665.51
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[30278], 10.00th=[31589], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[45351], 99.50th=[46400], 99.90th=[48497], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min=  824, max=  984, per=25.45%, avg=884.73, stdev=56.99, samples=11
   iops        : min=  206, max=  246, avg=221.18, stdev=14.25, samples=11
  lat (msec)   : 10=0.04%, 20=0.20%, 50=99.69%, 100=0.08%
  cpu          : usr=0.38%, sys=6.49%, ctx=4693, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182085: Sun Sep 13 23:00:34 2026
  read: IOPS=207, BW=830KiB/s (850kB/s)(4984KiB/6002msec)
    slat (usec): min=1114, max=8148, avg=2604.63, stdev=718.67
    clat (usec): min=10991, max=53526, avg=35874.66, stdev=4664.70
     lat (usec): min=13252, max=55938, avg=38480.60, stdev=4653.18
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28443], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41681], 95.00th=[43779],
     | 99.00th=[46924], 99.50th=[49021], 99.90th=[52167], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  696, max=  944, per=25.00%, avg=834.91, stdev=68.00, samples=11
   iops        : min=  174, max=  236, avg=208.73, stdev=17.00, samples=11
  write: IOPS=218, BW=876KiB/s (897kB/s)(5256KiB/6002msec); 0 zone resets
    slat (usec): min=49, max=637, avg=168.82, stdev=78.30
    clat (usec): min=19704, max=51589, avg=35985.95, stdev=3577.56
     lat (usec): min=19906, max=51726, avg=36155.92, stdev=3576.87
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[30540], 10.00th=[31589], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[44827], 99.50th=[45876], 99.90th=[51643], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min=  712, max=  960, per=25.03%, avg=870.45, stdev=76.20, samples=11
   iops        : min=  178, max=  240, avg=217.55, stdev=19.11, samples=11
  lat (msec)   : 20=0.12%, 50=99.61%, 100=0.27%
  cpu          : usr=0.38%, sys=6.53%, ctx=4799, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182086: Sun Sep 13 23:00:34 2026
  read: IOPS=210, BW=842KiB/s (863kB/s)(5060KiB/6007msec)
    slat (usec): min=1101, max=9851, avg=2564.60, stdev=714.34
    clat (usec): min=11745, max=49785, avg=35859.38, stdev=4587.59
     lat (usec): min=14059, max=52943, avg=38425.50, stdev=4590.80
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[28705], 10.00th=[29754], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36439], 60.00th=[37487],
     | 70.00th=[38536], 80.00th=[39584], 90.00th=[41157], 95.00th=[43254],
     | 99.00th=[45876], 99.50th=[46924], 99.90th=[49546], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min=  750, max=  936, per=25.23%, avg=842.73, stdev=57.15, samples=11
   iops        : min=  187, max=  234, avg=210.64, stdev=14.37, samples=11
  write: IOPS=215, BW=862KiB/s (883kB/s)(5180KiB/6007msec); 0 zone resets
    slat (usec): min=52, max=1822, avg=169.22, stdev=91.35
    clat (usec): min=8777, max=46907, avg=35871.06, stdev=3855.50
     lat (usec): min=9015, max=47007, avg=36041.39, stdev=3854.04
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[30278], 10.00th=[31327], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[45876], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  768, max=  952, per=24.82%, avg=863.00, stdev=61.95, samples=11
   iops        : min=  192, max=  238, avg=215.73, stdev=15.52, samples=11
  lat (msec)   : 10=0.04%, 20=0.35%, 50=99.61%
  cpu          : usr=0.45%, sys=6.41%, ctx=4808, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3341KiB/s (3421kB/s), 829KiB/s-845KiB/s (848kB/s-865kB/s), io=19.6MiB (20.5MB), run=5977-6007msec
  WRITE: bw=3478KiB/s (3561kB/s), 862KiB/s-876KiB/s (883kB/s-897kB/s), io=20.4MiB (21.4MB), run=5977-6007msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182090: Sun Sep 13 23:00:38 2026
  read: IOPS=8, BW=351MiB/s (368MB/s)(40.0MiB/114msec)
    slat (nsec): min=9246.0k, max=9246.0k, avg=9246996.00, stdev= 0.00
    clat (nsec): min=99061k, max=99061k, avg=99060961.00, stdev= 0.00
     lat (nsec): min=108314k, max=108314k, avg=108313682.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[99091],  5.00th=[99091], 10.00th=[99091], 20.00th=[99091],
     | 30.00th=[99091], 40.00th=[99091], 50.00th=[99091], 60.00th=[99091],
     | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091],
     | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091],
     | 99.99th=[99091]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.85%, ctx=4, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=40.0MiB (41.9MB), run=114-114msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout