== sanity test 398c: run fio to test AIO ================= 16:09:18 (1778357358)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.144595 s, 290 MB/s
osc.lustre-OST0000-osc-ffff938e0b906800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=193187: Sat May  9 16:09:27 2026
  write: IOPS=481, BW=1927KiB/s (1974kB/s)(10.0MiB/5313msec); 0 zone resets
    slat (usec): min=40, max=1231, avg=180.90, stdev=102.81
    clat (usec): min=13336, max=47512, avg=32963.63, stdev=4012.89
     lat (usec): min=13426, max=47750, avg=33145.67, stdev=4027.81
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[26084], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[31589], 40.00th=[32637], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[42730], 99.50th=[44827], 99.90th=[44827], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min= 1768, max= 2210, per=25.04%, avg=1915.40, stdev=119.52, samples=10
   iops        : min=  442, max=  552, avg=478.80, stdev=29.74, samples=10
  lat (msec)   : 20=1.33%, 50=98.67%
  cpu          : usr=0.38%, sys=6.57%, ctx=2677, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=193188: Sat May  9 16:09:27 2026
  write: IOPS=478, BW=1912KiB/s (1958kB/s)(10.0MiB/5355msec); 0 zone resets
    slat (usec): min=41, max=1701, avg=186.56, stdev=103.77
    clat (usec): min=3344, max=51537, avg=32863.88, stdev=3799.52
     lat (usec): min=3450, max=51643, avg=33051.53, stdev=3807.75
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[26084], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[41157], 99.50th=[42206], 99.90th=[45351], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min= 1744, max= 2051, per=24.82%, avg=1897.90, stdev=86.73, samples=10
   iops        : min=  436, max=  512, avg=474.40, stdev=21.54, samples=10
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.82%, 50=99.02%, 100=0.08%
  cpu          : usr=0.56%, sys=6.59%, ctx=2692, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=193189: Sat May  9 16:09:27 2026
  write: IOPS=478, BW=1913KiB/s (1959kB/s)(10.0MiB/5353msec); 0 zone resets
    slat (usec): min=40, max=874, avg=177.40, stdev=100.43
    clat (usec): min=4323, max=46297, avg=32881.51, stdev=3686.25
     lat (usec): min=4436, max=46424, avg=33060.10, stdev=3691.14
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[26084], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[41157], 99.50th=[42206], 99.90th=[43254], 99.95th=[43779],
     | 99.99th=[46400]
   bw (  KiB/s): min= 1720, max= 2051, per=24.83%, avg=1898.70, stdev=87.97, samples=10
   iops        : min=  430, max=  512, avg=474.60, stdev=21.85, samples=10
  lat (msec)   : 10=0.04%, 20=0.86%, 50=99.10%
  cpu          : usr=0.43%, sys=6.35%, ctx=2713, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=193190: Sat May  9 16:09:27 2026
  write: IOPS=478, BW=1915KiB/s (1961kB/s)(10.0MiB/5347msec); 0 zone resets
    slat (usec): min=41, max=799, avg=184.56, stdev=96.31
    clat (usec): min=5715, max=49448, avg=32864.50, stdev=3857.08
     lat (usec): min=5920, max=49759, avg=33050.20, stdev=3863.21
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[25822], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[41157], 99.50th=[42206], 99.90th=[46924], 99.95th=[49021],
     | 99.99th=[49546]
   bw (  KiB/s): min= 1776, max= 2083, per=24.88%, avg=1902.70, stdev=91.15, samples=10
   iops        : min=  444, max=  520, avg=475.60, stdev=22.62, samples=10
  lat (msec)   : 10=0.08%, 20=0.74%, 50=99.18%
  cpu          : usr=0.51%, sys=7.07%, ctx=2862, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7649KiB/s (7833kB/s), 1912KiB/s-1927KiB/s (1958kB/s-1974kB/s), io=40.0MiB (41.9MB), run=5313-5355msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=193200: Sat May  9 16:09:36 2026
  read: IOPS=210, BW=842KiB/s (863kB/s)(5048KiB/5992msec)
    slat (usec): min=1071, max=6143, avg=2572.99, stdev=871.47
    clat (usec): min=19150, max=66468, avg=35766.82, stdev=7315.72
     lat (usec): min=20707, max=68386, avg=38341.30, stdev=7628.85
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26084], 10.00th=[27657], 20.00th=[29754],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38536], 80.00th=[42206], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[54789], 99.50th=[56886], 99.90th=[64750], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  720, max= 1008, per=25.34%, avg=848.73, stdev=98.07, samples=11
   iops        : min=  180, max=  252, avg=212.18, stdev=24.52, samples=11
  write: IOPS=216, BW=866KiB/s (887kB/s)(5192KiB/5992msec); 0 zone resets
    slat (usec): min=49, max=2729, avg=181.58, stdev=124.10
    clat (usec): min=22598, max=60887, avg=35729.07, stdev=6594.51
     lat (usec): min=22729, max=61332, avg=35911.65, stdev=6591.68
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27395], 10.00th=[28705], 20.00th=[30016],
     | 30.00th=[31589], 40.00th=[32637], 50.00th=[33817], 60.00th=[35914],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45876], 95.00th=[47973],
     | 99.00th=[53216], 99.50th=[54264], 99.90th=[55837], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  584, max= 1104, per=25.09%, avg=874.73, stdev=159.82, samples=11
   iops        : min=  146, max=  276, avg=218.64, stdev=39.97, samples=11
  lat (msec)   : 20=0.04%, 50=96.25%, 100=3.71%
  cpu          : usr=0.37%, sys=6.84%, ctx=4670, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=193201: Sat May  9 16:09:36 2026
  read: IOPS=208, BW=833KiB/s (853kB/s)(4976KiB/5971msec)
    slat (usec): min=1019, max=7334, avg=2553.99, stdev=905.08
    clat (usec): min=7398, max=72977, avg=35407.56, stdev=7219.41
     lat (usec): min=9811, max=75627, avg=37963.31, stdev=7572.39
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25822], 10.00th=[27657], 20.00th=[29492],
     | 30.00th=[31065], 40.00th=[32375], 50.00th=[33817], 60.00th=[35914],
     | 70.00th=[38011], 80.00th=[41681], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[55313], 99.50th=[57934], 99.90th=[62653], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  680, max= 1032, per=25.12%, avg=841.27, stdev=111.71, samples=11
   iops        : min=  170, max=  258, avg=210.27, stdev=27.96, samples=11
  write: IOPS=220, BW=882KiB/s (903kB/s)(5264KiB/5971msec); 0 zone resets
    slat (usec): min=41, max=656, avg=172.52, stdev=85.45
    clat (usec): min=23488, max=72888, avg=35905.64, stdev=6841.19
     lat (usec): min=23617, max=72969, avg=36079.35, stdev=6841.00
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[27132], 10.00th=[28181], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[34341], 60.00th=[36439],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[46400], 95.00th=[48497],
     | 99.00th=[52691], 99.50th=[54789], 99.90th=[58983], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  680, max= 1104, per=25.64%, avg=893.91, stdev=133.96, samples=11
   iops        : min=  170, max=  276, avg=223.45, stdev=33.48, samples=11
  lat (msec)   : 10=0.04%, 20=0.04%, 50=96.37%, 100=3.55%
  cpu          : usr=0.47%, sys=6.63%, ctx=4682, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=193202: Sat May  9 16:09:36 2026
  read: IOPS=207, BW=832KiB/s (852kB/s)(4984KiB/5992msec)
    slat (usec): min=1026, max=7067, avg=2563.94, stdev=877.25
    clat (usec): min=17701, max=67627, avg=35270.06, stdev=7096.09
     lat (usec): min=19971, max=69704, avg=37835.43, stdev=7419.67
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26084], 10.00th=[27395], 20.00th=[29492],
     | 30.00th=[31065], 40.00th=[32375], 50.00th=[33817], 60.00th=[35390],
     | 70.00th=[37487], 80.00th=[41157], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[54789], 99.50th=[55837], 99.90th=[65274], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  632, max= 1016, per=25.20%, avg=843.91, stdev=133.66, samples=11
   iops        : min=  158, max=  254, avg=210.91, stdev=33.41, samples=11
  write: IOPS=219, BW=877KiB/s (898kB/s)(5256KiB/5992msec); 0 zone resets
    slat (usec): min=56, max=1986, avg=182.00, stdev=108.52
    clat (usec): min=23049, max=61176, avg=36059.37, stdev=6979.36
     lat (usec): min=23362, max=61326, avg=36242.44, stdev=6978.10
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27132], 10.00th=[28443], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32375], 50.00th=[34341], 60.00th=[36439],
     | 70.00th=[39584], 80.00th=[42730], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[53740], 99.50th=[54789], 99.90th=[57934], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  656, max= 1064, per=25.32%, avg=882.55, stdev=121.62, samples=11
   iops        : min=  164, max=  266, avg=220.64, stdev=30.40, samples=11
  lat (msec)   : 20=0.04%, 50=96.02%, 100=3.95%
  cpu          : usr=0.58%, sys=6.56%, ctx=4645, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=193203: Sat May  9 16:09:36 2026
  read: IOPS=211, BW=848KiB/s (868kB/s)(5060KiB/5967msec)
    slat (usec): min=94, max=8175, avg=2526.23, stdev=921.03
    clat (usec): min=8131, max=68005, avg=35613.30, stdev=7318.92
     lat (usec): min=8344, max=71376, avg=38141.05, stdev=7689.74
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[26084], 10.00th=[27657], 20.00th=[29754],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38536], 80.00th=[41681], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[54264], 99.50th=[57410], 99.90th=[66323], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  608, max= 1024, per=25.52%, avg=854.82, stdev=128.33, samples=11
   iops        : min=  152, max=  256, avg=213.64, stdev=32.05, samples=11
  write: IOPS=217, BW=868KiB/s (889kB/s)(5180KiB/5967msec); 0 zone resets
    slat (usec): min=49, max=757, avg=176.67, stdev=95.30
    clat (usec): min=22918, max=87554, avg=36101.83, stdev=7264.50
     lat (usec): min=23165, max=87822, avg=36279.64, stdev=7264.69
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[27395], 10.00th=[28443], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32375], 50.00th=[33817], 60.00th=[36439],
     | 70.00th=[39584], 80.00th=[42730], 90.00th=[45876], 95.00th=[49546],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[87557], 99.95th=[87557],
     | 99.99th=[87557]
   bw (  KiB/s): min=  616, max=  992, per=25.29%, avg=881.45, stdev=120.89, samples=11
   iops        : min=  154, max=  248, avg=220.36, stdev=30.22, samples=11
  lat (msec)   : 10=0.12%, 20=0.08%, 50=95.82%, 100=3.98%
  cpu          : usr=0.40%, sys=6.65%, ctx=4703, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3349KiB/s (3430kB/s), 832KiB/s-848KiB/s (852kB/s-868kB/s), io=19.6MiB (20.5MB), run=5967-5992msec
  WRITE: bw=3487KiB/s (3570kB/s), 866KiB/s-882KiB/s (887kB/s-903kB/s), io=20.4MiB (21.4MB), run=5967-5992msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=193208: Sat May  9 16:09:40 2026
  read: IOPS=8, BW=351MiB/s (368MB/s)(40.0MiB/114msec)
    slat (nsec): min=10500k, max=10500k, avg=10500268.00, stdev= 0.00
    clat (nsec): min=98044k, max=98044k, avg=98044101.00, stdev= 0.00
     lat (nsec): min=108548k, max=108548k, avg=108547864.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[98042],  5.00th=[98042], 10.00th=[98042], 20.00th=[98042],
     | 30.00th=[98042], 40.00th=[98042], 50.00th=[98042], 60.00th=[98042],
     | 70.00th=[98042], 80.00th=[98042], 90.00th=[98042], 95.00th=[98042],
     | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042],
     | 99.99th=[98042]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.96%, ctx=11, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=40.0MiB (41.9MB), run=114-114msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout