== sanity test 398c: run fio to test AIO ================= 14:53:41 (1781031221)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.131006 s, 320 MB/s
osc.lustre-OST0000-osc-ffff9e0e07840000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=184801: Tue Jun  9 14:53:50 2026
  write: IOPS=484, BW=1938KiB/s (1985kB/s)(10.0MiB/5283msec); 0 zone resets
    slat (usec): min=51, max=25460, avg=204.79, stdev=510.57
    clat (usec): min=6648, max=48397, avg=32492.94, stdev=3823.15
     lat (usec): min=6812, max=48510, avg=32698.92, stdev=3800.66
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[26870], 10.00th=[28967], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[31851], 50.00th=[32637], 60.00th=[33162],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[42206], 99.50th=[45351], 99.90th=[45876], 99.95th=[45876],
     | 99.99th=[48497]
   bw (  KiB/s): min= 1792, max= 2000, per=24.93%, avg=1931.80, stdev=64.82, samples=10
   iops        : min=  448, max=  500, avg=482.90, stdev=16.16, samples=10
  lat (msec)   : 10=0.20%, 20=1.17%, 50=98.63%
  cpu          : usr=0.57%, sys=7.46%, ctx=2760, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184802: Tue Jun  9 14:53:50 2026
  write: IOPS=484, BW=1937KiB/s (1984kB/s)(10.0MiB/5286msec); 0 zone resets
    slat (usec): min=41, max=1147, avg=192.40, stdev=101.07
    clat (usec): min=12593, max=45795, avg=32503.93, stdev=3658.35
     lat (usec): min=12746, max=45960, avg=32697.46, stdev=3667.66
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[26870], 10.00th=[28967], 20.00th=[30278],
     | 30.00th=[31065], 40.00th=[31851], 50.00th=[32637], 60.00th=[33162],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[41157], 99.50th=[44827], 99.90th=[45876], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min= 1792, max= 2000, per=24.87%, avg=1926.80, stdev=65.32, samples=10
   iops        : min=  448, max=  500, avg=481.70, stdev=16.33, samples=10
  lat (msec)   : 20=1.02%, 50=98.98%
  cpu          : usr=0.47%, sys=7.44%, ctx=2758, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184803: Tue Jun  9 14:53:50 2026
  write: IOPS=487, BW=1950KiB/s (1997kB/s)(10.0MiB/5252msec); 0 zone resets
    slat (usec): min=48, max=898, avg=200.87, stdev=101.34
    clat (usec): min=4606, max=54670, avg=32567.60, stdev=3952.47
     lat (usec): min=4704, max=54936, avg=32769.62, stdev=3954.86
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[26870], 10.00th=[28967], 20.00th=[30540],
     | 30.00th=[31327], 40.00th=[31851], 50.00th=[32637], 60.00th=[33424],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[44303], 99.50th=[44827], 99.90th=[47973], 99.95th=[47973],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1808, max= 2080, per=25.09%, avg=1944.00, stdev=80.62, samples=10
   iops        : min=  452, max=  520, avg=486.00, stdev=20.15, samples=10
  lat (msec)   : 10=0.20%, 20=1.09%, 50=98.67%, 100=0.04%
  cpu          : usr=0.72%, sys=7.08%, ctx=2683, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184804: Tue Jun  9 14:53:50 2026
  write: IOPS=485, BW=1942KiB/s (1989kB/s)(10.0MiB/5273msec); 0 zone resets
    slat (usec): min=45, max=726, avg=193.32, stdev=97.26
    clat (usec): min=10676, max=55142, avg=32631.20, stdev=3614.89
     lat (usec): min=10890, max=55227, avg=32825.67, stdev=3619.87
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[27395], 10.00th=[29230], 20.00th=[30540],
     | 30.00th=[31327], 40.00th=[31851], 50.00th=[32637], 60.00th=[33162],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[41681], 99.50th=[45351], 99.90th=[52167], 99.95th=[52691],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1784, max= 2024, per=25.03%, avg=1939.00, stdev=79.43, samples=10
   iops        : min=  446, max=  506, avg=484.70, stdev=19.80, samples=10
  lat (msec)   : 20=0.59%, 50=99.30%, 100=0.12%
  cpu          : usr=0.55%, sys=7.00%, ctx=2683, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7749KiB/s (7935kB/s), 1937KiB/s-1950KiB/s (1984kB/s-1997kB/s), io=40.0MiB (41.9MB), run=5252-5286msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=184814: Tue Jun  9 14:53:59 2026
  read: IOPS=214, BW=858KiB/s (879kB/s)(5048KiB/5884msec)
    slat (usec): min=153, max=10537, avg=2508.53, stdev=972.26
    clat (usec): min=13359, max=63839, avg=34889.84, stdev=7353.72
     lat (usec): min=13816, max=66962, avg=37400.45, stdev=7758.79
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[25822], 10.00th=[27132], 20.00th=[29230],
     | 30.00th=[30540], 40.00th=[31851], 50.00th=[33162], 60.00th=[34866],
     | 70.00th=[36963], 80.00th=[40109], 90.00th=[45876], 95.00th=[49546],
     | 99.00th=[55837], 99.50th=[57934], 99.90th=[63177], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  592, max= 1008, per=25.11%, avg=854.45, stdev=148.45, samples=11
   iops        : min=  148, max=  252, avg=213.55, stdev=37.15, samples=11
  write: IOPS=220, BW=882KiB/s (904kB/s)(5192KiB/5884msec); 0 zone resets
    slat (usec): min=43, max=3701, avg=176.63, stdev=154.13
    clat (usec): min=14861, max=63529, avg=35641.19, stdev=7115.38
     lat (usec): min=15015, max=63634, avg=35818.85, stdev=7126.52
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[26870], 10.00th=[28443], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[35390],
     | 70.00th=[37487], 80.00th=[40633], 90.00th=[45876], 95.00th=[51119],
     | 99.00th=[57410], 99.50th=[60031], 99.90th=[61080], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  688, max= 1080, per=24.88%, avg=881.36, stdev=123.89, samples=11
   iops        : min=  172, max=  270, avg=220.27, stdev=31.06, samples=11
  lat (msec)   : 20=0.16%, 50=94.53%, 100=5.31%
  cpu          : usr=0.56%, sys=6.49%, ctx=4711, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184815: Tue Jun  9 14:53:59 2026
  read: IOPS=212, BW=850KiB/s (870kB/s)(4976KiB/5854msec)
    slat (usec): min=64, max=11612, avg=2496.38, stdev=947.82
    clat (usec): min=7676, max=64804, avg=34843.14, stdev=7659.95
     lat (usec): min=7788, max=67711, avg=37341.28, stdev=8070.23
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[25297], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31851], 50.00th=[33162], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[40109], 90.00th=[46400], 95.00th=[50594],
     | 99.00th=[56361], 99.50th=[58983], 99.90th=[64226], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  592, max= 1000, per=24.92%, avg=847.91, stdev=146.57, samples=11
   iops        : min=  148, max=  250, avg=211.91, stdev=36.72, samples=11
  write: IOPS=224, BW=899KiB/s (921kB/s)(5264KiB/5854msec); 0 zone resets
    slat (usec): min=55, max=1853, avg=167.55, stdev=100.03
    clat (usec): min=14542, max=70596, avg=35554.82, stdev=7614.81
     lat (usec): min=14787, max=70896, avg=35723.37, stdev=7629.86
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26608], 10.00th=[28181], 20.00th=[29754],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[40109], 90.00th=[47449], 95.00th=[51643],
     | 99.00th=[57934], 99.50th=[59507], 99.90th=[62653], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  667, max= 1120, per=25.38%, avg=899.00, stdev=137.09, samples=11
   iops        : min=  166, max=  280, avg=224.64, stdev=34.41, samples=11
  lat (msec)   : 10=0.08%, 20=0.74%, 50=92.93%, 100=6.25%
  cpu          : usr=0.58%, sys=6.53%, ctx=4641, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184816: Tue Jun  9 14:53:59 2026
  read: IOPS=211, BW=846KiB/s (866kB/s)(4984KiB/5894msec)
    slat (usec): min=922, max=10268, avg=2512.10, stdev=930.21
    clat (usec): min=4963, max=64156, avg=35248.61, stdev=7399.38
     lat (usec): min=8336, max=67812, avg=37762.60, stdev=7787.22
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[25822], 10.00th=[27395], 20.00th=[29492],
     | 30.00th=[30802], 40.00th=[32113], 50.00th=[33817], 60.00th=[35390],
     | 70.00th=[37487], 80.00th=[40633], 90.00th=[45876], 95.00th=[50070],
     | 99.00th=[57410], 99.50th=[60556], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  648, max= 1016, per=24.90%, avg=847.18, stdev=118.75, samples=11
   iops        : min=  162, max=  254, avg=211.73, stdev=29.75, samples=11
  write: IOPS=222, BW=892KiB/s (913kB/s)(5256KiB/5894msec); 0 zone resets
    slat (usec): min=53, max=4577, avg=165.18, stdev=166.61
    clat (usec): min=19017, max=64635, avg=35237.94, stdev=7060.27
     lat (usec): min=19119, max=64770, avg=35404.07, stdev=7071.55
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[27132], 10.00th=[28181], 20.00th=[29754],
     | 30.00th=[31065], 40.00th=[32113], 50.00th=[33424], 60.00th=[34866],
     | 70.00th=[36963], 80.00th=[40109], 90.00th=[45351], 95.00th=[50594],
     | 99.00th=[59507], 99.50th=[60556], 99.90th=[62653], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  644, max= 1136, per=24.97%, avg=884.36, stdev=173.81, samples=11
   iops        : min=  161, max=  284, avg=221.09, stdev=43.45, samples=11
  lat (msec)   : 10=0.04%, 20=0.04%, 50=94.65%, 100=5.27%
  cpu          : usr=0.51%, sys=6.52%, ctx=4654, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184817: Tue Jun  9 14:53:59 2026
  read: IOPS=214, BW=858KiB/s (879kB/s)(5060KiB/5897msec)
    slat (usec): min=992, max=9554, avg=2492.51, stdev=887.73
    clat (usec): min=16396, max=68978, avg=35075.74, stdev=7529.08
     lat (usec): min=17627, max=71918, avg=37570.11, stdev=7907.90
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[25822], 10.00th=[27395], 20.00th=[29492],
     | 30.00th=[30802], 40.00th=[32113], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[36963], 80.00th=[40109], 90.00th=[45351], 95.00th=[50594],
     | 99.00th=[59507], 99.50th=[63177], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  620, max= 1016, per=25.03%, avg=851.91, stdev=147.87, samples=11
   iops        : min=  155, max=  254, avg=212.91, stdev=37.05, samples=11
  write: IOPS=219, BW=878KiB/s (899kB/s)(5180KiB/5897msec); 0 zone resets
    slat (usec): min=46, max=2898, avg=169.48, stdev=128.98
    clat (usec): min=7057, max=67390, avg=35371.69, stdev=7194.40
     lat (usec): min=7166, max=67520, avg=35542.21, stdev=7202.02
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28443], 20.00th=[30016],
     | 30.00th=[31065], 40.00th=[32375], 50.00th=[33817], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[39584], 90.00th=[45351], 95.00th=[51119],
     | 99.00th=[58983], 99.50th=[61604], 99.90th=[64750], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  616, max= 1048, per=24.68%, avg=874.27, stdev=136.84, samples=11
   iops        : min=  154, max=  262, avg=218.45, stdev=34.31, samples=11
  lat (msec)   : 10=0.04%, 20=0.23%, 50=93.98%, 100=5.74%
  cpu          : usr=0.49%, sys=6.51%, ctx=4711, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3403KiB/s (3485kB/s), 846KiB/s-858KiB/s (866kB/s-879kB/s), io=19.6MiB (20.5MB), run=5854-5897msec
  WRITE: bw=3543KiB/s (3628kB/s), 878KiB/s-899KiB/s (899kB/s-921kB/s), io=20.4MiB (21.4MB), run=5854-5897msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=184822: Tue Jun  9 14:54:03 2026
  read: IOPS=9, BW=385MiB/s (403MB/s)(40.0MiB/104msec)
    slat (nsec): min=9458.5k, max=9458.5k, avg=9458514.00, stdev= 0.00
    clat (nsec): min=87284k, max=87284k, avg=87283863.00, stdev= 0.00
     lat (nsec): min=96747k, max=96747k, avg=96746775.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[87557],  5.00th=[87557], 10.00th=[87557], 20.00th=[87557],
     | 30.00th=[87557], 40.00th=[87557], 50.00th=[87557], 60.00th=[87557],
     | 70.00th=[87557], 80.00th=[87557], 90.00th=[87557], 95.00th=[87557],
     | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557],
     | 99.99th=[87557]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.74%, ctx=9, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=385MiB/s (403MB/s), 385MiB/s-385MiB/s (403MB/s-403MB/s), io=40.0MiB (41.9MB), run=104-104msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout