== sanity test 398c: run fio to test AIO ================= 23:27:28 (1784086048)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.158155 s, 265 MB/s
osc.lustre-OST0000-osc-ffff94d1ad4db800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=189222: Tue Jul 14 23:27:37 2026
  write: IOPS=489, BW=1958KiB/s (2005kB/s)(10.0MiB/5229msec); 0 zone resets
    slat (usec): min=37, max=2778, avg=190.81, stdev=111.46
    clat (usec): min=6298, max=63016, avg=32426.92, stdev=4860.23
     lat (usec): min=6670, max=63093, avg=32618.92, stdev=4864.63
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[25560], 10.00th=[27395], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[32113], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[35390], 90.00th=[38011], 95.00th=[40633],
     | 99.00th=[49546], 99.50th=[52691], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[63177]
   bw (  KiB/s): min= 1744, max= 2160, per=25.19%, avg=1958.80, stdev=139.87, samples=10
   iops        : min=  436, max=  540, avg=489.70, stdev=34.97, samples=10
  lat (msec)   : 10=0.12%, 20=0.86%, 50=98.20%, 100=0.82%
  cpu          : usr=0.55%, sys=7.17%, ctx=2549, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189223: Tue Jul 14 23:27:37 2026
  write: IOPS=485, BW=1944KiB/s (1990kB/s)(10.0MiB/5268msec); 0 zone resets
    slat (usec): min=42, max=4392, avg=175.79, stdev=127.04
    clat (usec): min=3708, max=54128, avg=32407.07, stdev=4592.58
     lat (usec): min=3791, max=54306, avg=32584.03, stdev=4602.47
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[25297], 10.00th=[27919], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32900],
     | 70.00th=[34341], 80.00th=[35390], 90.00th=[37487], 95.00th=[39584],
     | 99.00th=[49021], 99.50th=[51119], 99.90th=[53740], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min= 1704, max= 2160, per=24.97%, avg=1941.10, stdev=137.86, samples=10
   iops        : min=  426, max=  540, avg=485.20, stdev=34.44, samples=10
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.55%, 50=98.71%, 100=0.66%
  cpu          : usr=0.51%, sys=7.08%, ctx=2618, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189224: Tue Jul 14 23:27:37 2026
  write: IOPS=486, BW=1945KiB/s (1992kB/s)(10.0MiB/5264msec); 0 zone resets
    slat (usec): min=46, max=24311, avg=188.97, stdev=486.78
    clat (usec): min=3564, max=54197, avg=32396.01, stdev=4736.77
     lat (usec): min=3903, max=54289, avg=32586.00, stdev=4712.80
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[25560], 10.00th=[27395], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31327], 50.00th=[32113], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[35390], 90.00th=[37487], 95.00th=[40109],
     | 99.00th=[49546], 99.50th=[52167], 99.90th=[53740], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min= 1768, max= 2152, per=25.02%, avg=1945.30, stdev=124.85, samples=10
   iops        : min=  442, max=  538, avg=486.30, stdev=31.19, samples=10
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.78%, 50=98.16%, 100=0.94%
  cpu          : usr=0.34%, sys=7.41%, ctx=2566, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189225: Tue Jul 14 23:27:37 2026
  write: IOPS=487, BW=1949KiB/s (1996kB/s)(10.0MiB/5254msec); 0 zone resets
    slat (usec): min=50, max=2850, avg=176.13, stdev=129.29
    clat (usec): min=17219, max=54247, avg=32531.69, stdev=4633.43
     lat (usec): min=17472, max=54420, avg=32708.94, stdev=4637.99
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[26084], 10.00th=[27657], 20.00th=[29492],
     | 30.00th=[30540], 40.00th=[31327], 50.00th=[32113], 60.00th=[32900],
     | 70.00th=[34341], 80.00th=[35390], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[49546], 99.50th=[50594], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[54264]
   bw (  KiB/s): min= 1736, max= 2160, per=25.06%, avg=1948.40, stdev=137.77, samples=10
   iops        : min=  434, max=  540, avg=487.10, stdev=34.44, samples=10
  lat (msec)   : 20=0.47%, 50=98.87%, 100=0.66%
  cpu          : usr=0.67%, sys=6.32%, ctx=2619, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7775KiB/s (7962kB/s), 1944KiB/s-1958KiB/s (1990kB/s-2005kB/s), io=40.0MiB (41.9MB), run=5229-5268msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=189235: Tue Jul 14 23:27:46 2026
  read: IOPS=192, BW=771KiB/s (789kB/s)(5048KiB/6549msec)
    slat (usec): min=1033, max=10801, avg=2749.78, stdev=990.77
    clat (usec): min=9595, max=65154, avg=38929.44, stdev=7709.03
     lat (usec): min=11587, max=71237, avg=41681.37, stdev=7998.90
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27657], 10.00th=[28967], 20.00th=[31851],
     | 30.00th=[34341], 40.00th=[36439], 50.00th=[38536], 60.00th=[40633],
     | 70.00th=[43254], 80.00th=[45351], 90.00th=[49021], 95.00th=[52167],
     | 99.00th=[57410], 99.50th=[60556], 99.90th=[64226], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  672, max=  904, per=25.07%, avg=768.08, stdev=75.25, samples=12
   iops        : min=  168, max=  226, avg=192.00, stdev=18.82, samples=12
  write: IOPS=198, BW=793KiB/s (812kB/s)(5192KiB/6549msec); 0 zone resets
    slat (usec): min=36, max=2797, avg=164.25, stdev=120.12
    clat (usec): min=13229, max=62630, avg=39318.11, stdev=7142.46
     lat (usec): min=13511, max=62790, avg=39483.38, stdev=7152.51
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28967], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[34866], 40.00th=[36963], 50.00th=[38536], 60.00th=[40633],
     | 70.00th=[43254], 80.00th=[45876], 90.00th=[49546], 95.00th=[51643],
     | 99.00th=[57410], 99.50th=[58983], 99.90th=[61080], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  592, max= 1104, per=25.01%, avg=797.75, stdev=129.54, samples=12
   iops        : min=  148, max=  276, avg=199.42, stdev=32.40, samples=12
  lat (msec)   : 10=0.04%, 20=0.08%, 50=91.37%, 100=8.52%
  cpu          : usr=0.53%, sys=5.59%, ctx=4754, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189236: Tue Jul 14 23:27:46 2026
  read: IOPS=191, BW=765KiB/s (784kB/s)(4976KiB/6502msec)
    slat (usec): min=967, max=8698, avg=2783.34, stdev=953.79
    clat (usec): min=4748, max=80179, avg=38859.91, stdev=7725.81
     lat (usec): min=7497, max=84164, avg=41645.19, stdev=8037.53
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27395], 10.00th=[29492], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35914], 50.00th=[38536], 60.00th=[41157],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[48497], 95.00th=[51643],
     | 99.00th=[58459], 99.50th=[61080], 99.90th=[68682], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  688, max=  944, per=24.84%, avg=761.08, stdev=74.35, samples=12
   iops        : min=  172, max=  236, avg=190.25, stdev=18.60, samples=12
  write: IOPS=202, BW=810KiB/s (829kB/s)(5264KiB/6502msec); 0 zone resets
    slat (usec): min=55, max=953, avg=158.28, stdev=82.73
    clat (usec): min=15332, max=76158, avg=38988.46, stdev=6914.83
     lat (usec): min=15482, max=76312, avg=39147.78, stdev=6918.56
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28967], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[36439], 50.00th=[38536], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[48497], 95.00th=[50070],
     | 99.00th=[54789], 99.50th=[58459], 99.90th=[64750], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  608, max= 1056, per=25.62%, avg=817.25, stdev=119.60, samples=12
   iops        : min=  152, max=  264, avg=204.25, stdev=29.88, samples=12
  lat (msec)   : 10=0.04%, 20=0.12%, 50=93.12%, 100=6.72%
  cpu          : usr=0.40%, sys=5.77%, ctx=4715, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189237: Tue Jul 14 23:27:46 2026
  read: IOPS=192, BW=771KiB/s (790kB/s)(4984KiB/6464msec)
    slat (usec): min=53, max=9979, avg=2745.84, stdev=1002.87
    clat (usec): min=4495, max=96498, avg=38915.53, stdev=8238.92
     lat (usec): min=6531, max=99640, avg=41663.37, stdev=8526.39
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27657], 10.00th=[29492], 20.00th=[31851],
     | 30.00th=[33817], 40.00th=[35914], 50.00th=[38011], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[45876], 90.00th=[49021], 95.00th=[52167],
     | 99.00th=[58983], 99.50th=[62653], 99.90th=[92799], 99.95th=[96994],
     | 99.99th=[96994]
   bw (  KiB/s): min=  632, max= 1040, per=25.34%, avg=776.33, stdev=136.00, samples=12
   iops        : min=  158, max=  260, avg=194.08, stdev=34.00, samples=12
  write: IOPS=203, BW=813KiB/s (833kB/s)(5256KiB/6464msec); 0 zone resets
    slat (usec): min=53, max=2819, avg=163.43, stdev=110.96
    clat (usec): min=14933, max=88143, avg=38883.41, stdev=7401.68
     lat (usec): min=15148, max=88322, avg=39047.86, stdev=7400.19
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28967], 10.00th=[30278], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[38011], 60.00th=[40109],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[48497], 95.00th=[51643],
     | 99.00th=[56886], 99.50th=[62653], 99.90th=[83362], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min=  672, max=  968, per=25.44%, avg=811.67, stdev=91.14, samples=12
   iops        : min=  168, max=  242, avg=202.92, stdev=22.79, samples=12
  lat (msec)   : 10=0.04%, 20=0.23%, 50=92.27%, 100=7.46%
  cpu          : usr=0.40%, sys=5.71%, ctx=4697, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189238: Tue Jul 14 23:27:46 2026
  read: IOPS=193, BW=773KiB/s (791kB/s)(5060KiB/6549msec)
    slat (usec): min=993, max=8720, avg=2765.32, stdev=956.19
    clat (usec): min=19831, max=65630, avg=39031.70, stdev=7587.47
     lat (usec): min=22270, max=68063, avg=41798.65, stdev=7868.04
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27657], 10.00th=[29754], 20.00th=[32113],
     | 30.00th=[34341], 40.00th=[36439], 50.00th=[38536], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[45876], 90.00th=[49546], 95.00th=[52167],
     | 99.00th=[56886], 99.50th=[58459], 99.90th=[63177], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  672, max=  960, per=25.25%, avg=773.75, stdev=87.00, samples=12
   iops        : min=  168, max=  240, avg=193.42, stdev=21.75, samples=12
  write: IOPS=197, BW=791KiB/s (810kB/s)(5180KiB/6549msec); 0 zone resets
    slat (usec): min=56, max=673, avg=157.38, stdev=79.24
    clat (usec): min=12582, max=81069, avg=39403.90, stdev=7170.82
     lat (usec): min=12724, max=81358, avg=39562.25, stdev=7173.93
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29230], 10.00th=[30802], 20.00th=[32637],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[38536], 60.00th=[41157],
     | 70.00th=[43254], 80.00th=[45876], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[56361], 99.50th=[57410], 99.90th=[68682], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min=  608, max= 1032, per=24.91%, avg=794.67, stdev=104.66, samples=12
   iops        : min=  152, max=  258, avg=198.67, stdev=26.16, samples=12
  lat (msec)   : 20=0.12%, 50=91.45%, 100=8.44%
  cpu          : usr=0.58%, sys=5.50%, ctx=4734, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3064KiB/s (3138kB/s), 765KiB/s-773KiB/s (784kB/s-791kB/s), io=19.6MiB (20.5MB), run=6464-6549msec
  WRITE: bw=3190KiB/s (3267kB/s), 791KiB/s-813KiB/s (810kB/s-833kB/s), io=20.4MiB (21.4MB), run=6464-6549msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=189242: Tue Jul 14 23:27:50 2026
  read: IOPS=8, BW=328MiB/s (344MB/s)(40.0MiB/122msec)
    slat (nsec): min=9352.7k, max=9352.7k, avg=9352679.00, stdev= 0.00
    clat (nsec): min=106903k, max=106903k, avg=106902822.00, stdev= 0.00
     lat (nsec): min=116261k, max=116261k, avg=116260666.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  107],  5.00th=[  107], 10.00th=[  107], 20.00th=[  107],
     | 30.00th=[  107], 40.00th=[  107], 50.00th=[  107], 60.00th=[  107],
     | 70.00th=[  107], 80.00th=[  107], 90.00th=[  107], 95.00th=[  107],
     | 99.00th=[  107], 99.50th=[  107], 99.90th=[  107], 99.95th=[  107],
     | 99.99th=[  107]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.44%, ctx=7, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=40.0MiB (41.9MB), run=122-122msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout