== sanity test 398c: run fio to test AIO ================= 07:40:39 (1786534839)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.138695 s, 302 MB/s
osc.lustre-OST0000-osc-ffff8a1eedaba000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183388: Wed Aug 12 07:40:48 2026
  write: IOPS=460, BW=1843KiB/s (1887kB/s)(10.0MiB/5556msec); 0 zone resets
    slat (usec): min=47, max=1015, avg=200.42, stdev=107.87
    clat (usec): min=6060, max=61494, avg=34137.98, stdev=3932.42
     lat (usec): min=6325, max=61647, avg=34339.60, stdev=3938.56
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28705], 10.00th=[30016], 20.00th=[31327],
     | 30.00th=[32637], 40.00th=[33424], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[35914], 90.00th=[37487], 95.00th=[39584],
     | 99.00th=[52691], 99.50th=[54789], 99.90th=[61080], 99.95th=[61080],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1752, max= 2016, per=25.01%, avg=1843.20, stdev=78.85, samples=10
   iops        : min=  438, max=  504, avg=460.80, stdev=19.71, samples=10
  lat (msec)   : 10=0.04%, 20=0.16%, 50=98.52%, 100=1.29%
  cpu          : usr=0.56%, sys=7.40%, ctx=2745, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183389: Wed Aug 12 07:40:48 2026
  write: IOPS=460, BW=1843KiB/s (1887kB/s)(10.0MiB/5557msec); 0 zone resets
    slat (usec): min=41, max=909, avg=188.69, stdev=110.19
    clat (usec): min=5746, max=56590, avg=34108.66, stdev=3769.88
     lat (usec): min=5891, max=56675, avg=34298.58, stdev=3778.47
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[29230], 10.00th=[30278], 20.00th=[31589],
     | 30.00th=[32637], 40.00th=[33424], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[35914], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[51119], 99.50th=[53216], 99.90th=[56361], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min= 1736, max= 2048, per=24.94%, avg=1837.90, stdev=99.26, samples=10
   iops        : min=  434, max=  512, avg=459.40, stdev=24.89, samples=10
  lat (msec)   : 10=0.04%, 20=0.23%, 50=98.55%, 100=1.17%
  cpu          : usr=0.61%, sys=6.32%, ctx=2662, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183390: Wed Aug 12 07:40:48 2026
  write: IOPS=464, BW=1857KiB/s (1902kB/s)(10.0MiB/5514msec); 0 zone resets
    slat (usec): min=44, max=866, avg=203.63, stdev=107.63
    clat (usec): min=11414, max=61561, avg=34192.73, stdev=4063.24
     lat (usec): min=11649, max=61706, avg=34397.52, stdev=4066.21
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28967], 10.00th=[30278], 20.00th=[31589],
     | 30.00th=[32637], 40.00th=[33424], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[37487], 95.00th=[39584],
     | 99.00th=[52691], 99.50th=[54264], 99.90th=[61080], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1736, max= 2000, per=25.22%, avg=1858.40, stdev=87.31, samples=10
   iops        : min=  434, max=  500, avg=464.60, stdev=21.83, samples=10
  lat (msec)   : 20=0.43%, 50=98.16%, 100=1.41%
  cpu          : usr=0.49%, sys=7.22%, ctx=2643, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183391: Wed Aug 12 07:40:48 2026
  write: IOPS=460, BW=1843KiB/s (1887kB/s)(10.0MiB/5556msec); 0 zone resets
    slat (usec): min=45, max=26766, avg=217.97, stdev=536.39
    clat (usec): min=18187, max=61076, avg=34181.37, stdev=3873.77
     lat (usec): min=18321, max=61227, avg=34400.44, stdev=3906.49
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28967], 10.00th=[30278], 20.00th=[31589],
     | 30.00th=[32637], 40.00th=[33424], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[52691], 99.50th=[54789], 99.90th=[56886], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min= 1736, max= 2048, per=25.05%, avg=1846.50, stdev=90.61, samples=10
   iops        : min=  434, max=  512, avg=461.60, stdev=22.66, samples=10
  lat (msec)   : 20=0.35%, 50=98.28%, 100=1.37%
  cpu          : usr=0.58%, sys=7.43%, ctx=2841, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7371KiB/s (7548kB/s), 1843KiB/s-1857KiB/s (1887kB/s-1902kB/s), io=40.0MiB (41.9MB), run=5514-5557msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183401: Wed Aug 12 07:40:59 2026
  read: IOPS=164, BW=659KiB/s (674kB/s)(5048KiB/7665msec)
    slat (usec): min=1128, max=8781, avg=3322.13, stdev=1080.84
    clat (usec): min=18468, max=68888, avg=45576.36, stdev=6934.50
     lat (usec): min=25971, max=73905, avg=48900.79, stdev=7098.97
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[33817], 10.00th=[36439], 20.00th=[39584],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[54264], 95.00th=[56886],
     | 99.00th=[61604], 99.50th=[63701], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  557, max=  848, per=25.16%, avg=657.93, stdev=81.48, samples=15
   iops        : min=  139, max=  212, avg=164.47, stdev=20.39, samples=15
  write: IOPS=169, BW=677KiB/s (694kB/s)(5192KiB/7665msec); 0 zone resets
    slat (usec): min=54, max=3244, avg=186.25, stdev=134.20
    clat (usec): min=29278, max=64431, avg=45998.32, stdev=6001.11
     lat (usec): min=29358, max=64647, avg=46185.81, stdev=5999.53
    clat percentiles (usec):
     |  1.00th=[31851],  5.00th=[35390], 10.00th=[38011], 20.00th=[41157],
     | 30.00th=[43254], 40.00th=[44827], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[53740], 95.00th=[56361],
     | 99.00th=[60031], 99.50th=[61080], 99.90th=[62653], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  568, max=  816, per=24.95%, avg=679.47, stdev=64.10, samples=15
   iops        : min=  142, max=  204, avg=169.87, stdev=16.03, samples=15
  lat (msec)   : 20=0.04%, 50=75.47%, 100=24.49%
  cpu          : usr=0.37%, sys=5.78%, ctx=4787, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183402: Wed Aug 12 07:40:59 2026
  read: IOPS=163, BW=653KiB/s (669kB/s)(4976KiB/7622msec)
    slat (usec): min=39, max=12262, avg=3365.20, stdev=1141.39
    clat (usec): min=10073, max=74966, avg=45739.31, stdev=6857.88
     lat (usec): min=10145, max=78102, avg=49106.68, stdev=7068.84
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[34341], 10.00th=[36963], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[53740], 95.00th=[56361],
     | 99.00th=[61080], 99.50th=[62653], 99.90th=[72877], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  544, max=  712, per=24.42%, avg=638.57, stdev=59.07, samples=14
   iops        : min=  136, max=  178, avg=159.64, stdev=14.77, samples=14
  write: IOPS=172, BW=691KiB/s (707kB/s)(5264KiB/7622msec); 0 zone resets
    slat (usec): min=41, max=731, avg=187.83, stdev=97.30
    clat (usec): min=19579, max=63895, avg=45891.76, stdev=5906.07
     lat (usec): min=19678, max=64025, avg=46080.84, stdev=5911.11
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[35914], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[43254], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[50594], 90.00th=[53740], 95.00th=[55313],
     | 99.00th=[59507], 99.50th=[61604], 99.90th=[62653], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  608, max=  808, per=25.24%, avg=687.36, stdev=59.17, samples=14
   iops        : min=  152, max=  202, avg=171.79, stdev=14.80, samples=14
  lat (msec)   : 20=0.27%, 50=74.69%, 100=25.04%
  cpu          : usr=0.37%, sys=5.81%, ctx=4741, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183403: Wed Aug 12 07:40:59 2026
  read: IOPS=162, BW=650KiB/s (666kB/s)(4984KiB/7665msec)
    slat (usec): min=1162, max=15999, avg=3385.17, stdev=1146.43
    clat (usec): min=8902, max=68168, avg=45587.39, stdev=7009.81
     lat (usec): min=11417, max=73498, avg=48974.82, stdev=7193.27
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[34341], 10.00th=[36439], 20.00th=[39584],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51643], 90.00th=[54264], 95.00th=[56886],
     | 99.00th=[61604], 99.50th=[65274], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  506, max=  856, per=24.97%, avg=652.93, stdev=74.45, samples=15
   iops        : min=  126, max=  214, avg=163.20, stdev=18.68, samples=15
  write: IOPS=171, BW=686KiB/s (702kB/s)(5256KiB/7665msec); 0 zone resets
    slat (usec): min=57, max=4101, avg=197.77, stdev=165.94
    clat (usec): min=27906, max=64346, avg=45847.42, stdev=5861.50
     lat (usec): min=28072, max=64541, avg=46046.24, stdev=5871.91
    clat percentiles (usec):
     |  1.00th=[32900],  5.00th=[35914], 10.00th=[38011], 20.00th=[41157],
     | 30.00th=[42730], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[50594], 90.00th=[53216], 95.00th=[55313],
     | 99.00th=[60556], 99.50th=[62653], 99.90th=[63701], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  576, max=  824, per=25.13%, avg=684.40, stdev=78.05, samples=15
   iops        : min=  144, max=  206, avg=171.07, stdev=19.55, samples=15
  lat (msec)   : 10=0.04%, 20=0.04%, 50=75.70%, 100=24.22%
  cpu          : usr=0.51%, sys=5.73%, ctx=4746, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183404: Wed Aug 12 07:40:59 2026
  read: IOPS=164, BW=660KiB/s (675kB/s)(5060KiB/7672msec)
    slat (usec): min=1121, max=8409, avg=3359.73, stdev=1125.94
    clat (usec): min=2997, max=65526, avg=45561.92, stdev=7110.69
     lat (usec): min=7013, max=68702, avg=48923.76, stdev=7342.50
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[33817], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[45876], 60.00th=[47973],
     | 70.00th=[49546], 80.00th=[51119], 90.00th=[54264], 95.00th=[56361],
     | 99.00th=[61604], 99.50th=[63177], 99.90th=[65274], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  544, max=  816, per=25.13%, avg=657.07, stdev=81.46, samples=15
   iops        : min=  136, max=  204, avg=164.27, stdev=20.37, samples=15
  write: IOPS=168, BW=675KiB/s (691kB/s)(5180KiB/7672msec); 0 zone resets
    slat (usec): min=58, max=899, avg=183.08, stdev=95.48
    clat (usec): min=7079, max=63513, avg=45852.78, stdev=6482.04
     lat (usec): min=7283, max=63670, avg=46037.08, stdev=6484.96
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[34341], 10.00th=[38011], 20.00th=[41157],
     | 30.00th=[43254], 40.00th=[44827], 50.00th=[46400], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[50594], 90.00th=[53740], 95.00th=[55837],
     | 99.00th=[59507], 99.50th=[60031], 99.90th=[62653], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  551, max=  864, per=24.68%, avg=671.93, stdev=77.08, samples=15
   iops        : min=  137, max=  216, avg=167.93, stdev=19.36, samples=15
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.31%, 50=72.89%, 100=26.72%
  cpu          : usr=0.48%, sys=5.64%, ctx=4721, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2616KiB/s (2679kB/s), 650KiB/s-660KiB/s (666kB/s-675kB/s), io=19.6MiB (20.5MB), run=7622-7672msec
  WRITE: bw=2723KiB/s (2789kB/s), 675KiB/s-691KiB/s (691kB/s-707kB/s), io=20.4MiB (21.4MB), run=7622-7672msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183409: Wed Aug 12 07:41:03 2026
  read: IOPS=8, BW=339MiB/s (355MB/s)(40.0MiB/118msec)
    slat (nsec): min=11027k, max=11027k, avg=11026936.00, stdev= 0.00
    clat (nsec): min=101297k, max=101297k, avg=101297049.00, stdev= 0.00
     lat (nsec): min=112329k, max=112329k, avg=112328944.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  102],  5.00th=[  102], 10.00th=[  102], 20.00th=[  102],
     | 30.00th=[  102], 40.00th=[  102], 50.00th=[  102], 60.00th=[  102],
     | 70.00th=[  102], 80.00th=[  102], 90.00th=[  102], 95.00th=[  102],
     | 99.00th=[  102], 99.50th=[  102], 99.90th=[  102], 99.95th=[  102],
     | 99.99th=[  102]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.69%, ctx=12, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=40.0MiB (41.9MB), run=118-118msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout