== sanity test 398c: run fio to test AIO ================= 10:06:59 (1773238019)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.132806 s, 316 MB/s
osc.lustre-OST0000-osc-ffff8a1cc9367800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=157053: Wed Mar 11 10:07:08 2026
  write: IOPS=446, BW=1786KiB/s (1829kB/s)(10.0MiB/5733msec); 0 zone resets
    slat (usec): min=25, max=437, avg=100.52, stdev=46.78
    clat (usec): min=10476, max=69681, avg=35582.64, stdev=4657.73
     lat (usec): min=10545, max=69748, avg=35684.46, stdev=4660.77
    clat percentiles (usec):
     |  1.00th=[16450],  5.00th=[29492], 10.00th=[31065], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[41157],
     | 99.00th=[44303], 99.50th=[46400], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1648, max= 2012, per=25.04%, avg=1784.36, stdev=93.51, samples=11
   iops        : min=  412, max=  503, avg=446.09, stdev=23.38, samples=11
  lat (msec)   : 20=1.09%, 50=98.52%, 100=0.39%
  cpu          : usr=0.54%, sys=3.77%, ctx=2391, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=157054: Wed Mar 11 10:07:08 2026
  write: IOPS=446, BW=1786KiB/s (1829kB/s)(10.0MiB/5733msec); 0 zone resets
    slat (usec): min=24, max=598, avg=102.73, stdev=52.86
    clat (usec): min=10630, max=69803, avg=35561.20, stdev=4722.78
     lat (usec): min=10729, max=69881, avg=35665.20, stdev=4725.65
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[29230], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[41157],
     | 99.00th=[44303], 99.50th=[45876], 99.90th=[66323], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1648, max= 2038, per=25.03%, avg=1783.09, stdev=100.59, samples=11
   iops        : min=  412, max=  509, avg=445.73, stdev=25.02, samples=11
  lat (msec)   : 20=1.21%, 50=98.36%, 100=0.43%
  cpu          : usr=0.49%, sys=3.86%, ctx=2472, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=157055: Wed Mar 11 10:07:08 2026
  write: IOPS=446, BW=1784KiB/s (1827kB/s)(10.0MiB/5739msec); 0 zone resets
    slat (usec): min=24, max=575, avg=105.01, stdev=55.56
    clat (usec): min=10648, max=72529, avg=35704.18, stdev=4818.17
     lat (usec): min=10729, max=72628, avg=35810.25, stdev=4818.16
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[29492], 10.00th=[31065], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[49546], 99.50th=[50070], 99.90th=[65274], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1664, max= 2025, per=25.01%, avg=1781.91, stdev=97.51, samples=11
   iops        : min=  416, max=  506, avg=445.45, stdev=24.32, samples=11
  lat (msec)   : 20=1.17%, 50=98.16%, 100=0.66%
  cpu          : usr=0.37%, sys=3.96%, ctx=2498, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=157056: Wed Mar 11 10:07:08 2026
  write: IOPS=445, BW=1781KiB/s (1824kB/s)(10.0MiB/5748msec); 0 zone resets
    slat (usec): min=28, max=26194, avg=114.07, stdev=518.30
    clat (usec): min=3891, max=50177, avg=35461.47, stdev=4399.41
     lat (usec): min=3949, max=50259, avg=35576.61, stdev=4374.41
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[29230], 10.00th=[31065], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[41157],
     | 99.00th=[43779], 99.50th=[45351], 99.90th=[46400], 99.95th=[46924],
     | 99.99th=[50070]
   bw (  KiB/s): min= 1648, max= 1942, per=24.87%, avg=1772.18, stdev=81.43, samples=11
   iops        : min=  412, max=  485, avg=443.00, stdev=20.25, samples=11
  lat (msec)   : 4=0.04%, 10=0.08%, 20=1.33%, 50=98.52%, 100=0.04%
  cpu          : usr=0.59%, sys=4.30%, ctx=2515, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7126KiB/s (7297kB/s), 1781KiB/s-1786KiB/s (1824kB/s-1829kB/s), io=40.0MiB (41.9MB), run=5733-5748msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=157066: Wed Mar 11 10:07:18 2026
  read: IOPS=172, BW=691KiB/s (707kB/s)(5048KiB/7307msec)
    slat (usec): min=1078, max=10723, avg=3338.38, stdev=1237.34
    clat (usec): min=4704, max=67450, avg=43560.53, stdev=7485.00
     lat (usec): min=8130, max=71139, avg=46900.77, stdev=7775.31
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[32375], 10.00th=[34866], 20.00th=[37487],
     | 30.00th=[39584], 40.00th=[41681], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[49546], 90.00th=[52691], 95.00th=[55837],
     | 99.00th=[61604], 99.50th=[63701], 99.90th=[65799], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  568, max=  944, per=24.89%, avg=683.50, stdev=92.47, samples=14
   iops        : min=  142, max=  236, avg=170.86, stdev=23.12, samples=14
  write: IOPS=177, BW=711KiB/s (728kB/s)(5192KiB/7307msec); 0 zone resets
    slat (usec): min=28, max=451, avg=93.00, stdev=49.35
    clat (usec): min=4548, max=64762, avg=43780.29, stdev=6683.80
     lat (usec): min=4694, max=64804, avg=43874.38, stdev=6678.47
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[33162], 10.00th=[35390], 20.00th=[38011],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[47449], 80.00th=[49021], 90.00th=[51643], 95.00th=[54789],
     | 99.00th=[58983], 99.50th=[59507], 99.90th=[61604], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  528, max=  792, per=24.49%, avg=700.07, stdev=70.21, samples=14
   iops        : min=  132, max=  198, avg=175.00, stdev=17.55, samples=14
  lat (msec)   : 10=0.27%, 20=0.16%, 50=82.19%, 100=17.38%
  cpu          : usr=0.42%, sys=3.79%, ctx=4649, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=157067: Wed Mar 11 10:07:18 2026
  read: IOPS=170, BW=681KiB/s (698kB/s)(4976KiB/7302msec)
    slat (usec): min=1127, max=11776, avg=3358.16, stdev=1192.38
    clat (usec): min=10076, max=67827, avg=43593.85, stdev=6855.88
     lat (usec): min=12998, max=71069, avg=46954.00, stdev=7086.65
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[32113], 10.00th=[34866], 20.00th=[38011],
     | 30.00th=[39584], 40.00th=[41681], 50.00th=[43254], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[49021], 90.00th=[52167], 95.00th=[54789],
     | 99.00th=[59507], 99.50th=[62129], 99.90th=[66847], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  576, max=  872, per=24.56%, avg=674.50, stdev=72.69, samples=14
   iops        : min=  144, max=  218, avg=168.57, stdev=18.19, samples=14
  write: IOPS=180, BW=721KiB/s (738kB/s)(5264KiB/7302msec); 0 zone resets
    slat (usec): min=30, max=1867, avg=95.77, stdev=72.94
    clat (usec): min=9965, max=61940, avg=43886.44, stdev=6274.01
     lat (usec): min=10045, max=62041, avg=43983.46, stdev=6273.52
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[33817], 10.00th=[35914], 20.00th=[38536],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[43779], 60.00th=[45876],
     | 70.00th=[47449], 80.00th=[49021], 90.00th=[51643], 95.00th=[54264],
     | 99.00th=[58459], 99.50th=[59507], 99.90th=[60556], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  608, max=  848, per=25.08%, avg=717.00, stdev=75.69, samples=14
   iops        : min=  152, max=  212, avg=179.21, stdev=18.91, samples=14
  lat (msec)   : 10=0.04%, 20=0.04%, 50=83.16%, 100=16.76%
  cpu          : usr=0.53%, sys=3.67%, ctx=4730, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=157068: Wed Mar 11 10:07:18 2026
  read: IOPS=171, BW=684KiB/s (701kB/s)(4984KiB/7282msec)
    slat (usec): min=1244, max=8924, avg=3359.76, stdev=1106.47
    clat (usec): min=3058, max=65199, avg=43462.74, stdev=6674.51
     lat (usec): min=5735, max=69337, avg=46824.14, stdev=6924.15
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[32900], 10.00th=[34866], 20.00th=[37487],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[46924], 80.00th=[49021], 90.00th=[51643], 95.00th=[53740],
     | 99.00th=[59507], 99.50th=[61604], 99.90th=[64750], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  576, max=  880, per=24.85%, avg=682.29, stdev=75.40, samples=14
   iops        : min=  144, max=  220, avg=170.57, stdev=18.85, samples=14
  write: IOPS=180, BW=722KiB/s (739kB/s)(5256KiB/7282msec); 0 zone resets
    slat (usec): min=35, max=515, avg=96.63, stdev=54.16
    clat (usec): min=11455, max=68150, avg=43511.67, stdev=6381.78
     lat (usec): min=11560, max=68244, avg=43609.21, stdev=6378.28
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[33162], 10.00th=[35390], 20.00th=[38536],
     | 30.00th=[40633], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[46924], 80.00th=[48497], 90.00th=[51643], 95.00th=[53216],
     | 99.00th=[57934], 99.50th=[60031], 99.90th=[62129], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  600, max=  856, per=24.92%, avg=712.43, stdev=83.88, samples=14
   iops        : min=  150, max=  214, avg=178.07, stdev=20.97, samples=14
  lat (msec)   : 4=0.04%, 20=0.20%, 50=84.61%, 100=15.16%
  cpu          : usr=0.47%, sys=3.76%, ctx=4652, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=157069: Wed Mar 11 10:07:18 2026
  read: IOPS=174, BW=696KiB/s (713kB/s)(5060KiB/7269msec)
    slat (usec): min=58, max=9661, avg=3374.45, stdev=1175.26
    clat (usec): min=2946, max=64957, avg=43415.93, stdev=7362.90
     lat (usec): min=4887, max=70780, avg=46792.04, stdev=7656.06
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[32375], 10.00th=[34866], 20.00th=[38011],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[49021], 90.00th=[51643], 95.00th=[54264],
     | 99.00th=[59507], 99.50th=[62129], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  520, max=  848, per=25.23%, avg=692.79, stdev=85.10, samples=14
   iops        : min=  130, max=  212, avg=173.14, stdev=21.23, samples=14
  write: IOPS=178, BW=713KiB/s (730kB/s)(5180KiB/7269msec); 0 zone resets
    slat (usec): min=35, max=530, avg=96.30, stdev=52.65
    clat (usec): min=16028, max=68521, avg=43861.91, stdev=6699.11
     lat (usec): min=16136, max=68644, avg=43959.25, stdev=6696.25
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[32900], 10.00th=[35390], 20.00th=[38536],
     | 30.00th=[40633], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[49546], 90.00th=[52167], 95.00th=[55313],
     | 99.00th=[60556], 99.50th=[63177], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  600, max=  904, per=24.72%, avg=706.71, stdev=81.49, samples=14
   iops        : min=  150, max=  226, avg=176.64, stdev=20.34, samples=14
  lat (msec)   : 4=0.08%, 10=0.20%, 20=0.16%, 50=83.01%, 100=16.56%
  cpu          : usr=0.48%, sys=3.78%, ctx=4656, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2746KiB/s (2812kB/s), 681KiB/s-696KiB/s (698kB/s-713kB/s), io=19.6MiB (20.5MB), run=7269-7307msec
  WRITE: bw=2859KiB/s (2928kB/s), 711KiB/s-722KiB/s (728kB/s-739kB/s), io=20.4MiB (21.4MB), run=7269-7307msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=157074: Wed Mar 11 10:07:22 2026
  read: IOPS=9, BW=381MiB/s (399MB/s)(40.0MiB/105msec)
    slat (nsec): min=12473k, max=12473k, avg=12472502.00, stdev= 0.00
    clat (nsec): min=86684k, max=86684k, avg=86683533.00, stdev= 0.00
     lat (nsec): min=99272k, max=99272k, avg=99271676.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[86508],  5.00th=[86508], 10.00th=[86508], 20.00th=[86508],
     | 30.00th=[86508], 40.00th=[86508], 50.00th=[86508], 60.00th=[86508],
     | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508],
     | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.58%, ctx=11, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=381MiB/s (399MB/s), 381MiB/s-381MiB/s (399MB/s-399MB/s), io=40.0MiB (41.9MB), run=105-105msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout