== sanity test 398c: run fio to test AIO ================= 05:56:53 (1786355813)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.125638 s, 334 MB/s
osc.lustre-OST0000-osc-ffff8ef718fb3000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182188: Mon Aug 10 05:57:02 2026
  write: IOPS=476, BW=1904KiB/s (1950kB/s)(10.0MiB/5377msec); 0 zone resets
    slat (usec): min=50, max=39732, avg=236.89, stdev=790.90
    clat (usec): min=15083, max=62400, avg=32952.69, stdev=4762.75
     lat (usec): min=15273, max=62746, avg=33190.85, stdev=4785.91
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[26870], 10.00th=[27919], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[32113], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[46400], 99.50th=[47973], 99.90th=[56361], 99.95th=[60031],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1640, max= 2248, per=25.04%, avg=1905.40, stdev=208.81, samples=10
   iops        : min=  410, max=  562, avg=476.30, stdev=52.26, samples=10
  lat (msec)   : 20=0.23%, 50=99.49%, 100=0.27%
  cpu          : usr=0.50%, sys=8.56%, ctx=2814, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182189: Mon Aug 10 05:57:02 2026
  write: IOPS=475, BW=1903KiB/s (1948kB/s)(10.0MiB/5382msec); 0 zone resets
    slat (usec): min=52, max=1213, avg=221.64, stdev=118.58
    clat (usec): min=23256, max=54948, avg=32946.47, stdev=4582.73
     lat (usec): min=24019, max=55385, avg=33169.34, stdev=4590.47
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[26870], 10.00th=[28181], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[32113], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[46400], 99.50th=[47973], 99.90th=[51643], 99.95th=[51643],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1624, max= 2232, per=24.98%, avg=1901.20, stdev=219.59, samples=10
   iops        : min=  406, max=  558, avg=475.30, stdev=54.90, samples=10
  lat (msec)   : 50=99.88%, 100=0.12%
  cpu          : usr=0.50%, sys=8.57%, ctx=2778, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182190: Mon Aug 10 05:57:02 2026
  write: IOPS=477, BW=1912KiB/s (1958kB/s)(10.0MiB/5356msec); 0 zone resets
    slat (usec): min=48, max=927, avg=219.71, stdev=123.41
    clat (usec): min=15151, max=63309, avg=33178.54, stdev=5228.91
     lat (usec): min=15468, max=63399, avg=33399.53, stdev=5232.95
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[26870], 10.00th=[28181], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[32113], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39584], 95.00th=[42730],
     | 99.00th=[50070], 99.50th=[60556], 99.90th=[63177], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min= 1632, max= 2248, per=25.12%, avg=1911.70, stdev=202.18, samples=10
   iops        : min=  408, max=  562, avg=477.90, stdev=50.56, samples=10
  lat (msec)   : 20=0.08%, 50=98.91%, 100=1.02%
  cpu          : usr=0.60%, sys=7.69%, ctx=2790, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182191: Mon Aug 10 05:57:02 2026
  write: IOPS=475, BW=1903KiB/s (1948kB/s)(10.0MiB/5382msec); 0 zone resets
    slat (usec): min=43, max=927, avg=219.57, stdev=124.19
    clat (usec): min=6732, max=48127, avg=32880.75, stdev=4634.88
     lat (usec): min=6943, max=48548, avg=33101.41, stdev=4652.96
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[26870], 10.00th=[28181], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[32113], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[45876], 99.50th=[46924], 99.90th=[47973], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min= 1620, max= 2248, per=24.89%, avg=1894.00, stdev=221.86, samples=10
   iops        : min=  405, max=  562, avg=473.50, stdev=55.46, samples=10
  lat (msec)   : 10=0.12%, 20=0.04%, 50=99.84%
  cpu          : usr=0.33%, sys=7.90%, ctx=2902, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7611KiB/s (7793kB/s), 1903KiB/s-1912KiB/s (1948kB/s-1958kB/s), io=40.0MiB (41.9MB), run=5356-5382msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182201: Mon Aug 10 05:57:12 2026
  read: IOPS=198, BW=793KiB/s (813kB/s)(5048KiB/6362msec)
    slat (usec): min=1099, max=16621, avg=2797.72, stdev=964.25
    clat (usec): min=6727, max=70433, avg=37454.75, stdev=6660.03
     lat (usec): min=8056, max=74871, avg=40254.04, stdev=6954.27
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[28181], 10.00th=[30016], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[56361], 99.50th=[56886], 99.90th=[63177], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  592, max=  928, per=24.94%, avg=786.50, stdev=112.24, samples=12
   iops        : min=  148, max=  232, avg=196.58, stdev=28.09, samples=12
  write: IOPS=204, BW=816KiB/s (836kB/s)(5192KiB/6362msec); 0 zone resets
    slat (usec): min=48, max=4668, avg=198.90, stdev=192.95
    clat (usec): min=9566, max=64684, avg=38061.65, stdev=6290.42
     lat (usec): min=9669, max=64750, avg=38261.75, stdev=6293.04
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30278], 10.00th=[31327], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[47449], 95.00th=[50070],
     | 99.00th=[55313], 99.50th=[58459], 99.90th=[63177], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  672, max=  968, per=24.70%, avg=810.92, stdev=82.73, samples=12
   iops        : min=  168, max=  242, avg=202.67, stdev=20.76, samples=12
  lat (msec)   : 10=0.16%, 20=0.16%, 50=94.80%, 100=4.88%
  cpu          : usr=0.42%, sys=6.90%, ctx=4787, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182202: Mon Aug 10 05:57:12 2026
  read: IOPS=195, BW=783KiB/s (801kB/s)(4976KiB/6358msec)
    slat (usec): min=1153, max=14638, avg=2814.50, stdev=977.94
    clat (usec): min=20234, max=63017, avg=37748.42, stdev=6473.90
     lat (usec): min=26714, max=64930, avg=40564.70, stdev=6728.04
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28443], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[56886], 99.50th=[58983], 99.90th=[61080], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  640, max=  920, per=24.71%, avg=779.42, stdev=92.46, samples=12
   iops        : min=  160, max=  230, avg=194.83, stdev=23.13, samples=12
  write: IOPS=206, BW=828KiB/s (848kB/s)(5264KiB/6358msec); 0 zone resets
    slat (usec): min=47, max=896, avg=187.57, stdev=103.77
    clat (usec): min=25847, max=62599, avg=38248.33, stdev=6063.52
     lat (usec): min=26003, max=62711, avg=38437.05, stdev=6065.11
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[30540], 10.00th=[31851], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[35914], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[46924], 95.00th=[51119],
     | 99.00th=[55837], 99.50th=[58459], 99.90th=[60556], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  616, max=  976, per=25.15%, avg=825.58, stdev=96.48, samples=12
   iops        : min=  154, max=  244, avg=206.33, stdev=24.10, samples=12
  lat (msec)   : 50=94.49%, 100=5.51%
  cpu          : usr=0.52%, sys=6.81%, ctx=4647, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182203: Mon Aug 10 05:57:12 2026
  read: IOPS=197, BW=790KiB/s (809kB/s)(4984KiB/6310msec)
    slat (usec): min=46, max=13489, avg=2781.27, stdev=993.98
    clat (usec): min=6964, max=63499, avg=37668.43, stdev=7066.82
     lat (usec): min=8523, max=67039, avg=40451.34, stdev=7337.65
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27919], 10.00th=[29754], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[47449], 95.00th=[50594],
     | 99.00th=[56886], 99.50th=[60556], 99.90th=[63177], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  664, max=  896, per=25.13%, avg=792.50, stdev=80.65, samples=12
   iops        : min=  166, max=  224, avg=198.08, stdev=20.16, samples=12
  write: IOPS=208, BW=833KiB/s (853kB/s)(5256KiB/6310msec); 0 zone resets
    slat (usec): min=47, max=4757, avg=208.72, stdev=202.08
    clat (usec): min=23347, max=71996, avg=38038.94, stdev=6355.49
     lat (usec): min=23542, max=72077, avg=38248.69, stdev=6358.75
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30016], 10.00th=[31327], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[46400], 95.00th=[50594],
     | 99.00th=[58459], 99.50th=[61080], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  616, max= 1016, per=25.25%, avg=828.92, stdev=115.37, samples=12
   iops        : min=  154, max=  254, avg=207.17, stdev=28.85, samples=12
  lat (msec)   : 10=0.12%, 20=0.12%, 50=94.22%, 100=5.55%
  cpu          : usr=0.36%, sys=7.16%, ctx=4763, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182204: Mon Aug 10 05:57:12 2026
  read: IOPS=198, BW=796KiB/s (815kB/s)(5060KiB/6358msec)
    slat (usec): min=1110, max=16056, avg=2777.12, stdev=985.52
    clat (usec): min=20474, max=66427, avg=37575.56, stdev=6578.11
     lat (usec): min=22462, max=70551, avg=40354.12, stdev=6853.32
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28443], 10.00th=[30278], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[46400], 95.00th=[50594],
     | 99.00th=[56361], 99.50th=[57934], 99.90th=[64226], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  528, max=  944, per=25.10%, avg=791.67, stdev=115.41, samples=12
   iops        : min=  132, max=  236, avg=197.92, stdev=28.85, samples=12
  write: IOPS=203, BW=815KiB/s (834kB/s)(5180KiB/6358msec); 0 zone resets
    slat (usec): min=62, max=1025, avg=192.21, stdev=107.25
    clat (usec): min=25535, max=57601, avg=38075.22, stdev=5893.99
     lat (usec): min=25631, max=57846, avg=38268.46, stdev=5892.17
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[30016], 10.00th=[31327], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[43254], 90.00th=[46400], 95.00th=[49546],
     | 99.00th=[53740], 99.50th=[54789], 99.90th=[57410], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  672, max=  912, per=24.73%, avg=812.00, stdev=74.38, samples=12
   iops        : min=  168, max=  228, avg=203.00, stdev=18.60, samples=12
  lat (msec)   : 50=95.23%, 100=4.77%
  cpu          : usr=0.33%, sys=7.06%, ctx=4811, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3154KiB/s (3230kB/s), 783KiB/s-796KiB/s (801kB/s-815kB/s), io=19.6MiB (20.5MB), run=6310-6362msec
  WRITE: bw=3284KiB/s (3363kB/s), 815KiB/s-833KiB/s (834kB/s-853kB/s), io=20.4MiB (21.4MB), run=6310-6362msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182208: Mon Aug 10 05:57:16 2026
  read: IOPS=6, BW=272MiB/s (285MB/s)(40.0MiB/147msec)
    slat (nsec): min=17022k, max=17022k, avg=17021933.00, stdev= 0.00
    clat (nsec): min=124389k, max=124389k, avg=124388715.00, stdev= 0.00
     lat (nsec): min=141417k, max=141417k, avg=141417183.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  125],  5.00th=[  125], 10.00th=[  125], 20.00th=[  125],
     | 30.00th=[  125], 40.00th=[  125], 50.00th=[  125], 60.00th=[  125],
     | 70.00th=[  125], 80.00th=[  125], 90.00th=[  125], 95.00th=[  125],
     | 99.00th=[  125], 99.50th=[  125], 99.90th=[  125], 99.95th=[  125],
     | 99.99th=[  125]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=9.59%, ctx=10, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=40.0MiB (41.9MB), run=147-147msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout