== sanity test 398c: run fio to test AIO ================= 08:23:16 (1717158196)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.123912 s, 338 MB/s
osc.lustre-OST0000-osc-ffff88006f9dd000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=14965: Fri May 31 08:23:23 2024
  write: IOPS=451, BW=1804KiB/s (1847kB/s)(10.0MiB/5676msec)
    slat (usec): min=16, max=336, avg=63.22, stdev=36.02
    clat (usec): min=3867, max=59311, avg=35363.48, stdev=4652.11
     lat (usec): min=3984, max=59457, avg=35428.27, stdev=4651.23
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28443], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33162], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[39584], 95.00th=[42206],
     | 99.00th=[54789], 99.50th=[56886], 99.90th=[57410], 99.95th=[58983],
     | 99.99th=[59507]
   bw (  KiB/s): min= 1664, max= 1992, per=24.97%, avg=1801.82, stdev=119.14, samples=11
   iops        : min=  416, max=  498, avg=450.45, stdev=29.78, samples=11
  lat (msec)   : 4=0.04%, 50=98.63%, 100=1.33%
  cpu          : usr=0.65%, sys=3.63%, ctx=831, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14966: Fri May 31 08:23:23 2024
  write: IOPS=451, BW=1807KiB/s (1850kB/s)(10.0MiB/5667msec)
    slat (usec): min=23, max=33048, avg=84.66, stdev=652.91
    clat (usec): min=4442, max=68527, avg=35058.27, stdev=4631.80
     lat (usec): min=4534, max=68620, avg=35144.62, stdev=4591.15
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27657], 10.00th=[29230], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[39584], 95.00th=[40633],
     | 99.00th=[48497], 99.50th=[50594], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1664, max= 2040, per=25.06%, avg=1808.00, stdev=122.11, samples=11
   iops        : min=  416, max=  510, avg=452.00, stdev=30.53, samples=11
  lat (msec)   : 10=0.16%, 20=0.20%, 50=98.98%, 100=0.66%
  cpu          : usr=0.60%, sys=4.84%, ctx=1314, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14967: Fri May 31 08:23:23 2024
  write: IOPS=454, BW=1817KiB/s (1860kB/s)(10.0MiB/5636msec)
    slat (usec): min=17, max=461, avg=77.49, stdev=38.59
    clat (usec): min=4536, max=78581, avg=35105.66, stdev=4762.59
     lat (usec): min=4584, max=78620, avg=35184.89, stdev=4762.68
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27657], 10.00th=[29230], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[39584], 95.00th=[40633],
     | 99.00th=[48497], 99.50th=[53216], 99.90th=[64226], 99.95th=[67634],
     | 99.99th=[78119]
   bw (  KiB/s): min= 1672, max= 2016, per=25.20%, avg=1818.18, stdev=121.21, samples=11
   iops        : min=  418, max=  504, avg=454.55, stdev=30.30, samples=11
  lat (msec)   : 10=0.16%, 20=0.20%, 50=98.71%, 100=0.94%
  cpu          : usr=0.96%, sys=4.37%, ctx=1528, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14968: Fri May 31 08:23:23 2024
  write: IOPS=451, BW=1805KiB/s (1848kB/s)(10.0MiB/5674msec)
    slat (usec): min=17, max=33799, avg=84.69, stdev=667.62
    clat (usec): min=21999, max=68304, avg=35098.31, stdev=4431.78
     lat (usec): min=22091, max=68360, avg=35184.76, stdev=4477.50
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27657], 10.00th=[29230], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[39584], 95.00th=[40109],
     | 99.00th=[48497], 99.50th=[53216], 99.90th=[64226], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1664, max= 2040, per=25.04%, avg=1806.55, stdev=127.64, samples=11
   iops        : min=  416, max=  510, avg=451.64, stdev=31.91, samples=11
  lat (msec)   : 50=99.22%, 100=0.78%
  cpu          : usr=0.51%, sys=4.92%, ctx=1223, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7216KiB/s (7390kB/s), 1804KiB/s-1817KiB/s (1847kB/s-1860kB/s), io=40.0MiB (41.9MB), run=5636-5676msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=14983: Fri May 31 08:23:26 2024
   read: IOPS=558, BW=2235KiB/s (2288kB/s)(5048KiB/2259msec)
    slat (usec): min=23, max=2872, avg=840.98, stdev=314.46
    clat (usec): min=1681, max=30572, avg=13266.44, stdev=4049.99
     lat (usec): min=1710, max=32249, avg=14109.29, stdev=4271.82
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[ 8979], 10.00th=[ 9634], 20.00th=[10290],
     | 30.00th=[10945], 40.00th=[11600], 50.00th=[12256], 60.00th=[13042],
     | 70.00th=[14091], 80.00th=[15664], 90.00th=[19006], 95.00th=[21627],
     | 99.00th=[27132], 99.50th=[28705], 99.90th=[30540], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1488, max= 2648, per=24.44%, avg=2164.00, stdev=569.99, samples=4
   iops        : min=  372, max=  662, avg=541.00, stdev=142.50, samples=4
  write: IOPS=574, BW=2298KiB/s (2354kB/s)(5192KiB/2259msec)
    slat (usec): min=12, max=817, avg=42.79, stdev=38.41
    clat (usec): min=2597, max=29960, avg=14005.05, stdev=4059.65
     lat (usec): min=2637, max=30022, avg=14048.86, stdev=4069.12
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[11994], 50.00th=[12518], 60.00th=[13304],
     | 70.00th=[15008], 80.00th=[16909], 90.00th=[19530], 95.00th=[23200],
     | 99.00th=[26870], 99.50th=[28181], 99.90th=[30016], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1544, max= 2824, per=24.47%, avg=2256.00, stdev=573.85, samples=4
   iops        : min=  386, max=  706, avg=564.00, stdev=143.46, samples=4
  lat (msec)   : 2=0.12%, 4=0.23%, 10=10.66%, 20=80.78%, 50=8.20%
  cpu          : usr=1.11%, sys=9.87%, ctx=2109, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14984: Fri May 31 08:23:26 2024
   read: IOPS=549, BW=2198KiB/s (2251kB/s)(4976KiB/2264msec)
    slat (usec): min=28, max=2267, avg=852.45, stdev=293.81
    clat (usec): min=4340, max=31360, avg=13259.83, stdev=3921.57
     lat (usec): min=6084, max=33164, avg=14114.27, stdev=4129.16
    clat percentiles (usec):
     |  1.00th=[ 8225],  5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10290],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[12125], 60.00th=[12780],
     | 70.00th=[13960], 80.00th=[15795], 90.00th=[19006], 95.00th=[21627],
     | 99.00th=[26870], 99.50th=[28181], 99.90th=[31327], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1336, max= 2632, per=23.89%, avg=2116.00, stdev=624.15, samples=4
   iops        : min=  334, max=  658, avg=529.00, stdev=156.04, samples=4
  write: IOPS=581, BW=2325KiB/s (2381kB/s)(5264KiB/2264msec)
    slat (usec): min=12, max=296, avg=41.38, stdev=26.05
    clat (usec): min=3749, max=30563, avg=13970.91, stdev=4086.46
     lat (usec): min=3797, max=30662, avg=14013.19, stdev=4094.70
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12387], 60.00th=[13304],
     | 70.00th=[15139], 80.00th=[17171], 90.00th=[19792], 95.00th=[23200],
     | 99.00th=[26346], 99.50th=[28181], 99.90th=[30278], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1616, max= 2856, per=24.71%, avg=2278.00, stdev=557.57, samples=4
   iops        : min=  404, max=  714, avg=569.50, stdev=139.39, samples=4
  lat (msec)   : 4=0.04%, 10=10.55%, 20=80.86%, 50=8.55%
  cpu          : usr=1.28%, sys=9.72%, ctx=2173, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14985: Fri May 31 08:23:26 2024
   read: IOPS=549, BW=2199KiB/s (2252kB/s)(4984KiB/2266msec)
    slat (usec): min=24, max=2506, avg=859.63, stdev=300.55
    clat (usec): min=4609, max=32140, avg=13362.47, stdev=4074.02
     lat (usec): min=6371, max=33871, avg=14224.37, stdev=4279.15
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 8979], 10.00th=[ 9503], 20.00th=[10290],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[12125], 60.00th=[13042],
     | 70.00th=[14222], 80.00th=[16057], 90.00th=[19006], 95.00th=[21627],
     | 99.00th=[27395], 99.50th=[28967], 99.90th=[30540], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1440, max= 2520, per=23.98%, avg=2124.00, stdev=506.66, samples=4
   iops        : min=  360, max=  630, avg=531.00, stdev=126.66, samples=4
  write: IOPS=579, BW=2320KiB/s (2375kB/s)(5256KiB/2266msec)
    slat (usec): min=12, max=238, avg=42.05, stdev=25.13
    clat (usec): min=6102, max=30728, avg=13896.78, stdev=4046.97
     lat (usec): min=6150, max=30790, avg=13939.86, stdev=4054.26
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12387], 60.00th=[13304],
     | 70.00th=[14615], 80.00th=[16450], 90.00th=[19268], 95.00th=[23462],
     | 99.00th=[27657], 99.50th=[28705], 99.90th=[30278], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1440, max= 2968, per=24.54%, avg=2262.00, stdev=713.43, samples=4
   iops        : min=  360, max=  742, avg=565.50, stdev=178.36, samples=4
  lat (msec)   : 10=10.51%, 20=81.29%, 50=8.20%
  cpu          : usr=1.28%, sys=9.67%, ctx=2141, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14986: Fri May 31 08:23:26 2024
   read: IOPS=559, BW=2237KiB/s (2291kB/s)(5060KiB/2262msec)
    slat (usec): min=28, max=2188, avg=850.01, stdev=306.79
    clat (usec): min=4185, max=32035, avg=13303.08, stdev=3967.51
     lat (usec): min=5690, max=33646, avg=14154.89, stdev=4188.66
    clat percentiles (usec):
     |  1.00th=[ 7832],  5.00th=[ 8979], 10.00th=[ 9634], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[12125], 60.00th=[12911],
     | 70.00th=[13960], 80.00th=[15926], 90.00th=[19006], 95.00th=[21890],
     | 99.00th=[26084], 99.50th=[28181], 99.90th=[30802], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1464, max= 2720, per=24.41%, avg=2162.00, stdev=581.14, samples=4
   iops        : min=  366, max=  680, avg=540.50, stdev=145.28, samples=4
  write: IOPS=572, BW=2290KiB/s (2345kB/s)(5180KiB/2262msec)
    slat (usec): min=12, max=218, avg=41.24, stdev=23.88
    clat (usec): min=7656, max=32086, avg=13910.59, stdev=4114.93
     lat (usec): min=7709, max=32120, avg=13952.88, stdev=4122.53
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9765], 10.00th=[10159], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12256], 60.00th=[13173],
     | 70.00th=[14877], 80.00th=[16712], 90.00th=[19792], 95.00th=[23200],
     | 99.00th=[26346], 99.50th=[27657], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1440, max= 2776, per=24.23%, avg=2234.00, stdev=624.80, samples=4
   iops        : min=  360, max=  694, avg=558.50, stdev=156.20, samples=4
  lat (msec)   : 10=11.41%, 20=80.43%, 50=8.16%
  cpu          : usr=0.84%, sys=10.00%, ctx=2221, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=8856KiB/s (9069kB/s), 2198KiB/s-2237KiB/s (2251kB/s-2291kB/s), io=19.6MiB (20.5MB), run=2259-2266msec
  WRITE: bw=9220KiB/s (9441kB/s), 2290KiB/s-2325KiB/s (2345kB/s-2381kB/s), io=20.4MiB (21.4MB), run=2259-2266msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=14991: Fri May 31 08:23:28 2024
   read: IOPS=7, BW=286MiB/s (300MB/s)(40.0MiB/140msec)
    slat (nsec): min=6824.0k, max=6824.0k, avg=6824986.00, stdev= 0.00
    clat (nsec): min=127800k, max=127800k, avg=127800342.00, stdev= 0.00
     lat (nsec): min=134634k, max=134634k, avg=134634093.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  128],  5.00th=[  128], 10.00th=[  128], 20.00th=[  128],
     | 30.00th=[  128], 40.00th=[  128], 50.00th=[  128], 60.00th=[  128],
     | 70.00th=[  128], 80.00th=[  128], 90.00th=[  128], 95.00th=[  128],
     | 99.00th=[  128], 99.50th=[  128], 99.90th=[  128], 99.95th=[  128],
     | 99.99th=[  128]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.04%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=286MiB/s (300MB/s), 286MiB/s-286MiB/s (300MB/s-300MB/s), io=40.0MiB (41.9MB), run=140-140msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck