== sanity test 398c: run fio to test AIO ================= 19:43:59 (1736988239)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0801462 s, 523 MB/s
osc.lustre-OST0000-osc-ffff8800a7e5e800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=20735: Wed Jan 15 19:44:03 2025
  write: IOPS=816, BW=3265KiB/s (3344kB/s)(10.0MiB/3136msec)
    slat (usec): min=12, max=341, avg=49.37, stdev=33.00
    clat (usec): min=5231, max=37955, avg=19512.08, stdev=7079.35
     lat (usec): min=5269, max=38076, avg=19562.87, stdev=7094.81
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[11469], 10.00th=[13042], 20.00th=[14484],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16909], 60.00th=[17957],
     | 70.00th=[19530], 80.00th=[26870], 90.00th=[32375], 95.00th=[33817],
     | 99.00th=[35914], 99.50th=[36963], 99.90th=[36963], 99.95th=[36963],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1936, max= 4384, per=24.61%, avg=3213.33, stdev=995.18, samples=6
   iops        : min=  484, max= 1096, avg=803.33, stdev=248.79, samples=6
  lat (msec)   : 10=1.33%, 20=69.80%, 50=28.87%
  cpu          : usr=0.80%, sys=4.50%, ctx=1374, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20736: Wed Jan 15 19:44:03 2025
  write: IOPS=817, BW=3269KiB/s (3348kB/s)(10.0MiB/3132msec)
    slat (usec): min=12, max=344, avg=57.24, stdev=37.91
    clat (usec): min=8087, max=37739, avg=19475.68, stdev=7065.80
     lat (usec): min=8148, max=37856, avg=19534.38, stdev=7084.48
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11338], 10.00th=[13042], 20.00th=[14484],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[19268], 80.00th=[25822], 90.00th=[32637], 95.00th=[33817],
     | 99.00th=[36439], 99.50th=[36963], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1960, max= 4392, per=24.67%, avg=3221.33, stdev=997.70, samples=6
   iops        : min=  490, max= 1098, avg=805.33, stdev=249.42, samples=6
  lat (msec)   : 10=0.70%, 20=71.17%, 50=28.12%
  cpu          : usr=0.73%, sys=5.01%, ctx=1950, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20737: Wed Jan 15 19:44:03 2025
  write: IOPS=821, BW=3284KiB/s (3363kB/s)(10.0MiB/3118msec)
    slat (usec): min=12, max=295, avg=53.44, stdev=35.65
    clat (usec): min=1683, max=37742, avg=19402.16, stdev=7237.60
     lat (usec): min=1757, max=37793, avg=19456.92, stdev=7253.07
    clat percentiles (usec):
     |  1.00th=[ 7046],  5.00th=[10945], 10.00th=[12911], 20.00th=[14484],
     | 30.00th=[15533], 40.00th=[16057], 50.00th=[16712], 60.00th=[17957],
     | 70.00th=[19530], 80.00th=[26084], 90.00th=[32375], 95.00th=[33817],
     | 99.00th=[36963], 99.50th=[36963], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1976, max= 4576, per=24.83%, avg=3242.67, stdev=1049.77, samples=6
   iops        : min=  494, max= 1144, avg=810.67, stdev=262.44, samples=6
  lat (msec)   : 2=0.04%, 4=0.23%, 10=2.54%, 20=68.63%, 50=28.55%
  cpu          : usr=0.90%, sys=4.81%, ctx=1681, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20738: Wed Jan 15 19:44:03 2025
  write: IOPS=816, BW=3264KiB/s (3343kB/s)(10.0MiB/3137msec)
    slat (usec): min=12, max=462, avg=53.05, stdev=35.77
    clat (usec): min=7209, max=37641, avg=19509.94, stdev=7054.79
     lat (usec): min=7309, max=37736, avg=19564.35, stdev=7070.23
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11469], 10.00th=[12911], 20.00th=[14484],
     | 30.00th=[15533], 40.00th=[16057], 50.00th=[16909], 60.00th=[17957],
     | 70.00th=[19530], 80.00th=[26084], 90.00th=[32637], 95.00th=[34341],
     | 99.00th=[35914], 99.50th=[36963], 99.90th=[36963], 99.95th=[36963],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1936, max= 4360, per=24.60%, avg=3212.00, stdev=991.57, samples=6
   iops        : min=  484, max= 1090, avg=803.00, stdev=247.89, samples=6
  lat (msec)   : 10=0.35%, 20=71.25%, 50=28.40%
  cpu          : usr=0.77%, sys=4.72%, ctx=1573, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.8MiB/s (13.4MB/s), 3264KiB/s-3284KiB/s (3343kB/s-3363kB/s), io=40.0MiB (41.9MB), run=3118-3137msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=20750: Wed Jan 15 19:44:06 2025
   read: IOPS=639, BW=2557KiB/s (2619kB/s)(5048KiB/1974msec)
    slat (usec): min=446, max=1345, avg=751.76, stdev=126.18
    clat (usec): min=6948, max=22689, avg=11713.39, stdev=1693.31
     lat (usec): min=7815, max=23398, avg=12466.88, stdev=1714.59
    clat percentiles (usec):
     |  1.00th=[ 8225],  5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10290],
     | 30.00th=[10945], 40.00th=[11207], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12387], 80.00th=[13042], 90.00th=[13698], 95.00th=[14484],
     | 99.00th=[16319], 99.50th=[17433], 99.90th=[21890], 99.95th=[22676],
     | 99.99th=[22676]
   bw (  KiB/s): min= 2299, max= 2632, per=24.26%, avg=2465.00, stdev=166.50, samples=3
   iops        : min=  574, max=  658, avg=616.00, stdev=42.00, samples=3
  write: IOPS=657, BW=2630KiB/s (2693kB/s)(5192KiB/1974msec)
    slat (usec): min=13, max=329, avg=43.63, stdev=25.08
    clat (usec): min=7757, max=20354, avg=12008.25, stdev=1424.09
     lat (usec): min=7793, max=20381, avg=12052.99, stdev=1423.39
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9765], 10.00th=[10290], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11600], 50.00th=[11994], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13042], 90.00th=[13698], 95.00th=[14353],
     | 99.00th=[16057], 99.50th=[17171], 99.90th=[19530], 99.95th=[20317],
     | 99.99th=[20317]
   bw (  KiB/s): min= 2499, max= 2656, per=24.59%, avg=2601.00, stdev=88.43, samples=3
   iops        : min=  624, max=  664, avg=650.00, stdev=22.54, samples=3
  lat (msec)   : 10=10.90%, 20=88.98%, 50=0.12%
  cpu          : usr=1.32%, sys=9.28%, ctx=3017, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20751: Wed Jan 15 19:44:06 2025
   read: IOPS=631, BW=2526KiB/s (2587kB/s)(4976KiB/1970msec)
    slat (usec): min=35, max=1504, avg=749.57, stdev=134.11
    clat (usec): min=1596, max=21458, avg=11687.70, stdev=1768.32
     lat (usec): min=2559, max=22369, avg=12439.02, stdev=1800.31
    clat percentiles (usec):
     |  1.00th=[ 8291],  5.00th=[ 9110], 10.00th=[ 9503], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11207], 50.00th=[11600], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[13829], 95.00th=[14353],
     | 99.00th=[16712], 99.50th=[17433], 99.90th=[20317], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2312, max= 2576, per=24.04%, avg=2442.67, stdev=132.02, samples=3
   iops        : min=  578, max=  644, avg=610.67, stdev=33.01, samples=3
  write: IOPS=668, BW=2672KiB/s (2736kB/s)(5264KiB/1970msec)
    slat (usec): min=13, max=569, avg=42.17, stdev=25.93
    clat (usec): min=3083, max=21419, avg=12085.59, stdev=1526.85
     lat (usec): min=3120, max=21453, avg=12128.79, stdev=1524.64
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11600], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13960], 95.00th=[14615],
     | 99.00th=[16712], 99.50th=[17433], 99.90th=[19792], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2512, max= 2712, per=25.01%, avg=2645.33, stdev=115.47, samples=3
   iops        : min=  628, max=  678, avg=661.33, stdev=28.87, samples=3
  lat (msec)   : 2=0.04%, 4=0.08%, 10=11.29%, 20=88.48%, 50=0.12%
  cpu          : usr=1.37%, sys=9.19%, ctx=2767, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20752: Wed Jan 15 19:44:06 2025
   read: IOPS=630, BW=2524KiB/s (2584kB/s)(4984KiB/1975msec)
    slat (usec): min=491, max=1250, avg=754.03, stdev=124.58
    clat (usec): min=5342, max=19564, avg=11705.64, stdev=1657.80
     lat (usec): min=5959, max=20289, avg=12461.34, stdev=1680.49
    clat percentiles (usec):
     |  1.00th=[ 8160],  5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10159],
     | 30.00th=[10814], 40.00th=[11338], 50.00th=[11731], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[13829], 95.00th=[14484],
     | 99.00th=[15664], 99.50th=[16057], 99.90th=[16909], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min= 2368, max= 2576, per=24.54%, avg=2493.33, stdev=110.37, samples=3
   iops        : min=  592, max=  644, avg=623.33, stdev=27.59, samples=3
  write: IOPS=665, BW=2661KiB/s (2725kB/s)(5256KiB/1975msec)
    slat (usec): min=12, max=385, avg=42.30, stdev=24.61
    clat (usec): min=2323, max=18564, avg=12051.80, stdev=1462.65
     lat (usec): min=2380, max=18602, avg=12095.10, stdev=1460.92
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11731], 50.00th=[11994], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13173], 90.00th=[13960], 95.00th=[14484],
     | 99.00th=[16057], 99.50th=[16909], 99.90th=[17957], 99.95th=[18482],
     | 99.99th=[18482]
   bw (  KiB/s): min= 2384, max= 2744, per=24.28%, avg=2568.00, stdev=180.13, samples=3
   iops        : min=  596, max=  686, avg=642.00, stdev=45.03, samples=3
  lat (msec)   : 4=0.04%, 10=10.98%, 20=88.98%
  cpu          : usr=1.72%, sys=8.71%, ctx=2738, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20753: Wed Jan 15 19:44:06 2025
   read: IOPS=643, BW=2572KiB/s (2634kB/s)(5060KiB/1967msec)
    slat (usec): min=12, max=1446, avg=750.06, stdev=139.61
    clat (usec): min=1094, max=19637, avg=11675.51, stdev=1816.56
     lat (usec): min=1129, max=20463, avg=12427.36, stdev=1849.62
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 8979], 10.00th=[ 9503], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11207], 50.00th=[11600], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[13829], 95.00th=[14746],
     | 99.00th=[16909], 99.50th=[17433], 99.90th=[18482], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min= 2320, max= 2624, per=24.70%, avg=2509.33, stdev=165.18, samples=3
   iops        : min=  580, max=  656, avg=627.33, stdev=41.30, samples=3
  write: IOPS=658, BW=2633KiB/s (2697kB/s)(5180KiB/1967msec)
    slat (usec): min=12, max=792, avg=45.20, stdev=36.12
    clat (usec): min=5130, max=24238, avg=12052.70, stdev=1476.92
     lat (usec): min=5155, max=24281, avg=12098.97, stdev=1478.40
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[10028], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11731], 50.00th=[11994], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13042], 90.00th=[13829], 95.00th=[14484],
     | 99.00th=[15533], 99.50th=[16057], 99.90th=[22676], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 2536, max= 2624, per=24.50%, avg=2592.00, stdev=48.66, samples=3
   iops        : min=  634, max=  656, avg=648.00, stdev=12.17, samples=3
  lat (msec)   : 2=0.08%, 10=11.02%, 20=88.79%, 50=0.12%
  cpu          : usr=0.92%, sys=9.61%, ctx=2917, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9.92MiB/s (10.4MB/s), 2524KiB/s-2572KiB/s (2584kB/s-2634kB/s), io=19.6MiB (20.5MB), run=1967-1975msec
  WRITE: bw=10.3MiB/s (10.8MB/s), 2630KiB/s-2672KiB/s (2693kB/s-2736kB/s), io=20.4MiB (21.4MB), run=1967-1975msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=20758: Wed Jan 15 19:44:07 2025
   read: IOPS=12, BW=519MiB/s (545MB/s)(40.0MiB/77msec)
    slat (nsec): min=11760k, max=11760k, avg=11760038.00, stdev= 0.00
    clat (nsec): min=61698k, max=61698k, avg=61698431.00, stdev= 0.00
     lat (nsec): min=73470k, max=73470k, avg=73470413.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[61604],  5.00th=[61604], 10.00th=[61604], 20.00th=[61604],
     | 30.00th=[61604], 40.00th=[61604], 50.00th=[61604], 60.00th=[61604],
     | 70.00th=[61604], 80.00th=[61604], 90.00th=[61604], 95.00th=[61604],
     | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[61604]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=15.79%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=40.0MiB (41.9MB), run=77-77msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck