== sanity test 398c: run fio to test AIO ================= 19:32:22 (1736987542)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0894721 s, 469 MB/s
osc.lustre-OST0000-osc-ffff880000021000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=17932: Wed Jan 15 19:32:26 2025
  write: IOPS=949, BW=3800KiB/s (3891kB/s)(10.0MiB/2695msec)
    slat (usec): min=11, max=300, avg=43.81, stdev=20.97
    clat (usec): min=3469, max=31445, avg=16757.97, stdev=2993.72
     lat (usec): min=3542, max=31512, avg=16803.03, stdev=2994.12
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[11469], 10.00th=[13829], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20317],
     | 99.00th=[29754], 99.50th=[31327], 99.90th=[31327], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 3344, max= 4464, per=24.97%, avg=3793.60, stdev=416.09, samples=5
   iops        : min=  836, max= 1116, avg=948.40, stdev=104.02, samples=5
  lat (msec)   : 4=0.04%, 10=2.58%, 20=91.09%, 50=6.29%
  cpu          : usr=1.00%, sys=5.01%, ctx=1331, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17933: Wed Jan 15 19:32:26 2025
  write: IOPS=954, BW=3817KiB/s (3908kB/s)(10.0MiB/2683msec)
    slat (usec): min=12, max=241, avg=42.87, stdev=19.69
    clat (usec): min=2092, max=32533, avg=16694.55, stdev=2997.17
     lat (usec): min=2128, max=32568, avg=16738.68, stdev=2998.66
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[11338], 10.00th=[13566], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[27657], 99.50th=[30540], 99.90th=[32637], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 3328, max= 4576, per=25.09%, avg=3811.20, stdev=463.95, samples=5
   iops        : min=  832, max= 1144, avg=952.80, stdev=115.99, samples=5
  lat (msec)   : 4=0.20%, 10=2.93%, 20=91.29%, 50=5.59%
  cpu          : usr=1.04%, sys=4.85%, ctx=1291, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17934: Wed Jan 15 19:32:26 2025
  write: IOPS=949, BW=3800KiB/s (3891kB/s)(10.0MiB/2695msec)
    slat (usec): min=11, max=388, avg=41.85, stdev=20.61
    clat (usec): min=2612, max=31302, avg=16759.74, stdev=2925.40
     lat (usec): min=2665, max=31414, avg=16802.76, stdev=2927.50
    clat percentiles (usec):
     |  1.00th=[ 8356],  5.00th=[11469], 10.00th=[13960], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20317],
     | 99.00th=[30278], 99.50th=[31065], 99.90th=[31327], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 3328, max= 4432, per=24.94%, avg=3788.80, stdev=403.45, samples=5
   iops        : min=  832, max= 1108, avg=947.20, stdev=100.86, samples=5
  lat (msec)   : 4=0.04%, 10=2.62%, 20=91.09%, 50=6.25%
  cpu          : usr=1.15%, sys=4.64%, ctx=1247, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17935: Wed Jan 15 19:32:26 2025
  write: IOPS=949, BW=3798KiB/s (3889kB/s)(10.0MiB/2696msec)
    slat (usec): min=12, max=283, avg=41.03, stdev=20.04
    clat (usec): min=5150, max=30623, avg=16763.17, stdev=2857.47
     lat (usec): min=5187, max=30688, avg=16805.32, stdev=2859.05
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[11469], 10.00th=[13829], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[28443], 99.50th=[30278], 99.90th=[30540], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 3328, max= 4424, per=24.87%, avg=3777.60, stdev=400.22, samples=5
   iops        : min=  832, max= 1106, avg=944.40, stdev=100.05, samples=5
  lat (msec)   : 10=2.58%, 20=91.88%, 50=5.55%
  cpu          : usr=0.74%, sys=4.75%, ctx=1046, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=14.8MiB/s (15.6MB/s), 3798KiB/s-3817KiB/s (3889kB/s-3908kB/s), io=40.0MiB (41.9MB), run=2683-2696msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=17947: Wed Jan 15 19:32:29 2025
   read: IOPS=615, BW=2461KiB/s (2520kB/s)(5048KiB/2051msec)
    slat (usec): min=511, max=1620, avg=790.77, stdev=133.93
    clat (usec): min=1764, max=17619, avg=12127.95, stdev=1732.37
     lat (usec): min=2760, max=18546, avg=12920.30, stdev=1734.83
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 9372], 10.00th=[10028], 20.00th=[10683],
     | 30.00th=[11207], 40.00th=[11600], 50.00th=[12125], 60.00th=[12649],
     | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353], 95.00th=[14877],
     | 99.00th=[15795], 99.50th=[16188], 99.90th=[17171], 99.95th=[17695],
     | 99.99th=[17695]
   bw (  KiB/s): min= 2336, max= 2504, per=25.07%, avg=2452.00, stdev=79.60, samples=4
   iops        : min=  584, max=  626, avg=613.00, stdev=19.90, samples=4
  write: IOPS=632, BW=2531KiB/s (2592kB/s)(5192KiB/2051msec)
    slat (usec): min=12, max=329, avg=43.20, stdev=23.62
    clat (usec): min=3247, max=17539, avg=12527.86, stdev=1259.91
     lat (usec): min=3359, max=17586, avg=12572.15, stdev=1258.74
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10552], 10.00th=[10945], 20.00th=[11469],
     | 30.00th=[11863], 40.00th=[12256], 50.00th=[12518], 60.00th=[12911],
     | 70.00th=[13173], 80.00th=[13566], 90.00th=[14091], 95.00th=[14484],
     | 99.00th=[15270], 99.50th=[15664], 99.90th=[16909], 99.95th=[17433],
     | 99.99th=[17433]
   bw (  KiB/s): min= 2440, max= 2680, per=24.83%, avg=2528.00, stdev=106.53, samples=4
   iops        : min=  610, max=  670, avg=632.00, stdev=26.63, samples=4
  lat (msec)   : 2=0.04%, 4=0.12%, 10=5.35%, 20=94.49%
  cpu          : usr=1.27%, sys=9.32%, ctx=2610, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17948: Wed Jan 15 19:32:29 2025
   read: IOPS=608, BW=2436KiB/s (2494kB/s)(4976KiB/2043msec)
    slat (usec): min=29, max=1481, avg=790.68, stdev=137.82
    clat (usec): min=992, max=21407, avg=12040.89, stdev=1768.53
     lat (usec): min=1023, max=22117, avg=12833.41, stdev=1784.26
    clat percentiles (usec):
     |  1.00th=[ 8225],  5.00th=[ 9372], 10.00th=[ 9896], 20.00th=[10552],
     | 30.00th=[11076], 40.00th=[11600], 50.00th=[11994], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[14222], 95.00th=[14877],
     | 99.00th=[16188], 99.50th=[16581], 99.90th=[19792], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2368, max= 2480, per=24.89%, avg=2434.00, stdev=47.27, samples=4
   iops        : min=  592, max=  620, avg=608.50, stdev=11.82, samples=4
  write: IOPS=644, BW=2577KiB/s (2638kB/s)(5264KiB/2043msec)
    slat (usec): min=11, max=189, avg=43.28, stdev=20.83
    clat (usec): min=4150, max=24671, avg=12603.37, stdev=1505.74
     lat (usec): min=4189, max=24696, avg=12647.95, stdev=1505.85
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[10290], 10.00th=[10814], 20.00th=[11600],
     | 30.00th=[11994], 40.00th=[12387], 50.00th=[12649], 60.00th=[13042],
     | 70.00th=[13304], 80.00th=[13698], 90.00th=[14222], 95.00th=[14615],
     | 99.00th=[15795], 99.50th=[17433], 99.90th=[22152], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 2464, max= 2760, per=25.30%, avg=2576.00, stdev=129.82, samples=4
   iops        : min=  616, max=  690, avg=644.00, stdev=32.46, samples=4
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.04%, 10=6.99%, 20=92.81%, 50=0.12%
  cpu          : usr=1.08%, sys=9.55%, ctx=2632, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17949: Wed Jan 15 19:32:29 2025
   read: IOPS=608, BW=2435KiB/s (2493kB/s)(4984KiB/2047msec)
    slat (usec): min=499, max=1535, avg=788.49, stdev=132.07
    clat (usec): min=791, max=21340, avg=12052.76, stdev=1751.97
     lat (usec): min=1427, max=22051, avg=12842.99, stdev=1755.63
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 9372], 10.00th=[10028], 20.00th=[10683],
     | 30.00th=[11207], 40.00th=[11600], 50.00th=[11994], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13566], 90.00th=[14222], 95.00th=[14746],
     | 99.00th=[15664], 99.50th=[16188], 99.90th=[17171], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2368, max= 2560, per=24.95%, avg=2440.00, stdev=85.42, samples=4
   iops        : min=  592, max=  640, avg=610.00, stdev=21.35, samples=4
  write: IOPS=641, BW=2568KiB/s (2629kB/s)(5256KiB/2047msec)
    slat (usec): min=12, max=754, avg=44.00, stdev=31.90
    clat (usec): min=8416, max=21192, avg=12575.79, stdev=1263.53
     lat (usec): min=8465, max=21211, avg=12620.92, stdev=1261.57
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[10552], 10.00th=[11076], 20.00th=[11600],
     | 30.00th=[11994], 40.00th=[12256], 50.00th=[12518], 60.00th=[12780],
     | 70.00th=[13173], 80.00th=[13566], 90.00th=[14091], 95.00th=[14484],
     | 99.00th=[15401], 99.50th=[15795], 99.90th=[21103], 99.95th=[21103],
     | 99.99th=[21103]
   bw (  KiB/s): min= 2432, max= 2624, per=25.14%, avg=2560.00, stdev=86.90, samples=4
   iops        : min=  608, max=  656, avg=640.00, stdev=21.73, samples=4
  lat (usec)   : 1000=0.04%
  lat (msec)   : 4=0.08%, 10=5.94%, 20=93.83%, 50=0.12%
  cpu          : usr=1.52%, sys=9.14%, ctx=2596, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17950: Wed Jan 15 19:32:29 2025
   read: IOPS=616, BW=2466KiB/s (2525kB/s)(5060KiB/2052msec)
    slat (usec): min=514, max=1773, avg=785.82, stdev=127.03
    clat (usec): min=835, max=17522, avg=12158.34, stdev=1758.24
     lat (usec): min=1681, max=18366, avg=12945.87, stdev=1766.06
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10683],
     | 30.00th=[11207], 40.00th=[11731], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353], 95.00th=[15008],
     | 99.00th=[16319], 99.50th=[16909], 99.90th=[17433], 99.95th=[17433],
     | 99.99th=[17433]
   bw (  KiB/s): min= 2328, max= 2568, per=25.09%, avg=2454.00, stdev=98.28, samples=4
   iops        : min=  582, max=  642, avg=613.50, stdev=24.57, samples=4
  write: IOPS=631, BW=2524KiB/s (2585kB/s)(5180KiB/2052msec)
    slat (usec): min=12, max=957, avg=47.39, stdev=52.56
    clat (usec): min=2913, max=16826, avg=12535.19, stdev=1367.58
     lat (usec): min=2984, max=16859, avg=12583.74, stdev=1367.72
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[10421], 10.00th=[10945], 20.00th=[11469],
     | 30.00th=[11863], 40.00th=[12256], 50.00th=[12518], 60.00th=[12911],
     | 70.00th=[13304], 80.00th=[13566], 90.00th=[14091], 95.00th=[14615],
     | 99.00th=[15926], 99.50th=[16319], 99.90th=[16712], 99.95th=[16909],
     | 99.99th=[16909]
   bw (  KiB/s): min= 2424, max= 2624, per=24.67%, avg=2512.00, stdev=83.39, samples=4
   iops        : min=  606, max=  656, avg=628.00, stdev=20.85, samples=4
  lat (usec)   : 1000=0.04%
  lat (msec)   : 4=0.04%, 10=5.94%, 20=93.98%
  cpu          : usr=1.12%, sys=9.51%, ctx=2762, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9780KiB/s (10.0MB/s), 2435KiB/s-2466KiB/s (2493kB/s-2525kB/s), io=19.6MiB (20.5MB), run=2043-2052msec
  WRITE: bw=9.94MiB/s (10.4MB/s), 2524KiB/s-2577KiB/s (2585kB/s-2638kB/s), io=20.4MiB (21.4MB), run=2043-2052msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=17955: Wed Jan 15 19:32:31 2025
   read: IOPS=12, BW=494MiB/s (518MB/s)(40.0MiB/81msec)
    slat (nsec): min=9437.3k, max=9437.3k, avg=9437274.00, stdev= 0.00
    clat (nsec): min=68299k, max=68299k, avg=68299308.00, stdev= 0.00
     lat (nsec): min=77756k, max=77756k, avg=77756054.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[68682],  5.00th=[68682], 10.00th=[68682], 20.00th=[68682],
     | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[68682],
     | 70.00th=[68682], 80.00th=[68682], 90.00th=[68682], 95.00th=[68682],
     | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.25%, ctx=10, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=40.0MiB (41.9MB), run=81-81msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck