== sanity test 398c: run fio to test AIO ================= 11:10:37 (1736266237)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0756133 s, 555 MB/s
osc.lustre-OST0000-osc-ffff8800a798c800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18322: Tue Jan  7 11:10:42 2025
  write: IOPS=822, BW=3289KiB/s (3368kB/s)(10.0MiB/3113msec)
    slat (usec): min=12, max=390, avg=50.74, stdev=29.61
    clat (usec): min=2258, max=39819, avg=19377.79, stdev=7072.53
     lat (usec): min=2327, max=39904, avg=19429.97, stdev=7084.58
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[11994], 10.00th=[12911], 20.00th=[14091],
     | 30.00th=[15008], 40.00th=[15926], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[19006], 80.00th=[27132], 90.00th=[31589], 95.00th=[34341],
     | 99.00th=[36439], 99.50th=[36439], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 2016, max= 4440, per=24.78%, avg=3249.33, stdev=1021.84, samples=6
   iops        : min=  504, max= 1110, avg=812.33, stdev=255.46, samples=6
  lat (msec)   : 4=0.04%, 10=0.51%, 20=71.68%, 50=27.77%
  cpu          : usr=0.87%, sys=4.79%, ctx=1535, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18323: Tue Jan  7 11:10:42 2025
  write: IOPS=819, BW=3279KiB/s (3358kB/s)(10.0MiB/3123msec)
    slat (usec): min=13, max=389, avg=51.34, stdev=32.98
    clat (usec): min=8189, max=39839, avg=19428.16, stdev=7073.24
     lat (usec): min=8240, max=39951, avg=19480.92, stdev=7088.36
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12125], 10.00th=[13042], 20.00th=[14222],
     | 30.00th=[15008], 40.00th=[15926], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[19006], 80.00th=[27132], 90.00th=[31589], 95.00th=[34341],
     | 99.00th=[36439], 99.50th=[37487], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 2048, max= 4464, per=24.64%, avg=3232.00, stdev=1006.82, samples=6
   iops        : min=  512, max= 1116, avg=808.00, stdev=251.70, samples=6
  lat (msec)   : 10=0.55%, 20=71.80%, 50=27.66%
  cpu          : usr=0.64%, sys=5.03%, ctx=1528, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18324: Tue Jan  7 11:10:42 2025
  write: IOPS=819, BW=3279KiB/s (3358kB/s)(10.0MiB/3123msec)
    slat (usec): min=13, max=20689, avg=61.09, stdev=409.21
    clat (usec): min=5935, max=39735, avg=19328.71, stdev=7046.55
     lat (usec): min=7521, max=39790, avg=19391.22, stdev=7054.17
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[12125], 10.00th=[12911], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15926], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[18744], 80.00th=[27132], 90.00th=[31589], 95.00th=[34341],
     | 99.00th=[35914], 99.50th=[36963], 99.90th=[39060], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 2000, max= 4432, per=24.61%, avg=3228.00, stdev=997.38, samples=6
   iops        : min=  500, max= 1108, avg=807.00, stdev=249.34, samples=6
  lat (msec)   : 10=0.70%, 20=72.07%, 50=27.23%
  cpu          : usr=0.74%, sys=5.51%, ctx=1804, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18325: Tue Jan  7 11:10:42 2025
  write: IOPS=820, BW=3281KiB/s (3360kB/s)(10.0MiB/3121msec)
    slat (usec): min=14, max=20464, avg=62.01, stdev=404.62
    clat (usec): min=2284, max=39906, avg=19314.17, stdev=7090.39
     lat (usec): min=2351, max=39998, avg=19377.77, stdev=7096.29
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[11994], 10.00th=[12780], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15926], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[19006], 80.00th=[26870], 90.00th=[31589], 95.00th=[34341],
     | 99.00th=[36439], 99.50th=[36963], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2040, max= 4456, per=24.66%, avg=3234.67, stdev=1006.22, samples=6
   iops        : min=  510, max= 1114, avg=808.67, stdev=251.55, samples=6
  lat (msec)   : 4=0.08%, 10=0.74%, 20=71.76%, 50=27.42%
  cpu          : usr=0.90%, sys=5.03%, ctx=1859, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.8MiB/s (13.4MB/s), 3279KiB/s-3289KiB/s (3358kB/s-3368kB/s), io=40.0MiB (41.9MB), run=3113-3123msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18337: Tue Jan  7 11:10:45 2025
   read: IOPS=635, BW=2543KiB/s (2604kB/s)(5048KiB/1985msec)
    slat (usec): min=14, max=1419, avg=752.16, stdev=143.45
    clat (usec): min=1102, max=23698, avg=11817.51, stdev=1781.84
     lat (usec): min=1134, max=24424, avg=12571.37, stdev=1813.95
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 9372], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[11863], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13173], 90.00th=[13829], 95.00th=[14353],
     | 99.00th=[15401], 99.50th=[15926], 99.90th=[22938], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 2514, max= 2544, per=25.15%, avg=2526.00, stdev=15.87, samples=3
   iops        : min=  628, max=  636, avg=631.33, stdev= 4.16, samples=3
  write: IOPS=653, BW=2616KiB/s (2678kB/s)(5192KiB/1985msec)
    slat (usec): min=13, max=844, avg=42.23, stdev=31.48
    clat (usec): min=1980, max=22817, avg=12138.41, stdev=1485.64
     lat (usec): min=2032, max=22858, avg=12181.70, stdev=1484.70
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9896], 10.00th=[10552], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12125], 60.00th=[12518],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13698], 95.00th=[14222],
     | 99.00th=[15139], 99.50th=[15533], 99.90th=[21365], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 2600, max= 2744, per=25.41%, avg=2656.67, stdev=76.74, samples=3
   iops        : min=  650, max=  686, avg=664.00, stdev=19.29, samples=3
  lat (msec)   : 2=0.16%, 4=0.23%, 10=8.01%, 20=91.33%, 50=0.27%
  cpu          : usr=1.01%, sys=9.53%, ctx=2479, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18338: Tue Jan  7 11:10:45 2025
   read: IOPS=623, BW=2494KiB/s (2554kB/s)(4976KiB/1995msec)
    slat (usec): min=470, max=1209, avg=754.63, stdev=118.46
    clat (usec): min=5707, max=22382, avg=11810.68, stdev=1658.66
     lat (usec): min=6434, max=23097, avg=12566.95, stdev=1666.80
    clat percentiles (usec):
     |  1.00th=[ 8160],  5.00th=[ 9241], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10814], 40.00th=[11338], 50.00th=[11863], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13173], 90.00th=[13829], 95.00th=[14353],
     | 99.00th=[15664], 99.50th=[16712], 99.90th=[21627], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 2408, max= 2504, per=24.45%, avg=2456.00, stdev=48.00, samples=3
   iops        : min=  602, max=  626, avg=614.00, stdev=12.00, samples=3
  write: IOPS=659, BW=2639KiB/s (2702kB/s)(5264KiB/1995msec)
    slat (usec): min=13, max=755, avg=42.29, stdev=29.68
    clat (usec): min=8562, max=17496, avg=12210.02, stdev=1248.03
     lat (usec): min=8599, max=17525, avg=12253.30, stdev=1246.71
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[10159], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[11863], 50.00th=[12256], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13304], 90.00th=[13829], 95.00th=[14353],
     | 99.00th=[15139], 99.50th=[15664], 99.90th=[16188], 99.95th=[17433],
     | 99.99th=[17433]
   bw (  KiB/s): min= 2664, max= 2712, per=25.63%, avg=2680.00, stdev=27.71, samples=3
   iops        : min=  666, max=  678, avg=670.00, stdev= 6.93, samples=3
  lat (msec)   : 10=8.05%, 20=91.88%, 50=0.08%
  cpu          : usr=1.35%, sys=8.98%, ctx=2532, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18339: Tue Jan  7 11:10:45 2025
   read: IOPS=623, BW=2494KiB/s (2554kB/s)(4984KiB/1998msec)
    slat (usec): min=486, max=1372, avg=754.44, stdev=117.17
    clat (usec): min=2886, max=16586, avg=11909.52, stdev=1545.86
     lat (usec): min=3597, max=17505, avg=12665.69, stdev=1548.48
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10552],
     | 30.00th=[11076], 40.00th=[11469], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13829], 95.00th=[14353],
     | 99.00th=[15401], 99.50th=[15926], 99.90th=[16581], 99.95th=[16581],
     | 99.99th=[16581]
   bw (  KiB/s): min= 2488, max= 2560, per=25.06%, avg=2517.33, stdev=37.81, samples=3
   iops        : min=  622, max=  640, avg=629.33, stdev= 9.45, samples=3
  write: IOPS=657, BW=2631KiB/s (2694kB/s)(5256KiB/1998msec)
    slat (usec): min=13, max=207, avg=39.89, stdev=17.81
    clat (usec): min=5832, max=17108, avg=12155.83, stdev=1341.50
     lat (usec): min=5885, max=17127, avg=12196.94, stdev=1340.59
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10028], 10.00th=[10552], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12125], 60.00th=[12518],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13698], 95.00th=[14222],
     | 99.00th=[15533], 99.50th=[15926], 99.90th=[16909], 99.95th=[17171],
     | 99.99th=[17171]
   bw (  KiB/s): min= 2544, max= 2632, per=24.76%, avg=2589.33, stdev=44.06, samples=3
   iops        : min=  636, max=  658, avg=647.33, stdev=11.02, samples=3
  lat (msec)   : 4=0.08%, 10=7.34%, 20=92.58%
  cpu          : usr=0.90%, sys=9.76%, ctx=2298, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18340: Tue Jan  7 11:10:45 2025
   read: IOPS=633, BW=2534KiB/s (2595kB/s)(5060KiB/1997msec)
    slat (usec): min=445, max=1379, avg=757.14, stdev=120.37
    clat (usec): min=7107, max=19827, avg=11855.13, stdev=1601.32
     lat (usec): min=8011, max=20671, avg=12613.95, stdev=1610.61
    clat percentiles (usec):
     |  1.00th=[ 8356],  5.00th=[ 9372], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[11863], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13829], 95.00th=[14353],
     | 99.00th=[15795], 99.50th=[16581], 99.90th=[19006], 99.95th=[19792],
     | 99.99th=[19792]
   bw (  KiB/s): min= 2496, max= 2592, per=25.18%, avg=2529.00, stdev=54.58, samples=3
   iops        : min=  624, max=  648, avg=632.00, stdev=13.86, samples=3
  write: IOPS=648, BW=2594KiB/s (2656kB/s)(5180KiB/1997msec)
    slat (usec): min=14, max=1076, avg=41.82, stdev=37.55
    clat (usec): min=7313, max=24312, avg=12180.43, stdev=1374.74
     lat (usec): min=7358, max=24398, avg=12223.26, stdev=1374.18
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10159], 10.00th=[10552], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[11863], 50.00th=[12125], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13698], 95.00th=[14353],
     | 99.00th=[15401], 99.50th=[15795], 99.90th=[22938], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 2584, max= 2632, per=24.95%, avg=2608.67, stdev=24.03, samples=3
   iops        : min=  646, max=  658, avg=652.00, stdev= 6.00, samples=3
  lat (msec)   : 10=8.09%, 20=91.80%, 50=0.12%
  cpu          : usr=0.85%, sys=9.72%, ctx=2374, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9.81MiB/s (10.3MB/s), 2494KiB/s-2543KiB/s (2554kB/s-2604kB/s), io=19.6MiB (20.5MB), run=1985-1998msec
  WRITE: bw=10.2MiB/s (10.7MB/s), 2594KiB/s-2639KiB/s (2656kB/s-2702kB/s), io=20.4MiB (21.4MB), run=1985-1998msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18345: Tue Jan  7 11:10:46 2025
   read: IOPS=14, BW=571MiB/s (599MB/s)(40.0MiB/70msec)
    slat (nsec): min=6965.4k, max=6965.4k, avg=6965368.00, stdev= 0.00
    clat (nsec): min=59231k, max=59231k, avg=59230883.00, stdev= 0.00
     lat (nsec): min=66209k, max=66209k, avg=66208711.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[58983],  5.00th=[58983], 10.00th=[58983], 20.00th=[58983],
     | 30.00th=[58983], 40.00th=[58983], 50.00th=[58983], 60.00th=[58983],
     | 70.00th=[58983], 80.00th=[58983], 90.00th=[58983], 95.00th=[58983],
     | 99.00th=[58983], 99.50th=[58983], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.14%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=571MiB/s (599MB/s), 571MiB/s-571MiB/s (599MB/s-599MB/s), io=40.0MiB (41.9MB), run=70-70msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck