== sanity test 398c: run fio to test AIO ================= 23:38:02 (1708835882)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0874332 s, 480 MB/s
osc.lustre-OST0000-osc-ffff8800b5933800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=9348: Sat Feb 24 23:38:07 2024
  write: IOPS=659, BW=2636KiB/s (2700kB/s)(10.0MiB/3884msec)
    slat (usec): min=12, max=825, avg=51.22, stdev=38.15
    clat (usec): min=2638, max=44082, avg=24130.18, stdev=4823.84
     lat (usec): min=2671, max=44107, avg=24182.92, stdev=4826.90
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[17957], 10.00th=[19006], 20.00th=[20317],
     | 30.00th=[21365], 40.00th=[22414], 50.00th=[23462], 60.00th=[24773],
     | 70.00th=[26084], 80.00th=[27395], 90.00th=[30278], 95.00th=[33162],
     | 99.00th=[38536], 99.50th=[39060], 99.90th=[44303], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min= 2227, max= 2848, per=24.65%, avg=2595.00, stdev=243.18, samples=7
   iops        : min=  556, max=  712, avg=648.57, stdev=61.00, samples=7
  lat (msec)   : 4=0.04%, 10=0.39%, 20=16.29%, 50=83.28%
  cpu          : usr=0.75%, sys=3.94%, ctx=1654, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9349: Sat Feb 24 23:38:07 2024
  write: IOPS=658, BW=2634KiB/s (2697kB/s)(10.0MiB/3888msec)
    slat (usec): min=11, max=786, avg=50.39, stdev=38.92
    clat (usec): min=3519, max=40638, avg=24146.53, stdev=4709.68
     lat (usec): min=3577, max=40696, avg=24198.38, stdev=4713.28
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[17957], 10.00th=[19006], 20.00th=[20317],
     | 30.00th=[21365], 40.00th=[22414], 50.00th=[23725], 60.00th=[24773],
     | 70.00th=[26084], 80.00th=[27395], 90.00th=[30278], 95.00th=[33162],
     | 99.00th=[38536], 99.50th=[39060], 99.90th=[40109], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 2251, max= 2800, per=24.58%, avg=2588.14, stdev=219.89, samples=7
   iops        : min=  562, max=  700, avg=646.86, stdev=55.15, samples=7
  lat (msec)   : 4=0.08%, 20=16.64%, 50=83.28%
  cpu          : usr=0.98%, sys=3.76%, ctx=1603, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9350: Sat Feb 24 23:38:07 2024
  write: IOPS=658, BW=2635KiB/s (2698kB/s)(10.0MiB/3886msec)
    slat (usec): min=11, max=1279, avg=52.79, stdev=46.09
    clat (usec): min=4756, max=43902, avg=24129.83, stdev=4686.62
     lat (usec): min=4835, max=43931, avg=24184.05, stdev=4690.40
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[17433], 10.00th=[19006], 20.00th=[20317],
     | 30.00th=[21365], 40.00th=[22414], 50.00th=[23725], 60.00th=[24773],
     | 70.00th=[25822], 80.00th=[27395], 90.00th=[30278], 95.00th=[33162],
     | 99.00th=[38536], 99.50th=[39060], 99.90th=[40633], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 2232, max= 2816, per=24.62%, avg=2592.00, stdev=227.92, samples=7
   iops        : min=  558, max=  704, avg=648.00, stdev=56.98, samples=7
  lat (msec)   : 10=0.08%, 20=16.95%, 50=82.97%
  cpu          : usr=0.90%, sys=3.84%, ctx=1704, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9351: Sat Feb 24 23:38:07 2024
  write: IOPS=658, BW=2632KiB/s (2696kB/s)(10.0MiB/3890msec)
    slat (usec): min=11, max=618, avg=50.10, stdev=34.65
    clat (usec): min=5044, max=43926, avg=24157.76, stdev=4697.49
     lat (usec): min=5116, max=43954, avg=24209.31, stdev=4701.62
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[17433], 10.00th=[18744], 20.00th=[20317],
     | 30.00th=[21365], 40.00th=[22414], 50.00th=[23462], 60.00th=[25035],
     | 70.00th=[26084], 80.00th=[27395], 90.00th=[30278], 95.00th=[33162],
     | 99.00th=[38536], 99.50th=[39060], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[43779]
   bw (  KiB/s): min= 2251, max= 2824, per=24.55%, avg=2584.71, stdev=224.91, samples=7
   iops        : min=  562, max=  706, avg=646.00, stdev=56.42, samples=7
  lat (msec)   : 10=0.04%, 20=16.88%, 50=83.09%
  cpu          : usr=0.67%, sys=4.01%, ctx=1649, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.3MiB/s (10.8MB/s), 2632KiB/s-2636KiB/s (2696kB/s-2700kB/s), io=40.0MiB (41.9MB), run=3884-3890msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=9363: Sat Feb 24 23:38:11 2024
   read: IOPS=449, BW=1799KiB/s (1842kB/s)(5048KiB/2806msec)
    slat (usec): min=17, max=2404, avg=1145.84, stdev=275.24
    clat (usec): min=1435, max=31931, avg=16528.04, stdev=2708.17
     lat (usec): min=1531, max=32832, avg=17675.55, stdev=2794.64
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[12780], 10.00th=[13435], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19530], 95.00th=[20841],
     | 99.00th=[23725], 99.50th=[24773], 99.90th=[28705], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 1680, max= 1880, per=24.99%, avg=1777.60, stdev=76.82, samples=5
   iops        : min=  420, max=  470, avg=444.40, stdev=19.20, samples=5
  write: IOPS=462, BW=1850KiB/s (1895kB/s)(5192KiB/2806msec)
    slat (usec): min=12, max=1393, avg=44.48, stdev=45.50
    clat (usec): min=5928, max=27676, avg=17275.14, stdev=2146.51
     lat (usec): min=5996, max=27708, avg=17320.92, stdev=2146.36
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14353], 10.00th=[14877], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19792], 95.00th=[21103],
     | 99.00th=[23987], 99.50th=[25822], 99.90th=[27395], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1744, max= 1960, per=25.13%, avg=1860.80, stdev=79.32, samples=5
   iops        : min=  436, max=  490, avg=465.20, stdev=19.83, samples=5
  lat (msec)   : 2=0.12%, 4=0.04%, 10=0.51%, 20=90.98%, 50=8.36%
  cpu          : usr=0.86%, sys=6.95%, ctx=2838, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9364: Sat Feb 24 23:38:11 2024
   read: IOPS=440, BW=1764KiB/s (1806kB/s)(4976KiB/2821msec)
    slat (usec): min=648, max=2413, avg=1154.34, stdev=254.62
    clat (usec): min=8588, max=27536, avg=16624.52, stdev=2481.77
     lat (usec): min=9684, max=28966, avg=17780.84, stdev=2532.90
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12780], 10.00th=[13566], 20.00th=[14484],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19530], 95.00th=[21103],
     | 99.00th=[23462], 99.50th=[23987], 99.90th=[27395], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1608, max= 1904, per=24.36%, avg=1732.80, stdev=120.08, samples=5
   iops        : min=  402, max=  476, avg=433.20, stdev=30.02, samples=5
  write: IOPS=466, BW=1866KiB/s (1911kB/s)(5264KiB/2821msec)
    slat (usec): min=11, max=1172, avg=42.69, stdev=35.76
    clat (usec): min=8539, max=27454, avg=17275.62, stdev=1991.14
     lat (usec): min=8580, max=27488, avg=17319.44, stdev=1990.90
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14353], 10.00th=[14877], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16712], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[23725], 99.90th=[24511], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1808, max= 1928, per=25.22%, avg=1867.20, stdev=49.51, samples=5
   iops        : min=  452, max=  482, avg=466.80, stdev=12.38, samples=5
  lat (msec)   : 10=0.16%, 20=91.72%, 50=8.12%
  cpu          : usr=0.57%, sys=7.13%, ctx=2687, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9365: Sat Feb 24 23:38:11 2024
   read: IOPS=442, BW=1769KiB/s (1812kB/s)(4984KiB/2817msec)
    slat (usec): min=678, max=2673, avg=1164.40, stdev=258.73
    clat (usec): min=8713, max=26644, avg=16611.50, stdev=2390.46
     lat (usec): min=9895, max=27777, avg=17777.78, stdev=2435.59
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13042], 10.00th=[13698], 20.00th=[14484],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[23725], 99.50th=[25560], 99.90th=[26346], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min= 1720, max= 1808, per=24.70%, avg=1756.80, stdev=36.04, samples=5
   iops        : min=  430, max=  452, avg=439.20, stdev= 9.01, samples=5
  write: IOPS=466, BW=1866KiB/s (1911kB/s)(5256KiB/2817msec)
    slat (usec): min=10, max=520, avg=44.46, stdev=28.64
    clat (usec): min=11124, max=26103, avg=17220.23, stdev=2058.14
     lat (usec): min=11193, max=26137, avg=17265.94, stdev=2060.15
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14091], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19792], 95.00th=[20841],
     | 99.00th=[23200], 99.50th=[23725], 99.90th=[25297], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 1624, max= 2072, per=25.02%, avg=1852.80, stdev=172.38, samples=5
   iops        : min=  406, max=  518, avg=463.20, stdev=43.10, samples=5
  lat (msec)   : 10=0.08%, 20=91.52%, 50=8.40%
  cpu          : usr=1.10%, sys=6.75%, ctx=2813, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9366: Sat Feb 24 23:38:11 2024
   read: IOPS=448, BW=1794KiB/s (1837kB/s)(5060KiB/2820msec)
    slat (usec): min=636, max=2150, avg=1147.26, stdev=253.01
    clat (usec): min=10271, max=27438, avg=16653.78, stdev=2419.59
     lat (usec): min=11011, max=28523, avg=17802.72, stdev=2465.30
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[12911], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19792], 95.00th=[20841],
     | 99.00th=[22938], 99.50th=[23200], 99.90th=[25035], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1680, max= 1920, per=25.06%, avg=1782.40, stdev=98.53, samples=5
   iops        : min=  420, max=  480, avg=445.60, stdev=24.63, samples=5
  write: IOPS=459, BW=1837KiB/s (1881kB/s)(5180KiB/2820msec)
    slat (usec): min=11, max=987, avg=45.97, stdev=42.06
    clat (usec): min=11135, max=27626, avg=17222.12, stdev=2126.16
     lat (usec): min=11203, max=27642, avg=17269.59, stdev=2125.84
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14222], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[23725], 99.50th=[25035], 99.90th=[26608], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1736, max= 1912, per=24.61%, avg=1822.40, stdev=83.41, samples=5
   iops        : min=  434, max=  478, avg=455.60, stdev=20.85, samples=5
  lat (msec)   : 20=90.90%, 50=9.10%
  cpu          : usr=0.85%, sys=6.99%, ctx=2912, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7114KiB/s (7285kB/s), 1764KiB/s-1799KiB/s (1806kB/s-1842kB/s), io=19.6MiB (20.5MB), run=2806-2821msec
  WRITE: bw=7406KiB/s (7584kB/s), 1837KiB/s-1866KiB/s (1881kB/s-1911kB/s), io=20.4MiB (21.4MB), run=2806-2821msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=9372: Sat Feb 24 23:38:12 2024
   read: IOPS=10, BW=426MiB/s (446MB/s)(40.0MiB/94msec)
    slat (nsec): min=6852.8k, max=6852.8k, avg=6852773.00, stdev= 0.00
    clat (nsec): min=83676k, max=83676k, avg=83676212.00, stdev= 0.00
     lat (nsec): min=90549k, max=90549k, avg=90549425.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[83362],  5.00th=[83362], 10.00th=[83362], 20.00th=[83362],
     | 30.00th=[83362], 40.00th=[83362], 50.00th=[83362], 60.00th=[83362],
     | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362],
     | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[83362]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.53%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=40.0MiB (41.9MB), run=94-94msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout