== sanity test 398c: run fio to test AIO ================= 07:20:49 (1715340049)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0919747 s, 456 MB/s
osc.lustre-OST0000-osc-ffff8801323fa000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18187: Fri May 10 07:20:55 2024
  write: IOPS=557, BW=2229KiB/s (2283kB/s)(10.0MiB/4593msec)
    slat (usec): min=13, max=489, avg=67.37, stdev=46.47
    clat (usec): min=4094, max=47081, avg=28558.22, stdev=10398.03
     lat (usec): min=4136, max=47229, avg=28627.31, stdev=10419.77
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[14746], 10.00th=[16581], 20.00th=[18482],
     | 30.00th=[20055], 40.00th=[22414], 50.00th=[26084], 60.00th=[32900],
     | 70.00th=[36439], 80.00th=[40109], 90.00th=[43254], 95.00th=[44827],
     | 99.00th=[45876], 99.50th=[46400], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1480, max= 3712, per=24.80%, avg=2209.78, stdev=819.08, samples=9
   iops        : min=  370, max=  928, avg=552.44, stdev=204.77, samples=9
  lat (msec)   : 10=0.23%, 20=30.08%, 50=69.69%
  cpu          : usr=0.68%, sys=3.92%, ctx=1751, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18188: Fri May 10 07:20:55 2024
  write: IOPS=558, BW=2232KiB/s (2286kB/s)(10.0MiB/4587msec)
    slat (usec): min=12, max=380, avg=67.45, stdev=45.45
    clat (usec): min=2488, max=47147, avg=28523.48, stdev=10554.36
     lat (usec): min=2541, max=47196, avg=28592.57, stdev=10575.43
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[14746], 10.00th=[16581], 20.00th=[18220],
     | 30.00th=[19792], 40.00th=[22676], 50.00th=[26084], 60.00th=[32900],
     | 70.00th=[36963], 80.00th=[40109], 90.00th=[43254], 95.00th=[44827],
     | 99.00th=[46400], 99.50th=[46400], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1504, max= 3792, per=24.94%, avg=2222.22, stdev=844.73, samples=9
   iops        : min=  376, max=  948, avg=555.56, stdev=211.18, samples=9
  lat (msec)   : 4=0.16%, 10=1.02%, 20=29.26%, 50=69.57%
  cpu          : usr=0.65%, sys=4.21%, ctx=1633, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18189: Fri May 10 07:20:55 2024
  write: IOPS=556, BW=2228KiB/s (2281kB/s)(10.0MiB/4597msec)
    slat (usec): min=12, max=669, avg=69.91, stdev=53.88
    clat (usec): min=2528, max=47060, avg=28583.25, stdev=10377.18
     lat (usec): min=2591, max=47253, avg=28654.90, stdev=10402.03
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[14877], 10.00th=[16712], 20.00th=[18482],
     | 30.00th=[20055], 40.00th=[22414], 50.00th=[26346], 60.00th=[32637],
     | 70.00th=[36963], 80.00th=[40109], 90.00th=[43254], 95.00th=[44827],
     | 99.00th=[46400], 99.50th=[46400], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1488, max= 3728, per=24.79%, avg=2209.22, stdev=816.29, samples=9
   iops        : min=  372, max=  932, avg=552.22, stdev=204.04, samples=9
  lat (msec)   : 4=0.04%, 10=0.16%, 20=30.04%, 50=69.77%
  cpu          : usr=0.52%, sys=4.44%, ctx=1734, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18190: Fri May 10 07:20:55 2024
  write: IOPS=556, BW=2228KiB/s (2281kB/s)(10.0MiB/4597msec)
    slat (usec): min=12, max=374, avg=64.65, stdev=44.16
    clat (usec): min=4660, max=47007, avg=28587.42, stdev=10396.04
     lat (usec): min=4753, max=47241, avg=28653.61, stdev=10416.59
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14877], 10.00th=[16581], 20.00th=[18482],
     | 30.00th=[19792], 40.00th=[22676], 50.00th=[26346], 60.00th=[32637],
     | 70.00th=[36439], 80.00th=[40109], 90.00th=[43254], 95.00th=[44827],
     | 99.00th=[46400], 99.50th=[46924], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1472, max= 3712, per=24.78%, avg=2208.33, stdev=821.13, samples=9
   iops        : min=  368, max=  928, avg=552.00, stdev=205.25, samples=9
  lat (msec)   : 10=0.04%, 20=30.70%, 50=69.26%
  cpu          : usr=0.57%, sys=3.96%, ctx=1510, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8910KiB/s (9124kB/s), 2228KiB/s-2232KiB/s (2281kB/s-2286kB/s), io=40.0MiB (41.9MB), run=4587-4597msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18204: Fri May 10 07:20:59 2024
   read: IOPS=429, BW=1719KiB/s (1760kB/s)(5048KiB/2937msec)
    slat (usec): min=28, max=2487, avg=1272.54, stdev=289.13
    clat (usec): min=3455, max=27347, avg=17306.18, stdev=2737.42
     lat (usec): min=3546, max=28556, avg=18580.64, stdev=2794.63
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13304], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19530], 90.00th=[20579], 95.00th=[21890],
     | 99.00th=[24511], 99.50th=[25822], 99.90th=[26870], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1624, max= 1816, per=24.76%, avg=1686.40, stdev=78.87, samples=5
   iops        : min=  406, max=  454, avg=421.60, stdev=19.72, samples=5
  write: IOPS=441, BW=1768KiB/s (1810kB/s)(5192KiB/2937msec)
    slat (usec): min=14, max=592, avg=52.67, stdev=31.12
    clat (usec): min=4043, max=29280, avg=18051.27, stdev=2127.70
     lat (usec): min=4088, max=29313, avg=18105.38, stdev=2129.13
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[14877], 10.00th=[15664], 20.00th=[16450],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19530], 90.00th=[20579], 95.00th=[21627],
     | 99.00th=[23987], 99.50th=[25035], 99.90th=[27395], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1688, max= 1848, per=25.05%, avg=1776.00, stdev=64.99, samples=5
   iops        : min=  422, max=  462, avg=444.00, stdev=16.25, samples=5
  lat (msec)   : 4=0.16%, 10=0.27%, 20=85.20%, 50=14.38%
  cpu          : usr=1.09%, sys=7.49%, ctx=2742, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18205: Fri May 10 07:20:59 2024
   read: IOPS=422, BW=1691KiB/s (1731kB/s)(4976KiB/2943msec)
    slat (usec): min=14, max=2700, avg=1277.21, stdev=274.12
    clat (usec): min=3457, max=29132, avg=17393.59, stdev=2535.78
     lat (usec): min=3531, max=30224, avg=18672.74, stdev=2561.51
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13566], 10.00th=[14222], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16712], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20579], 95.00th=[21627],
     | 99.00th=[23462], 99.50th=[24773], 99.90th=[28181], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1528, max= 1792, per=24.45%, avg=1665.60, stdev=112.20, samples=5
   iops        : min=  382, max=  448, avg=416.40, stdev=28.05, samples=5
  write: IOPS=447, BW=1789KiB/s (1832kB/s)(5264KiB/2943msec)
    slat (usec): min=13, max=300, avg=49.63, stdev=25.94
    clat (usec): min=6743, max=31184, avg=18041.23, stdev=2052.25
     lat (usec): min=6769, max=31211, avg=18092.18, stdev=2051.74
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15008], 10.00th=[15664], 20.00th=[16450],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19530], 90.00th=[20579], 95.00th=[21627],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[30278], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1744, max= 1872, per=25.25%, avg=1790.40, stdev=51.04, samples=5
   iops        : min=  436, max=  468, avg=447.60, stdev=12.76, samples=5
  lat (msec)   : 4=0.04%, 10=0.16%, 20=84.96%, 50=14.84%
  cpu          : usr=0.85%, sys=7.55%, ctx=2713, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18206: Fri May 10 07:20:59 2024
   read: IOPS=422, BW=1692KiB/s (1732kB/s)(4984KiB/2946msec)
    slat (usec): min=14, max=2378, avg=1261.96, stdev=271.11
    clat (usec): min=2326, max=27886, avg=17427.75, stdev=2536.68
     lat (usec): min=2358, max=28993, avg=18691.71, stdev=2571.07
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13698], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20841], 95.00th=[21627],
     | 99.00th=[24773], 99.50th=[25560], 99.90th=[26870], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1520, max= 1824, per=24.50%, avg=1668.80, stdev=117.79, samples=5
   iops        : min=  380, max=  456, avg=417.20, stdev=29.45, samples=5
  write: IOPS=446, BW=1784KiB/s (1827kB/s)(5256KiB/2946msec)
    slat (usec): min=13, max=336, avg=51.41, stdev=27.90
    clat (usec): min=4943, max=26969, avg=18061.36, stdev=2195.17
     lat (usec): min=4982, max=27066, avg=18113.91, stdev=2194.56
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14877], 10.00th=[15664], 20.00th=[16450],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19792], 90.00th=[20841], 95.00th=[21890],
     | 99.00th=[23987], 99.50th=[24511], 99.90th=[26346], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1528, max= 1960, per=24.96%, avg=1769.60, stdev=165.06, samples=5
   iops        : min=  382, max=  490, avg=442.40, stdev=41.26, samples=5
  lat (msec)   : 4=0.04%, 10=0.27%, 20=84.41%, 50=15.27%
  cpu          : usr=1.09%, sys=7.50%, ctx=2641, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18207: Fri May 10 07:20:59 2024
   read: IOPS=429, BW=1719KiB/s (1760kB/s)(5060KiB/2944msec)
    slat (usec): min=15, max=2797, avg=1269.89, stdev=279.19
    clat (usec): min=3410, max=26870, avg=17362.54, stdev=2408.70
     lat (usec): min=3429, max=28152, avg=18634.39, stdev=2445.69
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13698], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[16188], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20317], 95.00th=[21365],
     | 99.00th=[23200], 99.50th=[23987], 99.90th=[26870], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1648, max= 1832, per=25.04%, avg=1705.60, stdev=76.61, samples=5
   iops        : min=  412, max=  458, avg=426.40, stdev=19.15, samples=5
  write: IOPS=439, BW=1760KiB/s (1802kB/s)(5180KiB/2944msec)
    slat (usec): min=14, max=233, avg=49.85, stdev=24.14
    clat (usec): min=6730, max=28309, avg=18082.10, stdev=2118.34
     lat (usec): min=6846, max=28364, avg=18133.05, stdev=2118.19
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14877], 10.00th=[15664], 20.00th=[16319],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19792], 90.00th=[20579], 95.00th=[21627],
     | 99.00th=[23725], 99.50th=[24773], 99.90th=[25822], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1672, max= 1840, per=24.66%, avg=1748.80, stdev=76.86, samples=5
   iops        : min=  418, max=  460, avg=437.20, stdev=19.21, samples=5
  lat (msec)   : 4=0.08%, 10=0.12%, 20=85.70%, 50=14.10%
  cpu          : usr=0.71%, sys=7.61%, ctx=2728, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6812KiB/s (6975kB/s), 1691KiB/s-1719KiB/s (1731kB/s-1760kB/s), io=19.6MiB (20.5MB), run=2937-2946msec
  WRITE: bw=7092KiB/s (7262kB/s), 1760KiB/s-1789KiB/s (1802kB/s-1832kB/s), io=20.4MiB (21.4MB), run=2937-2946msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18213: Fri May 10 07:21:00 2024
   read: IOPS=11, BW=476MiB/s (499MB/s)(40.0MiB/84msec)
    slat (nsec): min=7913.2k, max=7913.2k, avg=7913198.00, stdev= 0.00
    clat (nsec): min=75749k, max=75749k, avg=75748946.00, stdev= 0.00
     lat (nsec): min=83673k, max=83673k, avg=83672814.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[76022],  5.00th=[76022], 10.00th=[76022], 20.00th=[76022],
     | 30.00th=[76022], 40.00th=[76022], 50.00th=[76022], 60.00th=[76022],
     | 70.00th=[76022], 80.00th=[76022], 90.00th=[76022], 95.00th=[76022],
     | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.33%, ctx=10, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=476MiB/s (499MB/s), 476MiB/s-476MiB/s (499MB/s-499MB/s), io=40.0MiB (41.9MB), run=84-84msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout