== sanity test 398c: run fio to test AIO ================= 10:41:52 (1724769712)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0954174 s, 440 MB/s
osc.lustre-OST0000-osc-ffff8800a6ee5000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=7287: Tue Aug 27 10:41:57 2024
  write: IOPS=608, BW=2435KiB/s (2494kB/s)(10.0MiB/4205msec)
    slat (usec): min=13, max=280, avg=59.55, stdev=34.42
    clat (usec): min=5890, max=47772, avg=26125.19, stdev=6887.03
     lat (usec): min=5950, max=47810, avg=26186.30, stdev=6899.36
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[19006], 10.00th=[20317], 20.00th=[21103],
     | 30.00th=[22152], 40.00th=[22938], 50.00th=[23725], 60.00th=[25035],
     | 70.00th=[27395], 80.00th=[30540], 90.00th=[36963], 95.00th=[41681],
     | 99.00th=[45876], 99.50th=[46400], 99.90th=[46924], 99.95th=[47449],
     | 99.99th=[47973]
   bw (  KiB/s): min= 1536, max= 2968, per=25.28%, avg=2452.25, stdev=479.60, samples=8
   iops        : min=  384, max=  742, avg=613.00, stdev=119.89, samples=8
  lat (msec)   : 10=0.31%, 20=8.16%, 50=91.52%
  cpu          : usr=0.69%, sys=4.00%, ctx=1812, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7288: Tue Aug 27 10:41:57 2024
  write: IOPS=606, BW=2427KiB/s (2485kB/s)(10.0MiB/4220msec)
    slat (usec): min=12, max=263, avg=48.91, stdev=30.09
    clat (usec): min=5377, max=47792, avg=26187.58, stdev=6747.60
     lat (usec): min=5420, max=47846, avg=26237.86, stdev=6759.50
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[19268], 10.00th=[20317], 20.00th=[21365],
     | 30.00th=[22152], 40.00th=[22938], 50.00th=[23725], 60.00th=[25035],
     | 70.00th=[27657], 80.00th=[30278], 90.00th=[37487], 95.00th=[41157],
     | 99.00th=[45876], 99.50th=[46924], 99.90th=[47973], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min= 1532, max= 2944, per=25.07%, avg=2431.75, stdev=458.63, samples=8
   iops        : min=  383, max=  736, avg=607.88, stdev=114.60, samples=8
  lat (msec)   : 10=0.04%, 20=7.54%, 50=92.42%
  cpu          : usr=0.57%, sys=3.27%, ctx=970, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7289: Tue Aug 27 10:41:57 2024
  write: IOPS=606, BW=2425KiB/s (2483kB/s)(10.0MiB/4223msec)
    slat (usec): min=13, max=286, avg=49.27, stdev=31.22
    clat (usec): min=4866, max=46945, avg=26182.43, stdev=6734.68
     lat (usec): min=4919, max=46983, avg=26232.94, stdev=6748.34
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[19530], 10.00th=[20317], 20.00th=[21365],
     | 30.00th=[22152], 40.00th=[22938], 50.00th=[23725], 60.00th=[25035],
     | 70.00th=[27395], 80.00th=[30540], 90.00th=[36439], 95.00th=[41157],
     | 99.00th=[46400], 99.50th=[46400], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1536, max= 2944, per=25.09%, avg=2433.38, stdev=462.97, samples=8
   iops        : min=  384, max=  736, avg=608.25, stdev=115.76, samples=8
  lat (msec)   : 10=0.04%, 20=7.46%, 50=92.50%
  cpu          : usr=0.57%, sys=3.34%, ctx=916, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7290: Tue Aug 27 10:41:57 2024
  write: IOPS=607, BW=2429KiB/s (2488kB/s)(10.0MiB/4215msec)
    slat (usec): min=13, max=262, avg=49.34, stdev=31.08
    clat (usec): min=6748, max=47597, avg=26179.52, stdev=6752.30
     lat (usec): min=6871, max=47697, avg=26230.09, stdev=6763.64
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[19268], 10.00th=[20317], 20.00th=[21103],
     | 30.00th=[22152], 40.00th=[22938], 50.00th=[23725], 60.00th=[25035],
     | 70.00th=[27657], 80.00th=[30016], 90.00th=[37487], 95.00th=[41681],
     | 99.00th=[45351], 99.50th=[47449], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min= 1536, max= 2928, per=25.13%, avg=2437.38, stdev=474.15, samples=8
   iops        : min=  384, max=  732, avg=609.25, stdev=118.53, samples=8
  lat (msec)   : 10=0.08%, 20=8.09%, 50=91.84%
  cpu          : usr=0.50%, sys=3.46%, ctx=1179, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9699KiB/s (9932kB/s), 2425KiB/s-2435KiB/s (2483kB/s-2494kB/s), io=40.0MiB (41.9MB), run=4205-4223msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=7303: Tue Aug 27 10:42:01 2024
   read: IOPS=477, BW=1909KiB/s (1955kB/s)(5048KiB/2644msec)
    slat (usec): min=628, max=2630, avg=1066.21, stdev=216.39
    clat (usec): min=2503, max=23004, avg=15597.74, stdev=2350.66
     lat (usec): min=3760, max=23972, avg=16665.76, stdev=2359.64
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[11994], 10.00th=[12780], 20.00th=[13566],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15664], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18482], 95.00th=[19530],
     | 99.00th=[21103], 99.50th=[21365], 99.90th=[21890], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 1760, max= 2008, per=24.96%, avg=1894.40, stdev=102.04, samples=5
   iops        : min=  440, max=  502, avg=473.60, stdev=25.51, samples=5
  write: IOPS=490, BW=1964KiB/s (2011kB/s)(5192KiB/2644msec)
    slat (usec): min=13, max=1026, avg=40.87, stdev=36.55
    clat (usec): min=3560, max=22064, avg=16106.75, stdev=1896.69
     lat (usec): min=3622, max=22105, avg=16148.67, stdev=1897.45
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13173], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[16188], 60.00th=[16581],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18482], 95.00th=[19006],
     | 99.00th=[20317], 99.50th=[20841], 99.90th=[22152], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 1872, max= 2064, per=24.75%, avg=1955.20, stdev=74.32, samples=5
   iops        : min=  468, max=  516, avg=488.80, stdev=18.58, samples=5
  lat (msec)   : 4=0.08%, 10=0.74%, 20=97.23%, 50=1.95%
  cpu          : usr=1.21%, sys=6.47%, ctx=2424, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7304: Tue Aug 27 10:42:01 2024
   read: IOPS=474, BW=1896KiB/s (1942kB/s)(4976KiB/2624msec)
    slat (usec): min=35, max=2191, avg=1070.32, stdev=224.31
    clat (usec): min=1323, max=29142, avg=15565.66, stdev=2397.52
     lat (usec): min=1361, max=30129, avg=16637.66, stdev=2428.55
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[11994], 10.00th=[12649], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14877], 50.00th=[15533], 60.00th=[16057],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[20841], 99.50th=[21627], 99.90th=[23200], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1808, max= 1944, per=24.79%, avg=1881.60, stdev=62.84, samples=5
   iops        : min=  452, max=  486, avg=470.40, stdev=15.71, samples=5
  write: IOPS=501, BW=2006KiB/s (2054kB/s)(5264KiB/2624msec)
    slat (usec): min=14, max=162, avg=39.81, stdev=18.06
    clat (usec): min=3589, max=25234, avg=16056.08, stdev=1951.05
     lat (usec): min=3618, max=25272, avg=16096.89, stdev=1950.46
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[13042], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[16057], 60.00th=[16581],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18482], 95.00th=[19006],
     | 99.00th=[20579], 99.50th=[20841], 99.90th=[22152], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 1896, max= 2128, per=25.46%, avg=2011.20, stdev=88.83, samples=5
   iops        : min=  474, max=  532, avg=502.80, stdev=22.21, samples=5
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.62%, 20=97.46%, 50=1.80%
  cpu          : usr=0.91%, sys=6.79%, ctx=2412, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7305: Tue Aug 27 10:42:01 2024
   read: IOPS=471, BW=1886KiB/s (1931kB/s)(4984KiB/2643msec)
    slat (usec): min=662, max=2291, avg=1066.29, stdev=208.36
    clat (usec): min=4058, max=23137, avg=15567.69, stdev=2331.59
     lat (usec): min=5264, max=24055, avg=16635.61, stdev=2348.96
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[11994], 10.00th=[12780], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14877], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[20841], 99.50th=[21103], 99.90th=[22414], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 1808, max= 2024, per=24.85%, avg=1886.40, stdev=82.05, samples=5
   iops        : min=  452, max=  506, avg=471.60, stdev=20.51, samples=5
  write: IOPS=497, BW=1989KiB/s (2036kB/s)(5256KiB/2643msec)
    slat (usec): min=12, max=304, avg=40.49, stdev=19.60
    clat (usec): min=9845, max=22775, avg=16096.14, stdev=1852.50
     lat (usec): min=9886, max=22801, avg=16137.77, stdev=1852.65
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13042], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[16188], 60.00th=[16581],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18482], 95.00th=[19006],
     | 99.00th=[20579], 99.50th=[21103], 99.90th=[22676], 99.95th=[22676],
     | 99.99th=[22676]
   bw (  KiB/s): min= 1800, max= 2032, per=24.95%, avg=1971.20, stdev=96.93, samples=5
   iops        : min=  450, max=  508, avg=492.80, stdev=24.23, samples=5
  lat (msec)   : 10=0.55%, 20=97.50%, 50=1.95%
  cpu          : usr=0.68%, sys=7.08%, ctx=2443, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7306: Tue Aug 27 10:42:01 2024
   read: IOPS=478, BW=1914KiB/s (1960kB/s)(5060KiB/2643msec)
    slat (usec): min=587, max=2405, avg=1077.18, stdev=216.13
    clat (usec): min=4942, max=22989, avg=15642.75, stdev=2338.88
     lat (usec): min=6019, max=24068, avg=16721.61, stdev=2350.09
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[11863], 10.00th=[12649], 20.00th=[13698],
     | 30.00th=[14484], 40.00th=[15139], 50.00th=[15664], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18482], 95.00th=[19530],
     | 99.00th=[21103], 99.50th=[21890], 99.90th=[22676], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 1816, max= 1968, per=25.04%, avg=1900.80, stdev=61.34, samples=5
   iops        : min=  454, max=  492, avg=475.20, stdev=15.34, samples=5
  write: IOPS=489, BW=1960KiB/s (2007kB/s)(5180KiB/2643msec)
    slat (usec): min=13, max=874, avg=41.93, stdev=31.31
    clat (usec): min=8241, max=25256, avg=16080.27, stdev=1823.16
     lat (usec): min=8261, max=25390, avg=16123.26, stdev=1823.21
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13173], 10.00th=[13829], 20.00th=[14615],
     | 30.00th=[15270], 40.00th=[15664], 50.00th=[16188], 60.00th=[16581],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18220], 95.00th=[19006],
     | 99.00th=[20317], 99.50th=[21103], 99.90th=[22414], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 1824, max= 2096, per=24.71%, avg=1952.00, stdev=98.47, samples=5
   iops        : min=  456, max=  524, avg=488.00, stdev=24.62, samples=5
  lat (msec)   : 10=0.98%, 20=96.91%, 50=2.11%
  cpu          : usr=0.87%, sys=6.85%, ctx=2496, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7590KiB/s (7772kB/s), 1886KiB/s-1914KiB/s (1931kB/s-1960kB/s), io=19.6MiB (20.5MB), run=2624-2644msec
  WRITE: bw=7902KiB/s (8091kB/s), 1960KiB/s-2006KiB/s (2007kB/s-2054kB/s), io=20.4MiB (21.4MB), run=2624-2644msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=7312: Tue Aug 27 10:42:02 2024
   read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec)
    slat (nsec): min=5497.4k, max=5497.4k, avg=5497352.00, stdev= 0.00
    clat (nsec): min=91426k, max=91426k, avg=91426287.00, stdev= 0.00
     lat (nsec): min=96936k, max=96936k, avg=96935717.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[91751],  5.00th=[91751], 10.00th=[91751], 20.00th=[91751],
     | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751],
     | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751],
     | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.88%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout