== sanity test 398c: run fio to test AIO ================= 23:42:13 (1708836133)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0925124 s, 453 MB/s
osc.lustre-OST0000-osc-ffff88012c460800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=10252: Sat Feb 24 23:42:17 2024
  write: IOPS=767, BW=3070KiB/s (3143kB/s)(10.0MiB/3336msec)
    slat (usec): min=11, max=331, avg=42.18, stdev=21.33
    clat (usec): min=3390, max=30801, avg=20715.79, stdev=2293.22
     lat (usec): min=3464, max=30841, avg=20759.25, stdev=2292.99
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[17433], 10.00th=[18482], 20.00th=[19530],
     | 30.00th=[20055], 40.00th=[20579], 50.00th=[21103], 60.00th=[21365],
     | 70.00th=[21890], 80.00th=[22414], 90.00th=[22938], 95.00th=[23200],
     | 99.00th=[24773], 99.50th=[29230], 99.90th=[30802], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 2928, max= 3368, per=25.05%, avg=3073.33, stdev=160.03, samples=6
   iops        : min=  732, max=  842, avg=768.33, stdev=40.01, samples=6
  lat (msec)   : 4=0.04%, 10=0.74%, 20=29.65%, 50=69.57%
  cpu          : usr=0.81%, sys=3.72%, ctx=946, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10253: Sat Feb 24 23:42:17 2024
  write: IOPS=769, BW=3080KiB/s (3154kB/s)(10.0MiB/3325msec)
    slat (usec): min=12, max=217, avg=45.50, stdev=21.54
    clat (usec): min=2326, max=38304, avg=20658.49, stdev=2709.90
     lat (usec): min=2389, max=38340, avg=20705.42, stdev=2711.28
    clat percentiles (usec):
     |  1.00th=[ 7242],  5.00th=[16909], 10.00th=[18220], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20579], 50.00th=[21103], 60.00th=[21365],
     | 70.00th=[21890], 80.00th=[22414], 90.00th=[22938], 95.00th=[23200],
     | 99.00th=[27132], 99.50th=[30540], 99.90th=[36439], 99.95th=[38011],
     | 99.99th=[38536]
   bw (  KiB/s): min= 2904, max= 3520, per=25.17%, avg=3088.00, stdev=222.22, samples=6
   iops        : min=  726, max=  880, avg=772.00, stdev=55.56, samples=6
  lat (msec)   : 4=0.08%, 10=1.17%, 20=30.08%, 50=68.67%
  cpu          : usr=0.99%, sys=3.82%, ctx=1410, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10254: Sat Feb 24 23:42:17 2024
  write: IOPS=767, BW=3070KiB/s (3143kB/s)(10.0MiB/3336msec)
    slat (usec): min=12, max=217, avg=46.09, stdev=22.25
    clat (usec): min=4112, max=36120, avg=20710.31, stdev=2371.60
     lat (usec): min=4172, max=36179, avg=20757.74, stdev=2373.46
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[17171], 10.00th=[18220], 20.00th=[19268],
     | 30.00th=[19792], 40.00th=[20579], 50.00th=[21103], 60.00th=[21365],
     | 70.00th=[21890], 80.00th=[22414], 90.00th=[22938], 95.00th=[23200],
     | 99.00th=[25035], 99.50th=[30278], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2920, max= 3392, per=25.05%, avg=3073.33, stdev=169.89, samples=6
   iops        : min=  730, max=  848, avg=768.33, stdev=42.47, samples=6
  lat (msec)   : 10=0.78%, 20=31.48%, 50=67.73%
  cpu          : usr=0.48%, sys=4.32%, ctx=1488, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10255: Sat Feb 24 23:42:17 2024
  write: IOPS=766, BW=3068KiB/s (3141kB/s)(10.0MiB/3338msec)
    slat (usec): min=12, max=379, avg=42.19, stdev=19.90
    clat (usec): min=2555, max=31224, avg=20727.13, stdev=2258.69
     lat (usec): min=2613, max=31288, avg=20770.63, stdev=2257.18
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[16909], 10.00th=[18482], 20.00th=[19530],
     | 30.00th=[20055], 40.00th=[20579], 50.00th=[21103], 60.00th=[21365],
     | 70.00th=[21890], 80.00th=[22414], 90.00th=[22938], 95.00th=[23200],
     | 99.00th=[24511], 99.50th=[29230], 99.90th=[30278], 99.95th=[31065],
     | 99.99th=[31327]
   bw (  KiB/s): min= 2944, max= 3360, per=24.97%, avg=3064.00, stdev=159.04, samples=6
   iops        : min=  736, max=  840, avg=766.00, stdev=39.76, samples=6
  lat (msec)   : 4=0.04%, 10=0.70%, 20=30.27%, 50=68.98%
  cpu          : usr=1.11%, sys=3.51%, ctx=1124, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.0MiB/s (12.6MB/s), 3068KiB/s-3080KiB/s (3141kB/s-3154kB/s), io=40.0MiB (41.9MB), run=3325-3338msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=10267: Sat Feb 24 23:42:21 2024
   read: IOPS=426, BW=1707KiB/s (1748kB/s)(5048KiB/2958msec)
    slat (usec): min=19, max=2726, avg=1221.44, stdev=270.43
    clat (usec): min=11453, max=36914, avg=17529.12, stdev=2545.21
     lat (usec): min=12382, max=38158, avg=18752.52, stdev=2564.20
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13829], 10.00th=[14484], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20579], 95.00th=[21627],
     | 99.00th=[24249], 99.50th=[25822], 99.90th=[35390], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1648, max= 1712, per=25.00%, avg=1696.00, stdev=27.71, samples=5
   iops        : min=  412, max=  428, avg=424.00, stdev= 6.93, samples=5
  write: IOPS=438, BW=1755KiB/s (1797kB/s)(5192KiB/2958msec)
    slat (usec): min=13, max=1126, avg=46.90, stdev=36.69
    clat (usec): min=5841, max=35500, avg=18135.53, stdev=1989.98
     lat (usec): min=5885, max=35547, avg=18183.80, stdev=1990.69
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15139], 10.00th=[15926], 20.00th=[16581],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19530], 90.00th=[20317], 95.00th=[21103],
     | 99.00th=[22676], 99.50th=[24249], 99.90th=[31065], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1736, max= 1832, per=25.26%, avg=1784.00, stdev=46.30, samples=5
   iops        : min=  434, max=  458, avg=446.00, stdev=11.58, samples=5
  lat (msec)   : 10=0.04%, 20=85.90%, 50=14.06%
  cpu          : usr=0.95%, sys=7.24%, ctx=2474, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10268: Sat Feb 24 23:42:21 2024
   read: IOPS=421, BW=1686KiB/s (1727kB/s)(4976KiB/2951msec)
    slat (usec): min=13, max=3453, avg=1228.20, stdev=258.01
    clat (usec): min=2178, max=39244, avg=17517.81, stdev=2625.24
     lat (usec): min=2275, max=40363, avg=18748.51, stdev=2650.16
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13829], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16712], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19268], 90.00th=[20579], 95.00th=[21627],
     | 99.00th=[23725], 99.50th=[24773], 99.90th=[35390], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1648, max= 1736, per=24.83%, avg=1684.80, stdev=33.75, samples=5
   iops        : min=  412, max=  434, avg=421.20, stdev= 8.44, samples=5
  write: IOPS=445, BW=1784KiB/s (1827kB/s)(5264KiB/2951msec)
    slat (usec): min=13, max=369, avg=46.80, stdev=22.60
    clat (usec): min=5565, max=37451, avg=18069.16, stdev=2094.27
     lat (usec): min=5606, max=37496, avg=18117.53, stdev=2094.43
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15139], 10.00th=[15795], 20.00th=[16581],
     | 30.00th=[17171], 40.00th=[17433], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19530], 90.00th=[20579], 95.00th=[21103],
     | 99.00th=[22414], 99.50th=[23200], 99.90th=[33817], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1704, max= 1904, per=25.56%, avg=1804.80, stdev=82.48, samples=5
   iops        : min=  426, max=  476, avg=451.20, stdev=20.62, samples=5
  lat (msec)   : 4=0.04%, 10=0.20%, 20=84.96%, 50=14.80%
  cpu          : usr=1.08%, sys=7.15%, ctx=2495, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10269: Sat Feb 24 23:42:21 2024
   read: IOPS=422, BW=1689KiB/s (1729kB/s)(4984KiB/2951msec)
    slat (usec): min=16, max=3452, avg=1215.24, stdev=254.36
    clat (usec): min=3440, max=42870, avg=17473.33, stdev=2740.47
     lat (usec): min=4651, max=44080, avg=18690.48, stdev=2754.54
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13829], 10.00th=[14484], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16712], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20579], 95.00th=[21103],
     | 99.00th=[22676], 99.50th=[25035], 99.90th=[41681], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1504, max= 1856, per=24.93%, avg=1691.20, stdev=141.38, samples=5
   iops        : min=  376, max=  464, avg=422.80, stdev=35.34, samples=5
  write: IOPS=445, BW=1781KiB/s (1824kB/s)(5256KiB/2951msec)
    slat (usec): min=14, max=1235, avg=47.59, stdev=46.08
    clat (usec): min=5952, max=36928, avg=18110.59, stdev=2145.59
     lat (usec): min=6073, max=36968, avg=18159.62, stdev=2145.19
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15008], 10.00th=[15795], 20.00th=[16450],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19792], 90.00th=[20579], 95.00th=[21365],
     | 99.00th=[23200], 99.50th=[25822], 99.90th=[35914], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1696, max= 1960, per=25.47%, avg=1798.40, stdev=113.90, samples=5
   iops        : min=  424, max=  490, avg=449.60, stdev=28.47, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=84.77%, 50=15.12%
  cpu          : usr=0.71%, sys=7.39%, ctx=2437, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10270: Sat Feb 24 23:42:21 2024
   read: IOPS=427, BW=1711KiB/s (1752kB/s)(5060KiB/2958msec)
    slat (usec): min=24, max=3440, avg=1223.89, stdev=265.93
    clat (usec): min=3443, max=28635, avg=17502.15, stdev=2443.98
     lat (usec): min=4269, max=30274, avg=18728.00, stdev=2457.70
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13829], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[16188], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19530], 90.00th=[20317], 95.00th=[21627],
     | 99.00th=[23987], 99.50th=[25035], 99.90th=[26346], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1672, max= 1752, per=25.26%, avg=1713.60, stdev=35.51, samples=5
   iops        : min=  418, max=  438, avg=428.40, stdev= 8.88, samples=5
  write: IOPS=437, BW=1751KiB/s (1793kB/s)(5180KiB/2958msec)
    slat (usec): min=12, max=429, avg=46.27, stdev=22.31
    clat (usec): min=5959, max=35915, avg=18163.77, stdev=2079.02
     lat (usec): min=6122, max=35953, avg=18211.39, stdev=2077.73
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15270], 10.00th=[15795], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19792], 90.00th=[20579], 95.00th=[21365],
     | 99.00th=[23200], 99.50th=[24511], 99.90th=[33817], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1696, max= 1840, per=25.01%, avg=1766.40, stdev=52.88, samples=5
   iops        : min=  424, max=  460, avg=441.60, stdev=13.22, samples=5
  lat (msec)   : 4=0.04%, 10=0.04%, 20=85.00%, 50=14.92%
  cpu          : usr=1.12%, sys=6.97%, ctx=2543, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6784KiB/s (6947kB/s), 1686KiB/s-1711KiB/s (1727kB/s-1752kB/s), io=19.6MiB (20.5MB), run=2951-2958msec
  WRITE: bw=7063KiB/s (7232kB/s), 1751KiB/s-1784KiB/s (1793kB/s-1827kB/s), io=20.4MiB (21.4MB), run=2951-2958msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=10276: Sat Feb 24 23:42:23 2024
   read: IOPS=10, BW=412MiB/s (432MB/s)(40.0MiB/97msec)
    slat (nsec): min=8211.8k, max=8211.8k, avg=8211837.00, stdev= 0.00
    clat (nsec): min=87906k, max=87906k, avg=87905964.00, stdev= 0.00
     lat (nsec): min=96140k, max=96140k, avg=96139532.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[87557],  5.00th=[87557], 10.00th=[87557], 20.00th=[87557],
     | 30.00th=[87557], 40.00th=[87557], 50.00th=[87557], 60.00th=[87557],
     | 70.00th=[87557], 80.00th=[87557], 90.00th=[87557], 95.00th=[87557],
     | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557],
     | 99.99th=[87557]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.29%, ctx=11, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=412MiB/s (432MB/s), 412MiB/s-412MiB/s (432MB/s-432MB/s), io=40.0MiB (41.9MB), run=97-97msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout