== sanity test 398c: run fio to test AIO ================= 19:11:41 (1743462701)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.166081 s, 253 MB/s
osc.lustre-OST0000-osc-ffff8800a8426000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30435: Mon Mar 31 19:11:47 2025
  write: IOPS=602, BW=2409KiB/s (2467kB/s)(10.0MiB/4250msec)
    slat (usec): min=14, max=452, avg=63.82, stdev=46.80
    clat (usec): min=2678, max=39380, avg=26382.08, stdev=4418.72
     lat (usec): min=2721, max=39405, avg=26447.21, stdev=4424.20
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[17695], 10.00th=[21365], 20.00th=[24249],
     | 30.00th=[25035], 40.00th=[25822], 50.00th=[26608], 60.00th=[27657],
     | 70.00th=[28443], 80.00th=[29754], 90.00th=[31327], 95.00th=[32375],
     | 99.00th=[34341], 99.50th=[35390], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 2176, max= 3000, per=25.05%, avg=2413.75, stdev=257.58, samples=8
   iops        : min=  544, max=  750, avg=603.25, stdev=64.52, samples=8
  lat (msec)   : 4=0.08%, 10=0.74%, 20=5.35%, 50=93.83%
  cpu          : usr=0.68%, sys=4.21%, ctx=1376, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30436: Mon Mar 31 19:11:47 2025
  write: IOPS=602, BW=2411KiB/s (2469kB/s)(10.0MiB/4247msec)
    slat (usec): min=14, max=645, avg=63.04, stdev=48.03
    clat (usec): min=6397, max=56818, avg=26380.79, stdev=4662.22
     lat (usec): min=6452, max=56851, avg=26445.09, stdev=4668.59
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[16909], 10.00th=[21365], 20.00th=[23987],
     | 30.00th=[25035], 40.00th=[25822], 50.00th=[26608], 60.00th=[27657],
     | 70.00th=[28443], 80.00th=[29492], 90.00th=[31589], 95.00th=[32637],
     | 99.00th=[34341], 99.50th=[35914], 99.90th=[49021], 99.95th=[55313],
     | 99.99th=[56886]
   bw (  KiB/s): min= 2160, max= 3080, per=25.15%, avg=2423.75, stdev=292.11, samples=8
   iops        : min=  540, max=  770, avg=605.75, stdev=73.15, samples=8
  lat (msec)   : 10=0.86%, 20=6.17%, 50=92.89%, 100=0.08%
  cpu          : usr=0.89%, sys=4.07%, ctx=1319, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30437: Mon Mar 31 19:11:47 2025
  write: IOPS=603, BW=2413KiB/s (2471kB/s)(10.0MiB/4244msec)
    slat (usec): min=14, max=566, avg=64.25, stdev=48.45
    clat (usec): min=4464, max=56405, avg=26380.25, stdev=4648.98
     lat (usec): min=4564, max=56443, avg=26445.73, stdev=4654.54
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[16581], 10.00th=[21365], 20.00th=[23987],
     | 30.00th=[25297], 40.00th=[25822], 50.00th=[26608], 60.00th=[27657],
     | 70.00th=[28443], 80.00th=[29754], 90.00th=[31589], 95.00th=[32637],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[48497], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min= 2115, max= 3096, per=25.17%, avg=2425.37, stdev=297.03, samples=8
   iops        : min=  528, max=  774, avg=606.25, stdev=74.37, samples=8
  lat (msec)   : 10=0.74%, 20=6.56%, 50=92.62%, 100=0.08%
  cpu          : usr=0.64%, sys=4.31%, ctx=1438, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30438: Mon Mar 31 19:11:47 2025
  write: IOPS=605, BW=2423KiB/s (2481kB/s)(10.0MiB/4226msec)
    slat (usec): min=14, max=611, avg=67.61, stdev=48.07
    clat (usec): min=3124, max=56777, avg=26271.24, stdev=4976.31
     lat (usec): min=3401, max=56820, avg=26340.23, stdev=4977.81
    clat percentiles (usec):
     |  1.00th=[ 8225],  5.00th=[16319], 10.00th=[21103], 20.00th=[23987],
     | 30.00th=[25035], 40.00th=[25822], 50.00th=[26608], 60.00th=[27657],
     | 70.00th=[28443], 80.00th=[29492], 90.00th=[31589], 95.00th=[32375],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[56886], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min= 2163, max= 3200, per=25.33%, avg=2441.38, stdev=318.72, samples=8
   iops        : min=  540, max=  800, avg=610.25, stdev=79.77, samples=8
  lat (msec)   : 4=0.16%, 10=1.41%, 20=6.02%, 50=92.23%, 100=0.20%
  cpu          : usr=0.80%, sys=4.40%, ctx=1622, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9638KiB/s (9869kB/s), 2409KiB/s-2423KiB/s (2467kB/s-2481kB/s), io=40.0MiB (41.9MB), run=4226-4250msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30451: Mon Mar 31 19:11:51 2025
   read: IOPS=385, BW=1540KiB/s (1577kB/s)(5048KiB/3277msec)
    slat (usec): min=14, max=2596, avg=1265.41, stdev=282.15
    clat (usec): min=1775, max=37527, avg=19417.86, stdev=3301.64
     lat (usec): min=1915, max=38531, avg=20685.47, stdev=3361.96
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14746], 10.00th=[15795], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18482], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20841], 80.00th=[21890], 90.00th=[23462], 95.00th=[24511],
     | 99.00th=[27657], 99.50th=[30278], 99.90th=[36439], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1400, max= 1704, per=25.24%, avg=1533.33, stdev=115.23, samples=6
   iops        : min=  350, max=  426, avg=383.33, stdev=28.81, samples=6
  write: IOPS=396, BW=1584KiB/s (1622kB/s)(5192KiB/3277msec)
    slat (usec): min=15, max=303, avg=56.30, stdev=31.14
    clat (usec): min=5509, max=32989, avg=20132.63, stdev=2690.11
     lat (usec): min=5620, max=33076, avg=20190.16, stdev=2688.10
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[16319], 10.00th=[17171], 20.00th=[17957],
     | 30.00th=[18744], 40.00th=[19530], 50.00th=[20055], 60.00th=[20579],
     | 70.00th=[21365], 80.00th=[22152], 90.00th=[23462], 95.00th=[24511],
     | 99.00th=[27132], 99.50th=[27919], 99.90th=[31327], 99.95th=[32900],
     | 99.99th=[32900]
   bw (  KiB/s): min= 1392, max= 1728, per=25.15%, avg=1590.67, stdev=113.86, samples=6
   iops        : min=  348, max=  432, avg=397.67, stdev=28.47, samples=6
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.39%, 20=53.44%, 50=46.02%
  cpu          : usr=0.98%, sys=7.72%, ctx=2583, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30452: Mon Mar 31 19:11:51 2025
   read: IOPS=378, BW=1513KiB/s (1549kB/s)(4976KiB/3289msec)
    slat (usec): min=696, max=2799, avg=1277.46, stdev=269.67
    clat (usec): min=11083, max=30546, avg=19441.90, stdev=2960.97
     lat (usec): min=12193, max=32159, avg=20721.94, stdev=2996.95
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14877], 10.00th=[15664], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18482], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20841], 80.00th=[21890], 90.00th=[23200], 95.00th=[24511],
     | 99.00th=[27132], 99.50th=[27657], 99.90th=[28967], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1424, max= 1672, per=24.69%, avg=1500.00, stdev=97.55, samples=6
   iops        : min=  356, max=  418, avg=375.00, stdev=24.39, samples=6
  write: IOPS=400, BW=1600KiB/s (1639kB/s)(5264KiB/3289msec)
    slat (usec): min=15, max=287, avg=54.79, stdev=28.65
    clat (usec): min=2504, max=28877, avg=20124.60, stdev=2486.70
     lat (usec): min=2545, max=28930, avg=20180.75, stdev=2487.37
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[16319], 10.00th=[17171], 20.00th=[18220],
     | 30.00th=[18744], 40.00th=[19268], 50.00th=[20055], 60.00th=[20579],
     | 70.00th=[21365], 80.00th=[22152], 90.00th=[23462], 95.00th=[24249],
     | 99.00th=[26084], 99.50th=[26346], 99.90th=[28443], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1448, max= 1696, per=25.40%, avg=1606.67, stdev=95.39, samples=6
   iops        : min=  362, max=  424, avg=401.67, stdev=23.85, samples=6
  lat (msec)   : 4=0.04%, 10=0.08%, 20=53.95%, 50=45.94%
  cpu          : usr=0.79%, sys=7.66%, ctx=2602, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30453: Mon Mar 31 19:11:51 2025
   read: IOPS=377, BW=1511KiB/s (1547kB/s)(4984KiB/3298msec)
    slat (usec): min=757, max=2874, avg=1292.70, stdev=264.50
    clat (usec): min=10849, max=37926, avg=19549.93, stdev=3011.45
     lat (usec): min=12110, max=39190, avg=20844.96, stdev=3048.53
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[15139], 10.00th=[15926], 20.00th=[16909],
     | 30.00th=[17957], 40.00th=[18744], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[21103], 80.00th=[21890], 90.00th=[23462], 95.00th=[24511],
     | 99.00th=[26870], 99.50th=[28443], 99.90th=[36963], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1432, max= 1592, per=24.84%, avg=1509.33, stdev=52.99, samples=6
   iops        : min=  358, max=  398, avg=377.33, stdev=13.25, samples=6
  write: IOPS=398, BW=1594KiB/s (1632kB/s)(5256KiB/3298msec)
    slat (usec): min=15, max=982, avg=55.23, stdev=39.59
    clat (usec): min=5500, max=34365, avg=20053.47, stdev=2412.05
     lat (usec): min=5530, max=34477, avg=20110.00, stdev=2413.46
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[16581], 10.00th=[17171], 20.00th=[18220],
     | 30.00th=[18744], 40.00th=[19530], 50.00th=[20055], 60.00th=[20579],
     | 70.00th=[21103], 80.00th=[21890], 90.00th=[22938], 95.00th=[23987],
     | 99.00th=[26346], 99.50th=[28443], 99.90th=[33424], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1424, max= 1816, per=25.21%, avg=1594.67, stdev=130.74, samples=6
   iops        : min=  356, max=  454, avg=398.67, stdev=32.68, samples=6
  lat (msec)   : 10=0.04%, 20=54.80%, 50=45.16%
  cpu          : usr=1.30%, sys=7.07%, ctx=2608, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30454: Mon Mar 31 19:11:51 2025
   read: IOPS=382, BW=1532KiB/s (1569kB/s)(5060KiB/3303msec)
    slat (usec): min=707, max=2665, avg=1289.93, stdev=262.98
    clat (usec): min=3116, max=29596, avg=19494.17, stdev=3062.11
     lat (usec): min=4250, max=30902, avg=20786.59, stdev=3096.30
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[14877], 10.00th=[15926], 20.00th=[17171],
     | 30.00th=[17695], 40.00th=[18744], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20841], 80.00th=[21890], 90.00th=[23462], 95.00th=[24511],
     | 99.00th=[27132], 99.50th=[27919], 99.90th=[28967], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1408, max= 1704, per=25.11%, avg=1525.33, stdev=100.26, samples=6
   iops        : min=  352, max=  426, avg=381.33, stdev=25.07, samples=6
  write: IOPS=392, BW=1568KiB/s (1606kB/s)(5180KiB/3303msec)
    slat (usec): min=16, max=1678, avg=60.05, stdev=72.92
    clat (usec): min=4254, max=28401, avg=20105.84, stdev=2519.83
     lat (usec): min=4291, max=28444, avg=20167.31, stdev=2521.10
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[16450], 10.00th=[17171], 20.00th=[18220],
     | 30.00th=[19006], 40.00th=[19530], 50.00th=[20055], 60.00th=[20579],
     | 70.00th=[21103], 80.00th=[22152], 90.00th=[23462], 95.00th=[24249],
     | 99.00th=[26346], 99.50th=[27132], 99.90th=[28181], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1480, max= 1624, per=24.50%, avg=1549.33, stdev=50.00, samples=6
   iops        : min=  370, max=  406, avg=387.33, stdev=12.50, samples=6
  lat (msec)   : 4=0.08%, 10=0.27%, 20=53.48%, 50=46.17%
  cpu          : usr=1.03%, sys=7.66%, ctx=2618, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6076KiB/s (6222kB/s), 1511KiB/s-1540KiB/s (1547kB/s-1577kB/s), io=19.6MiB (20.5MB), run=3277-3303msec
  WRITE: bw=6325KiB/s (6477kB/s), 1568KiB/s-1600KiB/s (1606kB/s-1639kB/s), io=20.4MiB (21.4MB), run=3277-3303msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30461: Mon Mar 31 19:11:53 2025
   read: IOPS=7, BW=317MiB/s (333MB/s)(40.0MiB/126msec)
    slat (nsec): min=8672.7k, max=8672.7k, avg=8672652.00, stdev= 0.00
    clat (nsec): min=110663k, max=110663k, avg=110662946.00, stdev= 0.00
     lat (nsec): min=119368k, max=119368k, avg=119368335.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  111],  5.00th=[  111], 10.00th=[  111], 20.00th=[  111],
     | 30.00th=[  111], 40.00th=[  111], 50.00th=[  111], 60.00th=[  111],
     | 70.00th=[  111], 80.00th=[  111], 90.00th=[  111], 95.00th=[  111],
     | 99.00th=[  111], 99.50th=[  111], 99.90th=[  111], 99.95th=[  111],
     | 99.99th=[  111]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.80%, sys=6.40%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=317MiB/s (333MB/s), 317MiB/s-317MiB/s (333MB/s-333MB/s), io=40.0MiB (41.9MB), run=126-126msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout