== sanity test 398c: run fio to test AIO ================= 14:42:38 (1744137758)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.114854 s, 365 MB/s
osc.lustre-OST0000-osc-ffff8800b6f17800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=2602: Tue Apr  8 14:42:43 2025
  write: IOPS=776, BW=3106KiB/s (3180kB/s)(10.0MiB/3297msec)
    slat (usec): min=13, max=454, avg=52.55, stdev=33.74
    clat (usec): min=3875, max=38249, avg=20470.62, stdev=2532.09
     lat (usec): min=3931, max=38295, avg=20524.52, stdev=2523.94
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15008], 10.00th=[17171], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21890], 80.00th=[22414], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[24511], 99.50th=[25035], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[38011]
   bw (  KiB/s): min= 2952, max= 3656, per=24.86%, avg=3086.67, stdev=279.90, samples=6
   iops        : min=  738, max=  914, avg=771.67, stdev=69.98, samples=6
  lat (msec)   : 4=0.04%, 10=0.08%, 20=33.28%, 50=66.60%
  cpu          : usr=0.82%, sys=4.52%, ctx=1682, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2603: Tue Apr  8 14:42:43 2025
  write: IOPS=776, BW=3106KiB/s (3180kB/s)(10.0MiB/3297msec)
    slat (usec): min=12, max=297, avg=41.92, stdev=28.30
    clat (usec): min=11115, max=31553, avg=20444.47, stdev=2469.59
     lat (usec): min=11218, max=31596, avg=20487.44, stdev=2464.77
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14877], 10.00th=[17171], 20.00th=[18744],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21627],
     | 70.00th=[21890], 80.00th=[22414], 90.00th=[22938], 95.00th=[23462],
     | 99.00th=[24249], 99.50th=[24511], 99.90th=[24773], 99.95th=[25035],
     | 99.99th=[31589]
   bw (  KiB/s): min= 2944, max= 3592, per=24.77%, avg=3074.67, stdev=256.38, samples=6
   iops        : min=  736, max=  898, avg=768.67, stdev=64.10, samples=6
  lat (msec)   : 20=35.23%, 50=64.77%
  cpu          : usr=0.76%, sys=3.70%, ctx=1044, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2604: Tue Apr  8 14:42:43 2025
  write: IOPS=779, BW=3116KiB/s (3191kB/s)(10.0MiB/3286msec)
    slat (usec): min=12, max=421, avg=51.92, stdev=35.27
    clat (usec): min=3041, max=40909, avg=20413.08, stdev=2869.65
     lat (usec): min=3082, max=40933, avg=20466.15, stdev=2859.90
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[14746], 10.00th=[16909], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21890], 80.00th=[22414], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[24511], 99.50th=[25560], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[41157]
   bw (  KiB/s): min= 2952, max= 3760, per=25.01%, avg=3105.33, stdev=321.45, samples=6
   iops        : min=  738, max=  940, avg=776.33, stdev=80.36, samples=6
  lat (msec)   : 4=0.12%, 10=0.43%, 20=33.71%, 50=65.74%
  cpu          : usr=0.73%, sys=4.75%, ctx=1557, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2605: Tue Apr  8 14:42:43 2025
  write: IOPS=775, BW=3104KiB/s (3178kB/s)(10.0MiB/3299msec)
    slat (usec): min=12, max=432, avg=49.12, stdev=30.31
    clat (usec): min=5006, max=32763, avg=20464.71, stdev=2493.76
     lat (usec): min=5073, max=32818, avg=20514.97, stdev=2489.89
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14746], 10.00th=[17171], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21890], 80.00th=[22414], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[24511], 99.50th=[24511], 99.90th=[30540], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 2944, max= 3632, per=24.80%, avg=3078.67, stdev=272.21, samples=6
   iops        : min=  736, max=  908, avg=769.67, stdev=68.05, samples=6
  lat (msec)   : 10=0.04%, 20=33.05%, 50=66.91%
  cpu          : usr=0.52%, sys=4.46%, ctx=1475, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.1MiB/s (12.7MB/s), 3104KiB/s-3116KiB/s (3178kB/s-3191kB/s), io=40.0MiB (41.9MB), run=3286-3299msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=2617: Tue Apr  8 14:42:47 2025
   read: IOPS=453, BW=1813KiB/s (1856kB/s)(5048KiB/2785msec)
    slat (usec): min=22, max=2704, avg=1099.62, stdev=257.56
    clat (usec): min=2127, max=28070, avg=16479.33, stdev=2520.99
     lat (usec): min=2156, max=29202, avg=17580.85, stdev=2545.02
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[13042], 10.00th=[13829], 20.00th=[14615],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[22152], 99.50th=[22938], 99.90th=[27395], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1704, max= 1864, per=24.92%, avg=1788.80, stdev=66.84, samples=5
   iops        : min=  426, max=  466, avg=447.20, stdev=16.71, samples=5
  write: IOPS=466, BW=1864KiB/s (1909kB/s)(5192KiB/2785msec)
    slat (usec): min=12, max=360, avg=39.17, stdev=20.77
    clat (usec): min=7615, max=31419, avg=17112.45, stdev=1984.91
     lat (usec): min=7644, max=31459, avg=17152.60, stdev=1985.50
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14353], 10.00th=[14877], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[22152], 99.50th=[23987], 99.90th=[27919], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1792, max= 1952, per=25.03%, avg=1870.40, stdev=61.03, samples=5
   iops        : min=  448, max=  488, avg=467.60, stdev=15.26, samples=5
  lat (msec)   : 4=0.35%, 10=0.27%, 20=92.97%, 50=6.41%
  cpu          : usr=1.22%, sys=6.32%, ctx=2445, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2618: Tue Apr  8 14:42:47 2025
   read: IOPS=445, BW=1783KiB/s (1826kB/s)(4976KiB/2791msec)
    slat (usec): min=638, max=2746, avg=1109.94, stdev=233.81
    clat (usec): min=3213, max=25732, avg=16367.73, stdev=2234.50
     lat (usec): min=4273, max=26700, avg=17479.36, stdev=2245.00
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13042], 10.00th=[13698], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[22152], 99.50th=[22676], 99.90th=[25297], 99.95th=[25822],
     | 99.99th=[25822]
   bw (  KiB/s): min= 1592, max= 1904, per=24.46%, avg=1755.20, stdev=119.81, samples=5
   iops        : min=  398, max=  476, avg=438.80, stdev=29.95, samples=5
  write: IOPS=471, BW=1886KiB/s (1931kB/s)(5264KiB/2791msec)
    slat (usec): min=13, max=1530, avg=39.66, stdev=44.94
    clat (usec): min=8671, max=24933, avg=17190.47, stdev=1828.07
     lat (usec): min=8705, max=24972, avg=17231.13, stdev=1827.20
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14353], 10.00th=[14877], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19530], 95.00th=[20055],
     | 99.00th=[21627], 99.50th=[22414], 99.90th=[23987], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1776, max= 1952, per=25.14%, avg=1878.40, stdev=80.48, samples=5
   iops        : min=  444, max=  488, avg=469.60, stdev=20.12, samples=5
  lat (msec)   : 4=0.04%, 10=0.16%, 20=93.91%, 50=5.90%
  cpu          : usr=0.90%, sys=6.74%, ctx=2424, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2619: Tue Apr  8 14:42:47 2025
   read: IOPS=445, BW=1783KiB/s (1825kB/s)(4984KiB/2796msec)
    slat (usec): min=594, max=2578, avg=1105.89, stdev=234.98
    clat (usec): min=1668, max=23637, avg=16459.14, stdev=2322.15
     lat (usec): min=2740, max=24625, avg=17566.68, stdev=2328.70
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[13173], 10.00th=[13698], 20.00th=[14484],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20579],
     | 99.00th=[22414], 99.50th=[22676], 99.90th=[23200], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 1608, max= 1928, per=24.48%, avg=1756.80, stdev=132.13, samples=5
   iops        : min=  402, max=  482, avg=439.20, stdev=33.03, samples=5
  write: IOPS=469, BW=1880KiB/s (1925kB/s)(5256KiB/2796msec)
    slat (usec): min=12, max=961, avg=38.68, stdev=32.14
    clat (usec): min=4659, max=26531, avg=17061.42, stdev=1981.92
     lat (usec): min=4681, max=26560, avg=17100.94, stdev=1980.02
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[14222], 10.00th=[14877], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16712], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19268], 95.00th=[20317],
     | 99.00th=[22152], 99.50th=[22938], 99.90th=[24249], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min= 1688, max= 2064, per=24.84%, avg=1856.00, stdev=161.49, samples=5
   iops        : min=  422, max=  516, avg=464.00, stdev=40.37, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.47%, 20=93.24%, 50=6.21%
  cpu          : usr=0.79%, sys=6.87%, ctx=2351, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2620: Tue Apr  8 14:42:47 2025
   read: IOPS=452, BW=1812KiB/s (1855kB/s)(5060KiB/2793msec)
    slat (usec): min=647, max=2637, avg=1106.82, stdev=220.79
    clat (usec): min=4577, max=23914, avg=16436.70, stdev=2227.02
     lat (usec): min=5662, max=25195, avg=17545.29, stdev=2235.48
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[12911], 10.00th=[13698], 20.00th=[14484],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19268], 95.00th=[20317],
     | 99.00th=[21365], 99.50th=[21627], 99.90th=[23462], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min= 1656, max= 1896, per=25.08%, avg=1800.00, stdev=91.04, samples=5
   iops        : min=  414, max=  474, avg=450.00, stdev=22.76, samples=5
  write: IOPS=463, BW=1855KiB/s (1899kB/s)(5180KiB/2793msec)
    slat (usec): min=12, max=1147, avg=39.53, stdev=36.54
    clat (usec): min=8324, max=24805, avg=17063.86, stdev=1809.63
     lat (usec): min=8352, max=24851, avg=17104.34, stdev=1811.72
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14353], 10.00th=[14746], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21103], 99.50th=[22676], 99.90th=[24773], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 1672, max= 1984, per=24.52%, avg=1832.00, stdev=117.44, samples=5
   iops        : min=  418, max=  496, avg=458.00, stdev=29.36, samples=5
  lat (msec)   : 10=0.23%, 20=94.18%, 50=5.59%
  cpu          : usr=0.97%, sys=6.66%, ctx=2371, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7177KiB/s (7350kB/s), 1783KiB/s-1813KiB/s (1825kB/s-1856kB/s), io=19.6MiB (20.5MB), run=2785-2796msec
  WRITE: bw=7472KiB/s (7651kB/s), 1855KiB/s-1886KiB/s (1899kB/s-1931kB/s), io=20.4MiB (21.4MB), run=2785-2796msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=2626: Tue Apr  8 14:42:48 2025
   read: IOPS=10, BW=426MiB/s (446MB/s)(40.0MiB/94msec)
    slat (nsec): min=5742.5k, max=5742.5k, avg=5742503.00, stdev= 0.00
    clat (nsec): min=83574k, max=83574k, avg=83573952.00, stdev= 0.00
     lat (nsec): min=89329k, max=89329k, avg=89329395.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[83362],  5.00th=[83362], 10.00th=[83362], 20.00th=[83362],
     | 30.00th=[83362], 40.00th=[83362], 50.00th=[83362], 60.00th=[83362],
     | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362],
     | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[83362]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.38%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=40.0MiB (41.9MB), run=94-94msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout