== sanity test 398c: run fio to test AIO ================= 05:40:10 (1727775610)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0979036 s, 428 MB/s
osc.lustre-OST0000-osc-ffff8800b5aed800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=6665: Tue Oct  1 05:40:16 2024
  write: IOPS=624, BW=2499KiB/s (2559kB/s)(10.0MiB/4097msec)
    slat (usec): min=13, max=273, avg=48.75, stdev=32.82
    clat (usec): min=5977, max=40562, avg=25444.95, stdev=4749.35
     lat (usec): min=6042, max=40587, avg=25494.98, stdev=4752.41
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[19530], 10.00th=[20317], 20.00th=[21627],
     | 30.00th=[22676], 40.00th=[23725], 50.00th=[24511], 60.00th=[25297],
     | 70.00th=[26870], 80.00th=[29230], 90.00th=[32900], 95.00th=[34341],
     | 99.00th=[39584], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1928, max= 2960, per=24.85%, avg=2483.00, stdev=358.18, samples=8
   iops        : min=  482, max=  740, avg=620.75, stdev=89.54, samples=8
  lat (msec)   : 10=0.04%, 20=8.20%, 50=91.76%
  cpu          : usr=0.66%, sys=3.32%, ctx=882, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6666: Tue Oct  1 05:40:16 2024
  write: IOPS=625, BW=2501KiB/s (2561kB/s)(10.0MiB/4095msec)
    slat (usec): min=12, max=451, avg=58.73, stdev=38.64
    clat (usec): min=2583, max=40286, avg=25444.01, stdev=4692.64
     lat (usec): min=2694, max=40347, avg=25504.16, stdev=4699.93
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[19530], 10.00th=[20317], 20.00th=[21627],
     | 30.00th=[22676], 40.00th=[23725], 50.00th=[24249], 60.00th=[25560],
     | 70.00th=[27132], 80.00th=[29230], 90.00th=[32900], 95.00th=[34341],
     | 99.00th=[37487], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2032, max= 2952, per=24.91%, avg=2489.00, stdev=328.03, samples=8
   iops        : min=  508, max=  738, avg=622.25, stdev=82.01, samples=8
  lat (msec)   : 4=0.12%, 10=0.08%, 20=6.91%, 50=92.89%
  cpu          : usr=0.61%, sys=3.88%, ctx=1694, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6667: Tue Oct  1 05:40:16 2024
  write: IOPS=625, BW=2502KiB/s (2563kB/s)(10.0MiB/4092msec)
    slat (usec): min=13, max=643, avg=64.83, stdev=43.42
    clat (usec): min=3461, max=41139, avg=25446.24, stdev=4805.96
     lat (usec): min=3516, max=41202, avg=25512.44, stdev=4818.97
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[19530], 10.00th=[20317], 20.00th=[21627],
     | 30.00th=[22676], 40.00th=[23462], 50.00th=[24511], 60.00th=[25560],
     | 70.00th=[26870], 80.00th=[29492], 90.00th=[32900], 95.00th=[34341],
     | 99.00th=[38536], 99.50th=[39584], 99.90th=[41157], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 1976, max= 2944, per=24.95%, avg=2493.00, stdev=336.90, samples=8
   iops        : min=  494, max=  736, avg=623.25, stdev=84.22, samples=8
  lat (msec)   : 4=0.04%, 10=0.27%, 20=7.70%, 50=91.99%
  cpu          : usr=0.93%, sys=3.98%, ctx=2069, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6668: Tue Oct  1 05:40:16 2024
  write: IOPS=624, BW=2498KiB/s (2558kB/s)(10.0MiB/4100msec)
    slat (usec): min=12, max=348, avg=60.78, stdev=41.19
    clat (usec): min=4624, max=40424, avg=25424.66, stdev=4642.99
     lat (usec): min=4680, max=40482, avg=25486.75, stdev=4653.73
    clat percentiles (usec):
     |  1.00th=[17695],  5.00th=[19530], 10.00th=[20317], 20.00th=[21627],
     | 30.00th=[22676], 40.00th=[23725], 50.00th=[24511], 60.00th=[25560],
     | 70.00th=[26870], 80.00th=[29230], 90.00th=[32637], 95.00th=[34866],
     | 99.00th=[38011], 99.50th=[39584], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 2016, max= 2944, per=24.82%, avg=2479.25, stdev=331.78, samples=8
   iops        : min=  504, max=  736, avg=619.75, stdev=82.86, samples=8
  lat (msec)   : 10=0.20%, 20=7.58%, 50=92.23%
  cpu          : usr=0.56%, sys=4.03%, ctx=1885, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9990KiB/s (10.2MB/s), 2498KiB/s-2502KiB/s (2558kB/s-2563kB/s), io=40.0MiB (41.9MB), run=4092-4100msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=6681: Tue Oct  1 05:40:20 2024
   read: IOPS=436, BW=1746KiB/s (1787kB/s)(5048KiB/2892msec)
    slat (usec): min=13, max=2315, avg=1152.65, stdev=249.07
    clat (usec): min=1679, max=30426, avg=17161.93, stdev=2677.19
     lat (usec): min=1708, max=31411, avg=18316.38, stdev=2736.31
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[13566], 10.00th=[14353], 20.00th=[15139],
     | 30.00th=[15795], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20317], 95.00th=[21103],
     | 99.00th=[24249], 99.50th=[25560], 99.90th=[28967], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1688, max= 1760, per=24.96%, avg=1721.60, stdev=29.61, samples=5
   iops        : min=  422, max=  440, avg=430.40, stdev= 7.40, samples=5
  write: IOPS=448, BW=1795KiB/s (1838kB/s)(5192KiB/2892msec)
    slat (usec): min=12, max=407, avg=46.88, stdev=25.81
    clat (usec): min=4694, max=28998, avg=17708.89, stdev=2225.07
     lat (usec): min=4709, max=29028, avg=17756.93, stdev=2225.28
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14746], 10.00th=[15401], 20.00th=[16188],
     | 30.00th=[16712], 40.00th=[17171], 50.00th=[17695], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20579], 95.00th=[21365],
     | 99.00th=[23725], 99.50th=[24773], 99.90th=[26870], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1736, max= 1944, per=25.02%, avg=1796.80, stdev=84.78, samples=5
   iops        : min=  434, max=  486, avg=449.20, stdev=21.19, samples=5
  lat (msec)   : 2=0.12%, 4=0.04%, 10=0.70%, 20=87.23%, 50=11.91%
  cpu          : usr=0.86%, sys=7.06%, ctx=2359, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6682: Tue Oct  1 05:40:20 2024
   read: IOPS=428, BW=1712KiB/s (1753kB/s)(4976KiB/2906msec)
    slat (usec): min=704, max=2037, avg=1163.28, stdev=224.01
    clat (usec): min=8501, max=27413, avg=17231.72, stdev=2360.38
     lat (usec): min=9495, max=28821, avg=18396.88, stdev=2379.04
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13566], 10.00th=[14222], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[22938], 99.50th=[24249], 99.90th=[26608], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1632, max= 1720, per=24.29%, avg=1675.20, stdev=35.60, samples=5
   iops        : min=  408, max=  430, avg=418.80, stdev= 8.90, samples=5
  write: IOPS=452, BW=1811KiB/s (1855kB/s)(5264KiB/2906msec)
    slat (usec): min=14, max=302, avg=46.44, stdev=23.44
    clat (usec): min=8441, max=24998, avg=17697.04, stdev=1858.72
     lat (usec): min=8492, max=25062, avg=17744.64, stdev=1859.18
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[14877], 10.00th=[15401], 20.00th=[16057],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22414], 99.50th=[23200], 99.90th=[24511], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1752, max= 1880, per=25.11%, avg=1803.20, stdev=52.34, samples=5
   iops        : min=  438, max=  470, avg=450.80, stdev=13.08, samples=5
  lat (msec)   : 10=0.16%, 20=88.87%, 50=10.98%
  cpu          : usr=0.90%, sys=7.19%, ctx=2355, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6683: Tue Oct  1 05:40:20 2024
   read: IOPS=428, BW=1715KiB/s (1756kB/s)(4984KiB/2906msec)
    slat (usec): min=692, max=2631, avg=1172.25, stdev=237.17
    clat (usec): min=9290, max=29559, avg=17222.36, stdev=2361.42
     lat (usec): min=10252, max=30473, avg=18396.43, stdev=2373.42
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13698], 10.00th=[14353], 20.00th=[15008],
     | 30.00th=[15795], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[23462], 99.50th=[24249], 99.90th=[25297], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1536, max= 1848, per=24.52%, avg=1691.20, stdev=119.41, samples=5
   iops        : min=  384, max=  462, avg=422.80, stdev=29.85, samples=5
  write: IOPS=452, BW=1809KiB/s (1852kB/s)(5256KiB/2906msec)
    slat (usec): min=13, max=258, avg=45.44, stdev=23.08
    clat (usec): min=4983, max=29407, avg=17710.04, stdev=1968.98
     lat (usec): min=5019, max=29424, avg=17756.65, stdev=1967.46
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14877], 10.00th=[15533], 20.00th=[16188],
     | 30.00th=[16712], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22414], 99.50th=[23462], 99.90th=[29492], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1664, max= 2000, per=25.02%, avg=1796.80, stdev=134.53, samples=5
   iops        : min=  416, max=  500, avg=449.20, stdev=33.63, samples=5
  lat (msec)   : 10=0.16%, 20=89.41%, 50=10.43%
  cpu          : usr=0.86%, sys=7.13%, ctx=2348, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6684: Tue Oct  1 05:40:20 2024
   read: IOPS=434, BW=1739KiB/s (1781kB/s)(5060KiB/2909msec)
    slat (usec): min=680, max=4258, avg=1160.87, stdev=251.01
    clat (usec): min=1819, max=25380, avg=17288.85, stdev=2459.48
     lat (usec): min=3022, max=26559, avg=18451.57, stdev=2484.54
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13698], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16712], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20317], 95.00th=[21103],
     | 99.00th=[23200], 99.50th=[23725], 99.90th=[25297], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 1680, max= 1736, per=24.78%, avg=1709.60, stdev=27.66, samples=5
   iops        : min=  420, max=  434, avg=427.40, stdev= 6.91, samples=5
  write: IOPS=445, BW=1781KiB/s (1823kB/s)(5180KiB/2909msec)
    slat (usec): min=14, max=447, avg=46.33, stdev=26.48
    clat (usec): min=4395, max=24934, avg=17609.81, stdev=1948.19
     lat (usec): min=4414, max=24976, avg=17657.28, stdev=1947.77
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14746], 10.00th=[15401], 20.00th=[16188],
     | 30.00th=[16712], 40.00th=[17171], 50.00th=[17695], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[23200], 99.90th=[24249], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1724, max= 1800, per=24.45%, avg=1756.00, stdev=29.66, samples=5
   iops        : min=  431, max=  450, avg=439.00, stdev= 7.42, samples=5
  lat (msec)   : 2=0.04%, 10=0.39%, 20=88.79%, 50=10.78%
  cpu          : usr=0.89%, sys=7.08%, ctx=2365, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6899KiB/s (7064kB/s), 1712KiB/s-1746KiB/s (1753kB/s-1787kB/s), io=19.6MiB (20.5MB), run=2892-2909msec
  WRITE: bw=7182KiB/s (7354kB/s), 1781KiB/s-1811KiB/s (1823kB/s-1855kB/s), io=20.4MiB (21.4MB), run=2892-2909msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=6690: Tue Oct  1 05:40:21 2024
   read: IOPS=13, BW=548MiB/s (575MB/s)(40.0MiB/73msec)
    slat (nsec): min=8792.2k, max=8792.2k, avg=8792200.00, stdev= 0.00
    clat (nsec): min=59815k, max=59815k, avg=59814655.00, stdev= 0.00
     lat (nsec): min=68626k, max=68626k, avg=68625742.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[60031],  5.00th=[60031], 10.00th=[60031], 20.00th=[60031],
     | 30.00th=[60031], 40.00th=[60031], 50.00th=[60031], 60.00th=[60031],
     | 70.00th=[60031], 80.00th=[60031], 90.00th=[60031], 95.00th=[60031],
     | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
  lat (msec)   : 100=100.00%
  cpu          : usr=1.39%, sys=11.11%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=548MiB/s (575MB/s), 548MiB/s-548MiB/s (575MB/s-575MB/s), io=40.0MiB (41.9MB), run=73-73msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout