== sanity test 398c: run fio to test AIO ================= 07:33:18 (1734438798)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.103384 s, 406 MB/s
osc.lustre-OST0000-osc-ffff88012e16c000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=24470: Tue Dec 17 07:33:24 2024
  write: IOPS=562, BW=2249KiB/s (2303kB/s)(10.0MiB/4553msec)
    slat (usec): min=12, max=362, avg=61.11, stdev=36.11
    clat (usec): min=4182, max=46737, avg=28262.72, stdev=6454.04
     lat (usec): min=4241, max=46756, avg=28325.46, stdev=6463.72
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[20317], 10.00th=[21365], 20.00th=[22414],
     | 30.00th=[23725], 40.00th=[25035], 50.00th=[26608], 60.00th=[28705],
     | 70.00th=[31065], 80.00th=[33817], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[43779], 99.50th=[45351], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1576, max= 2760, per=24.98%, avg=2246.22, stdev=382.26, samples=9
   iops        : min=  394, max=  690, avg=561.56, stdev=95.57, samples=9
  lat (msec)   : 10=0.04%, 20=3.91%, 50=96.05%
  cpu          : usr=0.64%, sys=4.09%, ctx=1477, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24471: Tue Dec 17 07:33:24 2024
  write: IOPS=563, BW=2255KiB/s (2309kB/s)(10.0MiB/4542msec)
    slat (usec): min=12, max=431, avg=60.81, stdev=36.47
    clat (usec): min=4560, max=48900, avg=28243.48, stdev=6503.98
     lat (usec): min=4641, max=48932, avg=28305.83, stdev=6516.53
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[20317], 10.00th=[21103], 20.00th=[22414],
     | 30.00th=[23725], 40.00th=[25035], 50.00th=[26608], 60.00th=[28967],
     | 70.00th=[31065], 80.00th=[33817], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[44303], 99.50th=[44827], 99.90th=[46400], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min= 1664, max= 2800, per=25.03%, avg=2250.67, stdev=359.64, samples=9
   iops        : min=  416, max=  700, avg=562.67, stdev=89.91, samples=9
  lat (msec)   : 10=0.16%, 20=3.71%, 50=96.13%
  cpu          : usr=0.81%, sys=3.90%, ctx=1374, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24472: Tue Dec 17 07:33:24 2024
  write: IOPS=562, BW=2248KiB/s (2302kB/s)(10.0MiB/4555msec)
    slat (usec): min=12, max=572, avg=53.83, stdev=34.16
    clat (usec): min=2182, max=44753, avg=28243.94, stdev=6485.87
     lat (usec): min=2233, max=44821, avg=28299.26, stdev=6495.01
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[20055], 10.00th=[21365], 20.00th=[22414],
     | 30.00th=[23725], 40.00th=[25035], 50.00th=[26870], 60.00th=[28967],
     | 70.00th=[31065], 80.00th=[33817], 90.00th=[38011], 95.00th=[41681],
     | 99.00th=[43779], 99.50th=[43779], 99.90th=[44303], 99.95th=[44303],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1536, max= 2800, per=24.89%, avg=2238.22, stdev=388.80, samples=9
   iops        : min=  384, max=  700, avg=559.56, stdev=97.20, samples=9
  lat (msec)   : 4=0.08%, 10=0.08%, 20=3.32%, 50=96.52%
  cpu          : usr=0.53%, sys=3.69%, ctx=927, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24473: Tue Dec 17 07:33:24 2024
  write: IOPS=562, BW=2251KiB/s (2305kB/s)(10.0MiB/4550msec)
    slat (usec): min=12, max=528, avg=56.02, stdev=37.18
    clat (usec): min=4336, max=46785, avg=28287.31, stdev=6461.35
     lat (usec): min=4492, max=46818, avg=28344.81, stdev=6471.39
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[20317], 10.00th=[21365], 20.00th=[22414],
     | 30.00th=[23725], 40.00th=[25035], 50.00th=[26870], 60.00th=[28705],
     | 70.00th=[30802], 80.00th=[33817], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[44303], 99.50th=[44827], 99.90th=[45351], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1664, max= 2736, per=24.97%, avg=2245.33, stdev=350.41, samples=9
   iops        : min=  416, max=  684, avg=561.33, stdev=87.60, samples=9
  lat (msec)   : 10=0.04%, 20=2.46%, 50=97.50%
  cpu          : usr=0.48%, sys=3.87%, ctx=1221, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8992KiB/s (9208kB/s), 2248KiB/s-2255KiB/s (2302kB/s-2309kB/s), io=40.0MiB (41.9MB), run=4542-4555msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=24486: Tue Dec 17 07:33:28 2024
   read: IOPS=461, BW=1848KiB/s (1892kB/s)(5048KiB/2732msec)
    slat (usec): min=643, max=3947, avg=1113.95, stdev=233.81
    clat (usec): min=9694, max=28004, avg=16137.32, stdev=2371.59
     lat (usec): min=10629, max=29275, avg=17252.99, stdev=2399.26
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12256], 10.00th=[13042], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15533], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19006], 95.00th=[20055],
     | 99.00th=[21890], 99.50th=[22414], 99.90th=[26870], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1720, max= 1920, per=25.05%, avg=1840.00, stdev=73.54, samples=5
   iops        : min=  430, max=  480, avg=460.00, stdev=18.38, samples=5
  write: IOPS=475, BW=1900KiB/s (1946kB/s)(5192KiB/2732msec)
    slat (usec): min=12, max=536, avg=45.23, stdev=27.18
    clat (usec): min=1931, max=26948, avg=16608.01, stdev=1987.97
     lat (usec): min=1959, max=26977, avg=16654.45, stdev=1987.91
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13435], 10.00th=[14353], 20.00th=[15008],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[21103], 99.50th=[22414], 99.90th=[25822], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1864, max= 1976, per=24.98%, avg=1910.40, stdev=46.44, samples=5
   iops        : min=  466, max=  494, avg=477.60, stdev=11.61, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.04%, 20=95.78%, 50=4.10%
  cpu          : usr=1.03%, sys=7.14%, ctx=2771, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24487: Tue Dec 17 07:33:28 2024
   read: IOPS=455, BW=1822KiB/s (1866kB/s)(4976KiB/2731msec)
    slat (usec): min=657, max=3727, avg=1114.04, stdev=232.81
    clat (usec): min=8793, max=25086, avg=16099.56, stdev=2285.45
     lat (usec): min=9996, max=25886, avg=17215.36, stdev=2295.38
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12649], 10.00th=[13304], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[16057], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[22152], 99.50th=[23200], 99.90th=[24511], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1632, max= 1896, per=24.57%, avg=1804.80, stdev=102.39, samples=5
   iops        : min=  408, max=  474, avg=451.20, stdev=25.60, samples=5
  write: IOPS=481, BW=1927KiB/s (1974kB/s)(5264KiB/2731msec)
    slat (usec): min=13, max=333, avg=44.97, stdev=23.58
    clat (usec): min=10681, max=25030, avg=16646.29, stdev=1987.52
     lat (usec): min=10727, max=25073, avg=16692.30, stdev=1988.05
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13435], 10.00th=[14222], 20.00th=[15008],
     | 30.00th=[15533], 40.00th=[16057], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[20055],
     | 99.00th=[21365], 99.50th=[22152], 99.90th=[23200], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1840, max= 2040, per=25.36%, avg=1939.20, stdev=86.27, samples=5
   iops        : min=  460, max=  510, avg=484.80, stdev=21.57, samples=5
  lat (msec)   : 10=0.12%, 20=95.00%, 50=4.88%
  cpu          : usr=0.88%, sys=7.36%, ctx=2758, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24488: Tue Dec 17 07:33:28 2024
   read: IOPS=456, BW=1828KiB/s (1872kB/s)(4984KiB/2727msec)
    slat (usec): min=676, max=3732, avg=1109.85, stdev=227.49
    clat (usec): min=3114, max=28713, avg=16059.59, stdev=2369.45
     lat (usec): min=4332, max=29623, avg=17171.04, stdev=2386.56
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12387], 10.00th=[13173], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15270], 50.00th=[15926], 60.00th=[16581],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21890], 99.50th=[22676], 99.90th=[26084], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1744, max= 1904, per=24.90%, avg=1828.80, stdev=61.86, samples=5
   iops        : min=  436, max=  476, avg=457.20, stdev=15.47, samples=5
  write: IOPS=481, BW=1927KiB/s (1974kB/s)(5256KiB/2727msec)
    slat (usec): min=12, max=307, avg=44.68, stdev=22.67
    clat (usec): min=8037, max=28525, avg=16685.04, stdev=2003.85
     lat (usec): min=8097, max=28562, avg=16730.88, stdev=2002.65
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13698], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21365], 99.50th=[25035], 99.90th=[28443], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1800, max= 2008, per=25.17%, avg=1924.80, stdev=80.32, samples=5
   iops        : min=  450, max=  502, avg=481.20, stdev=20.08, samples=5
  lat (msec)   : 4=0.04%, 10=0.16%, 20=95.43%, 50=4.38%
  cpu          : usr=0.73%, sys=7.48%, ctx=2743, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24489: Tue Dec 17 07:33:28 2024
   read: IOPS=466, BW=1867KiB/s (1912kB/s)(5060KiB/2710msec)
    slat (usec): min=20, max=3787, avg=1106.50, stdev=253.21
    clat (usec): min=1205, max=24825, avg=15968.91, stdev=2545.63
     lat (usec): min=1346, max=25757, avg=17077.13, stdev=2595.51
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[12256], 10.00th=[13042], 20.00th=[13960],
     | 30.00th=[14746], 40.00th=[15401], 50.00th=[16057], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21627], 99.50th=[22938], 99.90th=[24773], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 1816, max= 1928, per=25.38%, avg=1864.00, stdev=42.71, samples=5
   iops        : min=  454, max=  482, avg=466.00, stdev=10.68, samples=5
  write: IOPS=477, BW=1911KiB/s (1957kB/s)(5180KiB/2710msec)
    slat (usec): min=12, max=363, avg=44.45, stdev=25.88
    clat (usec): min=3015, max=29687, avg=16670.97, stdev=2068.20
     lat (usec): min=3073, max=29746, avg=16716.45, stdev=2067.67
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[13698], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21365], 99.50th=[22676], 99.90th=[24511], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1848, max= 1984, per=25.00%, avg=1912.00, stdev=56.28, samples=5
   iops        : min=  462, max=  496, avg=478.00, stdev=14.07, samples=5
  lat (msec)   : 2=0.08%, 4=0.20%, 10=0.70%, 20=95.55%, 50=3.48%
  cpu          : usr=1.37%, sys=6.68%, ctx=2762, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7346KiB/s (7522kB/s), 1822KiB/s-1867KiB/s (1866kB/s-1912kB/s), io=19.6MiB (20.5MB), run=2710-2732msec
  WRITE: bw=7647KiB/s (7831kB/s), 1900KiB/s-1927KiB/s (1946kB/s-1974kB/s), io=20.4MiB (21.4MB), run=2710-2732msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=24495: Tue Dec 17 07:33:29 2024
   read: IOPS=10, BW=440MiB/s (461MB/s)(40.0MiB/91msec)
    slat (nsec): min=5165.5k, max=5165.5k, avg=5165499.00, stdev= 0.00
    clat (nsec): min=81548k, max=81548k, avg=81547797.00, stdev= 0.00
     lat (nsec): min=86724k, max=86724k, avg=86723571.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[81265],  5.00th=[81265], 10.00th=[81265], 20.00th=[81265],
     | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265],
     | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265],
     | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.56%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=440MiB/s (461MB/s), 440MiB/s-440MiB/s (461MB/s-461MB/s), io=40.0MiB (41.9MB), run=91-91msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout