== sanity test 398c: run fio to test AIO ================= 05:23:51 (1713086631)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.088405 s, 474 MB/s
osc.lustre-OST0000-osc-ffff88012c298000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=11157: Sun Apr 14 05:23:56 2024
  write: IOPS=785, BW=3140KiB/s (3216kB/s)(10.0MiB/3261msec)
    slat (usec): min=12, max=194, avg=41.37, stdev=21.50
    clat (usec): min=4103, max=36079, avg=20237.74, stdev=2901.94
     lat (usec): min=4163, max=36121, avg=20280.29, stdev=2901.85
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[14222], 10.00th=[15926], 20.00th=[18220],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[23200], 95.00th=[23987],
     | 99.00th=[27395], 99.50th=[28443], 99.90th=[29492], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2832, max= 3792, per=25.04%, avg=3140.00, stdev=336.98, samples=6
   iops        : min=  708, max=  948, avg=785.00, stdev=84.24, samples=6
  lat (msec)   : 10=0.04%, 20=37.11%, 50=62.85%
  cpu          : usr=0.71%, sys=3.87%, ctx=1306, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11158: Sun Apr 14 05:23:56 2024
  write: IOPS=784, BW=3137KiB/s (3213kB/s)(10.0MiB/3264msec)
    slat (usec): min=12, max=336, avg=42.96, stdev=22.26
    clat (usec): min=2272, max=40543, avg=20251.31, stdev=3027.89
     lat (usec): min=2287, max=40574, avg=20295.37, stdev=3029.09
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14222], 10.00th=[15926], 20.00th=[18482],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[27919], 99.50th=[29492], 99.90th=[38536], 99.95th=[38536],
     | 99.99th=[40633]
   bw (  KiB/s): min= 2800, max= 3816, per=25.09%, avg=3146.67, stdev=349.58, samples=6
   iops        : min=  700, max=  954, avg=786.33, stdev=87.47, samples=6
  lat (msec)   : 4=0.04%, 10=0.04%, 20=38.36%, 50=61.56%
  cpu          : usr=0.64%, sys=4.11%, ctx=1390, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11159: Sun Apr 14 05:23:56 2024
  write: IOPS=786, BW=3145KiB/s (3220kB/s)(10.0MiB/3256msec)
    slat (usec): min=11, max=220, avg=42.40, stdev=22.03
    clat (usec): min=2073, max=41701, avg=20215.64, stdev=3226.97
     lat (usec): min=2128, max=41737, avg=20259.13, stdev=3228.66
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[14091], 10.00th=[15795], 20.00th=[18220],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[23200], 95.00th=[23987],
     | 99.00th=[27657], 99.50th=[29230], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 2816, max= 3880, per=25.13%, avg=3152.00, stdev=373.11, samples=6
   iops        : min=  704, max=  970, avg=788.00, stdev=93.28, samples=6
  lat (msec)   : 4=0.12%, 10=0.59%, 20=37.03%, 50=62.27%
  cpu          : usr=0.86%, sys=3.75%, ctx=1346, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11160: Sun Apr 14 05:23:56 2024
  write: IOPS=783, BW=3135KiB/s (3211kB/s)(10.0MiB/3266msec)
    slat (usec): min=12, max=258, avg=41.37, stdev=21.58
    clat (usec): min=2678, max=36396, avg=20269.31, stdev=2927.98
     lat (usec): min=2742, max=36429, avg=20311.95, stdev=2929.79
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[14222], 10.00th=[16319], 20.00th=[18220],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[27657], 99.50th=[28181], 99.90th=[36439], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 2784, max= 3768, per=25.05%, avg=3141.33, stdev=333.08, samples=6
   iops        : min=  696, max=  942, avg=785.33, stdev=83.27, samples=6
  lat (msec)   : 4=0.04%, 10=0.04%, 20=36.41%, 50=63.52%
  cpu          : usr=0.89%, sys=3.77%, ctx=1296, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.2MiB/s (12.8MB/s), 3135KiB/s-3145KiB/s (3211kB/s-3220kB/s), io=40.0MiB (41.9MB), run=3256-3266msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=11172: Sun Apr 14 05:23:59 2024
   read: IOPS=448, BW=1795KiB/s (1838kB/s)(5048KiB/2813msec)
    slat (usec): min=679, max=2708, avg=1180.05, stdev=245.50
    clat (usec): min=10673, max=28014, avg=16517.73, stdev=2339.90
     lat (usec): min=11816, max=28892, avg=17699.57, stdev=2371.43
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13042], 10.00th=[13698], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[23462], 99.90th=[25822], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1640, max= 1832, per=24.83%, avg=1771.20, stdev=78.10, samples=5
   iops        : min=  410, max=  458, avg=442.80, stdev=19.52, samples=5
  write: IOPS=461, BW=1846KiB/s (1890kB/s)(5192KiB/2813msec)
    slat (usec): min=13, max=421, avg=44.82, stdev=23.99
    clat (usec): min=12154, max=30071, avg=17221.07, stdev=1998.95
     lat (usec): min=12198, max=30097, avg=17267.13, stdev=1997.84
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14222], 10.00th=[14746], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22414], 99.50th=[22676], 99.90th=[27919], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1768, max= 1976, per=24.95%, avg=1852.80, stdev=79.92, samples=5
   iops        : min=  442, max=  494, avg=463.20, stdev=19.98, samples=5
  lat (msec)   : 20=92.07%, 50=7.93%
  cpu          : usr=1.24%, sys=6.97%, ctx=2658, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11173: Sun Apr 14 05:23:59 2024
   read: IOPS=442, BW=1769KiB/s (1811kB/s)(4976KiB/2813msec)
    slat (usec): min=650, max=2280, avg=1168.23, stdev=238.54
    clat (usec): min=9947, max=26523, avg=16676.12, stdev=2402.86
     lat (usec): min=10943, max=27625, avg=17846.13, stdev=2421.53
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13042], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18744], 90.00th=[19792], 95.00th=[20841],
     | 99.00th=[23200], 99.50th=[23462], 99.90th=[25560], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min= 1608, max= 1824, per=24.40%, avg=1740.80, stdev=80.52, samples=5
   iops        : min=  402, max=  456, avg=435.20, stdev=20.13, samples=5
  write: IOPS=467, BW=1871KiB/s (1916kB/s)(5264KiB/2813msec)
    slat (usec): min=12, max=1199, avg=45.35, stdev=43.47
    clat (usec): min=9886, max=23135, avg=17137.87, stdev=1933.08
     lat (usec): min=9938, max=23176, avg=17184.38, stdev=1932.59
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14091], 10.00th=[14615], 20.00th=[15533],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[21890], 99.50th=[22414], 99.90th=[22676], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 1776, max= 1968, per=25.21%, avg=1872.00, stdev=85.42, samples=5
   iops        : min=  444, max=  492, avg=468.00, stdev=21.35, samples=5
  lat (msec)   : 10=0.08%, 20=91.95%, 50=7.97%
  cpu          : usr=0.78%, sys=7.25%, ctx=2473, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11174: Sun Apr 14 05:23:59 2024
   read: IOPS=444, BW=1779KiB/s (1821kB/s)(4984KiB/2802msec)
    slat (usec): min=17, max=2452, avg=1169.59, stdev=248.75
    clat (usec): min=3464, max=24347, avg=16609.03, stdev=2269.88
     lat (usec): min=4966, max=25904, avg=17780.59, stdev=2291.99
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13173], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19268], 95.00th=[20579],
     | 99.00th=[22938], 99.50th=[23200], 99.90th=[24249], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 1672, max= 1928, per=24.87%, avg=1774.40, stdev=94.89, samples=5
   iops        : min=  418, max=  482, avg=443.60, stdev=23.72, samples=5
  write: IOPS=468, BW=1876KiB/s (1921kB/s)(5256KiB/2802msec)
    slat (usec): min=13, max=452, avg=45.56, stdev=25.33
    clat (usec): min=8690, max=27200, avg=17119.38, stdev=2023.34
     lat (usec): min=8758, max=27226, avg=17166.22, stdev=2022.16
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[13960], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[22152], 99.50th=[23200], 99.90th=[27132], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1760, max= 1928, per=25.10%, avg=1864.00, stdev=73.97, samples=5
   iops        : min=  440, max=  482, avg=466.00, stdev=18.49, samples=5
  lat (msec)   : 4=0.04%, 10=0.12%, 20=92.58%, 50=7.27%
  cpu          : usr=1.00%, sys=7.28%, ctx=2593, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11175: Sun Apr 14 05:23:59 2024
   read: IOPS=450, BW=1802KiB/s (1845kB/s)(5060KiB/2808msec)
    slat (usec): min=28, max=2697, avg=1161.06, stdev=258.04
    clat (usec): min=1668, max=26377, avg=16634.51, stdev=2506.53
     lat (usec): min=1705, max=27634, avg=17797.40, stdev=2548.92
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[13042], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[23725], 99.90th=[25560], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1760, max= 1888, per=25.30%, avg=1804.80, stdev=48.86, samples=5
   iops        : min=  440, max=  472, avg=451.20, stdev=12.21, samples=5
  write: IOPS=461, BW=1845KiB/s (1889kB/s)(5180KiB/2808msec)
    slat (usec): min=13, max=954, avg=47.19, stdev=45.45
    clat (usec): min=5833, max=32361, avg=17177.88, stdev=2085.10
     lat (usec): min=5880, max=32400, avg=17226.27, stdev=2084.23
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14091], 10.00th=[14746], 20.00th=[15664],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19792], 95.00th=[20317],
     | 99.00th=[21365], 99.50th=[21890], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1736, max= 1976, per=24.65%, avg=1830.40, stdev=89.16, samples=5
   iops        : min=  434, max=  494, avg=457.60, stdev=22.29, samples=5
  lat (msec)   : 2=0.08%, 4=0.08%, 10=0.27%, 20=91.80%, 50=7.77%
  cpu          : usr=0.96%, sys=7.09%, ctx=2671, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7134KiB/s (7305kB/s), 1769KiB/s-1802KiB/s (1811kB/s-1845kB/s), io=19.6MiB (20.5MB), run=2802-2813msec
  WRITE: bw=7427KiB/s (7605kB/s), 1845KiB/s-1876KiB/s (1889kB/s-1921kB/s), io=20.4MiB (21.4MB), run=2802-2813msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=11181: Sun Apr 14 05:24:01 2024
   read: IOPS=12, BW=488MiB/s (512MB/s)(40.0MiB/82msec)
    slat (nsec): min=5984.8k, max=5984.8k, avg=5984814.00, stdev= 0.00
    clat (nsec): min=71590k, max=71590k, avg=71589775.00, stdev= 0.00
     lat (nsec): min=77585k, max=77585k, avg=77585259.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[71828],  5.00th=[71828], 10.00th=[71828], 20.00th=[71828],
     | 30.00th=[71828], 40.00th=[71828], 50.00th=[71828], 60.00th=[71828],
     | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828],
     | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[71828]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.64%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=488MiB/s (512MB/s), 488MiB/s-488MiB/s (512MB/s-512MB/s), io=40.0MiB (41.9MB), run=82-82msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout