== sanity test 398c: run fio to test AIO ================= 07:35:28 (1734438928)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.136863 s, 306 MB/s
osc.lustre-OST0000-osc-ffff880131c84800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=29185: Tue Dec 17 07:35:36 2024
  write: IOPS=442, BW=1771KiB/s (1813kB/s)(10.0MiB/5783msec)
    slat (usec): min=12, max=392, avg=61.44, stdev=40.16
    clat (usec): min=5153, max=54940, avg=35842.19, stdev=8553.11
     lat (usec): min=5262, max=55094, avg=35905.30, stdev=8564.07
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[21890], 10.00th=[23200], 20.00th=[26346],
     | 30.00th=[30802], 40.00th=[34866], 50.00th=[36963], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[44303], 90.00th=[45876], 95.00th=[46924],
     | 99.00th=[50594], 99.50th=[53216], 99.90th=[53740], 99.95th=[54264],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1408, max= 2424, per=24.36%, avg=1725.00, stdev=330.72, samples=11
   iops        : min=  352, max=  606, avg=431.18, stdev=82.58, samples=11
  lat (msec)   : 10=0.04%, 20=2.93%, 50=95.86%, 100=1.17%
  cpu          : usr=0.61%, sys=3.08%, ctx=835, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=29186: Tue Dec 17 07:35:36 2024
  write: IOPS=443, BW=1775KiB/s (1818kB/s)(10.0MiB/5769msec)
    slat (usec): min=12, max=390, avg=82.80, stdev=49.60
    clat (usec): min=2078, max=55422, avg=35811.76, stdev=8525.32
     lat (usec): min=2148, max=55602, avg=35896.64, stdev=8543.42
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[21890], 10.00th=[23462], 20.00th=[26608],
     | 30.00th=[31065], 40.00th=[34866], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[44303], 90.00th=[45876], 95.00th=[46924],
     | 99.00th=[47973], 99.50th=[54264], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1384, max= 2408, per=24.59%, avg=1741.36, stdev=340.64, samples=11
   iops        : min=  346, max=  602, avg=435.27, stdev=85.10, samples=11
  lat (msec)   : 4=0.04%, 10=0.12%, 20=2.97%, 50=96.25%, 100=0.62%
  cpu          : usr=0.54%, sys=4.14%, ctx=1928, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=29187: Tue Dec 17 07:35:36 2024
  write: IOPS=442, BW=1771KiB/s (1814kB/s)(10.0MiB/5782msec)
    slat (usec): min=12, max=499, avg=75.34, stdev=47.19
    clat (usec): min=7977, max=55453, avg=35843.30, stdev=8497.98
     lat (usec): min=8078, max=55507, avg=35920.46, stdev=8513.70
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[21890], 10.00th=[23200], 20.00th=[26346],
     | 30.00th=[31327], 40.00th=[34866], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[44303], 90.00th=[45876], 95.00th=[46924],
     | 99.00th=[47973], 99.50th=[54264], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1360, max= 2424, per=24.46%, avg=1732.00, stdev=334.62, samples=11
   iops        : min=  340, max=  606, avg=433.00, stdev=83.66, samples=11
  lat (msec)   : 10=0.04%, 20=2.93%, 50=96.45%, 100=0.59%
  cpu          : usr=0.71%, sys=3.72%, ctx=1546, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=29188: Tue Dec 17 07:35:36 2024
  write: IOPS=442, BW=1771KiB/s (1814kB/s)(10.0MiB/5782msec)
    slat (usec): min=13, max=489, avg=76.92, stdev=47.37
    clat (usec): min=13582, max=55575, avg=35883.57, stdev=8438.63
     lat (usec): min=13638, max=55636, avg=35962.43, stdev=8454.10
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[21890], 10.00th=[23725], 20.00th=[26346],
     | 30.00th=[31327], 40.00th=[34866], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[44303], 90.00th=[45876], 95.00th=[46924],
     | 99.00th=[47973], 99.50th=[54264], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[55837]
   bw (  KiB/s): min= 1360, max= 2416, per=24.47%, avg=1732.73, stdev=332.78, samples=11
   iops        : min=  340, max=  604, avg=433.18, stdev=83.19, samples=11
  lat (msec)   : 20=2.54%, 50=96.76%, 100=0.70%
  cpu          : usr=0.47%, sys=4.08%, ctx=1650, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7083KiB/s (7253kB/s), 1771KiB/s-1775KiB/s (1813kB/s-1818kB/s), io=40.0MiB (41.9MB), run=5769-5783msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=29202: Tue Dec 17 07:35:39 2024
   read: IOPS=450, BW=1803KiB/s (1846kB/s)(5048KiB/2800msec)
    slat (usec): min=23, max=2909, avg=1130.74, stdev=252.57
    clat (usec): min=3536, max=27879, avg=16568.98, stdev=2732.75
     lat (usec): min=6448, max=29804, avg=17701.64, stdev=2765.97
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12780], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19792], 95.00th=[20841],
     | 99.00th=[25822], 99.50th=[26608], 99.90th=[27657], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1536, max= 1912, per=24.82%, avg=1779.20, stdev=144.51, samples=5
   iops        : min=  384, max=  478, avg=444.80, stdev=36.13, samples=5
  write: IOPS=463, BW=1854KiB/s (1899kB/s)(5192KiB/2800msec)
    slat (usec): min=12, max=576, avg=47.17, stdev=29.52
    clat (usec): min=9430, max=29766, avg=16971.20, stdev=2147.75
     lat (usec): min=9490, max=29816, avg=17019.57, stdev=2149.40
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[13960], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20317],
     | 99.00th=[24511], 99.50th=[26346], 99.90th=[28181], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1600, max= 1936, per=24.94%, avg=1860.80, stdev=145.94, samples=5
   iops        : min=  400, max=  484, avg=465.20, stdev=36.49, samples=5
  lat (msec)   : 4=0.04%, 10=0.12%, 20=92.58%, 50=7.27%
  cpu          : usr=0.96%, sys=7.25%, ctx=2674, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=29203: Tue Dec 17 07:35:39 2024
   read: IOPS=444, BW=1779KiB/s (1822kB/s)(4976KiB/2797msec)
    slat (usec): min=717, max=3064, avg=1144.84, stdev=239.66
    clat (usec): min=2034, max=28474, avg=16455.93, stdev=2468.52
     lat (usec): min=3382, max=30300, avg=17602.57, stdev=2516.68
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12780], 10.00th=[13566], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20317],
     | 99.00th=[24511], 99.50th=[24773], 99.90th=[28443], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1400, max= 1904, per=24.53%, avg=1758.40, stdev=205.32, samples=5
   iops        : min=  350, max=  476, avg=439.60, stdev=51.33, samples=5
  write: IOPS=470, BW=1882KiB/s (1927kB/s)(5264KiB/2797msec)
    slat (usec): min=13, max=1126, avg=48.37, stdev=38.49
    clat (usec): min=2438, max=26685, avg=16903.99, stdev=2171.00
     lat (usec): min=2527, max=26706, avg=16953.56, stdev=2171.82
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13960], 10.00th=[14615], 20.00th=[15270],
     | 30.00th=[15795], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20317],
     | 99.00th=[25035], 99.50th=[25822], 99.90th=[26608], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min= 1664, max= 2048, per=25.24%, avg=1883.20, stdev=140.35, samples=5
   iops        : min=  416, max=  512, avg=470.80, stdev=35.09, samples=5
  lat (msec)   : 4=0.08%, 10=0.08%, 20=93.95%, 50=5.90%
  cpu          : usr=0.89%, sys=7.47%, ctx=2797, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=29204: Tue Dec 17 07:35:39 2024
   read: IOPS=449, BW=1800KiB/s (1843kB/s)(4984KiB/2769msec)
    slat (usec): min=17, max=2892, avg=1131.70, stdev=275.44
    clat (usec): min=2689, max=29177, avg=16361.28, stdev=2777.75
     lat (usec): min=2872, max=30585, avg=17494.79, stdev=2837.33
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[12649], 10.00th=[13435], 20.00th=[14222],
     | 30.00th=[14877], 40.00th=[15664], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[25035], 99.50th=[25822], 99.90th=[27657], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1608, max= 1976, per=25.18%, avg=1804.80, stdev=143.96, samples=5
   iops        : min=  402, max=  494, avg=451.20, stdev=35.99, samples=5
  write: IOPS=474, BW=1898KiB/s (1944kB/s)(5256KiB/2769msec)
    slat (usec): min=13, max=173, avg=46.43, stdev=19.69
    clat (usec): min=2884, max=30829, avg=16971.38, stdev=2295.78
     lat (usec): min=2915, max=30852, avg=17019.12, stdev=2294.54
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13829], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19530], 95.00th=[20841],
     | 99.00th=[24249], 99.50th=[25560], 99.90th=[30802], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1696, max= 2040, per=25.33%, avg=1889.60, stdev=163.79, samples=5
   iops        : min=  424, max=  510, avg=472.40, stdev=40.95, samples=5
  lat (msec)   : 4=0.16%, 10=0.51%, 20=91.33%, 50=8.01%
  cpu          : usr=1.05%, sys=7.26%, ctx=2665, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=29205: Tue Dec 17 07:35:39 2024
   read: IOPS=452, BW=1810KiB/s (1853kB/s)(5060KiB/2796msec)
    slat (usec): min=21, max=2404, avg=1143.04, stdev=255.30
    clat (usec): min=10123, max=29279, avg=16470.44, stdev=2585.85
     lat (usec): min=10824, max=31021, avg=17615.27, stdev=2636.02
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[12780], 10.00th=[13435], 20.00th=[14222],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[25560], 99.50th=[26870], 99.90th=[28443], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1544, max= 1952, per=25.16%, avg=1803.20, stdev=160.56, samples=5
   iops        : min=  386, max=  488, avg=450.80, stdev=40.14, samples=5
  write: IOPS=463, BW=1853KiB/s (1897kB/s)(5180KiB/2796msec)
    slat (usec): min=13, max=762, avg=47.75, stdev=37.07
    clat (usec): min=8836, max=27478, avg=16984.10, stdev=2230.27
     lat (usec): min=9032, max=27541, avg=17033.05, stdev=2231.27
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13960], 10.00th=[14615], 20.00th=[15270],
     | 30.00th=[15795], 40.00th=[16319], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19530], 95.00th=[20841],
     | 99.00th=[24773], 99.50th=[26608], 99.90th=[27132], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1544, max= 1944, per=24.55%, avg=1832.00, stdev=164.05, samples=5
   iops        : min=  386, max=  486, avg=458.00, stdev=41.01, samples=5
  lat (msec)   : 10=0.08%, 20=92.50%, 50=7.42%
  cpu          : usr=1.32%, sys=6.83%, ctx=2743, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7167KiB/s (7339kB/s), 1779KiB/s-1810KiB/s (1822kB/s-1853kB/s), io=19.6MiB (20.5MB), run=2769-2800msec
  WRITE: bw=7461KiB/s (7641kB/s), 1853KiB/s-1898KiB/s (1897kB/s-1944kB/s), io=20.4MiB (21.4MB), run=2769-2800msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=29211: Tue Dec 17 07:35:41 2024
   read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec)
    slat (nsec): min=5869.8k, max=5869.8k, avg=5869753.00, stdev= 0.00
    clat (nsec): min=79831k, max=79831k, avg=79830503.00, stdev= 0.00
     lat (nsec): min=85713k, max=85713k, avg=85713235.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[80217],  5.00th=[80217], 10.00th=[80217], 20.00th=[80217],
     | 30.00th=[80217], 40.00th=[80217], 50.00th=[80217], 60.00th=[80217],
     | 70.00th=[80217], 80.00th=[80217], 90.00th=[80217], 95.00th=[80217],
     | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.74%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout