== sanity test 398c: run fio to test AIO ================= 12:16:23 (1746634583)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0822679 s, 510 MB/s
osc.lustre-OST0000-osc-ffff88012d395800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=28706: Wed May  7 12:16:27 2025
  write: IOPS=820, BW=3283KiB/s (3362kB/s)(10.0MiB/3119msec)
    slat (usec): min=11, max=265, avg=43.39, stdev=21.40
    clat (usec): min=6724, max=31939, avg=19393.32, stdev=2285.43
     lat (usec): min=6770, max=32042, avg=19437.85, stdev=2287.16
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[15270], 10.00th=[17171], 20.00th=[18220],
     | 30.00th=[18744], 40.00th=[19268], 50.00th=[19530], 60.00th=[20055],
     | 70.00th=[20317], 80.00th=[20841], 90.00th=[21627], 95.00th=[22414],
     | 99.00th=[24511], 99.50th=[27132], 99.90th=[31851], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 3072, max= 3664, per=24.99%, avg=3280.00, stdev=200.67, samples=6
   iops        : min=  768, max=  916, avg=820.00, stdev=50.17, samples=6
  lat (msec)   : 10=0.62%, 20=60.43%, 50=38.95%
  cpu          : usr=0.64%, sys=4.11%, ctx=1640, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28707: Wed May  7 12:16:27 2025
  write: IOPS=820, BW=3281KiB/s (3360kB/s)(10.0MiB/3121msec)
    slat (usec): min=10, max=343, avg=43.05, stdev=23.77
    clat (usec): min=1903, max=27608, avg=19368.34, stdev=2246.20
     lat (usec): min=1952, max=27649, avg=19412.45, stdev=2249.41
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[15008], 10.00th=[16909], 20.00th=[17957],
     | 30.00th=[18744], 40.00th=[19268], 50.00th=[19530], 60.00th=[20055],
     | 70.00th=[20317], 80.00th=[20841], 90.00th=[21627], 95.00th=[22414],
     | 99.00th=[24249], 99.50th=[25822], 99.90th=[27657], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 3072, max= 3624, per=24.96%, avg=3276.00, stdev=186.37, samples=6
   iops        : min=  768, max=  906, avg=819.00, stdev=46.59, samples=6
  lat (msec)   : 2=0.04%, 10=0.74%, 20=59.14%, 50=40.08%
  cpu          : usr=0.74%, sys=3.88%, ctx=1461, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28708: Wed May  7 12:16:27 2025
  write: IOPS=823, BW=3294KiB/s (3373kB/s)(10.0MiB/3109msec)
    slat (usec): min=10, max=369, avg=42.02, stdev=21.70
    clat (usec): min=2557, max=35944, avg=19336.21, stdev=2465.03
     lat (usec): min=2617, max=35984, avg=19379.44, stdev=2465.90
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[15270], 10.00th=[16909], 20.00th=[17957],
     | 30.00th=[18744], 40.00th=[19268], 50.00th=[19530], 60.00th=[20055],
     | 70.00th=[20317], 80.00th=[20841], 90.00th=[21627], 95.00th=[22414],
     | 99.00th=[24249], 99.50th=[26608], 99.90th=[31589], 99.95th=[33817],
     | 99.99th=[35914]
   bw (  KiB/s): min= 3064, max= 3720, per=25.08%, avg=3290.67, stdev=225.06, samples=6
   iops        : min=  766, max=  930, avg=822.67, stdev=56.27, samples=6
  lat (msec)   : 4=0.04%, 10=1.09%, 20=59.45%, 50=39.41%
  cpu          : usr=0.87%, sys=3.70%, ctx=1393, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28709: Wed May  7 12:16:27 2025
  write: IOPS=820, BW=3281KiB/s (3360kB/s)(10.0MiB/3121msec)
    slat (usec): min=10, max=561, avg=44.13, stdev=24.83
    clat (usec): min=8210, max=33581, avg=19384.44, stdev=2316.84
     lat (usec): min=8243, max=33635, avg=19429.70, stdev=2318.53
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[15401], 10.00th=[17171], 20.00th=[17957],
     | 30.00th=[18744], 40.00th=[19268], 50.00th=[19530], 60.00th=[20055],
     | 70.00th=[20317], 80.00th=[20841], 90.00th=[21627], 95.00th=[22414],
     | 99.00th=[24249], 99.50th=[27132], 99.90th=[27395], 99.95th=[33424],
     | 99.99th=[33817]
   bw (  KiB/s): min= 3064, max= 3664, per=24.97%, avg=3277.33, stdev=201.67, samples=6
   iops        : min=  766, max=  916, avg=819.33, stdev=50.42, samples=6
  lat (msec)   : 10=1.25%, 20=58.98%, 50=39.77%
  cpu          : usr=0.64%, sys=3.97%, ctx=1677, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.8MiB/s (13.4MB/s), 3281KiB/s-3294KiB/s (3360kB/s-3373kB/s), io=40.0MiB (41.9MB), run=3109-3121msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=28722: Wed May  7 12:16:31 2025
   read: IOPS=470, BW=1880KiB/s (1925kB/s)(5048KiB/2685msec)
    slat (usec): min=624, max=2831, avg=1075.37, stdev=232.49
    clat (usec): min=3012, max=27424, avg=15871.17, stdev=2493.62
     lat (usec): min=3692, max=28621, avg=16948.05, stdev=2542.15
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12256], 10.00th=[12780], 20.00th=[13829],
     | 30.00th=[14615], 40.00th=[15139], 50.00th=[15795], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18744], 95.00th=[20317],
     | 99.00th=[23200], 99.50th=[23987], 99.90th=[26346], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1776, max= 2032, per=25.20%, avg=1881.60, stdev=126.92, samples=5
   iops        : min=  444, max=  508, avg=470.40, stdev=31.73, samples=5
  write: IOPS=483, BW=1934KiB/s (1980kB/s)(5192KiB/2685msec)
    slat (usec): min=12, max=1194, avg=43.07, stdev=57.17
    clat (usec): min=2407, max=26385, avg=16364.51, stdev=2125.43
     lat (usec): min=2451, max=26422, avg=16408.62, stdev=2127.96
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13566], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15664], 50.00th=[16188], 60.00th=[16581],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19006], 95.00th=[20317],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[25822], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1768, max= 2064, per=25.09%, avg=1950.40, stdev=118.44, samples=5
   iops        : min=  442, max=  516, avg=487.60, stdev=29.61, samples=5
  lat (msec)   : 4=0.12%, 10=0.08%, 20=94.26%, 50=5.55%
  cpu          : usr=0.86%, sys=6.74%, ctx=2279, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28723: Wed May  7 12:16:31 2025
   read: IOPS=464, BW=1857KiB/s (1902kB/s)(4976KiB/2679msec)
    slat (usec): min=640, max=2500, avg=1066.32, stdev=230.42
    clat (usec): min=4283, max=24996, avg=15911.08, stdev=2482.30
     lat (usec): min=5099, max=25992, avg=16978.84, stdev=2528.98
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12256], 10.00th=[12780], 20.00th=[13829],
     | 30.00th=[14484], 40.00th=[15270], 50.00th=[15795], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19006], 95.00th=[20317],
     | 99.00th=[22676], 99.50th=[23462], 99.90th=[24511], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1688, max= 2080, per=24.75%, avg=1848.00, stdev=160.30, samples=5
   iops        : min=  422, max=  520, avg=462.00, stdev=40.07, samples=5
  write: IOPS=491, BW=1965KiB/s (2012kB/s)(5264KiB/2679msec)
    slat (usec): min=11, max=1013, avg=40.54, stdev=34.06
    clat (usec): min=3988, max=27189, avg=16340.33, stdev=2255.72
     lat (usec): min=4039, max=27213, avg=16381.90, stdev=2256.92
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13173], 10.00th=[13829], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[16188], 60.00th=[16581],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19268], 95.00th=[20317],
     | 99.00th=[23462], 99.50th=[24773], 99.90th=[27132], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1800, max= 2160, per=25.62%, avg=1992.00, stdev=129.37, samples=5
   iops        : min=  450, max=  540, avg=498.00, stdev=32.34, samples=5
  lat (msec)   : 4=0.04%, 10=0.23%, 20=93.95%, 50=5.78%
  cpu          : usr=0.67%, sys=6.87%, ctx=2240, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28724: Wed May  7 12:16:31 2025
   read: IOPS=468, BW=1873KiB/s (1918kB/s)(4984KiB/2661msec)
    slat (usec): min=20, max=3134, avg=1062.75, stdev=257.49
    clat (usec): min=2337, max=33527, avg=15773.74, stdev=2739.76
     lat (usec): min=2358, max=34634, avg=16837.96, stdev=2818.85
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[11994], 10.00th=[12780], 20.00th=[13698],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15664], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17957], 90.00th=[19006], 95.00th=[20317],
     | 99.00th=[22676], 99.50th=[22938], 99.90th=[24511], 99.95th=[33424],
     | 99.99th=[33424]
   bw (  KiB/s): min= 1720, max= 2032, per=25.32%, avg=1891.20, stdev=138.98, samples=5
   iops        : min=  430, max=  508, avg=472.80, stdev=34.74, samples=5
  write: IOPS=493, BW=1975KiB/s (2023kB/s)(5256KiB/2661msec)
    slat (usec): min=11, max=1221, avg=42.89, stdev=47.64
    clat (usec): min=7020, max=28141, avg=16316.68, stdev=2107.73
     lat (usec): min=7064, max=28180, avg=16360.64, stdev=2108.93
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13435], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[21890], 99.50th=[22676], 99.90th=[28181], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1856, max= 2088, per=25.44%, avg=1977.60, stdev=92.16, samples=5
   iops        : min=  464, max=  522, avg=494.40, stdev=23.04, samples=5
  lat (msec)   : 4=0.27%, 10=0.74%, 20=94.22%, 50=4.77%
  cpu          : usr=0.94%, sys=6.84%, ctx=2306, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28725: Wed May  7 12:16:31 2025
   read: IOPS=470, BW=1883KiB/s (1928kB/s)(5060KiB/2687msec)
    slat (usec): min=654, max=2791, avg=1066.01, stdev=231.02
    clat (usec): min=8004, max=26335, avg=15934.59, stdev=2488.38
     lat (usec): min=8939, max=27375, avg=17002.12, stdev=2537.00
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12387], 10.00th=[12911], 20.00th=[13698],
     | 30.00th=[14484], 40.00th=[15270], 50.00th=[15795], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19268], 95.00th=[20579],
     | 99.00th=[22414], 99.50th=[22676], 99.90th=[24511], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1792, max= 2080, per=25.28%, avg=1888.00, stdev=123.29, samples=5
   iops        : min=  448, max=  520, avg=472.00, stdev=30.82, samples=5
  write: IOPS=481, BW=1928KiB/s (1974kB/s)(5180KiB/2687msec)
    slat (usec): min=11, max=263, avg=41.02, stdev=20.70
    clat (usec): min=8944, max=25241, avg=16337.40, stdev=2002.01
     lat (usec): min=8971, max=25273, avg=16379.40, stdev=2003.04
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13435], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15664], 50.00th=[16057], 60.00th=[16581],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[22414], 99.50th=[23200], 99.90th=[23987], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 1776, max= 2128, per=24.84%, avg=1931.20, stdev=128.45, samples=5
   iops        : min=  444, max=  532, avg=482.80, stdev=32.11, samples=5
  lat (msec)   : 10=0.23%, 20=94.26%, 50=5.51%
  cpu          : usr=1.01%, sys=6.63%, ctx=2372, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7469KiB/s (7648kB/s), 1857KiB/s-1883KiB/s (1902kB/s-1928kB/s), io=19.6MiB (20.5MB), run=2661-2687msec
  WRITE: bw=7775KiB/s (7962kB/s), 1928KiB/s-1975KiB/s (1974kB/s-2023kB/s), io=20.4MiB (21.4MB), run=2661-2687msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=28730: Wed May  7 12:16:32 2025
   read: IOPS=11, BW=449MiB/s (471MB/s)(40.0MiB/89msec)
    slat (nsec): min=6756.1k, max=6756.1k, avg=6756095.00, stdev= 0.00
    clat (nsec): min=78201k, max=78201k, avg=78200520.00, stdev= 0.00
     lat (nsec): min=84977k, max=84977k, avg=84977100.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[78119],  5.00th=[78119], 10.00th=[78119], 20.00th=[78119],
     | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[78119],
     | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119],
     | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.09%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=449MiB/s (471MB/s), 449MiB/s-449MiB/s (471MB/s-471MB/s), io=40.0MiB (41.9MB), run=89-89msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout