== sanity test 398c: run fio to test AIO ================= 01:18:53 (1735107533)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.080493 s, 521 MB/s
osc.lustre-OST0000-osc-ffff880091fc1000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=25733: Wed Dec 25 01:18:59 2024
  write: IOPS=618, BW=2476KiB/s (2535kB/s)(10.0MiB/4136msec)
    slat (usec): min=12, max=509, avg=53.08, stdev=36.58
    clat (usec): min=3231, max=73922, avg=25682.45, stdev=6501.69
     lat (usec): min=3294, max=74087, avg=25736.99, stdev=6508.62
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[17957], 10.00th=[19268], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23725], 60.00th=[25297],
     | 70.00th=[27919], 80.00th=[31327], 90.00th=[35390], 95.00th=[38536],
     | 99.00th=[41157], 99.50th=[42206], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1840, max= 3072, per=25.30%, avg=2502.00, stdev=400.41, samples=8
   iops        : min=  460, max=  768, avg=625.50, stdev=100.10, samples=8
  lat (msec)   : 4=0.04%, 10=0.04%, 20=15.78%, 50=83.87%, 100=0.27%
  cpu          : usr=0.56%, sys=3.72%, ctx=1401, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25734: Wed Dec 25 01:18:59 2024
  write: IOPS=621, BW=2484KiB/s (2544kB/s)(10.0MiB/4122msec)
    slat (usec): min=12, max=346, avg=59.56, stdev=35.79
    clat (usec): min=2736, max=81689, avg=25621.14, stdev=6929.20
     lat (usec): min=2848, max=81748, avg=25682.17, stdev=6936.31
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[17957], 10.00th=[19268], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23725], 60.00th=[25297],
     | 70.00th=[27919], 80.00th=[31065], 90.00th=[35390], 95.00th=[38011],
     | 99.00th=[40633], 99.50th=[50594], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1880, max= 3112, per=25.38%, avg=2510.00, stdev=409.24, samples=8
   iops        : min=  470, max=  778, avg=627.50, stdev=102.31, samples=8
  lat (msec)   : 4=0.12%, 10=0.23%, 20=15.62%, 50=83.52%, 100=0.51%
  cpu          : usr=0.73%, sys=3.88%, ctx=1772, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25735: Wed Dec 25 01:18:59 2024
  write: IOPS=619, BW=2478KiB/s (2538kB/s)(10.0MiB/4132msec)
    slat (usec): min=12, max=417, avg=59.63, stdev=38.20
    clat (usec): min=4923, max=69816, avg=25665.89, stdev=6452.68
     lat (usec): min=4986, max=70028, avg=25726.99, stdev=6460.13
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[17957], 10.00th=[19268], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23725], 60.00th=[25297],
     | 70.00th=[27919], 80.00th=[31327], 90.00th=[35390], 95.00th=[38536],
     | 99.00th=[41157], 99.50th=[42730], 99.90th=[65274], 99.95th=[65274],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1848, max= 3088, per=25.31%, avg=2503.00, stdev=403.56, samples=8
   iops        : min=  462, max=  772, avg=625.75, stdev=100.89, samples=8
  lat (msec)   : 10=0.12%, 20=15.47%, 50=84.10%, 100=0.31%
  cpu          : usr=0.68%, sys=3.78%, ctx=1855, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25736: Wed Dec 25 01:18:59 2024
  write: IOPS=618, BW=2473KiB/s (2532kB/s)(10.0MiB/4141msec)
    slat (usec): min=12, max=311, avg=49.51, stdev=31.81
    clat (usec): min=5250, max=60343, avg=25699.01, stdev=6191.13
     lat (usec): min=5323, max=60432, avg=25749.77, stdev=6196.87
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[18220], 10.00th=[19530], 20.00th=[20579],
     | 30.00th=[21890], 40.00th=[22676], 50.00th=[23725], 60.00th=[25560],
     | 70.00th=[28443], 80.00th=[31589], 90.00th=[35390], 95.00th=[37487],
     | 99.00th=[40633], 99.50th=[42206], 99.90th=[51119], 99.95th=[55837],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1808, max= 2944, per=25.25%, avg=2497.00, stdev=403.74, samples=8
   iops        : min=  452, max=  736, avg=624.25, stdev=100.94, samples=8
  lat (msec)   : 10=0.04%, 20=15.31%, 50=84.53%, 100=0.12%
  cpu          : usr=0.72%, sys=3.21%, ctx=1173, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9891KiB/s (10.1MB/s), 2473KiB/s-2484KiB/s (2532kB/s-2544kB/s), io=40.0MiB (41.9MB), run=4122-4141msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=25749: Wed Dec 25 01:19:03 2024
   read: IOPS=467, BW=1871KiB/s (1916kB/s)(5048KiB/2698msec)
    slat (usec): min=617, max=3064, avg=1119.87, stdev=253.54
    clat (usec): min=9740, max=28004, avg=15825.15, stdev=2527.79
     lat (usec): min=10668, max=29886, avg=16946.82, stdev=2574.01
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12125], 10.00th=[12780], 20.00th=[13698],
     | 30.00th=[14484], 40.00th=[15139], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19006], 95.00th=[20055],
     | 99.00th=[23200], 99.50th=[25297], 99.90th=[26084], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1616, max= 2040, per=24.96%, avg=1849.60, stdev=178.65, samples=5
   iops        : min=  404, max=  510, avg=462.40, stdev=44.66, samples=5
  write: IOPS=481, BW=1924KiB/s (1971kB/s)(5192KiB/2698msec)
    slat (usec): min=12, max=955, avg=47.38, stdev=35.59
    clat (usec): min=2801, max=27950, avg=16365.80, stdev=2182.24
     lat (usec): min=2838, max=28004, avg=16414.26, stdev=2177.68
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13304], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[18744], 95.00th=[19792],
     | 99.00th=[24773], 99.50th=[25822], 99.90th=[27919], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1736, max= 2064, per=24.83%, avg=1915.20, stdev=131.04, samples=5
   iops        : min=  434, max=  516, avg=478.80, stdev=32.76, samples=5
  lat (msec)   : 4=0.04%, 10=0.20%, 20=95.59%, 50=4.18%
  cpu          : usr=0.82%, sys=7.56%, ctx=2842, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25750: Wed Dec 25 01:19:03 2024
   read: IOPS=463, BW=1855KiB/s (1899kB/s)(4976KiB/2683msec)
    slat (usec): min=18, max=3688, avg=1124.18, stdev=263.72
    clat (usec): min=2633, max=29632, avg=15835.89, stdev=2618.09
     lat (usec): min=2652, max=30771, avg=16961.94, stdev=2661.71
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12256], 10.00th=[12911], 20.00th=[13829],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15664], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18744], 95.00th=[19792],
     | 99.00th=[23987], 99.50th=[26346], 99.90th=[28443], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1632, max= 1976, per=24.66%, avg=1827.20, stdev=136.07, samples=5
   iops        : min=  408, max=  494, avg=456.80, stdev=34.02, samples=5
  write: IOPS=490, BW=1962KiB/s (2009kB/s)(5264KiB/2683msec)
    slat (usec): min=13, max=402, avg=46.95, stdev=26.42
    clat (usec): min=5174, max=32489, avg=16423.05, stdev=2354.61
     lat (usec): min=5207, max=32512, avg=16470.99, stdev=2353.95
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[13042], 10.00th=[13829], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19006], 95.00th=[20055],
     | 99.00th=[24511], 99.50th=[26084], 99.90th=[31327], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1752, max= 2128, per=25.51%, avg=1968.00, stdev=153.15, samples=5
   iops        : min=  438, max=  532, avg=492.00, stdev=38.29, samples=5
  lat (msec)   : 4=0.08%, 10=0.35%, 20=94.80%, 50=4.77%
  cpu          : usr=1.19%, sys=7.23%, ctx=2862, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25751: Wed Dec 25 01:19:03 2024
   read: IOPS=460, BW=1840KiB/s (1885kB/s)(4984KiB/2708msec)
    slat (usec): min=674, max=2712, avg=1121.44, stdev=242.28
    clat (usec): min=9036, max=28894, avg=15892.44, stdev=2454.83
     lat (usec): min=10189, max=30696, avg=17015.70, stdev=2493.79
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12387], 10.00th=[13042], 20.00th=[13829],
     | 30.00th=[14615], 40.00th=[15139], 50.00th=[15664], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[28443], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1624, max= 1984, per=24.51%, avg=1816.00, stdev=181.99, samples=5
   iops        : min=  406, max=  496, avg=454.00, stdev=45.50, samples=5
  write: IOPS=485, BW=1941KiB/s (1987kB/s)(5256KiB/2708msec)
    slat (usec): min=13, max=808, avg=47.13, stdev=31.32
    clat (usec): min=2988, max=27254, avg=16410.10, stdev=2147.60
     lat (usec): min=3041, max=27287, avg=16458.29, stdev=2148.26
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[13304], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19006], 95.00th=[20055],
     | 99.00th=[22676], 99.50th=[23987], 99.90th=[25297], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1696, max= 2128, per=24.87%, avg=1918.40, stdev=195.50, samples=5
   iops        : min=  424, max=  532, avg=479.60, stdev=48.88, samples=5
  lat (msec)   : 4=0.04%, 10=0.31%, 20=94.80%, 50=4.84%
  cpu          : usr=0.74%, sys=7.83%, ctx=2784, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25752: Wed Dec 25 01:19:03 2024
   read: IOPS=467, BW=1869KiB/s (1913kB/s)(5060KiB/2708msec)
    slat (usec): min=657, max=2518, avg=1112.33, stdev=223.56
    clat (usec): min=4739, max=28240, avg=15829.62, stdev=2439.02
     lat (usec): min=5785, max=30063, avg=16943.66, stdev=2471.09
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12125], 10.00th=[12780], 20.00th=[13829],
     | 30.00th=[14484], 40.00th=[15139], 50.00th=[15795], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18744], 95.00th=[20055],
     | 99.00th=[22414], 99.50th=[23987], 99.90th=[27395], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1648, max= 2040, per=25.03%, avg=1854.40, stdev=176.13, samples=5
   iops        : min=  412, max=  510, avg=463.60, stdev=44.03, samples=5
  write: IOPS=478, BW=1913KiB/s (1959kB/s)(5180KiB/2708msec)
    slat (usec): min=12, max=875, avg=47.88, stdev=37.36
    clat (usec): min=9540, max=28820, avg=16394.52, stdev=2092.57
     lat (usec): min=9572, max=28839, avg=16443.40, stdev=2093.69
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13304], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16319], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19792],
     | 99.00th=[24249], 99.50th=[24511], 99.90th=[27395], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1672, max= 2016, per=24.50%, avg=1889.60, stdev=138.03, samples=5
   iops        : min=  418, max=  504, avg=472.40, stdev=34.51, samples=5
  lat (msec)   : 10=0.23%, 20=95.27%, 50=4.49%
  cpu          : usr=1.03%, sys=7.39%, ctx=2746, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7411KiB/s (7588kB/s), 1840KiB/s-1871KiB/s (1885kB/s-1916kB/s), io=19.6MiB (20.5MB), run=2683-2708msec
  WRITE: bw=7715KiB/s (7900kB/s), 1913KiB/s-1962KiB/s (1959kB/s-2009kB/s), io=20.4MiB (21.4MB), run=2683-2708msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=25758: Wed Dec 25 01:19:04 2024
   read: IOPS=6, BW=272MiB/s (285MB/s)(40.0MiB/147msec)
    slat (nsec): min=5827.6k, max=5827.6k, avg=5827562.00, stdev= 0.00
    clat (nsec): min=133487k, max=133487k, avg=133487239.00, stdev= 0.00
     lat (nsec): min=139324k, max=139324k, avg=139324236.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  134],  5.00th=[  134], 10.00th=[  134], 20.00th=[  134],
     | 30.00th=[  134], 40.00th=[  134], 50.00th=[  134], 60.00th=[  134],
     | 70.00th=[  134], 80.00th=[  134], 90.00th=[  134], 95.00th=[  134],
     | 99.00th=[  134], 99.50th=[  134], 99.90th=[  134], 99.95th=[  134],
     | 99.99th=[  134]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.11%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=40.0MiB (41.9MB), run=147-147msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout