== sanity test 398c: run fio to test AIO ================= 07:29:58 (1715340598)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.144239 s, 291 MB/s
osc.lustre-OST0000-osc-ffff8801372f8800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18497: Fri May 10 07:30:05 2024
  write: IOPS=449, BW=1797KiB/s (1841kB/s)(10.0MiB/5697msec)
    slat (usec): min=15, max=671, avg=81.90, stdev=49.86
    clat (usec): min=5685, max=61353, avg=35398.85, stdev=7105.14
     lat (usec): min=5771, max=61467, avg=35482.89, stdev=7114.56
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[22676], 10.00th=[25822], 20.00th=[28967],
     | 30.00th=[31851], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[50070], 99.50th=[51119], 99.90th=[61080], 99.95th=[61080],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1592, max= 2192, per=25.03%, avg=1799.64, stdev=178.39, samples=11
   iops        : min=  398, max=  548, avg=449.91, stdev=44.60, samples=11
  lat (msec)   : 10=0.08%, 20=1.76%, 50=97.27%, 100=0.90%
  cpu          : usr=0.65%, sys=3.90%, ctx=2024, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18498: Fri May 10 07:30:05 2024
  write: IOPS=449, BW=1798KiB/s (1841kB/s)(10.0MiB/5695msec)
    slat (usec): min=13, max=717, avg=77.54, stdev=47.18
    clat (usec): min=7899, max=67150, avg=35387.74, stdev=7090.94
     lat (usec): min=8022, max=67294, avg=35467.18, stdev=7100.35
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[22938], 10.00th=[25560], 20.00th=[28967],
     | 30.00th=[31851], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[49546], 99.50th=[50594], 99.90th=[62653], 99.95th=[66847],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1600, max= 2216, per=25.06%, avg=1801.82, stdev=187.73, samples=11
   iops        : min=  400, max=  554, avg=450.45, stdev=46.93, samples=11
  lat (msec)   : 10=0.08%, 20=1.72%, 50=97.42%, 100=0.78%
  cpu          : usr=0.72%, sys=3.76%, ctx=1692, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18499: Fri May 10 07:30:05 2024
  write: IOPS=451, BW=1805KiB/s (1848kB/s)(10.0MiB/5674msec)
    slat (usec): min=14, max=518, avg=83.44, stdev=49.31
    clat (usec): min=3889, max=70800, avg=35269.89, stdev=7524.36
     lat (usec): min=3970, max=70912, avg=35355.43, stdev=7533.52
    clat percentiles (usec):
     |  1.00th=[15139],  5.00th=[22414], 10.00th=[25297], 20.00th=[28705],
     | 30.00th=[31589], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[50070], 99.50th=[51119], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min= 1584, max= 2304, per=25.17%, avg=1809.45, stdev=202.35, samples=11
   iops        : min=  396, max=  576, avg=452.36, stdev=50.59, samples=11
  lat (msec)   : 4=0.04%, 10=0.31%, 20=1.60%, 50=96.88%, 100=1.17%
  cpu          : usr=0.78%, sys=3.83%, ctx=2084, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18500: Fri May 10 07:30:05 2024
  write: IOPS=449, BW=1798KiB/s (1841kB/s)(10.0MiB/5695msec)
    slat (usec): min=14, max=722, avg=79.95, stdev=49.61
    clat (usec): min=13516, max=66950, avg=35380.83, stdev=7112.28
     lat (usec): min=13548, max=67152, avg=35462.78, stdev=7121.91
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[22938], 10.00th=[25560], 20.00th=[28705],
     | 30.00th=[31851], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[49546], 99.50th=[50070], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[66847]
   bw (  KiB/s): min= 1600, max= 2208, per=25.06%, avg=1801.82, stdev=185.24, samples=11
   iops        : min=  400, max=  552, avg=450.45, stdev=46.31, samples=11
  lat (msec)   : 20=1.88%, 50=97.34%, 100=0.78%
  cpu          : usr=0.60%, sys=3.71%, ctx=1982, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7190KiB/s (7362kB/s), 1797KiB/s-1805KiB/s (1841kB/s-1848kB/s), io=40.0MiB (41.9MB), run=5674-5697msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18515: Fri May 10 07:30:09 2024
   read: IOPS=456, BW=1827KiB/s (1871kB/s)(5048KiB/2763msec)
    slat (usec): min=669, max=3547, avg=1179.56, stdev=329.81
    clat (usec): min=4342, max=38960, avg=16212.95, stdev=3286.43
     lat (usec): min=5453, max=41388, avg=17394.38, stdev=3440.65
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12125], 10.00th=[12911], 20.00th=[13829],
     | 30.00th=[14484], 40.00th=[15270], 50.00th=[15926], 60.00th=[16581],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19268], 95.00th=[21103],
     | 99.00th=[29492], 99.50th=[33162], 99.90th=[37487], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1376, max= 1920, per=24.64%, avg=1789.60, stdev=233.92, samples=5
   iops        : min=  344, max=  480, avg=447.40, stdev=58.48, samples=5
  write: IOPS=469, BW=1879KiB/s (1924kB/s)(5192KiB/2763msec)
    slat (usec): min=13, max=1250, avg=43.87, stdev=39.86
    clat (usec): min=4304, max=40099, avg=16729.62, stdev=3369.10
     lat (usec): min=4336, max=40145, avg=16774.60, stdev=3376.11
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13566], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[19006], 95.00th=[21103],
     | 99.00th=[33817], 99.50th=[37487], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1432, max= 2051, per=24.83%, avg=1877.40, stdev=258.41, samples=5
   iops        : min=  358, max=  512, avg=469.20, stdev=64.48, samples=5
  lat (msec)   : 10=0.43%, 20=92.89%, 50=6.68%
  cpu          : usr=0.83%, sys=6.88%, ctx=2579, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18516: Fri May 10 07:30:09 2024
   read: IOPS=452, BW=1809KiB/s (1852kB/s)(4976KiB/2751msec)
    slat (usec): min=17, max=3994, avg=1175.93, stdev=336.74
    clat (usec): min=3737, max=39604, avg=16306.92, stdev=3593.99
     lat (usec): min=3806, max=41791, avg=17484.55, stdev=3753.37
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12518], 10.00th=[13173], 20.00th=[13960],
     | 30.00th=[14615], 40.00th=[15270], 50.00th=[15795], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19268], 95.00th=[21890],
     | 99.00th=[32637], 99.50th=[36439], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 1328, max= 2032, per=24.71%, avg=1794.40, stdev=278.45, samples=5
   iops        : min=  332, max=  508, avg=448.60, stdev=69.61, samples=5
  write: IOPS=478, BW=1913KiB/s (1959kB/s)(5264KiB/2751msec)
    slat (usec): min=14, max=383, avg=44.08, stdev=27.00
    clat (usec): min=4113, max=38937, avg=16766.55, stdev=3334.34
     lat (usec): min=4200, max=39036, avg=16811.67, stdev=3340.19
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13698], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[19006], 95.00th=[21627],
     | 99.00th=[33817], 99.50th=[36963], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1616, max= 2035, per=25.34%, avg=1915.80, stdev=170.19, samples=5
   iops        : min=  404, max=  508, avg=478.80, stdev=42.42, samples=5
  lat (msec)   : 4=0.08%, 10=0.08%, 20=92.77%, 50=7.07%
  cpu          : usr=0.58%, sys=7.16%, ctx=2526, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18517: Fri May 10 07:30:09 2024
   read: IOPS=452, BW=1808KiB/s (1852kB/s)(4984KiB/2756msec)
    slat (usec): min=694, max=3400, avg=1182.26, stdev=332.88
    clat (usec): min=2543, max=44534, avg=16019.44, stdev=3309.92
     lat (usec): min=3517, max=47544, avg=17203.45, stdev=3485.21
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12387], 10.00th=[13042], 20.00th=[13829],
     | 30.00th=[14484], 40.00th=[15008], 50.00th=[15664], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18744], 95.00th=[20841],
     | 99.00th=[29492], 99.50th=[33817], 99.90th=[41681], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min= 1392, max= 2051, per=24.86%, avg=1805.40, stdev=247.36, samples=5
   iops        : min=  348, max=  512, avg=451.20, stdev=61.65, samples=5
  write: IOPS=476, BW=1907KiB/s (1953kB/s)(5256KiB/2756msec)
    slat (usec): min=14, max=318, avg=43.37, stdev=23.42
    clat (usec): min=10206, max=42363, avg=16882.93, stdev=3398.09
     lat (usec): min=10236, max=42426, avg=16927.36, stdev=3405.83
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13829], 10.00th=[14484], 20.00th=[15008],
     | 30.00th=[15533], 40.00th=[15926], 50.00th=[16319], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[19006], 95.00th=[21890],
     | 99.00th=[32375], 99.50th=[37487], 99.90th=[41681], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1416, max= 2120, per=24.98%, avg=1888.80, stdev=283.33, samples=5
   iops        : min=  354, max=  530, avg=472.20, stdev=70.83, samples=5
  lat (msec)   : 4=0.12%, 10=0.12%, 20=92.97%, 50=6.80%
  cpu          : usr=1.02%, sys=6.72%, ctx=2571, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18518: Fri May 10 07:30:09 2024
   read: IOPS=458, BW=1832KiB/s (1876kB/s)(5060KiB/2762msec)
    slat (usec): min=690, max=3639, avg=1172.79, stdev=325.07
    clat (usec): min=9854, max=41449, avg=16232.54, stdev=3346.48
     lat (usec): min=10650, max=44155, avg=17407.14, stdev=3516.01
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12518], 10.00th=[13173], 20.00th=[13960],
     | 30.00th=[14615], 40.00th=[15270], 50.00th=[15795], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19006], 95.00th=[21365],
     | 99.00th=[31065], 99.50th=[33817], 99.90th=[41157], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1432, max= 2040, per=25.15%, avg=1826.40, stdev=230.47, samples=5
   iops        : min=  358, max=  510, avg=456.60, stdev=57.62, samples=5
  write: IOPS=468, BW=1875KiB/s (1920kB/s)(5180KiB/2762msec)
    slat (usec): min=13, max=908, avg=45.22, stdev=36.82
    clat (usec): min=10400, max=39868, avg=16760.93, stdev=3210.50
     lat (usec): min=10442, max=39907, avg=16807.16, stdev=3217.27
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13698], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16319], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18744], 95.00th=[22414],
     | 99.00th=[34341], 99.50th=[36439], 99.90th=[39060], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1408, max= 2024, per=24.53%, avg=1855.00, stdev=257.66, samples=5
   iops        : min=  352, max=  506, avg=463.60, stdev=64.30, samples=5
  lat (msec)   : 10=0.12%, 20=92.97%, 50=6.91%
  cpu          : usr=0.62%, sys=7.10%, ctx=2650, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7263KiB/s (7437kB/s), 1808KiB/s-1832KiB/s (1852kB/s-1876kB/s), io=19.6MiB (20.5MB), run=2751-2763msec
  WRITE: bw=7561KiB/s (7743kB/s), 1875KiB/s-1913KiB/s (1920kB/s-1959kB/s), io=20.4MiB (21.4MB), run=2751-2763msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18524: Fri May 10 07:30:10 2024
   read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec)
    slat (nsec): min=9751.1k, max=9751.1k, avg=9751149.00, stdev= 0.00
    clat (nsec): min=74608k, max=74608k, avg=74607563.00, stdev= 0.00
     lat (nsec): min=84368k, max=84368k, avg=84367697.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[74974],  5.00th=[74974], 10.00th=[74974], 20.00th=[74974],
     | 30.00th=[74974], 40.00th=[74974], 50.00th=[74974], 60.00th=[74974],
     | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974],
     | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.11%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout