== sanity test 398c: run fio to test AIO ================= 17:49:00 (1722548940)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0831275 s, 505 MB/s
osc.lustre-OST0000-osc-ffff8800b6f6b800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=23936: Thu Aug  1 17:49:04 2024
  write: IOPS=927, BW=3711KiB/s (3801kB/s)(10.0MiB/2759msec)
    slat (usec): min=11, max=247, avg=43.86, stdev=21.51
    clat (usec): min=2043, max=36210, avg=17157.16, stdev=2575.52
     lat (usec): min=2118, max=36227, avg=17202.28, stdev=2577.84
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[13698], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[26608], 99.50th=[26608], 99.90th=[34341], 99.95th=[34341],
     | 99.99th=[36439]
   bw (  KiB/s): min= 3432, max= 4096, per=25.17%, avg=3736.00, stdev=284.82, samples=5
   iops        : min=  858, max= 1024, avg=934.00, stdev=71.20, samples=5
  lat (msec)   : 4=0.08%, 10=0.74%, 20=92.66%, 50=6.52%
  cpu          : usr=1.16%, sys=4.50%, ctx=1539, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23937: Thu Aug  1 17:49:04 2024
  write: IOPS=928, BW=3716KiB/s (3805kB/s)(10.0MiB/2756msec)
    slat (usec): min=12, max=21229, avg=54.11, stdev=419.28
    clat (usec): min=1749, max=34797, avg=17075.60, stdev=2552.72
     lat (usec): min=1881, max=34821, avg=17131.01, stdev=2555.65
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[13566], 10.00th=[14353], 20.00th=[15401],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[26346], 99.50th=[26608], 99.90th=[32900], 99.95th=[32900],
     | 99.99th=[34866]
   bw (  KiB/s): min= 3456, max= 4056, per=25.18%, avg=3737.60, stdev=272.20, samples=5
   iops        : min=  864, max= 1014, avg=934.40, stdev=68.05, samples=5
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.78%, 20=93.24%, 50=5.86%
  cpu          : usr=0.58%, sys=5.81%, ctx=1652, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23938: Thu Aug  1 17:49:04 2024
  write: IOPS=931, BW=3725KiB/s (3814kB/s)(10.0MiB/2749msec)
    slat (usec): min=12, max=363, avg=43.87, stdev=23.02
    clat (usec): min=2138, max=36536, avg=17105.40, stdev=2575.09
     lat (usec): min=2192, max=36564, avg=17150.47, stdev=2575.52
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[13566], 10.00th=[14484], 20.00th=[15401],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[18482], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[26346], 99.50th=[26870], 99.90th=[34866], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 3464, max= 4128, per=25.25%, avg=3748.80, stdev=291.24, samples=5
   iops        : min=  866, max= 1032, avg=937.20, stdev=72.81, samples=5
  lat (msec)   : 4=0.08%, 10=0.82%, 20=92.73%, 50=6.37%
  cpu          : usr=0.98%, sys=4.51%, ctx=1519, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23939: Thu Aug  1 17:49:04 2024
  write: IOPS=928, BW=3714KiB/s (3803kB/s)(10.0MiB/2757msec)
    slat (usec): min=12, max=20780, avg=52.38, stdev=410.47
    clat (usec): min=2114, max=34759, avg=17053.76, stdev=2576.51
     lat (usec): min=2187, max=34812, avg=17107.36, stdev=2575.03
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[13435], 10.00th=[14484], 20.00th=[15401],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[26346], 99.50th=[26608], 99.90th=[32900], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 3464, max= 4080, per=25.21%, avg=3742.40, stdev=276.28, samples=5
   iops        : min=  866, max= 1020, avg=935.60, stdev=69.07, samples=5
  lat (msec)   : 4=0.23%, 10=0.78%, 20=92.73%, 50=6.25%
  cpu          : usr=0.87%, sys=5.52%, ctx=1516, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=14.5MiB/s (15.2MB/s), 3711KiB/s-3725KiB/s (3801kB/s-3814kB/s), io=40.0MiB (41.9MB), run=2749-2759msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=23951: Thu Aug  1 17:49:07 2024
   read: IOPS=449, BW=1799KiB/s (1842kB/s)(5048KiB/2806msec)
    slat (usec): min=591, max=2594, avg=1043.15, stdev=279.48
    clat (usec): min=2284, max=31497, avg=16678.84, stdev=3580.58
     lat (usec): min=3061, max=33317, avg=17724.21, stdev=3697.33
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11731], 10.00th=[12649], 20.00th=[13829],
     | 30.00th=[14615], 40.00th=[15401], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[21365], 95.00th=[22938],
     | 99.00th=[28705], 99.50th=[29492], 99.90th=[30016], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 1696, max= 2000, per=25.45%, avg=1819.20, stdev=129.81, samples=5
   iops        : min=  424, max=  500, avg=454.80, stdev=32.45, samples=5
  write: IOPS=462, BW=1850KiB/s (1895kB/s)(5192KiB/2806msec)
    slat (usec): min=12, max=534, avg=50.48, stdev=35.77
    clat (usec): min=8689, max=33482, avg=17126.39, stdev=3195.14
     lat (usec): min=8751, max=33572, avg=17178.36, stdev=3198.65
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12911], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19268], 90.00th=[21103], 95.00th=[23200],
     | 99.00th=[27657], 99.50th=[29754], 99.90th=[32900], 99.95th=[33424],
     | 99.99th=[33424]
   bw (  KiB/s): min= 1720, max= 2056, per=25.46%, avg=1894.40, stdev=122.30, samples=5
   iops        : min=  430, max=  514, avg=473.60, stdev=30.57, samples=5
  lat (msec)   : 4=0.04%, 10=0.43%, 20=83.87%, 50=15.66%
  cpu          : usr=0.93%, sys=8.77%, ctx=2420, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23952: Thu Aug  1 17:49:07 2024
   read: IOPS=446, BW=1786KiB/s (1829kB/s)(4976KiB/2786msec)
    slat (usec): min=17, max=3025, avg=1044.14, stdev=298.32
    clat (usec): min=1288, max=34228, avg=16609.36, stdev=3801.70
     lat (usec): min=1306, max=35380, avg=17655.73, stdev=3929.13
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11731], 10.00th=[12518], 20.00th=[13829],
     | 30.00th=[14615], 40.00th=[15270], 50.00th=[16057], 60.00th=[16909],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[21365], 95.00th=[23462],
     | 99.00th=[29754], 99.50th=[31065], 99.90th=[33162], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1712, max= 2056, per=25.34%, avg=1811.20, stdev=143.62, samples=5
   iops        : min=  428, max=  514, avg=452.80, stdev=35.91, samples=5
  write: IOPS=472, BW=1889KiB/s (1935kB/s)(5264KiB/2786msec)
    slat (usec): min=11, max=761, avg=50.07, stdev=37.66
    clat (usec): min=3335, max=34221, avg=17065.51, stdev=3490.57
     lat (usec): min=3377, max=34260, avg=17116.90, stdev=3494.13
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[12780], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[21365], 95.00th=[23725],
     | 99.00th=[26870], 99.50th=[32900], 99.90th=[33817], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1744, max= 2024, per=26.10%, avg=1942.40, stdev=113.90, samples=5
   iops        : min=  436, max=  506, avg=485.60, stdev=28.47, samples=5
  lat (msec)   : 2=0.08%, 4=0.20%, 10=0.62%, 20=83.09%, 50=16.02%
  cpu          : usr=0.93%, sys=8.94%, ctx=2337, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23953: Thu Aug  1 17:49:07 2024
   read: IOPS=443, BW=1776KiB/s (1818kB/s)(4984KiB/2807msec)
    slat (usec): min=570, max=2480, avg=1049.81, stdev=284.07
    clat (usec): min=8382, max=32206, avg=16728.59, stdev=3519.66
     lat (usec): min=9273, max=33302, avg=17780.45, stdev=3641.95
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11731], 10.00th=[12780], 20.00th=[13960],
     | 30.00th=[14746], 40.00th=[15533], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19006], 90.00th=[21365], 95.00th=[23200],
     | 99.00th=[28443], 99.50th=[30016], 99.90th=[31851], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1632, max= 1904, per=25.31%, avg=1809.60, stdev=105.59, samples=5
   iops        : min=  408, max=  476, avg=452.40, stdev=26.40, samples=5
  write: IOPS=468, BW=1872KiB/s (1917kB/s)(5256KiB/2807msec)
    slat (usec): min=15, max=811, avg=51.23, stdev=40.01
    clat (usec): min=8826, max=35070, avg=17129.42, stdev=3364.21
     lat (usec): min=8889, max=35104, avg=17182.10, stdev=3367.91
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12780], 10.00th=[13566], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[18482], 80.00th=[19792], 90.00th=[21627], 95.00th=[23462],
     | 99.00th=[25822], 99.50th=[30278], 99.90th=[33817], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1792, max= 2104, per=25.46%, avg=1894.40, stdev=140.45, samples=5
   iops        : min=  448, max=  526, avg=473.60, stdev=35.11, samples=5
  lat (msec)   : 10=0.66%, 20=82.70%, 50=16.64%
  cpu          : usr=1.67%, sys=8.09%, ctx=2324, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23954: Thu Aug  1 17:49:07 2024
   read: IOPS=451, BW=1805KiB/s (1849kB/s)(5060KiB/2803msec)
    slat (usec): min=538, max=2437, avg=1045.42, stdev=280.43
    clat (usec): min=1157, max=30210, avg=16678.19, stdev=3509.02
     lat (usec): min=2367, max=32070, avg=17725.61, stdev=3622.21
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11731], 10.00th=[12780], 20.00th=[13960],
     | 30.00th=[14746], 40.00th=[15533], 50.00th=[16188], 60.00th=[16909],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[21103], 95.00th=[23200],
     | 99.00th=[27657], 99.50th=[28705], 99.90th=[30016], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1760, max= 1992, per=25.78%, avg=1843.20, stdev=106.22, samples=5
   iops        : min=  440, max=  498, avg=460.80, stdev=26.56, samples=5
  write: IOPS=462, BW=1848KiB/s (1892kB/s)(5180KiB/2803msec)
    slat (usec): min=13, max=385, avg=49.51, stdev=30.37
    clat (usec): min=8874, max=33649, avg=17124.49, stdev=3427.32
     lat (usec): min=8902, max=33681, avg=17175.50, stdev=3428.78
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12649], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15926], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[21365], 95.00th=[23725],
     | 99.00th=[28181], 99.50th=[29754], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1688, max= 2032, per=25.18%, avg=1873.60, stdev=125.14, samples=5
   iops        : min=  422, max=  508, avg=468.40, stdev=31.29, samples=5
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.55%, 20=82.70%, 50=16.64%
  cpu          : usr=1.46%, sys=8.35%, ctx=2393, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7149KiB/s (7321kB/s), 1776KiB/s-1805KiB/s (1818kB/s-1849kB/s), io=19.6MiB (20.5MB), run=2786-2807msec
  WRITE: bw=7443KiB/s (7621kB/s), 1848KiB/s-1889KiB/s (1892kB/s-1935kB/s), io=20.4MiB (21.4MB), run=2786-2807msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=23960: Thu Aug  1 17:49:09 2024
   read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec)
    slat (nsec): min=8991.6k, max=8991.6k, avg=8991602.00, stdev= 0.00
    clat (nsec): min=77296k, max=77296k, avg=77296469.00, stdev= 0.00
     lat (nsec): min=86310k, max=86310k, avg=86310306.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[77071],  5.00th=[77071], 10.00th=[77071], 20.00th=[77071],
     | 30.00th=[77071], 40.00th=[77071], 50.00th=[77071], 60.00th=[77071],
     | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071],
     | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[77071]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.24%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck