== sanity test 398c: run fio to test AIO ================= 16:18:56 (1715458736)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.139491 s, 301 MB/s
osc.lustre-OST0000-osc-ffff8800a7b02800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=17334: Sat May 11 16:19:03 2024
  write: IOPS=475, BW=1902KiB/s (1948kB/s)(10.0MiB/5384msec)
    slat (usec): min=15, max=33385, avg=90.26, stdev=659.57
    clat (usec): min=3427, max=50673, avg=33332.79, stdev=4364.75
     lat (usec): min=3512, max=50752, avg=33425.05, stdev=4351.29
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[24773], 10.00th=[27395], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33817], 50.00th=[34866], 60.00th=[35390],
     | 70.00th=[35914], 80.00th=[36439], 90.00th=[37487], 95.00th=[38011],
     | 99.00th=[39060], 99.50th=[39584], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[50594]
   bw (  KiB/s): min= 1728, max= 2304, per=24.94%, avg=1896.40, stdev=167.79, samples=10
   iops        : min=  432, max=  576, avg=474.10, stdev=41.95, samples=10
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.62%, 50=99.22%, 100=0.04%
  cpu          : usr=0.69%, sys=5.13%, ctx=1583, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17335: Sat May 11 16:19:03 2024
  write: IOPS=475, BW=1903KiB/s (1949kB/s)(10.0MiB/5380msec)
    slat (usec): min=18, max=33394, avg=93.20, stdev=659.65
    clat (usec): min=5005, max=55043, avg=33346.24, stdev=4462.29
     lat (usec): min=5093, max=55064, avg=33441.63, stdev=4448.05
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[24511], 10.00th=[27132], 20.00th=[30016],
     | 30.00th=[31851], 40.00th=[33424], 50.00th=[34866], 60.00th=[35390],
     | 70.00th=[35914], 80.00th=[36439], 90.00th=[37487], 95.00th=[38011],
     | 99.00th=[39584], 99.50th=[40109], 99.90th=[51643], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1688, max= 2288, per=25.00%, avg=1901.20, stdev=167.70, samples=10
   iops        : min=  422, max=  572, avg=475.30, stdev=41.93, samples=10
  lat (msec)   : 10=0.12%, 20=0.59%, 50=99.06%, 100=0.23%
  cpu          : usr=0.87%, sys=5.22%, ctx=1838, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17336: Sat May 11 16:19:03 2024
  write: IOPS=476, BW=1907KiB/s (1953kB/s)(10.0MiB/5369msec)
    slat (usec): min=15, max=577, avg=73.49, stdev=41.32
    clat (usec): min=3967, max=53911, avg=33431.70, stdev=4353.32
     lat (usec): min=4010, max=54013, avg=33507.11, stdev=4352.10
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[25035], 10.00th=[27132], 20.00th=[30016],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[35390],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[37487], 95.00th=[38011],
     | 99.00th=[39584], 99.50th=[43254], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min= 1696, max= 2296, per=25.04%, avg=1904.40, stdev=175.01, samples=10
   iops        : min=  424, max=  574, avg=476.10, stdev=43.75, samples=10
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.43%, 50=99.30%, 100=0.20%
  cpu          : usr=0.65%, sys=4.40%, ctx=1487, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17337: Sat May 11 16:19:03 2024
  write: IOPS=475, BW=1901KiB/s (1947kB/s)(10.0MiB/5386msec)
    slat (usec): min=16, max=31235, avg=81.83, stdev=620.21
    clat (usec): min=19961, max=44569, avg=33409.49, stdev=4118.28
     lat (usec): min=20043, max=51828, avg=33493.28, stdev=4128.24
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[25297], 10.00th=[27395], 20.00th=[30016],
     | 30.00th=[31851], 40.00th=[33424], 50.00th=[34866], 60.00th=[35390],
     | 70.00th=[35914], 80.00th=[36439], 90.00th=[37487], 95.00th=[38011],
     | 99.00th=[39060], 99.50th=[39584], 99.90th=[41157], 99.95th=[44303],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1696, max= 2288, per=24.91%, avg=1894.00, stdev=165.24, samples=10
   iops        : min=  424, max=  572, avg=473.50, stdev=41.31, samples=10
  lat (msec)   : 20=0.04%, 50=99.96%
  cpu          : usr=0.58%, sys=4.10%, ctx=1163, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7605KiB/s (7787kB/s), 1901KiB/s-1907KiB/s (1947kB/s-1953kB/s), io=40.0MiB (41.9MB), run=5369-5386msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=17352: Sat May 11 16:19:06 2024
   read: IOPS=615, BW=2461KiB/s (2520kB/s)(5048KiB/2051msec)
    slat (usec): min=28, max=2306, avg=779.74, stdev=303.12
    clat (usec): min=1406, max=35963, avg=12114.36, stdev=3951.69
     lat (usec): min=2181, max=38147, avg=12896.05, stdev=4180.35
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[ 9634],
     | 30.00th=[10290], 40.00th=[10814], 50.00th=[11207], 60.00th=[11731],
     | 70.00th=[12256], 80.00th=[13042], 90.00th=[15926], 95.00th=[21365],
     | 99.00th=[27132], 99.50th=[28967], 99.90th=[35390], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1576, max= 2952, per=25.24%, avg=2448.50, stdev=608.86, samples=4
   iops        : min=  394, max=  738, avg=612.00, stdev=152.13, samples=4
  write: IOPS=632, BW=2531KiB/s (2592kB/s)(5192KiB/2051msec)
    slat (usec): min=12, max=241, avg=45.16, stdev=26.72
    clat (usec): min=2358, max=35603, avg=12605.22, stdev=3795.31
     lat (usec): min=2413, max=35639, avg=12651.58, stdev=3805.86
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9503], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12518], 80.00th=[13435], 90.00th=[15664], 95.00th=[22152],
     | 99.00th=[27919], 99.50th=[29754], 99.90th=[34341], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1672, max= 2906, per=24.96%, avg=2520.50, stdev=577.59, samples=4
   iops        : min=  418, max=  726, avg=630.00, stdev=144.29, samples=4
  lat (msec)   : 2=0.04%, 4=0.23%, 10=18.36%, 20=74.77%, 50=6.60%
  cpu          : usr=1.46%, sys=9.90%, ctx=2588, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17353: Sat May 11 16:19:06 2024
   read: IOPS=601, BW=2405KiB/s (2463kB/s)(4976KiB/2069msec)
    slat (usec): min=416, max=2368, avg=779.21, stdev=284.84
    clat (usec): min=4224, max=32037, avg=12135.38, stdev=3798.33
     lat (usec): min=4913, max=33535, avg=12916.74, stdev=4012.09
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10814], 50.00th=[11207], 60.00th=[11731],
     | 70.00th=[12125], 80.00th=[13173], 90.00th=[15795], 95.00th=[21627],
     | 99.00th=[26870], 99.50th=[28443], 99.90th=[31851], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1392, max= 2888, per=24.48%, avg=2374.00, stdev=678.11, samples=4
   iops        : min=  348, max=  722, avg=593.50, stdev=169.53, samples=4
  write: IOPS=636, BW=2544KiB/s (2605kB/s)(5264KiB/2069msec)
    slat (usec): min=13, max=275, avg=43.36, stdev=24.76
    clat (usec): min=4167, max=30996, avg=12729.78, stdev=3890.22
     lat (usec): min=4215, max=31085, avg=12774.34, stdev=3900.81
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9503], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11600], 60.00th=[12125],
     | 70.00th=[12649], 80.00th=[13566], 90.00th=[15795], 95.00th=[23200],
     | 99.00th=[27657], 99.50th=[29230], 99.90th=[30802], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1656, max= 2904, per=24.96%, avg=2520.00, stdev=588.20, samples=4
   iops        : min=  414, max=  726, avg=630.00, stdev=147.05, samples=4
  lat (msec)   : 10=17.62%, 20=75.55%, 50=6.84%
  cpu          : usr=1.16%, sys=10.25%, ctx=2405, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17354: Sat May 11 16:19:06 2024
   read: IOPS=603, BW=2412KiB/s (2470kB/s)(4984KiB/2066msec)
    slat (usec): min=397, max=2293, avg=782.63, stdev=280.79
    clat (usec): min=6191, max=33020, avg=12250.64, stdev=3963.62
     lat (usec): min=7073, max=34562, avg=13035.18, stdev=4169.38
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10814], 50.00th=[11207], 60.00th=[11731],
     | 70.00th=[12387], 80.00th=[13304], 90.00th=[16057], 95.00th=[21365],
     | 99.00th=[29230], 99.50th=[30278], 99.90th=[31851], 99.95th=[32900],
     | 99.99th=[32900]
   bw (  KiB/s): min= 1528, max= 2792, per=24.83%, avg=2408.00, stdev=591.93, samples=4
   iops        : min=  382, max=  698, avg=602.00, stdev=147.98, samples=4
  write: IOPS=636, BW=2544KiB/s (2605kB/s)(5256KiB/2066msec)
    slat (usec): min=12, max=889, avg=44.17, stdev=35.68
    clat (usec): min=7687, max=32283, avg=12570.31, stdev=3856.19
     lat (usec): min=7727, max=32331, avg=12615.64, stdev=3866.53
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9503], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10814], 40.00th=[11076], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12387], 80.00th=[13304], 90.00th=[15533], 95.00th=[22676],
     | 99.00th=[27919], 99.50th=[29492], 99.90th=[31327], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1536, max= 3064, per=24.86%, avg=2510.00, stdev=699.24, samples=4
   iops        : min=  384, max=  766, avg=627.50, stdev=174.81, samples=4
  lat (msec)   : 10=18.40%, 20=75.20%, 50=6.41%
  cpu          : usr=1.02%, sys=10.56%, ctx=2482, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17355: Sat May 11 16:19:06 2024
   read: IOPS=612, BW=2449KiB/s (2508kB/s)(5060KiB/2066msec)
    slat (usec): min=407, max=2443, avg=776.19, stdev=279.19
    clat (usec): min=666, max=36050, avg=12202.06, stdev=3930.26
     lat (usec): min=1386, max=37519, avg=12980.08, stdev=4137.73
    clat percentiles (usec):
     |  1.00th=[ 7504],  5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9896],
     | 30.00th=[10421], 40.00th=[10945], 50.00th=[11338], 60.00th=[11731],
     | 70.00th=[12256], 80.00th=[13304], 90.00th=[15664], 95.00th=[21627],
     | 99.00th=[27395], 99.50th=[29230], 99.90th=[34866], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1552, max= 2896, per=25.10%, avg=2434.00, stdev=630.07, samples=4
   iops        : min=  388, max=  724, avg=608.50, stdev=157.52, samples=4
  write: IOPS=626, BW=2507KiB/s (2567kB/s)(5180KiB/2066msec)
    slat (usec): min=13, max=1020, avg=46.35, stdev=46.28
    clat (usec): min=2796, max=34021, avg=12604.88, stdev=3714.96
     lat (usec): min=2837, max=34082, avg=12652.45, stdev=3730.39
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9503], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10814], 40.00th=[11338], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12518], 80.00th=[13304], 90.00th=[15795], 95.00th=[22414],
     | 99.00th=[27657], 99.50th=[29230], 99.90th=[32375], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1528, max= 2912, per=24.64%, avg=2488.00, stdev=644.88, samples=4
   iops        : min=  382, max=  728, avg=622.00, stdev=161.22, samples=4
  lat (usec)   : 750=0.04%
  lat (msec)   : 2=0.08%, 4=0.08%, 10=16.99%, 20=76.41%, 50=6.41%
  cpu          : usr=1.36%, sys=9.93%, ctx=2471, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9699KiB/s (9932kB/s), 2405KiB/s-2461KiB/s (2463kB/s-2520kB/s), io=19.6MiB (20.5MB), run=2051-2069msec
  WRITE: bw=9.86MiB/s (10.3MB/s), 2507KiB/s-2544KiB/s (2567kB/s-2605kB/s), io=20.4MiB (21.4MB), run=2051-2069msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=17361: Sat May 11 16:19:07 2024
   read: IOPS=7, BW=299MiB/s (313MB/s)(40.0MiB/134msec)
    slat (nsec): min=6730.4k, max=6730.4k, avg=6730444.00, stdev= 0.00
    clat (nsec): min=121896k, max=121896k, avg=121895573.00, stdev= 0.00
     lat (nsec): min=128635k, max=128635k, avg=128634911.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  123],  5.00th=[  123], 10.00th=[  123], 20.00th=[  123],
     | 30.00th=[  123], 40.00th=[  123], 50.00th=[  123], 60.00th=[  123],
     | 70.00th=[  123], 80.00th=[  123], 90.00th=[  123], 95.00th=[  123],
     | 99.00th=[  123], 99.50th=[  123], 99.90th=[  123], 99.95th=[  123],
     | 99.99th=[  123]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=6.02%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=299MiB/s (313MB/s), 299MiB/s-299MiB/s (313MB/s-313MB/s), io=40.0MiB (41.9MB), run=134-134msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck