== sanity test 398c: run fio to test AIO ================= 06:56:58 (1721386618)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.076556 s, 548 MB/s
osc.lustre-OST0000-osc-ffff88012d35e800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=22545: Fri Jul 19 06:57:02 2024
  write: IOPS=949, BW=3800KiB/s (3891kB/s)(10.0MiB/2695msec)
    slat (usec): min=12, max=20845, avg=51.59, stdev=411.80
    clat (usec): min=2181, max=24828, avg=16620.54, stdev=1890.96
     lat (usec): min=2248, max=35183, avg=16673.33, stdev=1924.01
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13435], 10.00th=[14222], 20.00th=[15139],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[17957], 90.00th=[18744], 95.00th=[19268],
     | 99.00th=[21103], 99.50th=[23462], 99.90th=[24773], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 3600, max= 4088, per=24.69%, avg=3752.00, stdev=196.86, samples=5
   iops        : min=  900, max= 1022, avg=938.00, stdev=49.21, samples=5
  lat (msec)   : 4=0.08%, 20=97.70%, 50=2.23%
  cpu          : usr=0.52%, sys=5.53%, ctx=1511, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22546: Fri Jul 19 06:57:02 2024
  write: IOPS=949, BW=3800KiB/s (3891kB/s)(10.0MiB/2695msec)
    slat (usec): min=12, max=18654, avg=49.90, stdev=368.53
    clat (usec): min=1776, max=24810, avg=16621.95, stdev=1936.04
     lat (usec): min=1819, max=24823, avg=16673.05, stdev=1930.30
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13435], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[15795], 40.00th=[16319], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[17957], 90.00th=[18744], 95.00th=[19268],
     | 99.00th=[21365], 99.50th=[23200], 99.90th=[23725], 99.95th=[23725],
     | 99.99th=[24773]
   bw (  KiB/s): min= 3576, max= 4112, per=24.68%, avg=3750.40, stdev=210.86, samples=5
   iops        : min=  894, max= 1028, avg=937.60, stdev=52.71, samples=5
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.12%, 20=97.23%, 50=2.54%
  cpu          : usr=0.82%, sys=5.31%, ctx=1395, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22547: Fri Jul 19 06:57:02 2024
  write: IOPS=956, BW=3825KiB/s (3917kB/s)(10.0MiB/2677msec)
    slat (usec): min=13, max=218, avg=43.89, stdev=22.50
    clat (usec): min=1957, max=30174, avg=16658.28, stdev=2152.07
     lat (usec): min=2003, max=30212, avg=16703.42, stdev=2150.35
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13435], 10.00th=[13960], 20.00th=[15008],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[23462], 99.50th=[24249], 99.90th=[28181], 99.95th=[30016],
     | 99.99th=[30278]
   bw (  KiB/s): min= 3592, max= 4080, per=24.95%, avg=3792.00, stdev=190.16, samples=5
   iops        : min=  898, max= 1020, avg=948.00, stdev=47.54, samples=5
  lat (msec)   : 2=0.04%, 4=0.16%, 10=0.12%, 20=96.88%, 50=2.81%
  cpu          : usr=0.52%, sys=5.08%, ctx=1558, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22548: Fri Jul 19 06:57:02 2024
  write: IOPS=950, BW=3802KiB/s (3894kB/s)(10.0MiB/2693msec)
    slat (usec): min=13, max=20419, avg=50.98, stdev=403.40
    clat (usec): min=5776, max=27985, avg=16610.19, stdev=1905.82
     lat (usec): min=5828, max=28584, avg=16662.36, stdev=1912.55
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13566], 10.00th=[14091], 20.00th=[15008],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[17957], 90.00th=[18744], 95.00th=[19268],
     | 99.00th=[20841], 99.50th=[23200], 99.90th=[24773], 99.95th=[26346],
     | 99.99th=[27919]
   bw (  KiB/s): min= 3576, max= 4048, per=24.69%, avg=3752.00, stdev=182.87, samples=5
   iops        : min=  894, max= 1012, avg=938.00, stdev=45.72, samples=5
  lat (msec)   : 10=0.12%, 20=97.54%, 50=2.34%
  cpu          : usr=0.67%, sys=5.50%, ctx=1463, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=14.8MiB/s (15.6MB/s), 3800KiB/s-3825KiB/s (3891kB/s-3917kB/s), io=40.0MiB (41.9MB), run=2677-2695msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=22560: Fri Jul 19 06:57:05 2024
   read: IOPS=635, BW=2543KiB/s (2604kB/s)(5048KiB/1985msec)
    slat (usec): min=481, max=1651, avg=760.55, stdev=135.14
    clat (usec): min=1635, max=20000, avg=11642.57, stdev=1788.18
     lat (usec): min=2521, max=21000, avg=12404.81, stdev=1814.20
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 9110], 10.00th=[ 9503], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11207], 50.00th=[11600], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[13698], 95.00th=[14615],
     | 99.00th=[16450], 99.50th=[17171], 99.90th=[18744], 99.95th=[20055],
     | 99.99th=[20055]
   bw (  KiB/s): min= 2299, max= 2584, per=24.41%, avg=2467.67, stdev=149.53, samples=3
   iops        : min=  574, max=  646, avg=616.67, stdev=37.81, samples=3
  write: IOPS=653, BW=2616KiB/s (2678kB/s)(5192KiB/1985msec)
    slat (usec): min=12, max=324, avg=41.87, stdev=23.59
    clat (usec): min=2062, max=20541, avg=12163.22, stdev=1510.50
     lat (usec): min=2091, max=20567, avg=12206.24, stdev=1510.60
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10028], 10.00th=[10552], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[11731], 50.00th=[12125], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13829], 95.00th=[14615],
     | 99.00th=[16581], 99.50th=[17433], 99.90th=[20579], 99.95th=[20579],
     | 99.99th=[20579]
   bw (  KiB/s): min= 2499, max= 2672, per=24.82%, avg=2611.67, stdev=97.65, samples=3
   iops        : min=  624, max=  668, avg=652.67, stdev=24.85, samples=3
  lat (msec)   : 2=0.04%, 4=0.20%, 10=10.39%, 20=89.26%, 50=0.12%
  cpu          : usr=1.06%, sys=9.58%, ctx=2511, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22561: Fri Jul 19 06:57:05 2024
   read: IOPS=627, BW=2512KiB/s (2572kB/s)(4976KiB/1981msec)
    slat (usec): min=452, max=1829, avg=757.49, stdev=133.77
    clat (usec): min=1753, max=19548, avg=11697.05, stdev=1715.68
     lat (usec): min=2393, max=20285, avg=12456.18, stdev=1737.43
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10290],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11600], 60.00th=[12125],
     | 70.00th=[12387], 80.00th=[12911], 90.00th=[13698], 95.00th=[14484],
     | 99.00th=[16909], 99.50th=[17433], 99.90th=[18744], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min= 2347, max= 2560, per=24.25%, avg=2451.67, stdev=106.55, samples=3
   iops        : min=  586, max=  640, avg=612.67, stdev=27.01, samples=3
  write: IOPS=664, BW=2657KiB/s (2721kB/s)(5264KiB/1981msec)
    slat (usec): min=12, max=180, avg=41.34, stdev=21.02
    clat (usec): min=4115, max=23527, avg=12136.75, stdev=1510.72
     lat (usec): min=4157, max=23576, avg=12179.17, stdev=1510.42
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[11731], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13829], 95.00th=[14484],
     | 99.00th=[17171], 99.50th=[17957], 99.90th=[22414], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 2538, max= 2760, per=25.32%, avg=2664.67, stdev=114.27, samples=3
   iops        : min=  634, max=  690, avg=666.00, stdev=28.84, samples=3
  lat (msec)   : 2=0.04%, 4=0.08%, 10=9.73%, 20=90.08%, 50=0.08%
  cpu          : usr=1.77%, sys=8.59%, ctx=2596, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22562: Fri Jul 19 06:57:05 2024
   read: IOPS=628, BW=2515KiB/s (2575kB/s)(4984KiB/1982msec)
    slat (usec): min=21, max=1715, avg=756.11, stdev=148.85
    clat (usec): min=1900, max=21618, avg=11781.11, stdev=1919.68
     lat (usec): min=1938, max=22356, avg=12538.88, stdev=1950.12
    clat percentiles (usec):
     |  1.00th=[ 8291],  5.00th=[ 9241], 10.00th=[ 9634], 20.00th=[10290],
     | 30.00th=[10814], 40.00th=[11338], 50.00th=[11731], 60.00th=[12125],
     | 70.00th=[12649], 80.00th=[13173], 90.00th=[13829], 95.00th=[14746],
     | 99.00th=[18220], 99.50th=[20055], 99.90th=[21365], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 2472, max= 2560, per=24.98%, avg=2525.33, stdev=46.88, samples=3
   iops        : min=  618, max=  640, avg=631.33, stdev=11.72, samples=3
  write: IOPS=662, BW=2652KiB/s (2716kB/s)(5256KiB/1982msec)
    slat (usec): min=13, max=352, avg=41.28, stdev=22.06
    clat (usec): min=3647, max=24981, avg=12139.77, stdev=1709.13
     lat (usec): min=3719, max=25016, avg=12182.36, stdev=1708.61
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10028], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11207], 40.00th=[11600], 50.00th=[11994], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13173], 90.00th=[13829], 95.00th=[14615],
     | 99.00th=[17957], 99.50th=[18744], 99.90th=[25035], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 2464, max= 2728, per=24.76%, avg=2605.33, stdev=132.99, samples=3
   iops        : min=  616, max=  682, avg=651.33, stdev=33.25, samples=3
  lat (msec)   : 2=0.04%, 4=0.20%, 10=9.61%, 20=89.73%, 50=0.43%
  cpu          : usr=1.31%, sys=8.99%, ctx=2647, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22563: Fri Jul 19 06:57:05 2024
   read: IOPS=639, BW=2557KiB/s (2618kB/s)(5060KiB/1979msec)
    slat (usec): min=437, max=1747, avg=765.18, stdev=137.43
    clat (usec): min=2503, max=20603, avg=11657.92, stdev=1786.72
     lat (usec): min=3366, max=21828, avg=12424.82, stdev=1808.69
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 8979], 10.00th=[ 9503], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11207], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12387], 80.00th=[12911], 90.00th=[13960], 95.00th=[14615],
     | 99.00th=[16712], 99.50th=[17433], 99.90th=[20579], 99.95th=[20579],
     | 99.99th=[20579]
   bw (  KiB/s): min= 2299, max= 2640, per=24.83%, avg=2510.33, stdev=184.59, samples=3
   iops        : min=  574, max=  660, avg=627.33, stdev=46.58, samples=3
  write: IOPS=654, BW=2617KiB/s (2680kB/s)(5180KiB/1979msec)
    slat (usec): min=12, max=632, avg=41.01, stdev=26.20
    clat (usec): min=8130, max=22935, avg=12103.84, stdev=1491.58
     lat (usec): min=8157, max=22960, avg=12145.87, stdev=1491.34
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11731], 50.00th=[11994], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13042], 90.00th=[13698], 95.00th=[14484],
     | 99.00th=[17433], 99.50th=[18744], 99.90th=[21103], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 2506, max= 2656, per=24.66%, avg=2595.33, stdev=79.00, samples=3
   iops        : min=  626, max=  664, avg=648.67, stdev=20.03, samples=3
  lat (msec)   : 4=0.08%, 10=11.09%, 20=88.59%, 50=0.23%
  cpu          : usr=1.11%, sys=9.20%, ctx=2686, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9.87MiB/s (10.4MB/s), 2512KiB/s-2557KiB/s (2572kB/s-2618kB/s), io=19.6MiB (20.5MB), run=1979-1985msec
  WRITE: bw=10.3MiB/s (10.8MB/s), 2616KiB/s-2657KiB/s (2678kB/s-2721kB/s), io=20.4MiB (21.4MB), run=1979-1985msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=22568: Fri Jul 19 06:57:06 2024
   read: IOPS=15, BW=615MiB/s (645MB/s)(40.0MiB/65msec)
    slat (nsec): min=6956.6k, max=6956.6k, avg=6956568.00, stdev= 0.00
    clat (nsec): min=54940k, max=54940k, avg=54940241.00, stdev= 0.00
     lat (nsec): min=61907k, max=61907k, avg=61906638.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[54789],  5.00th=[54789], 10.00th=[54789], 20.00th=[54789],
     | 30.00th=[54789], 40.00th=[54789], 50.00th=[54789], 60.00th=[54789],
     | 70.00th=[54789], 80.00th=[54789], 90.00th=[54789], 95.00th=[54789],
     | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[54789]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.77%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=615MiB/s (645MB/s), 615MiB/s-615MiB/s (645MB/s-645MB/s), io=40.0MiB (41.9MB), run=65-65msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck