== sanity test 398c: run fio to test AIO ================= 19:38:21 (1736987901)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0948002 s, 442 MB/s
osc.lustre-OST0000-osc-ffff88012b17c000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=21082: Wed Jan 15 19:38:25 2025
  write: IOPS=1015, BW=4060KiB/s (4158kB/s)(10.0MiB/2522msec)
    slat (usec): min=11, max=208, avg=39.23, stdev=19.07
    clat (usec): min=1598, max=30559, avg=15678.43, stdev=2172.36
     lat (usec): min=1657, max=30602, avg=15718.77, stdev=2172.77
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12649], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15401], 50.00th=[15795], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17433], 95.00th=[17957],
     | 99.00th=[20055], 99.50th=[30540], 99.90th=[30540], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 3864, max= 4472, per=25.01%, avg=4057.60, stdev=244.78, samples=5
   iops        : min=  966, max= 1118, avg=1014.40, stdev=61.19, samples=5
  lat (msec)   : 2=0.08%, 4=0.12%, 10=0.55%, 20=98.32%, 50=0.94%
  cpu          : usr=1.03%, sys=4.60%, ctx=1313, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21083: Wed Jan 15 19:38:25 2025
  write: IOPS=1013, BW=4055KiB/s (4153kB/s)(10.0MiB/2525msec)
    slat (usec): min=12, max=225, avg=40.02, stdev=20.74
    clat (usec): min=3529, max=30623, avg=15700.62, stdev=2138.84
     lat (usec): min=3575, max=30636, avg=15741.70, stdev=2139.85
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12649], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15401], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17433], 95.00th=[17957],
     | 99.00th=[20055], 99.50th=[30540], 99.90th=[30540], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 3856, max= 4488, per=24.95%, avg=4046.40, stdev=253.83, samples=5
   iops        : min=  964, max= 1122, avg=1011.60, stdev=63.46, samples=5
  lat (msec)   : 4=0.04%, 10=0.74%, 20=98.09%, 50=1.13%
  cpu          : usr=0.79%, sys=4.91%, ctx=1429, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21084: Wed Jan 15 19:38:25 2025
  write: IOPS=1019, BW=4080KiB/s (4178kB/s)(10.0MiB/2510msec)
    slat (usec): min=12, max=193, avg=39.90, stdev=20.04
    clat (usec): min=1725, max=31851, avg=15618.74, stdev=2299.52
     lat (usec): min=1776, max=31914, avg=15659.70, stdev=2300.50
    clat percentiles (usec):
     |  1.00th=[ 6652],  5.00th=[12649], 10.00th=[13173], 20.00th=[14222],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17433], 95.00th=[17957],
     | 99.00th=[23200], 99.50th=[25297], 99.90th=[31851], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 3840, max= 4600, per=25.09%, avg=4070.40, stdev=304.81, samples=5
   iops        : min=  960, max= 1150, avg=1017.60, stdev=76.20, samples=5
  lat (msec)   : 2=0.04%, 4=0.31%, 10=0.98%, 20=97.54%, 50=1.13%
  cpu          : usr=0.92%, sys=4.74%, ctx=1329, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21085: Wed Jan 15 19:38:25 2025
  write: IOPS=1013, BW=4055KiB/s (4153kB/s)(10.0MiB/2525msec)
    slat (usec): min=12, max=299, avg=38.34, stdev=18.90
    clat (usec): min=2137, max=30517, avg=15701.22, stdev=2139.26
     lat (usec): min=2171, max=30551, avg=15740.56, stdev=2139.86
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12649], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15401], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16581], 80.00th=[16909], 90.00th=[17433], 95.00th=[17957],
     | 99.00th=[20055], 99.50th=[30540], 99.90th=[30540], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 3848, max= 4488, per=24.96%, avg=4048.00, stdev=252.41, samples=5
   iops        : min=  962, max= 1122, avg=1012.00, stdev=63.10, samples=5
  lat (msec)   : 4=0.16%, 10=0.51%, 20=98.28%, 50=1.05%
  cpu          : usr=0.75%, sys=4.83%, ctx=1179, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=15.8MiB/s (16.6MB/s), 4055KiB/s-4080KiB/s (4153kB/s-4178kB/s), io=40.0MiB (41.9MB), run=2510-2525msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=21097: Wed Jan 15 19:38:28 2025
   read: IOPS=607, BW=2430KiB/s (2489kB/s)(5048KiB/2077msec)
    slat (usec): min=23, max=1480, avg=800.09, stdev=159.27
    clat (usec): min=1115, max=24062, avg=12277.86, stdev=1934.34
     lat (usec): min=1149, max=24696, avg=13079.68, stdev=1968.50
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[ 9503], 10.00th=[10159], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12256], 60.00th=[12780],
     | 70.00th=[13173], 80.00th=[13829], 90.00th=[14484], 95.00th=[15008],
     | 99.00th=[16057], 99.50th=[16581], 99.90th=[23200], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min= 2344, max= 2544, per=25.41%, avg=2440.00, stdev=81.85, samples=4
   iops        : min=  586, max=  636, avg=610.00, stdev=20.46, samples=4
  write: IOPS=624, BW=2500KiB/s (2560kB/s)(5192KiB/2077msec)
    slat (usec): min=12, max=247, avg=40.71, stdev=20.91
    clat (usec): min=3075, max=23272, avg=12776.72, stdev=1533.42
     lat (usec): min=3097, max=23297, avg=12818.37, stdev=1533.47
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10552], 10.00th=[11076], 20.00th=[11600],
     | 30.00th=[12125], 40.00th=[12518], 50.00th=[12780], 60.00th=[13173],
     | 70.00th=[13566], 80.00th=[13829], 90.00th=[14353], 95.00th=[14877],
     | 99.00th=[16188], 99.50th=[17433], 99.90th=[22414], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 2376, max= 2696, per=24.99%, avg=2498.00, stdev=146.26, samples=4
   iops        : min=  594, max=  674, avg=624.50, stdev=36.57, samples=4
  lat (msec)   : 2=0.16%, 4=0.12%, 10=5.08%, 20=94.30%, 50=0.35%
  cpu          : usr=1.30%, sys=9.39%, ctx=2305, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21098: Wed Jan 15 19:38:28 2025
   read: IOPS=595, BW=2382KiB/s (2439kB/s)(4976KiB/2089msec)
    slat (usec): min=464, max=1477, avg=800.20, stdev=142.60
    clat (usec): min=2562, max=26418, avg=12316.20, stdev=1905.71
     lat (usec): min=3537, max=27136, avg=13118.18, stdev=1916.18
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[10683],
     | 30.00th=[11207], 40.00th=[11863], 50.00th=[12387], 60.00th=[12780],
     | 70.00th=[13173], 80.00th=[13829], 90.00th=[14353], 95.00th=[15008],
     | 99.00th=[16909], 99.50th=[18220], 99.90th=[25560], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 2304, max= 2464, per=24.83%, avg=2384.00, stdev=65.32, samples=4
   iops        : min=  576, max=  616, avg=596.00, stdev=16.33, samples=4
  write: IOPS=629, BW=2520KiB/s (2580kB/s)(5264KiB/2089msec)
    slat (usec): min=13, max=609, avg=42.06, stdev=27.00
    clat (usec): min=4355, max=23124, avg=12838.07, stdev=1470.26
     lat (usec): min=4434, max=23160, avg=12881.07, stdev=1468.39
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10683], 10.00th=[11207], 20.00th=[11731],
     | 30.00th=[12125], 40.00th=[12387], 50.00th=[12780], 60.00th=[13173],
     | 70.00th=[13566], 80.00th=[13960], 90.00th=[14746], 95.00th=[15139],
     | 99.00th=[16450], 99.50th=[16712], 99.90th=[21365], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 2432, max= 2656, per=25.31%, avg=2530.00, stdev=108.20, samples=4
   iops        : min=  608, max=  664, avg=632.50, stdev=27.05, samples=4
  lat (msec)   : 4=0.04%, 10=4.88%, 20=94.73%, 50=0.35%
  cpu          : usr=1.01%, sys=9.82%, ctx=2281, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21099: Wed Jan 15 19:38:28 2025
   read: IOPS=596, BW=2385KiB/s (2442kB/s)(4984KiB/2090msec)
    slat (usec): min=416, max=1542, avg=795.53, stdev=137.94
    clat (usec): min=713, max=20982, avg=12236.83, stdev=1811.46
     lat (usec): min=1508, max=22012, avg=13034.13, stdev=1817.02
    clat percentiles (usec):
     |  1.00th=[ 8291],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[10814],
     | 30.00th=[11207], 40.00th=[11731], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[13173], 80.00th=[13698], 90.00th=[14484], 95.00th=[15008],
     | 99.00th=[16450], 99.50th=[16909], 99.90th=[20055], 99.95th=[21103],
     | 99.99th=[21103]
   bw (  KiB/s): min= 2256, max= 2488, per=24.98%, avg=2398.00, stdev=102.74, samples=4
   iops        : min=  564, max=  622, avg=599.50, stdev=25.68, samples=4
  write: IOPS=628, BW=2515KiB/s (2575kB/s)(5256KiB/2090msec)
    slat (usec): min=12, max=537, avg=42.89, stdev=26.91
    clat (usec): min=7787, max=22404, avg=12912.29, stdev=1439.27
     lat (usec): min=7845, max=22448, avg=12956.25, stdev=1438.12
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[10814], 10.00th=[11207], 20.00th=[11731],
     | 30.00th=[12125], 40.00th=[12518], 50.00th=[12780], 60.00th=[13173],
     | 70.00th=[13435], 80.00th=[14091], 90.00th=[14746], 95.00th=[15270],
     | 99.00th=[16450], 99.50th=[17433], 99.90th=[20579], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 2480, max= 2544, per=24.99%, avg=2498.00, stdev=30.90, samples=4
   iops        : min=  620, max=  636, avg=624.50, stdev= 7.72, samples=4
  lat (usec)   : 750=0.04%
  lat (msec)   : 4=0.08%, 10=4.38%, 20=95.27%, 50=0.23%
  cpu          : usr=0.96%, sys=9.96%, ctx=2384, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21100: Wed Jan 15 19:38:28 2025
   read: IOPS=605, BW=2423KiB/s (2482kB/s)(5060KiB/2088msec)
    slat (usec): min=482, max=1592, avg=799.29, stdev=146.26
    clat (usec): min=1964, max=19270, avg=12260.74, stdev=1711.97
     lat (usec): min=2607, max=19915, avg=13061.76, stdev=1731.08
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9503], 10.00th=[10159], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[13173], 80.00th=[13698], 90.00th=[14353], 95.00th=[14877],
     | 99.00th=[16188], 99.50th=[16712], 99.90th=[18744], 99.95th=[19268],
     | 99.99th=[19268]
   bw (  KiB/s): min= 2376, max= 2440, per=25.16%, avg=2416.00, stdev=30.64, samples=4
   iops        : min=  594, max=  610, avg=604.00, stdev= 7.66, samples=4
  write: IOPS=620, BW=2481KiB/s (2540kB/s)(5180KiB/2088msec)
    slat (usec): min=12, max=348, avg=43.02, stdev=26.25
    clat (usec): min=6928, max=19985, avg=12862.50, stdev=1432.09
     lat (usec): min=6978, max=20028, avg=12906.51, stdev=1431.46
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[10814], 10.00th=[11207], 20.00th=[11600],
     | 30.00th=[11994], 40.00th=[12387], 50.00th=[12780], 60.00th=[13173],
     | 70.00th=[13566], 80.00th=[14091], 90.00th=[14615], 95.00th=[15139],
     | 99.00th=[16188], 99.50th=[16909], 99.90th=[20055], 99.95th=[20055],
     | 99.99th=[20055]
   bw (  KiB/s): min= 2328, max= 2560, per=24.83%, avg=2482.00, stdev=104.59, samples=4
   iops        : min=  582, max=  640, avg=620.50, stdev=26.15, samples=4
  lat (msec)   : 2=0.04%, 10=5.27%, 20=94.69%
  cpu          : usr=1.29%, sys=9.44%, ctx=2443, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9602KiB/s (9832kB/s), 2382KiB/s-2430KiB/s (2439kB/s-2489kB/s), io=19.6MiB (20.5MB), run=2077-2090msec
  WRITE: bw=9996KiB/s (10.2MB/s), 2481KiB/s-2520KiB/s (2540kB/s-2580kB/s), io=20.4MiB (21.4MB), run=2077-2090msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=21106: Wed Jan 15 19:38:30 2025
   read: IOPS=11, BW=460MiB/s (482MB/s)(40.0MiB/87msec)
    slat (nsec): min=7583.6k, max=7583.6k, avg=7583619.00, stdev= 0.00
    clat (nsec): min=75808k, max=75808k, avg=75808136.00, stdev= 0.00
     lat (nsec): min=83417k, max=83417k, avg=83416844.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[76022],  5.00th=[76022], 10.00th=[76022], 20.00th=[76022],
     | 30.00th=[76022], 40.00th=[76022], 50.00th=[76022], 60.00th=[76022],
     | 70.00th=[76022], 80.00th=[76022], 90.00th=[76022], 95.00th=[76022],
     | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.47%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=460MiB/s (482MB/s), 460MiB/s-460MiB/s (482MB/s-482MB/s), io=40.0MiB (41.9MB), run=87-87msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck