== sanity test 398c: run fio to test AIO ================= 18:06:06 (1718402766)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0863194 s, 486 MB/s
osc.lustre-OST0000-osc-ffff88013012c000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=14944: Fri Jun 14 18:06:09 2024
  write: IOPS=1017, BW=4072KiB/s (4169kB/s)(10.0MiB/2515msec)
    slat (usec): min=11, max=272, avg=43.46, stdev=23.89
    clat (usec): min=6395, max=26398, avg=15630.09, stdev=2144.97
     lat (usec): min=6414, max=26438, avg=15674.77, stdev=2147.57
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[12649], 10.00th=[13304], 20.00th=[14222],
     | 30.00th=[14746], 40.00th=[15270], 50.00th=[15664], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744],
     | 99.00th=[20317], 99.50th=[26084], 99.90th=[26346], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 3584, max= 4384, per=24.97%, avg=4067.00, stdev=300.53, samples=5
   iops        : min=  896, max= 1096, avg=1016.60, stdev=75.13, samples=5
  lat (msec)   : 10=1.68%, 20=97.19%, 50=1.13%
  cpu          : usr=0.88%, sys=4.85%, ctx=1704, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14945: Fri Jun 14 18:06:09 2024
  write: IOPS=1019, BW=4078KiB/s (4176kB/s)(10.0MiB/2511msec)
    slat (usec): min=12, max=241, avg=44.19, stdev=23.77
    clat (usec): min=2180, max=27410, avg=15614.07, stdev=2261.39
     lat (usec): min=2232, max=27455, avg=15659.41, stdev=2262.70
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[12518], 10.00th=[13304], 20.00th=[14091],
     | 30.00th=[14615], 40.00th=[15139], 50.00th=[15664], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744],
     | 99.00th=[20579], 99.50th=[27132], 99.90th=[27395], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 3616, max= 4424, per=24.99%, avg=4070.20, stdev=299.61, samples=5
   iops        : min=  904, max= 1106, avg=1017.40, stdev=74.90, samples=5
  lat (msec)   : 4=0.12%, 10=1.72%, 20=96.72%, 50=1.45%
  cpu          : usr=0.76%, sys=4.86%, ctx=1852, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14946: Fri Jun 14 18:06:09 2024
  write: IOPS=1017, BW=4072KiB/s (4169kB/s)(10.0MiB/2515msec)
    slat (usec): min=12, max=340, avg=44.30, stdev=25.29
    clat (usec): min=8232, max=26429, avg=15630.06, stdev=2112.48
     lat (usec): min=8288, max=26450, avg=15675.54, stdev=2113.76
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[12649], 10.00th=[13304], 20.00th=[14222],
     | 30.00th=[14615], 40.00th=[15139], 50.00th=[15664], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744],
     | 99.00th=[20317], 99.50th=[26346], 99.90th=[26346], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 3608, max= 4368, per=24.94%, avg=4062.20, stdev=286.00, samples=5
   iops        : min=  902, max= 1092, avg=1015.40, stdev=71.49, samples=5
  lat (msec)   : 10=1.33%, 20=97.27%, 50=1.41%
  cpu          : usr=0.99%, sys=4.46%, ctx=1740, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14947: Fri Jun 14 18:06:09 2024
  write: IOPS=1020, BW=4081KiB/s (4179kB/s)(10.0MiB/2509msec)
    slat (usec): min=12, max=271, avg=45.23, stdev=25.03
    clat (usec): min=2153, max=30472, avg=15604.54, stdev=2346.08
     lat (usec): min=2262, max=30565, avg=15650.91, stdev=2348.77
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[12256], 10.00th=[13435], 20.00th=[14091],
     | 30.00th=[14615], 40.00th=[15139], 50.00th=[15533], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744],
     | 99.00th=[20841], 99.50th=[26608], 99.90th=[30278], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 3616, max= 4464, per=25.00%, avg=4072.00, stdev=314.81, samples=5
   iops        : min=  904, max= 1116, avg=1018.00, stdev=78.70, samples=5
  lat (msec)   : 4=0.12%, 10=2.50%, 20=95.82%, 50=1.56%
  cpu          : usr=0.64%, sys=5.14%, ctx=1891, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=15.9MiB/s (16.7MB/s), 4072KiB/s-4081KiB/s (4169kB/s-4179kB/s), io=40.0MiB (41.9MB), run=2509-2515msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=14959: Fri Jun 14 18:06:12 2024
   read: IOPS=623, BW=2494KiB/s (2554kB/s)(5048KiB/2024msec)
    slat (usec): min=454, max=1727, avg=770.20, stdev=126.25
    clat (usec): min=1629, max=17534, avg=12001.21, stdev=1709.54
     lat (usec): min=2342, max=18509, avg=12773.39, stdev=1722.30
    clat percentiles (usec):
     |  1.00th=[ 8291],  5.00th=[ 9372], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[11076], 40.00th=[11469], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[14222], 95.00th=[15008],
     | 99.00th=[16057], 99.50th=[16450], 99.90th=[16909], 99.95th=[17433],
     | 99.99th=[17433]
   bw (  KiB/s): min= 2264, max= 2618, per=25.02%, avg=2480.50, stdev=155.03, samples=4
   iops        : min=  566, max=  654, avg=620.00, stdev=38.61, samples=4
  write: IOPS=641, BW=2565KiB/s (2627kB/s)(5192KiB/2024msec)
    slat (usec): min=13, max=734, avg=42.80, stdev=34.31
    clat (usec): min=3044, max=18932, avg=12328.45, stdev=1303.71
     lat (usec): min=3081, max=18975, avg=12372.31, stdev=1303.06
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10290], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[11994], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[12911], 80.00th=[13304], 90.00th=[13960], 95.00th=[14353],
     | 99.00th=[15533], 99.50th=[15926], 99.90th=[17957], 99.95th=[19006],
     | 99.99th=[19006]
   bw (  KiB/s): min= 2432, max= 2664, per=24.81%, avg=2560.50, stdev=95.84, samples=4
   iops        : min=  608, max=  666, avg=640.00, stdev=23.94, samples=4
  lat (msec)   : 2=0.04%, 4=0.04%, 10=6.91%, 20=93.01%
  cpu          : usr=1.14%, sys=9.00%, ctx=2789, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14960: Fri Jun 14 18:06:12 2024
   read: IOPS=615, BW=2462KiB/s (2521kB/s)(4976KiB/2021msec)
    slat (usec): min=460, max=1649, avg=773.29, stdev=133.36
    clat (usec): min=2529, max=19734, avg=12025.82, stdev=1716.69
     lat (usec): min=3167, max=20399, avg=12800.98, stdev=1728.69
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10552],
     | 30.00th=[11076], 40.00th=[11600], 50.00th=[11994], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[14091], 95.00th=[14746],
     | 99.00th=[16319], 99.50th=[16909], 99.90th=[19006], 99.95th=[19792],
     | 99.99th=[19792]
   bw (  KiB/s): min= 2264, max= 2610, per=24.76%, avg=2454.50, stdev=143.29, samples=4
   iops        : min=  566, max=  652, avg=613.50, stdev=35.64, samples=4
  write: IOPS=651, BW=2605KiB/s (2667kB/s)(5264KiB/2021msec)
    slat (usec): min=13, max=762, avg=42.92, stdev=32.15
    clat (usec): min=2684, max=19692, avg=12295.62, stdev=1394.98
     lat (usec): min=2766, max=19727, avg=12339.94, stdev=1394.02
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10159], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[11994], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[13960], 95.00th=[14353],
     | 99.00th=[15401], 99.50th=[15926], 99.90th=[18482], 99.95th=[19792],
     | 99.99th=[19792]
   bw (  KiB/s): min= 2448, max= 2688, per=25.25%, avg=2606.50, stdev=109.83, samples=4
   iops        : min=  612, max=  672, avg=651.50, stdev=27.44, samples=4
  lat (msec)   : 4=0.12%, 10=7.73%, 20=92.15%
  cpu          : usr=0.89%, sys=9.41%, ctx=2870, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14961: Fri Jun 14 18:06:12 2024
   read: IOPS=618, BW=2475KiB/s (2534kB/s)(4984KiB/2014msec)
    slat (usec): min=38, max=1729, avg=768.14, stdev=136.57
    clat (usec): min=1201, max=17039, avg=11960.47, stdev=1763.27
     lat (usec): min=1267, max=18093, avg=12730.45, stdev=1793.91
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 9241], 10.00th=[ 9896], 20.00th=[10552],
     | 30.00th=[11076], 40.00th=[11600], 50.00th=[12125], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13304], 90.00th=[14091], 95.00th=[14615],
     | 99.00th=[15795], 99.50th=[16057], 99.90th=[16712], 99.95th=[16909],
     | 99.99th=[16909]
   bw (  KiB/s): min= 2392, max= 2576, per=24.98%, avg=2476.75, stdev=79.33, samples=4
   iops        : min=  598, max=  644, avg=619.00, stdev=19.77, samples=4
  write: IOPS=652, BW=2610KiB/s (2672kB/s)(5256KiB/2014msec)
    slat (usec): min=12, max=278, avg=42.40, stdev=23.99
    clat (usec): min=4319, max=19088, avg=12341.87, stdev=1412.70
     lat (usec): min=4359, max=19110, avg=12385.36, stdev=1411.50
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[10159], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[11994], 50.00th=[12387], 60.00th=[12649],
     | 70.00th=[13042], 80.00th=[13435], 90.00th=[13960], 95.00th=[14484],
     | 99.00th=[15926], 99.50th=[16581], 99.90th=[19006], 99.95th=[19006],
     | 99.99th=[19006]
   bw (  KiB/s): min= 2416, max= 2720, per=25.17%, avg=2598.50, stdev=146.49, samples=4
   iops        : min=  604, max=  680, avg=649.50, stdev=36.49, samples=4
  lat (msec)   : 2=0.12%, 4=0.12%, 10=7.77%, 20=91.99%
  cpu          : usr=1.19%, sys=8.99%, ctx=2945, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14962: Fri Jun 14 18:06:12 2024
   read: IOPS=626, BW=2506KiB/s (2566kB/s)(5060KiB/2019msec)
    slat (usec): min=471, max=1577, avg=767.28, stdev=129.95
    clat (usec): min=2521, max=17104, avg=11960.70, stdev=1628.35
     lat (usec): min=3454, max=18024, avg=12729.95, stdev=1645.33
    clat percentiles (usec):
     |  1.00th=[ 8160],  5.00th=[ 9372], 10.00th=[ 9896], 20.00th=[10552],
     | 30.00th=[11207], 40.00th=[11600], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13304], 90.00th=[13960], 95.00th=[14615],
     | 99.00th=[15795], 99.50th=[16057], 99.90th=[16712], 99.95th=[17171],
     | 99.99th=[17171]
   bw (  KiB/s): min= 2280, max= 2616, per=25.23%, avg=2502.00, stdev=153.52, samples=4
   iops        : min=  570, max=  654, avg=625.50, stdev=38.38, samples=4
  write: IOPS=641, BW=2566KiB/s (2627kB/s)(5180KiB/2019msec)
    slat (usec): min=12, max=227, avg=42.03, stdev=21.51
    clat (usec): min=7129, max=19139, avg=12323.61, stdev=1350.09
     lat (usec): min=7230, max=19187, avg=12366.97, stdev=1350.60
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[10159], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[11994], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[13042], 80.00th=[13304], 90.00th=[13960], 95.00th=[14615],
     | 99.00th=[15533], 99.50th=[15664], 99.90th=[18482], 99.95th=[19268],
     | 99.99th=[19268]
   bw (  KiB/s): min= 2472, max= 2608, per=24.78%, avg=2558.00, stdev=59.64, samples=4
   iops        : min=  618, max=  652, avg=639.50, stdev=14.91, samples=4
  lat (msec)   : 4=0.08%, 10=7.50%, 20=92.42%
  cpu          : usr=1.14%, sys=8.92%, ctx=3001, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9915KiB/s (10.2MB/s), 2462KiB/s-2506KiB/s (2521kB/s-2566kB/s), io=19.6MiB (20.5MB), run=2014-2024msec
  WRITE: bw=10.1MiB/s (10.6MB/s), 2565KiB/s-2610KiB/s (2627kB/s-2672kB/s), io=20.4MiB (21.4MB), run=2014-2024msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=14967: Fri Jun 14 18:06:14 2024
   read: IOPS=13, BW=533MiB/s (559MB/s)(40.0MiB/75msec)
    slat (nsec): min=7938.1k, max=7938.1k, avg=7938146.00, stdev= 0.00
    clat (nsec): min=63745k, max=63745k, avg=63744862.00, stdev= 0.00
     lat (nsec): min=71695k, max=71695k, avg=71695398.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[63701],  5.00th=[63701], 10.00th=[63701], 20.00th=[63701],
     | 30.00th=[63701], 40.00th=[63701], 50.00th=[63701], 60.00th=[63701],
     | 70.00th=[63701], 80.00th=[63701], 90.00th=[63701], 95.00th=[63701],
     | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701],
     | 99.99th=[63701]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=12.16%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=533MiB/s (559MB/s), 533MiB/s-533MiB/s (559MB/s-559MB/s), io=40.0MiB (41.9MB), run=75-75msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck