== sanity test 398c: run fio to test AIO ================= 18:11:33 (1718403093)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0947458 s, 443 MB/s
osc.lustre-OST0000-osc-ffff880081bd8800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=14924: Fri Jun 14 18:11:37 2024
  write: IOPS=1063, BW=4252KiB/s (4355kB/s)(10.0MiB/2408msec)
    slat (usec): min=12, max=259, avg=41.29, stdev=22.68
    clat (usec): min=1828, max=28955, avg=14961.30, stdev=2522.80
     lat (usec): min=2020, max=28998, avg=15003.60, stdev=2524.11
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[10159], 10.00th=[12387], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14615], 50.00th=[15008], 60.00th=[15533],
     | 70.00th=[16057], 80.00th=[16712], 90.00th=[17433], 95.00th=[18482],
     | 99.00th=[20055], 99.50th=[24511], 99.90th=[28967], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 3816, max= 4736, per=25.53%, avg=4328.00, stdev=380.48, samples=4
   iops        : min=  954, max= 1184, avg=1082.00, stdev=95.12, samples=4
  lat (msec)   : 2=0.04%, 4=0.08%, 10=4.02%, 20=94.88%, 50=0.98%
  cpu          : usr=0.96%, sys=4.69%, ctx=1635, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14925: Fri Jun 14 18:11:37 2024
  write: IOPS=1066, BW=4267KiB/s (4369kB/s)(10.0MiB/2400msec)
    slat (usec): min=12, max=264, avg=43.95, stdev=23.44
    clat (usec): min=2950, max=27662, avg=14925.10, stdev=2570.18
     lat (usec): min=2994, max=27716, avg=14970.10, stdev=2572.53
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[10028], 10.00th=[12256], 20.00th=[13435],
     | 30.00th=[14222], 40.00th=[14615], 50.00th=[15008], 60.00th=[15533],
     | 70.00th=[16057], 80.00th=[16712], 90.00th=[17433], 95.00th=[18482],
     | 99.00th=[20579], 99.50th=[23725], 99.90th=[26870], 99.95th=[26870],
     | 99.99th=[27657]
   bw (  KiB/s): min= 3808, max= 4768, per=25.59%, avg=4338.00, stdev=396.16, samples=4
   iops        : min=  952, max= 1192, avg=1084.50, stdev=99.04, samples=4
  lat (msec)   : 4=0.12%, 10=4.65%, 20=94.02%, 50=1.21%
  cpu          : usr=0.63%, sys=5.63%, ctx=1862, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14926: Fri Jun 14 18:11:37 2024
  write: IOPS=1060, BW=4240KiB/s (4342kB/s)(10.0MiB/2415msec)
    slat (usec): min=13, max=354, avg=41.66, stdev=23.87
    clat (usec): min=2298, max=31285, avg=15003.47, stdev=2532.57
     lat (usec): min=2330, max=31303, avg=15046.22, stdev=2534.29
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[10290], 10.00th=[12387], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14746], 50.00th=[15139], 60.00th=[15533],
     | 70.00th=[16057], 80.00th=[16712], 90.00th=[17695], 95.00th=[18482],
     | 99.00th=[20317], 99.50th=[24511], 99.90th=[26608], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 3784, max= 4744, per=25.44%, avg=4312.00, stdev=395.82, samples=4
   iops        : min=  946, max= 1186, avg=1078.00, stdev=98.95, samples=4
  lat (msec)   : 4=0.04%, 10=4.30%, 20=94.61%, 50=1.05%
  cpu          : usr=0.75%, sys=4.89%, ctx=1668, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14927: Fri Jun 14 18:11:37 2024
  write: IOPS=1059, BW=4238KiB/s (4340kB/s)(10.0MiB/2416msec)
    slat (usec): min=12, max=18224, avg=46.99, stdev=362.85
    clat (usec): min=1701, max=25048, avg=14943.78, stdev=2350.81
     lat (usec): min=1754, max=25235, avg=14991.82, stdev=2346.17
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[10290], 10.00th=[12387], 20.00th=[13435],
     | 30.00th=[14222], 40.00th=[14746], 50.00th=[15139], 60.00th=[15533],
     | 70.00th=[16057], 80.00th=[16712], 90.00th=[17433], 95.00th=[18220],
     | 99.00th=[19792], 99.50th=[19792], 99.90th=[25035], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 3776, max= 4624, per=25.32%, avg=4292.00, stdev=363.07, samples=4
   iops        : min=  944, max= 1156, avg=1073.00, stdev=90.77, samples=4
  lat (msec)   : 2=0.04%, 4=0.04%, 10=4.10%, 20=95.51%, 50=0.31%
  cpu          : usr=0.95%, sys=5.09%, ctx=1286, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=16.6MiB/s (17.4MB/s), 4238KiB/s-4267KiB/s (4340kB/s-4369kB/s), io=40.0MiB (41.9MB), run=2400-2416msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=14939: Fri Jun 14 18:11:40 2024
   read: IOPS=637, BW=2549KiB/s (2611kB/s)(5048KiB/1980msec)
    slat (usec): min=15, max=1452, avg=747.18, stdev=138.53
    clat (usec): min=1244, max=23673, avg=11842.76, stdev=1803.19
     lat (usec): min=1285, max=24572, avg=12591.70, stdev=1837.05
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 9110], 10.00th=[ 9765], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[11863], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13173], 90.00th=[13829], 95.00th=[14484],
     | 99.00th=[15664], 99.50th=[16712], 99.90th=[22938], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 2456, max= 2608, per=25.03%, avg=2525.33, stdev=76.87, samples=3
   iops        : min=  614, max=  652, avg=631.33, stdev=19.22, samples=3
  write: IOPS=655, BW=2622KiB/s (2685kB/s)(5192KiB/1980msec)
    slat (usec): min=13, max=825, avg=43.42, stdev=35.08
    clat (usec): min=3213, max=20129, avg=12047.49, stdev=1403.36
     lat (usec): min=3259, max=20239, avg=12091.95, stdev=1404.47
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10028], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[11731], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12649], 80.00th=[13042], 90.00th=[13829], 95.00th=[14222],
     | 99.00th=[15270], 99.50th=[15401], 99.90th=[19268], 99.95th=[20055],
     | 99.99th=[20055]
   bw (  KiB/s): min= 2584, max= 2728, per=25.29%, avg=2656.00, stdev=72.00, samples=3
   iops        : min=  646, max=  682, avg=664.00, stdev=18.00, samples=3
  lat (msec)   : 2=0.12%, 4=0.16%, 10=8.48%, 20=91.05%, 50=0.20%
  cpu          : usr=1.57%, sys=8.79%, ctx=2778, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14940: Fri Jun 14 18:11:40 2024
   read: IOPS=626, BW=2504KiB/s (2564kB/s)(4976KiB/1987msec)
    slat (usec): min=428, max=1411, avg=748.28, stdev=126.78
    clat (usec): min=1818, max=22557, avg=11783.71, stdev=1670.07
     lat (usec): min=2967, max=23521, avg=12533.67, stdev=1688.51
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 9372], 10.00th=[ 9896], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11863], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[13698], 95.00th=[14222],
     | 99.00th=[16319], 99.50th=[16712], 99.90th=[21890], 99.95th=[22676],
     | 99.99th=[22676]
   bw (  KiB/s): min= 2384, max= 2536, per=24.40%, avg=2461.33, stdev=76.04, samples=3
   iops        : min=  596, max=  634, avg=615.33, stdev=19.01, samples=3
  write: IOPS=662, BW=2649KiB/s (2713kB/s)(5264KiB/1987msec)
    slat (usec): min=13, max=545, avg=42.53, stdev=26.54
    clat (usec): min=4756, max=22522, avg=12142.89, stdev=1359.12
     lat (usec): min=4777, max=22550, avg=12186.47, stdev=1358.89
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10159], 10.00th=[10552], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11731], 50.00th=[12125], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13829], 95.00th=[14353],
     | 99.00th=[15664], 99.50th=[16319], 99.90th=[21890], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 2664, max= 2696, per=25.54%, avg=2682.67, stdev=16.65, samples=3
   iops        : min=  666, max=  674, avg=670.67, stdev= 4.16, samples=3
  lat (msec)   : 2=0.04%, 4=0.12%, 10=7.93%, 20=91.76%, 50=0.16%
  cpu          : usr=1.16%, sys=9.06%, ctx=2878, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14941: Fri Jun 14 18:11:40 2024
   read: IOPS=626, BW=2506KiB/s (2566kB/s)(4984KiB/1989msec)
    slat (usec): min=425, max=1476, avg=748.58, stdev=124.35
    clat (usec): min=1757, max=20141, avg=11773.01, stdev=1626.93
     lat (usec): min=2609, max=20901, avg=12523.56, stdev=1634.24
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11863], 60.00th=[12125],
     | 70.00th=[12649], 80.00th=[13042], 90.00th=[13829], 95.00th=[14353],
     | 99.00th=[15533], 99.50th=[15926], 99.90th=[17957], 99.95th=[20055],
     | 99.99th=[20055]
   bw (  KiB/s): min= 2520, max= 2536, per=25.03%, avg=2525.33, stdev= 9.24, samples=3
   iops        : min=  630, max=  634, avg=631.33, stdev= 2.31, samples=3
  write: IOPS=660, BW=2643KiB/s (2706kB/s)(5256KiB/1989msec)
    slat (usec): min=12, max=518, avg=40.69, stdev=23.91
    clat (usec): min=8657, max=23108, avg=12153.14, stdev=1396.29
     lat (usec): min=8690, max=23129, avg=12194.84, stdev=1394.99
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10159], 10.00th=[10552], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[11731], 50.00th=[12125], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13698], 95.00th=[14222],
     | 99.00th=[15401], 99.50th=[16712], 99.90th=[23200], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 2512, max= 2664, per=24.70%, avg=2594.67, stdev=76.87, samples=3
   iops        : min=  628, max=  666, avg=648.67, stdev=19.22, samples=3
  lat (msec)   : 2=0.04%, 10=8.75%, 20=90.98%, 50=0.23%
  cpu          : usr=1.46%, sys=8.85%, ctx=2644, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14942: Fri Jun 14 18:11:40 2024
   read: IOPS=635, BW=2544KiB/s (2605kB/s)(5060KiB/1989msec)
    slat (usec): min=454, max=1488, avg=746.26, stdev=126.74
    clat (usec): min=1194, max=16694, avg=11768.19, stdev=1647.80
     lat (usec): min=1849, max=17208, avg=12516.07, stdev=1653.04
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11863], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13042], 90.00th=[13829], 95.00th=[14222],
     | 99.00th=[15533], 99.50th=[15795], 99.90th=[16450], 99.95th=[16712],
     | 99.99th=[16712]
   bw (  KiB/s): min= 2432, max= 2600, per=24.92%, avg=2514.67, stdev=84.03, samples=3
   iops        : min=  608, max=  650, avg=628.67, stdev=21.01, samples=3
  write: IOPS=651, BW=2604KiB/s (2667kB/s)(5180KiB/1989msec)
    slat (usec): min=12, max=722, avg=42.83, stdev=31.98
    clat (usec): min=1857, max=16298, avg=12168.08, stdev=1305.28
     lat (usec): min=1925, max=16336, avg=12212.02, stdev=1305.27
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10290], 10.00th=[10683], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12256], 60.00th=[12518],
     | 70.00th=[12780], 80.00th=[13173], 90.00th=[13698], 95.00th=[14353],
     | 99.00th=[15270], 99.50th=[15533], 99.90th=[16188], 99.95th=[16319],
     | 99.99th=[16319]
   bw (  KiB/s): min= 2584, max= 2616, per=24.73%, avg=2597.33, stdev=16.65, samples=3
   iops        : min=  646, max=  654, avg=649.33, stdev= 4.16, samples=3
  lat (msec)   : 2=0.08%, 4=0.08%, 10=8.55%, 20=91.29%
  cpu          : usr=1.51%, sys=8.80%, ctx=2782, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9.85MiB/s (10.3MB/s), 2504KiB/s-2549KiB/s (2564kB/s-2611kB/s), io=19.6MiB (20.5MB), run=1980-1989msec
  WRITE: bw=10.3MiB/s (10.8MB/s), 2604KiB/s-2649KiB/s (2667kB/s-2713kB/s), io=20.4MiB (21.4MB), run=1980-1989msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=14947: Fri Jun 14 18:11:41 2024
   read: IOPS=12, BW=494MiB/s (518MB/s)(40.0MiB/81msec)
    slat (nsec): min=8025.3k, max=8025.3k, avg=8025312.00, stdev= 0.00
    clat (nsec): min=69886k, max=69886k, avg=69885681.00, stdev= 0.00
     lat (nsec): min=77929k, max=77929k, avg=77929446.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[69731],  5.00th=[69731], 10.00th=[69731], 20.00th=[69731],
     | 30.00th=[69731], 40.00th=[69731], 50.00th=[69731], 60.00th=[69731],
     | 70.00th=[69731], 80.00th=[69731], 90.00th=[69731], 95.00th=[69731],
     | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[69731]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.25%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=40.0MiB (41.9MB), run=81-81msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck