== sanity test 398c: run fio to test AIO ================= 19:11:38 (1717024298)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0919764 s, 456 MB/s
osc.lustre-OST0000-osc-ffff88012c40d800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=16792: Wed May 29 19:11:42 2024
  write: IOPS=953, BW=3815KiB/s (3907kB/s)(10.0MiB/2684msec)
    slat (usec): min=12, max=311, avg=43.90, stdev=23.01
    clat (usec): min=2655, max=29319, avg=16693.67, stdev=2251.53
     lat (usec): min=2698, max=29362, avg=16738.84, stdev=2251.76
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12911], 10.00th=[14222], 20.00th=[15008],
     | 30.00th=[15533], 40.00th=[16319], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[19792],
     | 99.00th=[22414], 99.50th=[26084], 99.90th=[26346], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 3424, max= 4120, per=25.04%, avg=3820.80, stdev=276.24, samples=5
   iops        : min=  856, max= 1030, avg=955.20, stdev=69.06, samples=5
  lat (msec)   : 4=0.04%, 10=0.43%, 20=95.35%, 50=4.18%
  cpu          : usr=1.16%, sys=4.51%, ctx=1451, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16793: Wed May 29 19:11:42 2024
  write: IOPS=953, BW=3815KiB/s (3907kB/s)(10.0MiB/2684msec)
    slat (usec): min=12, max=297, avg=43.46, stdev=21.51
    clat (usec): min=1957, max=29320, avg=16694.46, stdev=2270.02
     lat (usec): min=1996, max=29375, avg=16739.12, stdev=2271.55
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12911], 10.00th=[14222], 20.00th=[15008],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[19792],
     | 99.00th=[21890], 99.50th=[26346], 99.90th=[27657], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 3440, max= 4128, per=25.05%, avg=3822.40, stdev=273.14, samples=5
   iops        : min=  860, max= 1032, avg=955.60, stdev=68.28, samples=5
  lat (msec)   : 2=0.04%, 10=0.51%, 20=94.80%, 50=4.65%
  cpu          : usr=0.75%, sys=4.92%, ctx=1412, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16794: Wed May 29 19:11:42 2024
  write: IOPS=960, BW=3841KiB/s (3933kB/s)(10.0MiB/2666msec)
    slat (usec): min=13, max=324, avg=43.24, stdev=21.59
    clat (usec): min=1906, max=27358, avg=16592.01, stdev=2391.26
     lat (usec): min=1969, max=27395, avg=16636.42, stdev=2391.35
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[12780], 10.00th=[14222], 20.00th=[15008],
     | 30.00th=[15533], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[19792],
     | 99.00th=[22152], 99.50th=[23200], 99.90th=[27395], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 3440, max= 4264, per=25.24%, avg=3851.20, stdev=310.59, samples=5
   iops        : min=  860, max= 1066, avg=962.80, stdev=77.65, samples=5
  lat (msec)   : 2=0.08%, 4=0.27%, 10=0.74%, 20=94.22%, 50=4.69%
  cpu          : usr=1.09%, sys=4.80%, ctx=1348, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16795: Wed May 29 19:11:42 2024
  write: IOPS=954, BW=3818KiB/s (3910kB/s)(10.0MiB/2682msec)
    slat (usec): min=12, max=19236, avg=51.89, stdev=379.94
    clat (usec): min=1887, max=27313, avg=16667.32, stdev=2289.34
     lat (usec): min=1929, max=27359, avg=16720.47, stdev=2275.98
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12780], 10.00th=[14222], 20.00th=[15008],
     | 30.00th=[15533], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[19792],
     | 99.00th=[22676], 99.50th=[25560], 99.90th=[27132], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 3440, max= 4136, per=25.07%, avg=3825.60, stdev=277.96, samples=5
   iops        : min=  860, max= 1034, avg=956.40, stdev=69.49, samples=5
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.47%, 20=94.77%, 50=4.65%
  cpu          : usr=1.04%, sys=5.52%, ctx=1467, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=14.9MiB/s (15.6MB/s), 3815KiB/s-3841KiB/s (3907kB/s-3933kB/s), io=40.0MiB (41.9MB), run=2666-2684msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16807: Wed May 29 19:11:46 2024
   read: IOPS=551, BW=2204KiB/s (2257kB/s)(5048KiB/2290msec)
    slat (usec): min=504, max=2156, avg=887.27, stdev=183.13
    clat (usec): min=1734, max=22798, avg=13557.84, stdev=2243.42
     lat (usec): min=2573, max=23815, avg=14446.87, stdev=2290.72
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[10290], 10.00th=[10814], 20.00th=[11600],
     | 30.00th=[12256], 40.00th=[12911], 50.00th=[13435], 60.00th=[14091],
     | 70.00th=[14615], 80.00th=[15401], 90.00th=[16581], 95.00th=[17433],
     | 99.00th=[19268], 99.50th=[20317], 99.90th=[22152], 99.95th=[22676],
     | 99.99th=[22676]
   bw (  KiB/s): min= 2024, max= 2392, per=25.24%, avg=2206.00, stdev=177.61, samples=4
   iops        : min=  506, max=  598, avg=551.50, stdev=44.40, samples=4
  write: IOPS=566, BW=2267KiB/s (2322kB/s)(5192KiB/2290msec)
    slat (usec): min=13, max=725, avg=45.63, stdev=29.62
    clat (usec): min=2804, max=21932, avg=13977.78, stdev=2029.57
     lat (usec): min=2889, max=22056, avg=14024.70, stdev=2032.20
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11207], 10.00th=[11731], 20.00th=[12387],
     | 30.00th=[12780], 40.00th=[13304], 50.00th=[13698], 60.00th=[14222],
     | 70.00th=[14746], 80.00th=[15533], 90.00th=[16712], 95.00th=[17433],
     | 99.00th=[19530], 99.50th=[20317], 99.90th=[20841], 99.95th=[21890],
     | 99.99th=[21890]
   bw (  KiB/s): min= 2104, max= 2496, per=25.39%, avg=2310.00, stdev=178.57, samples=4
   iops        : min=  526, max=  624, avg=577.50, stdev=44.64, samples=4
  lat (msec)   : 2=0.04%, 4=0.08%, 10=2.07%, 20=97.03%, 50=0.78%
  cpu          : usr=1.18%, sys=9.35%, ctx=2351, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16808: Wed May 29 19:11:46 2024
   read: IOPS=544, BW=2177KiB/s (2229kB/s)(4976KiB/2286msec)
    slat (usec): min=26, max=2239, avg=893.79, stdev=207.14
    clat (usec): min=885, max=23863, avg=13696.71, stdev=2343.33
     lat (usec): min=911, max=25037, avg=14592.35, stdev=2402.50
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[10290], 10.00th=[10945], 20.00th=[11863],
     | 30.00th=[12518], 40.00th=[13042], 50.00th=[13566], 60.00th=[14222],
     | 70.00th=[14746], 80.00th=[15533], 90.00th=[16581], 95.00th=[17695],
     | 99.00th=[20055], 99.50th=[20317], 99.90th=[22676], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min= 2096, max= 2336, per=25.03%, avg=2188.00, stdev=115.65, samples=4
   iops        : min=  524, max=  584, avg=547.00, stdev=28.91, samples=4
  write: IOPS=575, BW=2303KiB/s (2358kB/s)(5264KiB/2286msec)
    slat (usec): min=13, max=346, avg=45.03, stdev=24.76
    clat (usec): min=2483, max=23814, avg=13884.07, stdev=2095.91
     lat (usec): min=2544, max=23856, avg=13930.34, stdev=2098.32
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[11076], 10.00th=[11731], 20.00th=[12387],
     | 30.00th=[12780], 40.00th=[13304], 50.00th=[13698], 60.00th=[14222],
     | 70.00th=[14746], 80.00th=[15401], 90.00th=[16581], 95.00th=[17171],
     | 99.00th=[19792], 99.50th=[20317], 99.90th=[21890], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 2096, max= 2528, per=25.72%, avg=2340.00, stdev=195.69, samples=4
   iops        : min=  524, max=  632, avg=585.00, stdev=48.92, samples=4
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.04%, 4=0.16%, 10=2.70%, 20=96.33%, 50=0.74%
  cpu          : usr=1.18%, sys=9.28%, ctx=2344, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16809: Wed May 29 19:11:46 2024
   read: IOPS=543, BW=2173KiB/s (2225kB/s)(4984KiB/2294msec)
    slat (usec): min=534, max=2189, avg=886.21, stdev=188.20
    clat (usec): min=3026, max=21446, avg=13700.15, stdev=2261.79
     lat (usec): min=4043, max=22429, avg=14588.15, stdev=2307.62
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10290], 10.00th=[10945], 20.00th=[11863],
     | 30.00th=[12387], 40.00th=[12911], 50.00th=[13435], 60.00th=[13960],
     | 70.00th=[14877], 80.00th=[15664], 90.00th=[16909], 95.00th=[17695],
     | 99.00th=[19268], 99.50th=[20317], 99.90th=[20841], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min= 1896, max= 2400, per=24.94%, avg=2180.00, stdev=222.14, samples=4
   iops        : min=  474, max=  600, avg=545.00, stdev=55.53, samples=4
  write: IOPS=572, BW=2291KiB/s (2346kB/s)(5256KiB/2294msec)
    slat (usec): min=12, max=725, avg=44.34, stdev=30.78
    clat (usec): min=2345, max=21485, avg=13918.61, stdev=1894.38
     lat (usec): min=2490, max=21547, avg=13964.10, stdev=1894.02
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[11207], 10.00th=[11600], 20.00th=[12387],
     | 30.00th=[12911], 40.00th=[13304], 50.00th=[13829], 60.00th=[14353],
     | 70.00th=[14746], 80.00th=[15533], 90.00th=[16450], 95.00th=[17171],
     | 99.00th=[18482], 99.50th=[19268], 99.90th=[20579], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2080, max= 2480, per=25.34%, avg=2306.00, stdev=166.07, samples=4
   iops        : min=  520, max=  620, avg=576.50, stdev=41.52, samples=4
  lat (msec)   : 4=0.08%, 10=1.95%, 20=97.58%, 50=0.39%
  cpu          : usr=1.26%, sys=9.03%, ctx=2277, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16810: Wed May 29 19:11:46 2024
   read: IOPS=550, BW=2204KiB/s (2257kB/s)(5060KiB/2296msec)
    slat (usec): min=545, max=1919, avg=890.26, stdev=189.08
    clat (usec): min=5327, max=20951, avg=13613.42, stdev=2276.09
     lat (usec): min=6395, max=21817, avg=14505.52, stdev=2331.00
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10028], 10.00th=[10814], 20.00th=[11600],
     | 30.00th=[12256], 40.00th=[12911], 50.00th=[13566], 60.00th=[14091],
     | 70.00th=[14746], 80.00th=[15533], 90.00th=[16712], 95.00th=[17433],
     | 99.00th=[19268], 99.50th=[19792], 99.90th=[20317], 99.95th=[20841],
     | 99.99th=[20841]
   bw (  KiB/s): min= 2080, max= 2384, per=25.35%, avg=2216.00, stdev=155.26, samples=4
   iops        : min=  520, max=  596, avg=554.00, stdev=38.82, samples=4
  write: IOPS=564, BW=2256KiB/s (2310kB/s)(5180KiB/2296msec)
    slat (usec): min=12, max=1244, avg=48.11, stdev=50.91
    clat (usec): min=6400, max=22340, avg=14005.01, stdev=1903.93
     lat (usec): min=6436, max=22368, avg=14054.41, stdev=1903.73
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11207], 10.00th=[11731], 20.00th=[12518],
     | 30.00th=[12911], 40.00th=[13435], 50.00th=[13829], 60.00th=[14353],
     | 70.00th=[14746], 80.00th=[15533], 90.00th=[16581], 95.00th=[17171],
     | 99.00th=[19006], 99.50th=[19792], 99.90th=[21103], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 2096, max= 2448, per=24.90%, avg=2266.00, stdev=172.25, samples=4
   iops        : min=  524, max=  612, avg=566.50, stdev=43.06, samples=4
  lat (msec)   : 10=2.50%, 20=97.11%, 50=0.39%
  cpu          : usr=1.48%, sys=9.02%, ctx=2381, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=8740KiB/s (8950kB/s), 2173KiB/s-2204KiB/s (2225kB/s-2257kB/s), io=19.6MiB (20.5MB), run=2286-2296msec
  WRITE: bw=9099KiB/s (9318kB/s), 2256KiB/s-2303KiB/s (2310kB/s-2358kB/s), io=20.4MiB (21.4MB), run=2286-2296msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=16816: Wed May 29 19:11:47 2024
   read: IOPS=10, BW=435MiB/s (456MB/s)(40.0MiB/92msec)
    slat (nsec): min=8227.6k, max=8227.6k, avg=8227635.00, stdev= 0.00
    clat (nsec): min=80764k, max=80764k, avg=80764305.00, stdev= 0.00
     lat (nsec): min=89016k, max=89016k, avg=89015619.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[81265],  5.00th=[81265], 10.00th=[81265], 20.00th=[81265],
     | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265],
     | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265],
     | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.79%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=435MiB/s (456MB/s), 435MiB/s-435MiB/s (456MB/s-456MB/s), io=40.0MiB (41.9MB), run=92-92msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck