== sanity test 398c: run fio to test AIO ================= 10:03:54 (1747404234)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.104382 s, 402 MB/s
osc.lustre-OST0000-osc-ffff8800b6259000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=14458: Fri May 16 10:03:59 2025
  write: IOPS=777, BW=3109KiB/s (3183kB/s)(10.0MiB/3294msec)
    slat (usec): min=12, max=290, avg=46.12, stdev=27.78
    clat (usec): min=5554, max=35954, avg=20478.02, stdev=4555.25
     lat (usec): min=5651, max=36007, avg=20525.36, stdev=4560.89
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15270], 10.00th=[16188], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[18744], 60.00th=[19530],
     | 70.00th=[21627], 80.00th=[24773], 90.00th=[28443], 95.00th=[30016],
     | 99.00th=[31327], 99.50th=[31851], 99.90th=[33817], 99.95th=[34341],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2488, max= 3616, per=24.52%, avg=3046.67, stdev=569.60, samples=6
   iops        : min=  622, max=  904, avg=761.67, stdev=142.40, samples=6
  lat (msec)   : 10=0.04%, 20=63.75%, 50=36.21%
  cpu          : usr=0.76%, sys=4.62%, ctx=946, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14459: Fri May 16 10:03:59 2025
  write: IOPS=776, BW=3107KiB/s (3181kB/s)(10.0MiB/3296msec)
    slat (usec): min=11, max=182, avg=45.22, stdev=25.31
    clat (usec): min=12226, max=36067, avg=20489.19, stdev=4607.68
     lat (usec): min=12276, max=36177, avg=20535.68, stdev=4614.19
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15270], 10.00th=[16057], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[21627], 80.00th=[24773], 90.00th=[28443], 95.00th=[30016],
     | 99.00th=[31589], 99.50th=[33424], 99.90th=[35914], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2496, max= 3592, per=24.42%, avg=3034.67, stdev=570.50, samples=6
   iops        : min=  624, max=  898, avg=758.67, stdev=142.62, samples=6
  lat (msec)   : 20=63.59%, 50=36.41%
  cpu          : usr=0.67%, sys=4.61%, ctx=981, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14460: Fri May 16 10:03:59 2025
  write: IOPS=779, BW=3116KiB/s (3191kB/s)(10.0MiB/3286msec)
    slat (usec): min=11, max=251, avg=46.12, stdev=30.18
    clat (usec): min=2565, max=35942, avg=20452.75, stdev=4546.99
     lat (usec): min=2691, max=36000, avg=20500.21, stdev=4556.75
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15401], 10.00th=[16057], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[18744], 60.00th=[19530],
     | 70.00th=[21627], 80.00th=[24511], 90.00th=[28443], 95.00th=[30016],
     | 99.00th=[31589], 99.50th=[32900], 99.90th=[34866], 99.95th=[34866],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2483, max= 3584, per=24.58%, avg=3055.17, stdev=560.36, samples=6
   iops        : min=  620, max=  896, avg=763.67, stdev=140.24, samples=6
  lat (msec)   : 4=0.08%, 10=0.04%, 20=63.28%, 50=36.60%
  cpu          : usr=0.82%, sys=4.60%, ctx=809, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14461: Fri May 16 10:03:59 2025
  write: IOPS=776, BW=3107KiB/s (3181kB/s)(10.0MiB/3296msec)
    slat (usec): min=11, max=269, avg=48.17, stdev=28.79
    clat (usec): min=5020, max=36055, avg=20501.75, stdev=4637.63
     lat (usec): min=5116, max=36090, avg=20551.20, stdev=4643.88
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[15270], 10.00th=[16057], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[21365], 80.00th=[24511], 90.00th=[28705], 95.00th=[30016],
     | 99.00th=[31851], 99.50th=[33817], 99.90th=[35914], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2499, max= 3584, per=24.48%, avg=3041.83, stdev=576.86, samples=6
   iops        : min=  624, max=  896, avg=760.33, stdev=144.36, samples=6
  lat (msec)   : 10=0.04%, 20=63.48%, 50=36.48%
  cpu          : usr=0.64%, sys=4.98%, ctx=1116, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.1MiB/s (12.7MB/s), 3107KiB/s-3116KiB/s (3181kB/s-3191kB/s), io=40.0MiB (41.9MB), run=3286-3296msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=14473: Fri May 16 10:04:02 2025
   read: IOPS=597, BW=2391KiB/s (2449kB/s)(5048KiB/2111msec)
    slat (usec): min=461, max=2126, avg=807.23, stdev=209.16
    clat (usec): min=4967, max=32429, avg=12517.85, stdev=3378.24
     lat (usec): min=5670, max=33752, avg=13326.91, stdev=3516.16
    clat percentiles (usec):
     |  1.00th=[ 8160],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11863], 60.00th=[12256],
     | 70.00th=[12780], 80.00th=[13566], 90.00th=[15008], 95.00th=[20055],
     | 99.00th=[26608], 99.50th=[28967], 99.90th=[30016], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1696, max= 2664, per=24.95%, avg=2372.00, stdev=453.23, samples=4
   iops        : min=  424, max=  666, avg=593.00, stdev=113.31, samples=4
  write: IOPS=614, BW=2459KiB/s (2519kB/s)(5192KiB/2111msec)
    slat (usec): min=12, max=327, avg=41.29, stdev=22.16
    clat (usec): min=4913, max=30704, avg=12810.11, stdev=3255.69
     lat (usec): min=4959, max=30741, avg=12852.41, stdev=3261.52
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10028], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12125], 60.00th=[12387],
     | 70.00th=[12911], 80.00th=[13304], 90.00th=[14484], 95.00th=[21103],
     | 99.00th=[27132], 99.50th=[28181], 99.90th=[29230], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1856, max= 2712, per=24.70%, avg=2444.00, stdev=397.57, samples=4
   iops        : min=  464, max=  678, avg=611.00, stdev=99.39, samples=4
  lat (msec)   : 10=8.63%, 20=86.13%, 50=5.23%
  cpu          : usr=1.33%, sys=9.67%, ctx=2183, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14474: Fri May 16 10:04:02 2025
   read: IOPS=590, BW=2363KiB/s (2419kB/s)(4976KiB/2106msec)
    slat (usec): min=461, max=2190, avg=806.15, stdev=218.19
    clat (usec): min=7837, max=31676, avg=12489.34, stdev=3464.57
     lat (usec): min=8611, max=33060, avg=13297.28, stdev=3610.62
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10421],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11731], 60.00th=[12256],
     | 70.00th=[12649], 80.00th=[13304], 90.00th=[15008], 95.00th=[20579],
     | 99.00th=[27395], 99.50th=[28705], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 1680, max= 2600, per=24.51%, avg=2330.00, stdev=435.17, samples=4
   iops        : min=  420, max=  650, avg=582.50, stdev=108.79, samples=4
  write: IOPS=624, BW=2500KiB/s (2560kB/s)(5264KiB/2106msec)
    slat (usec): min=12, max=339, avg=41.72, stdev=22.47
    clat (usec): min=7744, max=31393, avg=12822.48, stdev=3232.03
     lat (usec): min=7783, max=31435, avg=12865.22, stdev=3238.46
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10028], 10.00th=[10552], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[11731], 50.00th=[12125], 60.00th=[12387],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[15008], 95.00th=[20055],
     | 99.00th=[27657], 99.50th=[28967], 99.90th=[29754], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1936, max= 2760, per=25.30%, avg=2504.00, stdev=383.94, samples=4
   iops        : min=  484, max=  690, avg=626.00, stdev=95.99, samples=4
  lat (msec)   : 10=8.67%, 20=86.25%, 50=5.08%
  cpu          : usr=1.38%, sys=9.93%, ctx=2188, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14475: Fri May 16 10:04:02 2025
   read: IOPS=591, BW=2368KiB/s (2425kB/s)(4984KiB/2105msec)
    slat (usec): min=26, max=2619, avg=806.61, stdev=237.36
    clat (usec): min=2202, max=31709, avg=12460.52, stdev=3424.10
     lat (usec): min=2244, max=33956, avg=13268.98, stdev=3580.74
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 9241], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11731], 60.00th=[12256],
     | 70.00th=[12780], 80.00th=[13566], 90.00th=[15270], 95.00th=[19268],
     | 99.00th=[27919], 99.50th=[29230], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 1872, max= 2632, per=24.87%, avg=2364.00, stdev=338.06, samples=4
   iops        : min=  468, max=  658, avg=591.00, stdev=84.51, samples=4
  write: IOPS=624, BW=2497KiB/s (2557kB/s)(5256KiB/2105msec)
    slat (usec): min=12, max=848, avg=42.05, stdev=33.04
    clat (usec): min=6956, max=30059, avg=12933.08, stdev=3399.87
     lat (usec): min=7007, max=30092, avg=12976.10, stdev=3407.32
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10028], 10.00th=[10552], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12125], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[15139], 95.00th=[21627],
     | 99.00th=[27657], 99.50th=[28181], 99.90th=[29492], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1808, max= 2776, per=25.02%, avg=2476.00, stdev=452.95, samples=4
   iops        : min=  452, max=  694, avg=619.00, stdev=113.24, samples=4
  lat (msec)   : 4=0.16%, 10=7.77%, 20=86.88%, 50=5.20%
  cpu          : usr=1.28%, sys=9.55%, ctx=2197, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14476: Fri May 16 10:04:02 2025
   read: IOPS=599, BW=2398KiB/s (2456kB/s)(5060KiB/2110msec)
    slat (usec): min=478, max=2246, avg=798.93, stdev=217.01
    clat (usec): min=6980, max=35428, avg=12472.89, stdev=3372.48
     lat (usec): min=7756, max=36764, avg=13273.68, stdev=3513.60
    clat percentiles (usec):
     |  1.00th=[ 8291],  5.00th=[ 9241], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11731], 60.00th=[12256],
     | 70.00th=[12780], 80.00th=[13435], 90.00th=[15008], 95.00th=[19530],
     | 99.00th=[27132], 99.50th=[28443], 99.90th=[33424], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1768, max= 2640, per=25.02%, avg=2378.00, stdev=408.78, samples=4
   iops        : min=  442, max=  660, avg=594.50, stdev=102.19, samples=4
  write: IOPS=613, BW=2455KiB/s (2514kB/s)(5180KiB/2110msec)
    slat (usec): min=12, max=477, avg=40.71, stdev=27.02
    clat (usec): min=3221, max=31454, avg=12852.00, stdev=3240.36
     lat (usec): min=3256, max=31505, avg=12893.78, stdev=3247.22
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10028], 10.00th=[10421], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12256], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[14877], 95.00th=[20317],
     | 99.00th=[26346], 99.50th=[27919], 99.90th=[30540], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1824, max= 2704, per=24.74%, avg=2448.00, stdev=418.20, samples=4
   iops        : min=  456, max=  676, avg=612.00, stdev=104.55, samples=4
  lat (msec)   : 4=0.04%, 10=8.71%, 20=86.37%, 50=4.88%
  cpu          : usr=1.56%, sys=9.20%, ctx=2235, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9506KiB/s (9735kB/s), 2363KiB/s-2398KiB/s (2419kB/s-2456kB/s), io=19.6MiB (20.5MB), run=2105-2111msec
  WRITE: bw=9897KiB/s (10.1MB/s), 2455KiB/s-2500KiB/s (2514kB/s-2560kB/s), io=20.4MiB (21.4MB), run=2105-2111msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=14482: Fri May 16 10:04:03 2025
   read: IOPS=9, BW=396MiB/s (415MB/s)(40.0MiB/101msec)
    slat (nsec): min=5948.9k, max=5948.9k, avg=5948928.00, stdev= 0.00
    clat (nsec): min=90100k, max=90100k, avg=90100460.00, stdev= 0.00
     lat (nsec): min=96059k, max=96059k, avg=96058534.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[89654],  5.00th=[89654], 10.00th=[89654], 20.00th=[89654],
     | 30.00th=[89654], 40.00th=[89654], 50.00th=[89654], 60.00th=[89654],
     | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654],
     | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.00%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=396MiB/s (415MB/s), 396MiB/s-396MiB/s (415MB/s-415MB/s), io=40.0MiB (41.9MB), run=101-101msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck