== sanity test 398c: run fio to test AIO ================= 10:17:54 (1747405074)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0832307 s, 504 MB/s
osc.lustre-OST0000-osc-ffff88012eb17800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18765: Fri May 16 10:18:01 2025
  write: IOPS=440, BW=1763KiB/s (1806kB/s)(10.0MiB/5807msec)
    slat (usec): min=12, max=524, avg=73.26, stdev=41.20
    clat (usec): min=4817, max=74011, avg=36172.02, stdev=6781.86
     lat (usec): min=4982, max=74045, avg=36246.95, stdev=6794.47
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[20317], 10.00th=[26608], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[36963], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[40109], 90.00th=[41157], 95.00th=[41681],
     | 99.00th=[42730], 99.50th=[44303], 99.90th=[63177], 99.95th=[70779],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1576, max= 2232, per=25.03%, avg=1765.09, stdev=231.53, samples=11
   iops        : min=  394, max=  558, avg=441.27, stdev=57.88, samples=11
  lat (msec)   : 10=1.48%, 20=3.44%, 50=94.80%, 100=0.27%
  cpu          : usr=0.86%, sys=3.77%, ctx=1453, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18766: Fri May 16 10:18:01 2025
  write: IOPS=440, BW=1763KiB/s (1806kB/s)(10.0MiB/5807msec)
    slat (usec): min=11, max=298, avg=70.36, stdev=36.70
    clat (usec): min=6439, max=63335, avg=36172.43, stdev=6795.11
     lat (usec): min=6490, max=63375, avg=36244.51, stdev=6804.77
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[20055], 10.00th=[26870], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[36963], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[40109], 90.00th=[41157], 95.00th=[41681],
     | 99.00th=[43254], 99.50th=[44827], 99.90th=[63177], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min= 1600, max= 2224, per=25.07%, avg=1768.00, stdev=222.62, samples=11
   iops        : min=  400, max=  556, avg=442.00, stdev=55.66, samples=11
  lat (msec)   : 10=1.52%, 20=3.55%, 50=94.45%, 100=0.47%
  cpu          : usr=0.83%, sys=3.79%, ctx=1278, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18767: Fri May 16 10:18:01 2025
  write: IOPS=440, BW=1763KiB/s (1806kB/s)(10.0MiB/5807msec)
    slat (usec): min=13, max=355, avg=75.13, stdev=38.85
    clat (usec): min=6536, max=74019, avg=36168.97, stdev=6750.35
     lat (usec): min=6579, max=74049, avg=36245.83, stdev=6761.27
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[20579], 10.00th=[26870], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[36963], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[40109], 90.00th=[41157], 95.00th=[41681],
     | 99.00th=[42730], 99.50th=[44303], 99.90th=[66847], 99.95th=[70779],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1576, max= 2216, per=25.03%, avg=1765.09, stdev=224.09, samples=11
   iops        : min=  394, max=  554, avg=441.27, stdev=56.02, samples=11
  lat (msec)   : 10=1.45%, 20=3.44%, 50=94.80%, 100=0.31%
  cpu          : usr=0.81%, sys=3.96%, ctx=1622, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18768: Fri May 16 10:18:01 2025
  write: IOPS=443, BW=1773KiB/s (1815kB/s)(10.0MiB/5777msec)
    slat (usec): min=12, max=277, avg=70.35, stdev=38.21
    clat (usec): min=2350, max=67665, avg=35996.00, stdev=7142.54
     lat (usec): min=2367, max=67740, avg=36068.03, stdev=7152.36
    clat percentiles (usec):
     |  1.00th=[ 7504],  5.00th=[18744], 10.00th=[26346], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[36963], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[40109], 90.00th=[41157], 95.00th=[41681],
     | 99.00th=[42730], 99.50th=[43779], 99.90th=[63701], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1576, max= 2328, per=25.16%, avg=1774.55, stdev=256.93, samples=11
   iops        : min=  394, max=  582, avg=443.64, stdev=64.23, samples=11
  lat (msec)   : 4=0.27%, 10=1.84%, 20=3.40%, 50=94.18%, 100=0.31%
  cpu          : usr=0.52%, sys=3.86%, ctx=1315, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7054KiB/s (7223kB/s), 1763KiB/s-1773KiB/s (1806kB/s-1815kB/s), io=40.0MiB (41.9MB), run=5777-5807msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18783: Fri May 16 10:18:04 2025
   read: IOPS=647, BW=2589KiB/s (2651kB/s)(5048KiB/1950msec)
    slat (usec): min=420, max=1605, avg=715.97, stdev=177.79
    clat (usec): min=4781, max=25029, avg=11615.51, stdev=2911.16
     lat (usec): min=5688, max=26172, avg=12333.28, stdev=3013.56
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[ 8291], 10.00th=[ 8717], 20.00th=[ 9241],
     | 30.00th=[ 9765], 40.00th=[10290], 50.00th=[10814], 60.00th=[11338],
     | 70.00th=[12125], 80.00th=[14222], 90.00th=[16319], 95.00th=[17695],
     | 99.00th=[19268], 99.50th=[20055], 99.90th=[22152], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 2152, max= 2562, per=23.46%, avg=2411.33, stdev=225.57, samples=3
   iops        : min=  538, max=  640, avg=602.67, stdev=56.23, samples=3
  write: IOPS=665, BW=2663KiB/s (2726kB/s)(5192KiB/1950msec)
    slat (usec): min=12, max=200, avg=35.87, stdev=18.00
    clat (usec): min=7480, max=22262, avg=11876.89, stdev=2801.58
     lat (usec): min=7505, max=22341, avg=11913.55, stdev=2803.40
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 8848], 10.00th=[ 9241], 20.00th=[ 9765],
     | 30.00th=[10159], 40.00th=[10552], 50.00th=[10814], 60.00th=[11338],
     | 70.00th=[12387], 80.00th=[14615], 90.00th=[16450], 95.00th=[17433],
     | 99.00th=[19792], 99.50th=[20317], 99.90th=[22152], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 2248, max= 2786, per=23.80%, avg=2547.33, stdev=274.08, samples=3
   iops        : min=  562, max=  696, avg=636.67, stdev=68.30, samples=3
  lat (msec)   : 10=29.92%, 20=69.41%, 50=0.66%
  cpu          : usr=1.49%, sys=8.36%, ctx=2479, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18784: Fri May 16 10:18:04 2025
   read: IOPS=637, BW=2549KiB/s (2610kB/s)(4976KiB/1952msec)
    slat (usec): min=354, max=1666, avg=722.29, stdev=189.96
    clat (usec): min=2800, max=25993, avg=11600.47, stdev=3096.39
     lat (usec): min=3701, max=27219, avg=12324.45, stdev=3212.02
    clat percentiles (usec):
     |  1.00th=[ 7242],  5.00th=[ 8029], 10.00th=[ 8586], 20.00th=[ 9241],
     | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10683], 60.00th=[11207],
     | 70.00th=[12256], 80.00th=[14222], 90.00th=[16450], 95.00th=[17957],
     | 99.00th=[20579], 99.50th=[21103], 99.90th=[24773], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 2136, max= 2506, per=23.07%, avg=2371.33, stdev=204.51, samples=3
   iops        : min=  534, max=  626, avg=592.67, stdev=50.96, samples=3
  write: IOPS=674, BW=2697KiB/s (2761kB/s)(5264KiB/1952msec)
    slat (usec): min=12, max=624, avg=36.14, stdev=26.18
    clat (usec): min=3759, max=22059, avg=11950.09, stdev=2776.55
     lat (usec): min=3855, max=22088, avg=11986.98, stdev=2778.04
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 8848], 10.00th=[ 9241], 20.00th=[ 9765],
     | 30.00th=[10159], 40.00th=[10552], 50.00th=[10945], 60.00th=[11469],
     | 70.00th=[12518], 80.00th=[14615], 90.00th=[16450], 95.00th=[17433],
     | 99.00th=[19530], 99.50th=[20055], 99.90th=[21103], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 2264, max= 2826, per=24.13%, avg=2582.00, stdev=288.22, samples=3
   iops        : min=  566, max=  706, avg=645.33, stdev=71.84, samples=3
  lat (msec)   : 4=0.08%, 10=30.04%, 20=68.95%, 50=0.94%
  cpu          : usr=0.87%, sys=8.82%, ctx=2523, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18785: Fri May 16 10:18:04 2025
   read: IOPS=639, BW=2560KiB/s (2621kB/s)(4984KiB/1947msec)
    slat (usec): min=13, max=1773, avg=720.60, stdev=194.96
    clat (usec): min=1068, max=25010, avg=11599.07, stdev=3031.95
     lat (usec): min=1098, max=26239, avg=12321.20, stdev=3150.55
    clat percentiles (usec):
     |  1.00th=[ 7111],  5.00th=[ 8160], 10.00th=[ 8717], 20.00th=[ 9241],
     | 30.00th=[ 9765], 40.00th=[10290], 50.00th=[10814], 60.00th=[11338],
     | 70.00th=[12125], 80.00th=[13829], 90.00th=[16450], 95.00th=[17695],
     | 99.00th=[20055], 99.50th=[20841], 99.90th=[23987], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 2120, max= 2770, per=23.79%, avg=2446.00, stdev=325.00, samples=3
   iops        : min=  530, max=  692, avg=611.33, stdev=81.00, samples=3
  write: IOPS=674, BW=2700KiB/s (2764kB/s)(5256KiB/1947msec)
    slat (usec): min=13, max=355, avg=36.40, stdev=22.41
    clat (usec): min=3338, max=24836, avg=11926.87, stdev=2889.13
     lat (usec): min=3386, max=24926, avg=11964.04, stdev=2890.92
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[ 8717], 10.00th=[ 9241], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10552], 50.00th=[10945], 60.00th=[11338],
     | 70.00th=[12387], 80.00th=[14615], 90.00th=[16581], 95.00th=[17695],
     | 99.00th=[19530], 99.50th=[20317], 99.90th=[24511], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 2248, max= 2696, per=23.70%, avg=2536.67, stdev=250.44, samples=3
   iops        : min=  562, max=  674, avg=634.00, stdev=62.48, samples=3
  lat (msec)   : 2=0.08%, 4=0.20%, 10=28.20%, 20=70.62%, 50=0.90%
  cpu          : usr=1.28%, sys=8.53%, ctx=2451, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18786: Fri May 16 10:18:04 2025
   read: IOPS=648, BW=2592KiB/s (2654kB/s)(5060KiB/1952msec)
    slat (usec): min=407, max=1659, avg=725.60, stdev=185.09
    clat (usec): min=5996, max=22654, avg=11650.09, stdev=2910.18
     lat (usec): min=6886, max=24110, avg=12377.33, stdev=3021.65
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[ 8356], 10.00th=[ 8717], 20.00th=[ 9241],
     | 30.00th=[ 9765], 40.00th=[10421], 50.00th=[10814], 60.00th=[11338],
     | 70.00th=[12125], 80.00th=[14091], 90.00th=[16319], 95.00th=[17433],
     | 99.00th=[20055], 99.50th=[21103], 99.90th=[22152], 99.95th=[22676],
     | 99.99th=[22676]
   bw (  KiB/s): min= 2216, max= 2634, per=23.77%, avg=2443.33, stdev=211.40, samples=3
   iops        : min=  554, max=  658, avg=610.67, stdev=52.62, samples=3
  write: IOPS=663, BW=2654KiB/s (2717kB/s)(5180KiB/1952msec)
    slat (usec): min=12, max=733, avg=36.90, stdev=31.36
    clat (usec): min=6381, max=23132, avg=11872.18, stdev=2855.16
     lat (usec): min=6404, max=23242, avg=11909.89, stdev=2858.12
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 8848], 10.00th=[ 9241], 20.00th=[ 9634],
     | 30.00th=[10159], 40.00th=[10552], 50.00th=[10814], 60.00th=[11338],
     | 70.00th=[12125], 80.00th=[14484], 90.00th=[16712], 95.00th=[17695],
     | 99.00th=[19792], 99.50th=[20055], 99.90th=[21365], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 2192, max= 2746, per=23.50%, avg=2515.33, stdev=288.39, samples=3
   iops        : min=  548, max=  686, avg=628.67, stdev=71.90, samples=3
  lat (msec)   : 10=30.04%, 20=69.10%, 50=0.86%
  cpu          : usr=1.03%, sys=8.82%, ctx=2556, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=10.0MiB/s (10.5MB/s), 2549KiB/s-2592KiB/s (2610kB/s-2654kB/s), io=19.6MiB (20.5MB), run=1947-1952msec
  WRITE: bw=10.5MiB/s (10.0MB/s), 2654KiB/s-2700KiB/s (2717kB/s-2764kB/s), io=20.4MiB (21.4MB), run=1947-1952msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18791: Fri May 16 10:18:05 2025
   read: IOPS=12, BW=506MiB/s (531MB/s)(40.0MiB/79msec)
    slat (nsec): min=5958.0k, max=5958.0k, avg=5958001.00, stdev= 0.00
    clat (nsec): min=68171k, max=68171k, avg=68171224.00, stdev= 0.00
     lat (nsec): min=74138k, max=74138k, avg=74138005.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[68682],  5.00th=[68682], 10.00th=[68682], 20.00th=[68682],
     | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[68682],
     | 70.00th=[68682], 80.00th=[68682], 90.00th=[68682], 95.00th=[68682],
     | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.97%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=506MiB/s (531MB/s), 506MiB/s-506MiB/s (531MB/s-531MB/s), io=40.0MiB (41.9MB), run=79-79msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck