== sanity test 398c: run fio to test AIO ================= 12:05:35 (1747411535)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0843443 s, 497 MB/s
osc.lustre-OST0000-osc-ffff8800a9069000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18127: Fri May 16 12:05:39 2025
  write: IOPS=1008, BW=4033KiB/s (4130kB/s)(10.0MiB/2539msec)
    slat (usec): min=10, max=328, avg=40.69, stdev=23.75
    clat (usec): min=3558, max=26554, avg=15789.94, stdev=3057.00
     lat (usec): min=3602, max=26593, avg=15831.76, stdev=3059.58
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[11469], 10.00th=[12387], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14877], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19530], 95.00th=[21890],
     | 99.00th=[25560], 99.50th=[25560], 99.90th=[25822], 99.95th=[26346],
     | 99.99th=[26608]
   bw (  KiB/s): min= 3312, max= 4616, per=24.98%, avg=4030.20, stdev=485.21, samples=5
   iops        : min=  828, max= 1154, avg=1007.40, stdev=121.24, samples=5
  lat (msec)   : 4=0.23%, 10=1.05%, 20=90.59%, 50=8.12%
  cpu          : usr=0.79%, sys=4.89%, ctx=1389, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18128: Fri May 16 12:05:39 2025
  write: IOPS=1009, BW=4038KiB/s (4135kB/s)(10.0MiB/2536msec)
    slat (usec): min=11, max=350, avg=39.89, stdev=23.81
    clat (usec): min=5165, max=26637, avg=15774.89, stdev=3058.39
     lat (usec): min=5182, max=26686, avg=15815.91, stdev=3061.47
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[11600], 10.00th=[12256], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14746], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19530], 95.00th=[21627],
     | 99.00th=[25560], 99.50th=[25822], 99.90th=[26608], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min= 3296, max= 4608, per=24.97%, avg=4028.60, stdev=490.17, samples=5
   iops        : min=  824, max= 1152, avg=1007.00, stdev=122.48, samples=5
  lat (msec)   : 10=1.76%, 20=90.08%, 50=8.16%
  cpu          : usr=0.91%, sys=4.69%, ctx=1339, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18129: Fri May 16 12:05:39 2025
  write: IOPS=1012, BW=4051KiB/s (4148kB/s)(10.0MiB/2528msec)
    slat (usec): min=10, max=287, avg=40.45, stdev=22.14
    clat (usec): min=1978, max=37930, avg=15735.29, stdev=3261.60
     lat (usec): min=1990, max=37966, avg=15776.90, stdev=3263.06
    clat percentiles (usec):
     |  1.00th=[ 7767],  5.00th=[11338], 10.00th=[12125], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14746], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19530], 95.00th=[21627],
     | 99.00th=[25297], 99.50th=[26608], 99.90th=[35914], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 3312, max= 4720, per=25.13%, avg=4054.20, stdev=518.87, samples=5
   iops        : min=  828, max= 1180, avg=1013.40, stdev=129.67, samples=5
  lat (msec)   : 2=0.04%, 4=0.47%, 10=1.64%, 20=89.30%, 50=8.55%
  cpu          : usr=1.07%, sys=4.79%, ctx=1459, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18130: Fri May 16 12:05:39 2025
  write: IOPS=1008, BW=4033KiB/s (4130kB/s)(10.0MiB/2539msec)
    slat (usec): min=10, max=276, avg=39.01, stdev=21.22
    clat (usec): min=3207, max=28332, avg=15791.97, stdev=3083.67
     lat (usec): min=3250, max=28364, avg=15832.02, stdev=3085.59
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[11600], 10.00th=[12387], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14746], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19268], 95.00th=[21890],
     | 99.00th=[25297], 99.50th=[25822], 99.90th=[28181], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 3328, max= 4608, per=24.99%, avg=4031.80, stdev=482.94, samples=5
   iops        : min=  832, max= 1152, avg=1007.80, stdev=120.65, samples=5
  lat (msec)   : 4=0.16%, 10=1.09%, 20=90.43%, 50=8.32%
  cpu          : usr=0.51%, sys=5.04%, ctx=1265, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=15.8MiB/s (16.5MB/s), 4033KiB/s-4051KiB/s (4130kB/s-4148kB/s), io=40.0MiB (41.9MB), run=2528-2539msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18142: Fri May 16 12:05:42 2025
   read: IOPS=568, BW=2274KiB/s (2328kB/s)(5048KiB/2220msec)
    slat (usec): min=14, max=1921, avg=841.01, stdev=202.94
    clat (usec): min=1241, max=31282, avg=13286.28, stdev=2405.60
     lat (usec): min=1262, max=32094, avg=14129.03, stdev=2466.03
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[10159], 10.00th=[10814], 20.00th=[11469],
     | 30.00th=[12125], 40.00th=[12649], 50.00th=[13173], 60.00th=[13698],
     | 70.00th=[14353], 80.00th=[15008], 90.00th=[15926], 95.00th=[16909],
     | 99.00th=[18744], 99.50th=[19530], 99.90th=[30540], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 2200, max= 2328, per=24.90%, avg=2242.75, stdev=59.67, samples=4
   iops        : min=  550, max=  582, avg=560.50, stdev=15.09, samples=4
  write: IOPS=584, BW=2339KiB/s (2395kB/s)(5192KiB/2220msec)
    slat (usec): min=11, max=442, avg=40.98, stdev=23.64
    clat (usec): min=2244, max=30479, avg=13524.62, stdev=2104.24
     lat (usec): min=2282, max=30529, avg=13566.70, stdev=2104.48
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10814], 10.00th=[11338], 20.00th=[11994],
     | 30.00th=[12518], 40.00th=[13042], 50.00th=[13435], 60.00th=[13829],
     | 70.00th=[14353], 80.00th=[14877], 90.00th=[15926], 95.00th=[16909],
     | 99.00th=[18482], 99.50th=[19268], 99.90th=[28967], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 2232, max= 2427, per=25.09%, avg=2352.75, stdev=85.02, samples=4
   iops        : min=  558, max=  606, avg=588.00, stdev=21.04, samples=4
  lat (msec)   : 2=0.12%, 4=0.27%, 10=2.77%, 20=96.45%, 50=0.39%
  cpu          : usr=1.13%, sys=8.79%, ctx=2337, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18143: Fri May 16 12:05:42 2025
   read: IOPS=558, BW=2235KiB/s (2289kB/s)(4976KiB/2226msec)
    slat (usec): min=18, max=2109, avg=842.86, stdev=193.35
    clat (usec): min=3360, max=22286, avg=13260.12, stdev=2245.20
     lat (usec): min=3379, max=23353, avg=14104.45, stdev=2302.15
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10159], 10.00th=[10683], 20.00th=[11338],
     | 30.00th=[11994], 40.00th=[12518], 50.00th=[13173], 60.00th=[13698],
     | 70.00th=[14353], 80.00th=[15139], 90.00th=[16188], 95.00th=[17433],
     | 99.00th=[18744], 99.50th=[19268], 99.90th=[21365], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 2144, max= 2328, per=24.43%, avg=2200.75, stdev=86.14, samples=4
   iops        : min=  536, max=  582, avg=550.00, stdev=21.60, samples=4
  write: IOPS=591, BW=2365KiB/s (2422kB/s)(5264KiB/2226msec)
    slat (usec): min=12, max=1302, avg=43.43, stdev=43.51
    clat (usec): min=5234, max=22242, avg=13616.25, stdev=1840.11
     lat (usec): min=5269, max=22275, avg=13660.80, stdev=1841.03
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10945], 10.00th=[11600], 20.00th=[12125],
     | 30.00th=[12649], 40.00th=[13042], 50.00th=[13435], 60.00th=[13960],
     | 70.00th=[14484], 80.00th=[15139], 90.00th=[16057], 95.00th=[16909],
     | 99.00th=[17957], 99.50th=[18220], 99.90th=[20055], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 2304, max= 2419, per=25.18%, avg=2360.75, stdev=47.41, samples=4
   iops        : min=  576, max=  604, avg=590.00, stdev=11.55, samples=4
  lat (msec)   : 4=0.12%, 10=3.01%, 20=96.68%, 50=0.20%
  cpu          : usr=0.63%, sys=9.35%, ctx=2440, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18144: Fri May 16 12:05:42 2025
   read: IOPS=559, BW=2239KiB/s (2293kB/s)(4984KiB/2226msec)
    slat (usec): min=529, max=1868, avg=848.09, stdev=185.24
    clat (usec): min=3878, max=31013, avg=13215.31, stdev=2247.84
     lat (usec): min=4864, max=32225, avg=14064.94, stdev=2295.48
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9896], 10.00th=[10552], 20.00th=[11338],
     | 30.00th=[11863], 40.00th=[12518], 50.00th=[13042], 60.00th=[13566],
     | 70.00th=[14353], 80.00th=[15008], 90.00th=[16057], 95.00th=[16909],
     | 99.00th=[18482], 99.50th=[19792], 99.90th=[26084], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 2152, max= 2331, per=24.81%, avg=2234.75, stdev=84.87, samples=4
   iops        : min=  538, max=  582, avg=558.50, stdev=20.94, samples=4
  write: IOPS=590, BW=2361KiB/s (2418kB/s)(5256KiB/2226msec)
    slat (usec): min=11, max=703, avg=43.06, stdev=33.00
    clat (usec): min=2951, max=30893, avg=13600.24, stdev=1959.78
     lat (usec): min=2984, max=30924, avg=13644.73, stdev=1959.23
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[10814], 10.00th=[11469], 20.00th=[12256],
     | 30.00th=[12649], 40.00th=[13173], 50.00th=[13435], 60.00th=[13829],
     | 70.00th=[14353], 80.00th=[15008], 90.00th=[15795], 95.00th=[16450],
     | 99.00th=[18744], 99.50th=[20055], 99.90th=[28967], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 2216, max= 2568, per=25.01%, avg=2344.75, stdev=153.72, samples=4
   iops        : min=  554, max=  642, avg=586.00, stdev=38.51, samples=4
  lat (msec)   : 4=0.08%, 10=3.79%, 20=95.59%, 50=0.55%
  cpu          : usr=1.17%, sys=8.90%, ctx=2403, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18145: Fri May 16 12:05:42 2025
   read: IOPS=567, BW=2271KiB/s (2326kB/s)(5060KiB/2228msec)
    slat (usec): min=487, max=1685, avg=845.42, stdev=182.25
    clat (usec): min=4737, max=21358, avg=13274.90, stdev=2186.55
     lat (usec): min=5360, max=22399, avg=14121.98, stdev=2231.49
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10028], 10.00th=[10552], 20.00th=[11469],
     | 30.00th=[11994], 40.00th=[12649], 50.00th=[13173], 60.00th=[13698],
     | 70.00th=[14484], 80.00th=[15139], 90.00th=[16057], 95.00th=[16909],
     | 99.00th=[18744], 99.50th=[19792], 99.90th=[20841], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2144, max= 2352, per=25.08%, avg=2258.75, stdev=99.43, samples=4
   iops        : min=  536, max=  588, avg=564.50, stdev=24.68, samples=4
  write: IOPS=581, BW=2325KiB/s (2381kB/s)(5180KiB/2228msec)
    slat (usec): min=10, max=757, avg=41.98, stdev=35.70
    clat (usec): min=2286, max=22145, avg=13574.51, stdev=1860.33
     lat (usec): min=2340, max=22181, avg=13617.60, stdev=1861.59
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[10814], 10.00th=[11338], 20.00th=[11994],
     | 30.00th=[12518], 40.00th=[13042], 50.00th=[13566], 60.00th=[13960],
     | 70.00th=[14484], 80.00th=[15139], 90.00th=[15795], 95.00th=[16712],
     | 99.00th=[18744], 99.50th=[19006], 99.90th=[20055], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 2200, max= 2323, per=24.43%, avg=2290.75, stdev=60.52, samples=4
   iops        : min=  550, max=  580, avg=572.50, stdev=15.00, samples=4
  lat (msec)   : 4=0.04%, 10=3.28%, 20=96.33%, 50=0.35%
  cpu          : usr=1.26%, sys=8.62%, ctx=2436, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9007KiB/s (9223kB/s), 2235KiB/s-2274KiB/s (2289kB/s-2328kB/s), io=19.6MiB (20.5MB), run=2220-2228msec
  WRITE: bw=9377KiB/s (9602kB/s), 2325KiB/s-2365KiB/s (2381kB/s-2422kB/s), io=20.4MiB (21.4MB), run=2220-2228msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18150: Fri May 16 12:05:44 2025
   read: IOPS=10, BW=421MiB/s (442MB/s)(40.0MiB/95msec)
    slat (nsec): min=10176k, max=10176k, avg=10175769.00, stdev= 0.00
    clat (nsec): min=81197k, max=81197k, avg=81196854.00, stdev= 0.00
     lat (nsec): min=91394k, max=91394k, avg=91394450.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[81265],  5.00th=[81265], 10.00th=[81265], 20.00th=[81265],
     | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265],
     | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265],
     | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.64%, ctx=12, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=421MiB/s (442MB/s), 421MiB/s-421MiB/s (442MB/s-442MB/s), io=40.0MiB (41.9MB), run=95-95msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck