== sanity test 398c: run fio to test AIO ================= 10:08:54 (1747404534)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0813104 s, 516 MB/s
osc.lustre-OST0000-osc-ffff8800a797a800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=21425: Fri May 16 10:08:58 2025
  write: IOPS=779, BW=3118KiB/s (3193kB/s)(10.0MiB/3284msec)
    slat (usec): min=11, max=269, avg=38.29, stdev=23.28
    clat (usec): min=1917, max=35626, avg=20413.56, stdev=3722.72
     lat (usec): min=1981, max=35656, avg=20452.76, stdev=3721.77
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15795], 10.00th=[16450], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[19530], 60.00th=[20317],
     | 70.00th=[21627], 80.00th=[23200], 90.00th=[25822], 95.00th=[27657],
     | 99.00th=[31327], 99.50th=[31851], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 2704, max= 3433, per=25.22%, avg=3140.17, stdev=261.41, samples=6
   iops        : min=  676, max=  858, avg=785.00, stdev=65.30, samples=6
  lat (msec)   : 2=0.04%, 10=0.08%, 20=55.90%, 50=43.98%
  cpu          : usr=0.67%, sys=3.59%, ctx=1173, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21426: Fri May 16 10:08:58 2025
  write: IOPS=781, BW=3127KiB/s (3202kB/s)(10.0MiB/3275msec)
    slat (usec): min=11, max=364, avg=42.13, stdev=26.37
    clat (usec): min=2615, max=34096, avg=20384.58, stdev=3809.14
     lat (usec): min=2670, max=34145, avg=20427.77, stdev=3809.87
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15664], 10.00th=[16450], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[19530], 60.00th=[20579],
     | 70.00th=[21627], 80.00th=[23200], 90.00th=[25822], 95.00th=[27919],
     | 99.00th=[30802], 99.50th=[31851], 99.90th=[32375], 99.95th=[33817],
     | 99.99th=[34341]
   bw (  KiB/s): min= 2768, max= 3401, per=25.31%, avg=3152.17, stdev=238.12, samples=6
   iops        : min=  692, max=  850, avg=788.00, stdev=59.48, samples=6
  lat (msec)   : 4=0.20%, 10=0.04%, 20=55.04%, 50=44.73%
  cpu          : usr=0.73%, sys=3.91%, ctx=1465, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21427: Fri May 16 10:08:58 2025
  write: IOPS=778, BW=3113KiB/s (3188kB/s)(10.0MiB/3289msec)
    slat (usec): min=12, max=273, avg=42.51, stdev=25.40
    clat (usec): min=10442, max=33121, avg=20440.95, stdev=3677.95
     lat (usec): min=10489, max=33167, avg=20484.48, stdev=3676.97
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15795], 10.00th=[16450], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[19792], 60.00th=[20579],
     | 70.00th=[21627], 80.00th=[23200], 90.00th=[26084], 95.00th=[27919],
     | 99.00th=[30540], 99.50th=[31589], 99.90th=[31851], 99.95th=[32113],
     | 99.99th=[33162]
   bw (  KiB/s): min= 2696, max= 3409, per=25.12%, avg=3128.17, stdev=260.28, samples=6
   iops        : min=  674, max=  852, avg=782.00, stdev=65.02, samples=6
  lat (msec)   : 20=55.35%, 50=44.65%
  cpu          : usr=0.58%, sys=3.95%, ctx=1591, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21428: Fri May 16 10:08:58 2025
  write: IOPS=778, BW=3113KiB/s (3188kB/s)(10.0MiB/3289msec)
    slat (usec): min=12, max=298, avg=40.86, stdev=26.54
    clat (usec): min=2530, max=33273, avg=20442.85, stdev=3721.78
     lat (usec): min=2583, max=33414, avg=20484.74, stdev=3722.64
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15664], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[19530], 60.00th=[20579],
     | 70.00th=[21627], 80.00th=[23200], 90.00th=[26084], 95.00th=[27919],
     | 99.00th=[30540], 99.50th=[31851], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[33162]
   bw (  KiB/s): min= 2688, max= 3369, per=25.10%, avg=3125.50, stdev=248.20, samples=6
   iops        : min=  672, max=  842, avg=781.33, stdev=62.00, samples=6
  lat (msec)   : 4=0.04%, 10=0.04%, 20=55.08%, 50=44.84%
  cpu          : usr=0.64%, sys=3.74%, ctx=1210, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.2MiB/s (12.8MB/s), 3113KiB/s-3127KiB/s (3188kB/s-3202kB/s), io=40.0MiB (41.9MB), run=3275-3289msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=21440: Fri May 16 10:09:01 2025
   read: IOPS=588, BW=2356KiB/s (2412kB/s)(5048KiB/2143msec)
    slat (usec): min=501, max=1501, avg=816.82, stdev=145.88
    clat (usec): min=6737, max=19181, avg=12716.53, stdev=1768.40
     lat (usec): min=7671, max=19974, avg=13535.11, stdev=1780.14
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[10028], 10.00th=[10552], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[12125], 50.00th=[12649], 60.00th=[13042],
     | 70.00th=[13698], 80.00th=[14222], 90.00th=[14877], 95.00th=[15664],
     | 99.00th=[17171], 99.50th=[17957], 99.90th=[19006], 99.95th=[19268],
     | 99.99th=[19268]
   bw (  KiB/s): min= 2248, max= 2384, per=24.95%, avg=2336.00, stdev=62.31, samples=4
   iops        : min=  562, max=  596, avg=584.00, stdev=15.58, samples=4
  write: IOPS=605, BW=2423KiB/s (2481kB/s)(5192KiB/2143msec)
    slat (usec): min=12, max=355, avg=40.27, stdev=21.63
    clat (usec): min=7045, max=18439, avg=13069.82, stdev=1352.67
     lat (usec): min=7077, max=18476, avg=13111.19, stdev=1351.04
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10814], 10.00th=[11338], 20.00th=[11994],
     | 30.00th=[12387], 40.00th=[12780], 50.00th=[13173], 60.00th=[13435],
     | 70.00th=[13698], 80.00th=[14091], 90.00th=[14746], 95.00th=[15270],
     | 99.00th=[16581], 99.50th=[16909], 99.90th=[17433], 99.95th=[18482],
     | 99.99th=[18482]
   bw (  KiB/s): min= 2384, max= 2504, per=24.89%, avg=2426.00, stdev=53.62, samples=4
   iops        : min=  596, max=  626, avg=606.50, stdev=13.40, samples=4
  lat (msec)   : 10=2.89%, 20=97.11%
  cpu          : usr=0.84%, sys=9.43%, ctx=2282, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21441: Fri May 16 10:09:01 2025
   read: IOPS=583, BW=2335KiB/s (2391kB/s)(4976KiB/2131msec)
    slat (usec): min=13, max=1834, avg=811.23, stdev=152.23
    clat (usec): min=1375, max=25542, avg=12634.88, stdev=1873.73
     lat (usec): min=1462, max=26298, avg=13447.94, stdev=1899.96
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[11076],
     | 30.00th=[11731], 40.00th=[12125], 50.00th=[12649], 60.00th=[13042],
     | 70.00th=[13566], 80.00th=[14091], 90.00th=[14877], 95.00th=[15533],
     | 99.00th=[16909], 99.50th=[17957], 99.90th=[22152], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 2224, max= 2384, per=24.73%, avg=2316.00, stdev=67.09, samples=4
   iops        : min=  556, max=  596, avg=579.00, stdev=16.77, samples=4
  write: IOPS=617, BW=2470KiB/s (2529kB/s)(5264KiB/2131msec)
    slat (usec): min=12, max=1041, avg=42.79, stdev=38.53
    clat (usec): min=3201, max=22919, avg=13079.66, stdev=1522.86
     lat (usec): min=3232, max=22964, avg=13123.54, stdev=1521.78
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10945], 10.00th=[11469], 20.00th=[11994],
     | 30.00th=[12387], 40.00th=[12780], 50.00th=[13173], 60.00th=[13435],
     | 70.00th=[13698], 80.00th=[14091], 90.00th=[14877], 95.00th=[15270],
     | 99.00th=[16712], 99.50th=[17695], 99.90th=[21365], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 2400, max= 2576, per=25.54%, avg=2490.00, stdev=71.96, samples=4
   iops        : min=  600, max=  644, avg=622.50, stdev=17.99, samples=4
  lat (msec)   : 2=0.08%, 4=0.12%, 10=3.40%, 20=96.17%, 50=0.23%
  cpu          : usr=0.99%, sys=9.34%, ctx=2299, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21442: Fri May 16 10:09:01 2025
   read: IOPS=581, BW=2327KiB/s (2383kB/s)(4984KiB/2142msec)
    slat (usec): min=491, max=1894, avg=816.07, stdev=141.22
    clat (usec): min=3116, max=19681, avg=12663.25, stdev=1794.92
     lat (usec): min=3795, max=20771, avg=13481.19, stdev=1807.72
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[12125], 50.00th=[12649], 60.00th=[13042],
     | 70.00th=[13566], 80.00th=[14222], 90.00th=[14877], 95.00th=[15533],
     | 99.00th=[17171], 99.50th=[17433], 99.90th=[19268], 99.95th=[19792],
     | 99.99th=[19792]
   bw (  KiB/s): min= 2240, max= 2448, per=24.93%, avg=2334.00, stdev=88.69, samples=4
   iops        : min=  560, max=  612, avg=583.50, stdev=22.17, samples=4
  write: IOPS=613, BW=2454KiB/s (2513kB/s)(5256KiB/2142msec)
    slat (usec): min=12, max=293, avg=41.39, stdev=21.62
    clat (usec): min=6383, max=19108, avg=13123.68, stdev=1375.58
     lat (usec): min=6454, max=19140, avg=13166.20, stdev=1374.43
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[11076], 10.00th=[11469], 20.00th=[11994],
     | 30.00th=[12387], 40.00th=[12780], 50.00th=[13173], 60.00th=[13435],
     | 70.00th=[13829], 80.00th=[14222], 90.00th=[14746], 95.00th=[15401],
     | 99.00th=[16450], 99.50th=[16909], 99.90th=[19006], 99.95th=[19006],
     | 99.99th=[19006]
   bw (  KiB/s): min= 2336, max= 2512, per=24.97%, avg=2434.00, stdev=87.73, samples=4
   iops        : min=  584, max=  628, avg=608.50, stdev=21.93, samples=4
  lat (msec)   : 4=0.04%, 10=3.32%, 20=96.64%
  cpu          : usr=1.87%, sys=8.50%, ctx=2285, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21443: Fri May 16 10:09:01 2025
   read: IOPS=590, BW=2363KiB/s (2420kB/s)(5060KiB/2141msec)
    slat (usec): min=499, max=1522, avg=807.46, stdev=145.41
    clat (usec): min=6646, max=17749, avg=12683.08, stdev=1697.99
     lat (usec): min=7795, max=18480, avg=13492.24, stdev=1702.27
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[11207],
     | 30.00th=[11863], 40.00th=[12256], 50.00th=[12649], 60.00th=[13173],
     | 70.00th=[13698], 80.00th=[14091], 90.00th=[14746], 95.00th=[15401],
     | 99.00th=[16450], 99.50th=[16581], 99.90th=[17695], 99.95th=[17695],
     | 99.99th=[17695]
   bw (  KiB/s): min= 2232, max= 2416, per=25.22%, avg=2362.00, stdev=86.99, samples=4
   iops        : min=  558, max=  604, avg=590.50, stdev=21.75, samples=4
  write: IOPS=604, BW=2419KiB/s (2477kB/s)(5180KiB/2141msec)
    slat (usec): min=12, max=673, avg=42.04, stdev=29.65
    clat (usec): min=7619, max=21713, avg=13084.36, stdev=1352.59
     lat (usec): min=7673, max=21737, avg=13127.67, stdev=1351.71
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10945], 10.00th=[11469], 20.00th=[11994],
     | 30.00th=[12387], 40.00th=[12780], 50.00th=[13042], 60.00th=[13435],
     | 70.00th=[13698], 80.00th=[14091], 90.00th=[14746], 95.00th=[15270],
     | 99.00th=[16581], 99.50th=[16909], 99.90th=[17695], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 2384, max= 2440, per=24.76%, avg=2414.00, stdev=23.89, samples=4
   iops        : min=  596, max=  610, avg=603.50, stdev= 5.97, samples=4
  lat (msec)   : 10=3.48%, 20=96.48%, 50=0.04%
  cpu          : usr=0.98%, sys=9.49%, ctx=2376, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9364KiB/s (9589kB/s), 2327KiB/s-2363KiB/s (2383kB/s-2420kB/s), io=19.6MiB (20.5MB), run=2131-2143msec
  WRITE: bw=9749KiB/s (9983kB/s), 2419KiB/s-2470KiB/s (2477kB/s-2529kB/s), io=20.4MiB (21.4MB), run=2131-2143msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=21448: Fri May 16 10:09:03 2025
   read: IOPS=12, BW=488MiB/s (512MB/s)(40.0MiB/82msec)
    slat (nsec): min=6831.7k, max=6831.7k, avg=6831730.00, stdev= 0.00
    clat (nsec): min=71882k, max=71882k, avg=71882293.00, stdev= 0.00
     lat (nsec): min=78727k, max=78727k, avg=78726786.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[71828],  5.00th=[71828], 10.00th=[71828], 20.00th=[71828],
     | 30.00th=[71828], 40.00th=[71828], 50.00th=[71828], 60.00th=[71828],
     | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828],
     | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[71828]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.64%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=488MiB/s (512MB/s), 488MiB/s-488MiB/s (512MB/s-512MB/s), io=40.0MiB (41.9MB), run=82-82msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck