== sanity test 398c: run fio to test AIO ================= 20:02:39 (1736989359)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0886549 s, 473 MB/s
osc.lustre-OST0000-osc-ffff88007e9a5000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=19798: Wed Jan 15 20:02:43 2025
  write: IOPS=947, BW=3791KiB/s (3882kB/s)(10.0MiB/2701msec)
    slat (usec): min=11, max=256, avg=41.13, stdev=20.45
    clat (usec): min=8618, max=31291, avg=16800.37, stdev=1969.64
     lat (usec): min=8669, max=31310, avg=16842.63, stdev=1969.69
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[14222], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[21890], 99.50th=[31065], 99.90th=[31327], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 3656, max= 3912, per=25.00%, avg=3790.40, stdev=101.57, samples=5
   iops        : min=  914, max=  978, avg=947.60, stdev=25.39, samples=5
  lat (msec)   : 10=0.16%, 20=97.97%, 50=1.88%
  cpu          : usr=1.04%, sys=4.48%, ctx=1179, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19799: Wed Jan 15 20:02:43 2025
  write: IOPS=947, BW=3791KiB/s (3882kB/s)(10.0MiB/2701msec)
    slat (usec): min=11, max=21105, avg=50.03, stdev=416.96
    clat (usec): min=1634, max=30403, avg=16731.26, stdev=1808.33
     lat (usec): min=1777, max=33697, avg=16782.38, stdev=1836.61
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[14222], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[20841], 99.50th=[22676], 99.90th=[30278], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 3672, max= 3888, per=24.96%, avg=3785.60, stdev=98.53, samples=5
   iops        : min=  918, max=  972, avg=946.40, stdev=24.63, samples=5
  lat (msec)   : 2=0.08%, 4=0.04%, 20=98.40%, 50=1.48%
  cpu          : usr=0.81%, sys=5.37%, ctx=1275, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19800: Wed Jan 15 20:02:43 2025
  write: IOPS=953, BW=3815KiB/s (3907kB/s)(10.0MiB/2684msec)
    slat (usec): min=11, max=306, avg=43.01, stdev=22.02
    clat (usec): min=3287, max=32672, avg=16703.57, stdev=2089.93
     lat (usec): min=3302, max=32707, avg=16747.82, stdev=2090.31
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[13829], 10.00th=[14746], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[23462], 99.50th=[26608], 99.90th=[32637], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 3664, max= 3928, per=25.18%, avg=3817.60, stdev=95.57, samples=5
   iops        : min=  916, max=  982, avg=954.40, stdev=23.89, samples=5
  lat (msec)   : 4=0.31%, 10=0.39%, 20=97.46%, 50=1.84%
  cpu          : usr=0.86%, sys=4.88%, ctx=1423, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19801: Wed Jan 15 20:02:43 2025
  write: IOPS=948, BW=3793KiB/s (3884kB/s)(10.0MiB/2700msec)
    slat (usec): min=12, max=20698, avg=52.42, stdev=408.81
    clat (usec): min=1839, max=33407, avg=16666.41, stdev=1956.82
     lat (usec): min=1893, max=33455, avg=16719.94, stdev=1950.34
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[13829], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17433], 80.00th=[17695], 90.00th=[18482], 95.00th=[19006],
     | 99.00th=[21890], 99.50th=[24511], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[33424]
   bw (  KiB/s): min= 3648, max= 3952, per=25.04%, avg=3796.80, stdev=111.23, samples=5
   iops        : min=  912, max=  988, avg=949.20, stdev=27.81, samples=5
  lat (msec)   : 2=0.04%, 4=0.16%, 10=0.27%, 20=97.77%, 50=1.76%
  cpu          : usr=1.00%, sys=5.56%, ctx=1525, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=14.8MiB/s (15.5MB/s), 3791KiB/s-3815KiB/s (3882kB/s-3907kB/s), io=40.0MiB (41.9MB), run=2684-2701msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=19813: Wed Jan 15 20:02:46 2025
   read: IOPS=580, BW=2321KiB/s (2377kB/s)(5048KiB/2175msec)
    slat (usec): min=479, max=1898, avg=818.29, stdev=141.75
    clat (usec): min=6535, max=19352, avg=12847.13, stdev=1767.79
     lat (usec): min=7177, max=20182, avg=13667.06, stdev=1784.12
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10028], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11863], 40.00th=[12387], 50.00th=[12911], 60.00th=[13304],
     | 70.00th=[13829], 80.00th=[14353], 90.00th=[15139], 95.00th=[15795],
     | 99.00th=[16909], 99.50th=[17433], 99.90th=[18482], 99.95th=[19268],
     | 99.99th=[19268]
   bw (  KiB/s): min= 2184, max= 2376, per=24.86%, avg=2294.00, stdev=93.84, samples=4
   iops        : min=  546, max=  594, avg=573.50, stdev=23.46, samples=4
  write: IOPS=596, BW=2387KiB/s (2444kB/s)(5192KiB/2175msec)
    slat (usec): min=13, max=288, avg=42.44, stdev=21.23
    clat (usec): min=7244, max=18424, avg=13341.35, stdev=1369.02
     lat (usec): min=7272, max=18474, avg=13384.94, stdev=1367.69
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[11207], 10.00th=[11600], 20.00th=[12256],
     | 30.00th=[12649], 40.00th=[13042], 50.00th=[13304], 60.00th=[13698],
     | 70.00th=[13960], 80.00th=[14484], 90.00th=[15139], 95.00th=[15664],
     | 99.00th=[16581], 99.50th=[16909], 99.90th=[17695], 99.95th=[18482],
     | 99.99th=[18482]
   bw (  KiB/s): min= 2336, max= 2464, per=24.82%, avg=2384.00, stdev=55.43, samples=4
   iops        : min=  584, max=  616, avg=596.00, stdev=13.86, samples=4
  lat (msec)   : 10=2.85%, 20=97.15%
  cpu          : usr=1.56%, sys=8.79%, ctx=2354, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19814: Wed Jan 15 20:02:46 2025
   read: IOPS=574, BW=2296KiB/s (2351kB/s)(4976KiB/2167msec)
    slat (usec): min=523, max=1393, avg=814.17, stdev=136.31
    clat (usec): min=1629, max=20410, avg=12796.52, stdev=1820.19
     lat (usec): min=2280, max=21251, avg=13612.39, stdev=1831.89
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[10159], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[12256], 50.00th=[12649], 60.00th=[13173],
     | 70.00th=[13698], 80.00th=[14353], 90.00th=[15270], 95.00th=[15795],
     | 99.00th=[17433], 99.50th=[18482], 99.90th=[19792], 99.95th=[20317],
     | 99.99th=[20317]
   bw (  KiB/s): min= 2224, max= 2360, per=24.60%, avg=2270.00, stdev=61.41, samples=4
   iops        : min=  556, max=  590, avg=567.50, stdev=15.35, samples=4
  write: IOPS=607, BW=2429KiB/s (2487kB/s)(5264KiB/2167msec)
    slat (usec): min=13, max=221, avg=42.20, stdev=20.62
    clat (usec): min=4838, max=23593, avg=13323.95, stdev=1467.90
     lat (usec): min=4862, max=23616, avg=13367.22, stdev=1466.85
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[11207], 10.00th=[11600], 20.00th=[12125],
     | 30.00th=[12518], 40.00th=[12911], 50.00th=[13304], 60.00th=[13566],
     | 70.00th=[13960], 80.00th=[14484], 90.00th=[15139], 95.00th=[15795],
     | 99.00th=[16909], 99.50th=[17695], 99.90th=[20317], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 2392, max= 2496, per=25.40%, avg=2440.00, stdev=48.44, samples=4
   iops        : min=  598, max=  624, avg=610.00, stdev=12.11, samples=4
  lat (msec)   : 2=0.04%, 10=2.23%, 20=97.62%, 50=0.12%
  cpu          : usr=1.48%, sys=8.91%, ctx=2369, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19815: Wed Jan 15 20:02:46 2025
   read: IOPS=577, BW=2311KiB/s (2366kB/s)(4984KiB/2157msec)
    slat (usec): min=12, max=1531, avg=817.75, stdev=150.55
    clat (usec): min=824, max=23411, avg=12737.69, stdev=2006.65
     lat (usec): min=845, max=24133, avg=13557.13, stdev=2037.73
    clat percentiles (usec):
     |  1.00th=[ 5997],  5.00th=[10028], 10.00th=[10552], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[12256], 50.00th=[12780], 60.00th=[13304],
     | 70.00th=[13698], 80.00th=[14222], 90.00th=[15139], 95.00th=[15795],
     | 99.00th=[16909], 99.50th=[17957], 99.90th=[19006], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 2208, max= 2384, per=25.04%, avg=2310.00, stdev=88.45, samples=4
   iops        : min=  552, max=  596, avg=577.50, stdev=22.11, samples=4
  write: IOPS=609, BW=2437KiB/s (2495kB/s)(5256KiB/2157msec)
    slat (usec): min=12, max=412, avg=44.21, stdev=26.92
    clat (usec): min=2196, max=21576, avg=13295.21, stdev=1582.11
     lat (usec): min=2245, max=21615, avg=13340.52, stdev=1580.69
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[11207], 10.00th=[11600], 20.00th=[12125],
     | 30.00th=[12649], 40.00th=[13042], 50.00th=[13304], 60.00th=[13566],
     | 70.00th=[13960], 80.00th=[14353], 90.00th=[15139], 95.00th=[15664],
     | 99.00th=[16909], 99.50th=[17695], 99.90th=[20579], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 2288, max= 2504, per=25.17%, avg=2418.00, stdev=105.60, samples=4
   iops        : min=  572, max=  626, avg=604.50, stdev=26.40, samples=4
  lat (usec)   : 1000=0.08%
  lat (msec)   : 2=0.08%, 4=0.23%, 10=2.66%, 20=96.80%, 50=0.16%
  cpu          : usr=1.35%, sys=9.18%, ctx=2347, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19816: Wed Jan 15 20:02:46 2025
   read: IOPS=582, BW=2329KiB/s (2384kB/s)(5060KiB/2173msec)
    slat (usec): min=487, max=1897, avg=822.46, stdev=147.74
    clat (usec): min=7167, max=19643, avg=12844.94, stdev=1711.30
     lat (usec): min=7852, max=20333, avg=13669.04, stdev=1726.24
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10290], 10.00th=[10683], 20.00th=[11338],
     | 30.00th=[11863], 40.00th=[12387], 50.00th=[12780], 60.00th=[13304],
     | 70.00th=[13698], 80.00th=[14222], 90.00th=[15139], 95.00th=[15664],
     | 99.00th=[17171], 99.50th=[17433], 99.90th=[18744], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min= 2203, max= 2376, per=25.09%, avg=2314.75, stdev=76.48, samples=4
   iops        : min=  550, max=  594, avg=578.50, stdev=19.49, samples=4
  write: IOPS=595, BW=2384KiB/s (2441kB/s)(5180KiB/2173msec)
    slat (usec): min=11, max=356, avg=44.04, stdev=26.53
    clat (usec): min=7395, max=20440, avg=13324.56, stdev=1367.40
     lat (usec): min=7495, max=20492, avg=13369.74, stdev=1367.14
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11338], 10.00th=[11731], 20.00th=[12256],
     | 30.00th=[12649], 40.00th=[12911], 50.00th=[13304], 60.00th=[13566],
     | 70.00th=[13960], 80.00th=[14353], 90.00th=[15008], 95.00th=[15664],
     | 99.00th=[16909], 99.50th=[17171], 99.90th=[18744], 99.95th=[20317],
     | 99.99th=[20317]
   bw (  KiB/s): min= 2328, max= 2416, per=24.66%, avg=2368.75, stdev=37.77, samples=4
   iops        : min=  582, max=  604, avg=592.00, stdev= 9.38, samples=4
  lat (msec)   : 10=2.11%, 20=97.85%, 50=0.04%
  cpu          : usr=1.34%, sys=8.98%, ctx=2378, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9227KiB/s (9448kB/s), 2296KiB/s-2329KiB/s (2351kB/s-2384kB/s), io=19.6MiB (20.5MB), run=2157-2175msec
  WRITE: bw=9606KiB/s (9836kB/s), 2384KiB/s-2437KiB/s (2441kB/s-2495kB/s), io=20.4MiB (21.4MB), run=2157-2175msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=19822: Wed Jan 15 20:02:48 2025
   read: IOPS=11, BW=465MiB/s (488MB/s)(40.0MiB/86msec)
    slat (nsec): min=6976.6k, max=6976.6k, avg=6976556.00, stdev= 0.00
    clat (nsec): min=75176k, max=75176k, avg=75175950.00, stdev= 0.00
     lat (nsec): min=82164k, max=82164k, avg=82164435.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[74974],  5.00th=[74974], 10.00th=[74974], 20.00th=[74974],
     | 30.00th=[74974], 40.00th=[74974], 50.00th=[74974], 60.00th=[74974],
     | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974],
     | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.41%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=465MiB/s (488MB/s), 465MiB/s-465MiB/s (488MB/s-488MB/s), io=40.0MiB (41.9MB), run=86-86msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck