== sanity test 398c: run fio to test AIO ================= 06:45:33 (1747219533)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0767443 s, 547 MB/s
osc.lustre-OST0000-osc-ffff8800b6119000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=324: Wed May 14 06:45:38 2025
  write: IOPS=666, BW=2666KiB/s (2730kB/s)(10.0MiB/3841msec)
    slat (usec): min=14, max=386, avg=53.31, stdev=32.60
    clat (usec): min=11313, max=41740, avg=23825.54, stdev=5149.59
     lat (usec): min=11430, max=41969, avg=23880.15, stdev=5158.51
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[17171], 10.00th=[19006], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21627], 50.00th=[22414], 60.00th=[23462],
     | 70.00th=[25297], 80.00th=[27395], 90.00th=[31065], 95.00th=[34341],
     | 99.00th=[39584], 99.50th=[41157], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 2072, max= 3088, per=24.62%, avg=2625.57, stdev=386.06, samples=7
   iops        : min=  518, max=  772, avg=656.29, stdev=96.60, samples=7
  lat (msec)   : 20=19.84%, 50=80.16%
  cpu          : usr=0.49%, sys=4.17%, ctx=1411, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=325: Wed May 14 06:45:38 2025
  write: IOPS=669, BW=2679KiB/s (2743kB/s)(10.0MiB/3823msec)
    slat (usec): min=13, max=417, avg=53.44, stdev=30.52
    clat (usec): min=3175, max=41825, avg=23719.12, stdev=5361.12
     lat (usec): min=3237, max=41925, avg=23773.86, stdev=5368.61
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[16450], 10.00th=[18744], 20.00th=[19792],
     | 30.00th=[20841], 40.00th=[21627], 50.00th=[22414], 60.00th=[23200],
     | 70.00th=[25297], 80.00th=[27657], 90.00th=[31327], 95.00th=[34866],
     | 99.00th=[39060], 99.50th=[40109], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 2080, max= 3200, per=24.78%, avg=2642.71, stdev=406.38, samples=7
   iops        : min=  520, max=  800, avg=660.57, stdev=101.68, samples=7
  lat (msec)   : 4=0.08%, 10=0.43%, 20=20.39%, 50=79.10%
  cpu          : usr=0.68%, sys=4.19%, ctx=1330, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=326: Wed May 14 06:45:38 2025
  write: IOPS=666, BW=2667KiB/s (2731kB/s)(10.0MiB/3839msec)
    slat (usec): min=13, max=304, avg=49.08, stdev=29.18
    clat (usec): min=3364, max=41963, avg=23773.61, stdev=5246.77
     lat (usec): min=3427, max=42023, avg=23823.91, stdev=5255.04
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[16581], 10.00th=[19006], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21890], 50.00th=[22414], 60.00th=[23462],
     | 70.00th=[25297], 80.00th=[27132], 90.00th=[31327], 95.00th=[34866],
     | 99.00th=[40109], 99.50th=[40633], 99.90th=[41157], 99.95th=[41681],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2056, max= 3088, per=24.59%, avg=2622.14, stdev=387.21, samples=7
   iops        : min=  514, max=  772, avg=655.43, stdev=96.87, samples=7
  lat (msec)   : 4=0.12%, 10=0.04%, 20=20.12%, 50=79.73%
  cpu          : usr=0.76%, sys=3.65%, ctx=1028, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=327: Wed May 14 06:45:38 2025
  write: IOPS=666, BW=2666KiB/s (2730kB/s)(10.0MiB/3841msec)
    slat (usec): min=13, max=319, avg=54.75, stdev=30.12
    clat (usec): min=8899, max=42389, avg=23802.53, stdev=5275.16
     lat (usec): min=9003, max=42427, avg=23858.56, stdev=5284.13
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[16581], 10.00th=[19006], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21627], 50.00th=[22414], 60.00th=[23200],
     | 70.00th=[25297], 80.00th=[27395], 90.00th=[31327], 95.00th=[34341],
     | 99.00th=[39584], 99.50th=[41157], 99.90th=[41681], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2064, max= 3096, per=24.67%, avg=2630.86, stdev=393.61, samples=7
   iops        : min=  516, max=  774, avg=657.71, stdev=98.40, samples=7
  lat (msec)   : 10=0.62%, 20=19.30%, 50=80.08%
  cpu          : usr=0.70%, sys=4.32%, ctx=1554, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.4MiB/s (10.9MB/s), 2666KiB/s-2679KiB/s (2730kB/s-2743kB/s), io=40.0MiB (41.9MB), run=3823-3841msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=340: Wed May 14 06:45:42 2025
   read: IOPS=465, BW=1861KiB/s (1906kB/s)(5048KiB/2712msec)
    slat (usec): min=719, max=2576, avg=1146.09, stdev=232.40
    clat (usec): min=8001, max=22603, avg=15902.79, stdev=2063.39
     lat (usec): min=8915, max=23658, avg=17050.43, stdev=2067.07
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[12911], 10.00th=[13304], 20.00th=[14091],
     | 30.00th=[14615], 40.00th=[15270], 50.00th=[15795], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[20579], 99.50th=[21627], 99.90th=[22414], 99.95th=[22676],
     | 99.99th=[22676]
   bw (  KiB/s): min= 1712, max= 1944, per=25.02%, avg=1851.20, stdev=94.76, samples=5
   iops        : min=  428, max=  486, avg=462.80, stdev=23.69, samples=5
  write: IOPS=478, BW=1914KiB/s (1960kB/s)(5192KiB/2712msec)
    slat (usec): min=14, max=1192, avg=48.50, stdev=47.84
    clat (usec): min=3975, max=23429, avg=16580.71, stdev=1730.94
     lat (usec): min=4011, max=23462, avg=16630.39, stdev=1730.65
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[13960], 10.00th=[14484], 20.00th=[15139],
     | 30.00th=[15664], 40.00th=[16057], 50.00th=[16581], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[20841], 99.50th=[21627], 99.90th=[22676], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 1856, max= 1984, per=24.97%, avg=1923.20, stdev=46.85, samples=5
   iops        : min=  464, max=  496, avg=480.80, stdev=11.71, samples=5
  lat (msec)   : 4=0.04%, 10=0.16%, 20=97.27%, 50=2.54%
  cpu          : usr=0.96%, sys=7.23%, ctx=2841, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=341: Wed May 14 06:45:42 2025
   read: IOPS=461, BW=1846KiB/s (1890kB/s)(4976KiB/2696msec)
    slat (usec): min=25, max=2500, avg=1130.27, stdev=235.12
    clat (usec): min=2218, max=34872, avg=15858.47, stdev=2286.26
     lat (usec): min=2244, max=35805, avg=16990.39, stdev=2291.57
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12780], 10.00th=[13304], 20.00th=[13960],
     | 30.00th=[14615], 40.00th=[15139], 50.00th=[15664], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[21103], 99.50th=[21365], 99.90th=[29754], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1712, max= 1896, per=24.82%, avg=1836.80, stdev=76.23, samples=5
   iops        : min=  428, max=  474, avg=459.20, stdev=19.06, samples=5
  write: IOPS=488, BW=1953KiB/s (1999kB/s)(5264KiB/2696msec)
    slat (usec): min=14, max=1022, avg=48.14, stdev=41.25
    clat (usec): min=5096, max=33454, avg=16580.87, stdev=1846.54
     lat (usec): min=5116, max=33556, avg=16630.20, stdev=1847.16
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13829], 10.00th=[14484], 20.00th=[15139],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16581], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19268],
     | 99.00th=[21103], 99.50th=[21890], 99.90th=[28705], 99.95th=[33424],
     | 99.99th=[33424]
   bw (  KiB/s): min= 1848, max= 2048, per=25.65%, avg=1976.00, stdev=80.40, samples=5
   iops        : min=  462, max=  512, avg=494.00, stdev=20.10, samples=5
  lat (msec)   : 4=0.08%, 10=0.12%, 20=96.72%, 50=3.09%
  cpu          : usr=0.93%, sys=7.31%, ctx=2718, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=342: Wed May 14 06:45:42 2025
   read: IOPS=459, BW=1838KiB/s (1883kB/s)(4984KiB/2711msec)
    slat (usec): min=611, max=2061, avg=1137.24, stdev=225.81
    clat (usec): min=6190, max=24666, avg=15788.69, stdev=2080.28
     lat (usec): min=7175, max=25500, avg=16927.55, stdev=2084.84
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12518], 10.00th=[13304], 20.00th=[13960],
     | 30.00th=[14615], 40.00th=[15139], 50.00th=[15664], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[21103], 99.50th=[21365], 99.90th=[23200], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 1744, max= 1952, per=25.00%, avg=1849.60, stdev=81.07, samples=5
   iops        : min=  436, max=  488, avg=462.40, stdev=20.27, samples=5
  write: IOPS=484, BW=1939KiB/s (1985kB/s)(5256KiB/2711msec)
    slat (usec): min=13, max=1469, avg=48.01, stdev=47.10
    clat (usec): min=11494, max=26557, avg=16634.88, stdev=1734.77
     lat (usec): min=11558, max=26614, avg=16683.98, stdev=1732.80
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14091], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16581], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[21103], 99.50th=[21890], 99.90th=[26608], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min= 1760, max= 2032, per=25.05%, avg=1929.60, stdev=112.77, samples=5
   iops        : min=  440, max=  508, avg=482.40, stdev=28.19, samples=5
  lat (msec)   : 10=0.04%, 20=96.95%, 50=3.01%
  cpu          : usr=0.85%, sys=7.27%, ctx=2758, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=343: Wed May 14 06:45:42 2025
   read: IOPS=466, BW=1866KiB/s (1911kB/s)(5060KiB/2711msec)
    slat (usec): min=686, max=2275, avg=1144.42, stdev=237.84
    clat (usec): min=2386, max=23829, avg=15823.60, stdev=2192.38
     lat (usec): min=3997, max=25129, avg=16969.64, stdev=2194.98
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[12518], 10.00th=[13173], 20.00th=[14091],
     | 30.00th=[14615], 40.00th=[15139], 50.00th=[15664], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18482], 95.00th=[19530],
     | 99.00th=[22152], 99.50th=[22676], 99.90th=[23725], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 1760, max= 1968, per=25.24%, avg=1867.20, stdev=88.29, samples=5
   iops        : min=  440, max=  492, avg=466.80, stdev=22.07, samples=5
  write: IOPS=477, BW=1911KiB/s (1957kB/s)(5180KiB/2711msec)
    slat (usec): min=13, max=1104, avg=47.19, stdev=42.55
    clat (usec): min=12230, max=24584, avg=16622.04, stdev=1746.33
     lat (usec): min=12293, max=24632, avg=16670.34, stdev=1745.03
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[13960], 10.00th=[14615], 20.00th=[15139],
     | 30.00th=[15664], 40.00th=[16057], 50.00th=[16581], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[21627], 99.50th=[22414], 99.90th=[24511], 99.95th=[24511],
     | 99.99th=[24511]
   bw (  KiB/s): min= 1776, max= 2040, per=24.82%, avg=1912.00, stdev=94.15, samples=5
   iops        : min=  444, max=  510, avg=478.00, stdev=23.54, samples=5
  lat (msec)   : 4=0.08%, 10=0.04%, 20=96.56%, 50=3.32%
  cpu          : usr=0.96%, sys=7.34%, ctx=2724, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7400KiB/s (7577kB/s), 1838KiB/s-1866KiB/s (1883kB/s-1911kB/s), io=19.6MiB (20.5MB), run=2696-2712msec
  WRITE: bw=7704KiB/s (7888kB/s), 1911KiB/s-1953KiB/s (1957kB/s-1999kB/s), io=20.4MiB (21.4MB), run=2696-2712msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=349: Wed May 14 06:45:43 2025
   read: IOPS=10, BW=430MiB/s (451MB/s)(40.0MiB/93msec)
    slat (nsec): min=6129.4k, max=6129.4k, avg=6129384.00, stdev= 0.00
    clat (nsec): min=82893k, max=82893k, avg=82893187.00, stdev= 0.00
     lat (nsec): min=89035k, max=89035k, avg=89035051.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[83362],  5.00th=[83362], 10.00th=[83362], 20.00th=[83362],
     | 30.00th=[83362], 40.00th=[83362], 50.00th=[83362], 60.00th=[83362],
     | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362],
     | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[83362]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.52%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=430MiB/s (451MB/s), 430MiB/s-430MiB/s (451MB/s-451MB/s), io=40.0MiB (41.9MB), run=93-93msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout