== sanity test 398c: run fio to test AIO ================= 23:41:30 (1708836090)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0808732 s, 519 MB/s
osc.lustre-OST0000-osc-ffff8800ab917000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=7605: Sat Feb 24 23:41:35 2024
  write: IOPS=729, BW=2920KiB/s (2990kB/s)(10.0MiB/3507msec)
    slat (usec): min=12, max=398, avg=50.35, stdev=25.62
    clat (usec): min=2414, max=40327, avg=21786.16, stdev=2432.04
     lat (usec): min=2475, max=40352, avg=21837.92, stdev=2432.35
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[18744], 10.00th=[19530], 20.00th=[20317],
     | 30.00th=[21103], 40.00th=[21627], 50.00th=[22152], 60.00th=[22414],
     | 70.00th=[22676], 80.00th=[23200], 90.00th=[23987], 95.00th=[24773],
     | 99.00th=[26346], 99.50th=[27132], 99.90th=[38011], 99.95th=[38011],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2760, max= 3144, per=24.97%, avg=2908.57, stdev=127.32, samples=7
   iops        : min=  690, max=  786, avg=727.14, stdev=31.83, samples=7
  lat (msec)   : 4=0.08%, 10=0.35%, 20=14.10%, 50=85.47%
  cpu          : usr=0.43%, sys=4.48%, ctx=1740, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7606: Sat Feb 24 23:41:35 2024
  write: IOPS=730, BW=2922KiB/s (2993kB/s)(10.0MiB/3504msec)
    slat (usec): min=12, max=391, avg=45.51, stdev=26.34
    clat (usec): min=3779, max=42143, avg=21779.03, stdev=2682.76
     lat (usec): min=3827, max=42183, avg=21825.93, stdev=2682.85
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[18220], 10.00th=[19530], 20.00th=[20317],
     | 30.00th=[21103], 40.00th=[21627], 50.00th=[22152], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23200], 90.00th=[23987], 95.00th=[25035],
     | 99.00th=[25822], 99.50th=[26870], 99.90th=[42206], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2720, max= 3224, per=24.97%, avg=2908.57, stdev=165.50, samples=7
   iops        : min=  680, max=  806, avg=727.14, stdev=41.37, samples=7
  lat (msec)   : 4=0.04%, 10=0.66%, 20=14.02%, 50=85.27%
  cpu          : usr=0.77%, sys=3.57%, ctx=1281, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7607: Sat Feb 24 23:41:35 2024
  write: IOPS=728, BW=2912KiB/s (2982kB/s)(10.0MiB/3516msec)
    slat (usec): min=12, max=251, avg=49.04, stdev=25.53
    clat (usec): min=2226, max=33996, avg=21838.80, stdev=2148.59
     lat (usec): min=2387, max=34027, avg=21889.28, stdev=2149.41
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[18744], 10.00th=[19792], 20.00th=[20317],
     | 30.00th=[21103], 40.00th=[21627], 50.00th=[22152], 60.00th=[22414],
     | 70.00th=[22676], 80.00th=[23200], 90.00th=[23987], 95.00th=[24773],
     | 99.00th=[26346], 99.50th=[26608], 99.90th=[32637], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 2728, max= 3072, per=24.93%, avg=2904.00, stdev=115.47, samples=7
   iops        : min=  682, max=  768, avg=726.00, stdev=28.87, samples=7
  lat (msec)   : 4=0.04%, 20=13.79%, 50=86.17%
  cpu          : usr=0.65%, sys=3.98%, ctx=1524, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7608: Sat Feb 24 23:41:35 2024
  write: IOPS=728, BW=2913KiB/s (2983kB/s)(10.0MiB/3515msec)
    slat (usec): min=12, max=375, avg=48.93, stdev=27.92
    clat (usec): min=2930, max=35515, avg=21835.62, stdev=2154.73
     lat (usec): min=3021, max=35566, avg=21886.05, stdev=2155.53
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[19006], 10.00th=[19792], 20.00th=[20579],
     | 30.00th=[21103], 40.00th=[21627], 50.00th=[22152], 60.00th=[22414],
     | 70.00th=[22676], 80.00th=[23200], 90.00th=[23987], 95.00th=[24773],
     | 99.00th=[26346], 99.50th=[26608], 99.90th=[28443], 99.95th=[30540],
     | 99.99th=[35390]
   bw (  KiB/s): min= 2744, max= 3088, per=24.95%, avg=2906.86, stdev=119.63, samples=7
   iops        : min=  686, max=  772, avg=726.57, stdev=29.97, samples=7
  lat (msec)   : 4=0.04%, 10=0.04%, 20=13.20%, 50=86.72%
  cpu          : usr=0.77%, sys=3.70%, ctx=1549, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.4MiB/s (11.9MB/s), 2912KiB/s-2922KiB/s (2982kB/s-2993kB/s), io=40.0MiB (41.9MB), run=3504-3516msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=7620: Sat Feb 24 23:41:39 2024
   read: IOPS=405, BW=1622KiB/s (1661kB/s)(5048KiB/3113msec)
    slat (usec): min=14, max=2447, avg=1276.18, stdev=276.45
    clat (usec): min=2211, max=27565, avg=18631.50, stdev=2887.89
     lat (usec): min=2238, max=29157, avg=19909.77, stdev=2944.66
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[14353], 10.00th=[15270], 20.00th=[16319],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[21103], 90.00th=[22152], 95.00th=[23200],
     | 99.00th=[25297], 99.50th=[26346], 99.90th=[26870], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1480, max= 1720, per=25.00%, avg=1611.33, stdev=88.16, samples=6
   iops        : min=  370, max=  430, avg=402.83, stdev=22.04, samples=6
  write: IOPS=416, BW=1668KiB/s (1708kB/s)(5192KiB/3113msec)
    slat (usec): min=12, max=362, avg=46.76, stdev=21.05
    clat (usec): min=4854, max=26470, avg=18939.87, stdev=2231.22
     lat (usec): min=4868, max=26515, avg=18988.08, stdev=2232.29
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15664], 10.00th=[16319], 20.00th=[17171],
     | 30.00th=[17957], 40.00th=[18482], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20579], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[23987], 99.50th=[25035], 99.90th=[25560], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1512, max= 1792, per=24.69%, avg=1656.67, stdev=92.69, samples=6
   iops        : min=  378, max=  448, avg=414.17, stdev=23.17, samples=6
  lat (msec)   : 4=0.08%, 10=0.59%, 20=69.49%, 50=29.84%
  cpu          : usr=0.84%, sys=7.26%, ctx=2171, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7621: Sat Feb 24 23:41:39 2024
   read: IOPS=400, BW=1604KiB/s (1642kB/s)(4976KiB/3103msec)
    slat (usec): min=27, max=2255, avg=1287.78, stdev=266.10
    clat (usec): min=2898, max=34813, avg=18471.95, stdev=2601.83
     lat (usec): min=2968, max=36547, avg=19761.74, stdev=2634.94
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14746], 10.00th=[15270], 20.00th=[16319],
     | 30.00th=[16909], 40.00th=[17695], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[25035], 99.50th=[25560], 99.90th=[30540], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1528, max= 1632, per=24.77%, avg=1596.67, stdev=42.51, samples=6
   iops        : min=  382, max=  408, avg=399.17, stdev=10.63, samples=6
  write: IOPS=424, BW=1696KiB/s (1737kB/s)(5264KiB/3103msec)
    slat (usec): min=13, max=771, avg=47.33, stdev=29.01
    clat (usec): min=2087, max=32229, avg=18959.07, stdev=2168.92
     lat (usec): min=2129, max=32298, avg=19007.80, stdev=2169.44
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[15795], 10.00th=[16450], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18482], 50.00th=[19006], 60.00th=[19268],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21627], 95.00th=[22414],
     | 99.00th=[24511], 99.50th=[25560], 99.90th=[28443], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1608, max= 1808, per=25.30%, avg=1698.00, stdev=81.97, samples=6
   iops        : min=  402, max=  452, avg=424.50, stdev=20.49, samples=6
  lat (msec)   : 4=0.08%, 10=0.20%, 20=71.48%, 50=28.24%
  cpu          : usr=1.29%, sys=7.12%, ctx=2221, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7622: Sat Feb 24 23:41:39 2024
   read: IOPS=401, BW=1607KiB/s (1645kB/s)(4984KiB/3102msec)
    slat (usec): min=26, max=2414, avg=1276.70, stdev=270.35
    clat (usec): min=5917, max=37327, avg=18389.58, stdev=2636.71
     lat (usec): min=5950, max=38432, avg=19668.38, stdev=2677.82
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14484], 10.00th=[15270], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[24249], 99.50th=[24511], 99.90th=[33817], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1392, max= 1792, per=25.04%, avg=1614.00, stdev=137.86, samples=6
   iops        : min=  348, max=  448, avg=403.50, stdev=34.47, samples=6
  write: IOPS=423, BW=1694KiB/s (1735kB/s)(5256KiB/3102msec)
    slat (usec): min=16, max=1215, avg=48.15, stdev=37.90
    clat (usec): min=5104, max=37172, avg=19029.73, stdev=2292.16
     lat (usec): min=5135, max=37212, avg=19079.23, stdev=2292.66
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[15926], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18220], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[20579], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[24773], 99.50th=[25822], 99.90th=[35390], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1600, max= 1784, per=25.12%, avg=1686.00, stdev=68.43, samples=6
   iops        : min=  400, max=  446, avg=421.50, stdev=17.11, samples=6
  lat (msec)   : 10=0.20%, 20=72.23%, 50=27.58%
  cpu          : usr=1.06%, sys=7.19%, ctx=2178, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7623: Sat Feb 24 23:41:39 2024
   read: IOPS=407, BW=1628KiB/s (1667kB/s)(5060KiB/3108msec)
    slat (usec): min=18, max=2761, avg=1278.44, stdev=274.35
    clat (usec): min=12496, max=27399, avg=18431.19, stdev=2508.94
     lat (usec): min=13563, max=29836, avg=19711.71, stdev=2545.24
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14615], 10.00th=[15270], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17695], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21890], 95.00th=[22676],
     | 99.00th=[24511], 99.50th=[25560], 99.90th=[26870], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1524, max= 1680, per=25.20%, avg=1624.67, stdev=63.32, samples=6
   iops        : min=  381, max=  420, avg=406.17, stdev=15.83, samples=6
  write: IOPS=416, BW=1667KiB/s (1707kB/s)(5180KiB/3108msec)
    slat (usec): min=12, max=511, avg=46.79, stdev=22.48
    clat (usec): min=11204, max=30393, avg=19057.98, stdev=1942.61
     lat (usec): min=11721, max=30437, avg=19106.15, stdev=1940.92
    clat percentiles (usec):
     |  1.00th=[15008],  5.00th=[15926], 10.00th=[16712], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18482], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20579], 90.00th=[21365], 95.00th=[22152],
     | 99.00th=[24249], 99.50th=[24773], 99.90th=[27132], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1616, max= 1712, per=24.75%, avg=1660.67, stdev=42.21, samples=6
   iops        : min=  404, max=  428, avg=415.17, stdev=10.55, samples=6
  lat (msec)   : 20=71.41%, 50=28.59%
  cpu          : usr=1.06%, sys=7.27%, ctx=2258, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6447KiB/s (6601kB/s), 1604KiB/s-1628KiB/s (1642kB/s-1667kB/s), io=19.6MiB (20.5MB), run=3102-3113msec
  WRITE: bw=6711KiB/s (6872kB/s), 1667KiB/s-1696KiB/s (1707kB/s-1737kB/s), io=20.4MiB (21.4MB), run=3102-3113msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=7630: Sat Feb 24 23:41:41 2024
   read: IOPS=10, BW=404MiB/s (424MB/s)(40.0MiB/99msec)
    slat (nsec): min=6418.8k, max=6418.8k, avg=6418795.00, stdev= 0.00
    clat (nsec): min=91740k, max=91740k, avg=91740471.00, stdev= 0.00
     lat (nsec): min=98177k, max=98177k, avg=98176796.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[91751],  5.00th=[91751], 10.00th=[91751], 20.00th=[91751],
     | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751],
     | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751],
     | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.14%, ctx=3, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=40.0MiB (41.9MB), run=99-99msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout