== sanity test 398c: run fio to test AIO ================= 04:05:04 (1749542704)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.103835 s, 404 MB/s
osc.lustre-OST0000-osc-ffff8800aa455000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=21521: Tue Jun 10 04:05:11 2025
  write: IOPS=486, BW=1946KiB/s (1993kB/s)(10.0MiB/5261msec)
    slat (usec): min=13, max=583, avg=73.02, stdev=48.29
    clat (usec): min=2925, max=56619, avg=32652.11, stdev=9615.50
     lat (usec): min=2978, max=56696, avg=32726.97, stdev=9633.17
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[17695], 10.00th=[21365], 20.00th=[23987],
     | 30.00th=[25822], 40.00th=[28181], 50.00th=[31589], 60.00th=[33817],
     | 70.00th=[39060], 80.00th=[43779], 90.00th=[46400], 95.00th=[47449],
     | 99.00th=[50070], 99.50th=[51119], 99.90th=[55837], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min= 1344, max= 2856, per=24.75%, avg=1927.00, stdev=491.62, samples=10
   iops        : min=  336, max=  714, avg=481.70, stdev=122.96, samples=10
  lat (msec)   : 4=0.08%, 10=0.04%, 20=6.68%, 50=92.19%, 100=1.02%
  cpu          : usr=0.80%, sys=3.90%, ctx=1440, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21522: Tue Jun 10 04:05:11 2025
  write: IOPS=486, BW=1947KiB/s (1994kB/s)(10.0MiB/5259msec)
    slat (usec): min=14, max=551, avg=73.60, stdev=49.96
    clat (usec): min=6519, max=54009, avg=32662.81, stdev=9625.69
     lat (usec): min=6583, max=54116, avg=32738.35, stdev=9643.06
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[18482], 10.00th=[21103], 20.00th=[23725],
     | 30.00th=[26084], 40.00th=[28443], 50.00th=[31589], 60.00th=[33817],
     | 70.00th=[39060], 80.00th=[43779], 90.00th=[46400], 95.00th=[47449],
     | 99.00th=[50070], 99.50th=[51119], 99.90th=[53216], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min= 1408, max= 2944, per=24.83%, avg=1933.30, stdev=509.23, samples=10
   iops        : min=  352, max=  736, avg=483.30, stdev=127.33, samples=10
  lat (msec)   : 10=0.04%, 20=6.76%, 50=92.30%, 100=0.90%
  cpu          : usr=0.63%, sys=3.88%, ctx=1529, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21523: Tue Jun 10 04:05:11 2025
  write: IOPS=487, BW=1949KiB/s (1996kB/s)(10.0MiB/5253msec)
    slat (usec): min=13, max=791, avg=75.59, stdev=51.49
    clat (usec): min=5662, max=55715, avg=32655.89, stdev=9578.48
     lat (usec): min=5711, max=55829, avg=32733.33, stdev=9595.30
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[19268], 10.00th=[21103], 20.00th=[23725],
     | 30.00th=[26084], 40.00th=[28443], 50.00th=[31589], 60.00th=[33817],
     | 70.00th=[39060], 80.00th=[43779], 90.00th=[46400], 95.00th=[47449],
     | 99.00th=[50070], 99.50th=[52167], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[55837]
   bw (  KiB/s): min= 1389, max= 2960, per=24.91%, avg=1939.20, stdev=510.01, samples=10
   iops        : min=  347, max=  740, avg=484.70, stdev=127.47, samples=10
  lat (msec)   : 10=0.08%, 20=6.25%, 50=92.46%, 100=1.21%
  cpu          : usr=0.53%, sys=4.28%, ctx=1667, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21524: Tue Jun 10 04:05:11 2025
  write: IOPS=488, BW=1955KiB/s (2002kB/s)(10.0MiB/5237msec)
    slat (usec): min=15, max=677, avg=76.75, stdev=50.51
    clat (usec): min=2691, max=56026, avg=32558.93, stdev=9729.15
     lat (usec): min=2761, max=56058, avg=32637.56, stdev=9736.78
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[18482], 10.00th=[20841], 20.00th=[23725],
     | 30.00th=[26084], 40.00th=[28181], 50.00th=[31851], 60.00th=[33817],
     | 70.00th=[39060], 80.00th=[43779], 90.00th=[46400], 95.00th=[47449],
     | 99.00th=[50070], 99.50th=[50070], 99.90th=[55837], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min= 1309, max= 3040, per=24.99%, avg=1945.60, stdev=529.02, samples=10
   iops        : min=  327, max=  760, avg=486.30, stdev=132.23, samples=10
  lat (msec)   : 4=0.04%, 10=0.70%, 20=6.37%, 50=91.60%, 100=1.29%
  cpu          : usr=0.76%, sys=3.99%, ctx=1697, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7786KiB/s (7972kB/s), 1946KiB/s-1955KiB/s (1993kB/s-2002kB/s), io=40.0MiB (41.9MB), run=5237-5261msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=21538: Tue Jun 10 04:05:15 2025
   read: IOPS=370, BW=1482KiB/s (1518kB/s)(5048KiB/3406msec)
    slat (usec): min=29, max=3936, avg=1382.75, stdev=501.63
    clat (usec): min=2683, max=50314, avg=19911.03, stdev=6246.51
     lat (usec): min=4544, max=53014, avg=21295.92, stdev=6590.09
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13698], 10.00th=[14615], 20.00th=[15664],
     | 30.00th=[16450], 40.00th=[17171], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[20055], 80.00th=[22676], 90.00th=[30278], 95.00th=[33817],
     | 99.00th=[40109], 99.50th=[40633], 99.90th=[49546], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min=  816, max= 1784, per=24.14%, avg=1420.00, stdev=407.57, samples=6
   iops        : min=  204, max=  446, avg=355.00, stdev=101.89, samples=6
  write: IOPS=381, BW=1524KiB/s (1561kB/s)(5192KiB/3406msec)
    slat (usec): min=13, max=308, avg=52.30, stdev=32.65
    clat (usec): min=6035, max=44485, avg=20906.36, stdev=6309.26
     lat (usec): min=6246, max=44560, avg=20960.09, stdev=6324.63
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15008], 10.00th=[15795], 20.00th=[16581],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18482], 60.00th=[19530],
     | 70.00th=[20579], 80.00th=[24511], 90.00th=[32375], 95.00th=[35390],
     | 99.00th=[40633], 99.50th=[41157], 99.90th=[42730], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min=  864, max= 1816, per=24.36%, avg=1492.00, stdev=395.52, samples=6
   iops        : min=  216, max=  454, avg=373.00, stdev=98.88, samples=6
  lat (msec)   : 4=0.08%, 10=0.08%, 20=67.62%, 50=32.19%, 100=0.04%
  cpu          : usr=0.85%, sys=7.20%, ctx=2290, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21539: Tue Jun 10 04:05:15 2025
   read: IOPS=364, BW=1459KiB/s (1494kB/s)(4976KiB/3411msec)
    slat (usec): min=34, max=3935, avg=1381.16, stdev=493.62
    clat (usec): min=3246, max=46788, avg=19881.03, stdev=6142.24
     lat (usec): min=4325, max=50447, avg=21264.30, stdev=6493.71
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13829], 10.00th=[14615], 20.00th=[15664],
     | 30.00th=[16450], 40.00th=[17171], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[20055], 80.00th=[22938], 90.00th=[30016], 95.00th=[33817],
     | 99.00th=[39060], 99.50th=[40633], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  744, max= 1752, per=23.68%, avg=1393.33, stdev=417.85, samples=6
   iops        : min=  186, max=  438, avg=348.33, stdev=104.46, samples=6
  write: IOPS=385, BW=1543KiB/s (1580kB/s)(5264KiB/3411msec)
    slat (usec): min=13, max=1325, avg=53.91, stdev=53.78
    clat (usec): min=3173, max=47838, avg=20996.50, stdev=6377.96
     lat (usec): min=3238, max=47881, avg=21051.67, stdev=6395.12
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15270], 10.00th=[15926], 20.00th=[16712],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18744], 60.00th=[19530],
     | 70.00th=[20579], 80.00th=[24773], 90.00th=[32113], 95.00th=[34866],
     | 99.00th=[41157], 99.50th=[43779], 99.90th=[46924], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min=  864, max= 1808, per=24.56%, avg=1504.00, stdev=398.24, samples=6
   iops        : min=  216, max=  452, avg=376.00, stdev=99.56, samples=6
  lat (msec)   : 4=0.08%, 10=0.20%, 20=66.99%, 50=32.73%
  cpu          : usr=1.00%, sys=6.92%, ctx=2215, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21540: Tue Jun 10 04:05:15 2025
   read: IOPS=367, BW=1472KiB/s (1507kB/s)(4984KiB/3387msec)
    slat (usec): min=27, max=5467, avg=1402.27, stdev=521.19
    clat (usec): min=2413, max=45792, avg=19979.23, stdev=6351.87
     lat (usec): min=3226, max=47623, avg=21383.51, stdev=6711.77
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[13435], 10.00th=[14615], 20.00th=[15664],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[20317], 80.00th=[23725], 90.00th=[30278], 95.00th=[33817],
     | 99.00th=[39060], 99.50th=[40109], 99.90th=[43779], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min=  888, max= 1760, per=24.18%, avg=1422.67, stdev=332.61, samples=6
   iops        : min=  222, max=  440, avg=355.67, stdev=83.15, samples=6
  write: IOPS=387, BW=1552KiB/s (1589kB/s)(5256KiB/3387msec)
    slat (usec): min=13, max=1738, avg=52.47, stdev=62.09
    clat (usec): min=9036, max=43631, avg=20799.62, stdev=6194.74
     lat (usec): min=9188, max=43703, avg=20853.45, stdev=6210.26
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[14877], 10.00th=[15664], 20.00th=[16712],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18482], 60.00th=[19530],
     | 70.00th=[20579], 80.00th=[23725], 90.00th=[31589], 95.00th=[34341],
     | 99.00th=[41157], 99.50th=[41681], 99.90th=[43254], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  952, max= 2024, per=24.78%, avg=1517.33, stdev=450.27, samples=6
   iops        : min=  238, max=  506, avg=379.33, stdev=112.57, samples=6
  lat (msec)   : 4=0.16%, 10=0.43%, 20=66.64%, 50=32.77%
  cpu          : usr=0.89%, sys=7.06%, ctx=2257, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21541: Tue Jun 10 04:05:15 2025
   read: IOPS=370, BW=1483KiB/s (1519kB/s)(5060KiB/3411msec)
    slat (usec): min=697, max=3915, avg=1404.06, stdev=506.37
    clat (usec): min=7890, max=46810, avg=19866.67, stdev=6062.02
     lat (usec): min=10425, max=50350, avg=21272.94, stdev=6445.35
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13566], 10.00th=[14484], 20.00th=[15664],
     | 30.00th=[16450], 40.00th=[17171], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19792], 80.00th=[23200], 90.00th=[30016], 95.00th=[34341],
     | 99.00th=[38536], 99.50th=[40109], 99.90th=[42730], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  760, max= 1768, per=24.25%, avg=1426.67, stdev=419.48, samples=6
   iops        : min=  190, max=  442, avg=356.67, stdev=104.87, samples=6
  write: IOPS=379, BW=1519KiB/s (1555kB/s)(5180KiB/3411msec)
    slat (usec): min=14, max=406, avg=52.13, stdev=32.84
    clat (usec): min=10453, max=43540, avg=20809.57, stdev=6288.18
     lat (usec): min=10474, max=43605, avg=20863.10, stdev=6303.03
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15139], 10.00th=[15795], 20.00th=[16712],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20317], 80.00th=[24249], 90.00th=[31851], 95.00th=[35390],
     | 99.00th=[40633], 99.50th=[41681], 99.90th=[43254], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  808, max= 1840, per=24.06%, avg=1473.33, stdev=418.74, samples=6
   iops        : min=  202, max=  460, avg=368.33, stdev=104.69, samples=6
  lat (msec)   : 10=0.08%, 20=68.91%, 50=31.02%
  cpu          : usr=1.23%, sys=6.77%, ctx=2293, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5883KiB/s (6025kB/s), 1459KiB/s-1483KiB/s (1494kB/s-1519kB/s), io=19.6MiB (20.5MB), run=3387-3411msec
  WRITE: bw=6125KiB/s (6272kB/s), 1519KiB/s-1552KiB/s (1555kB/s-1589kB/s), io=20.4MiB (21.4MB), run=3387-3411msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=21547: Tue Jun 10 04:05:17 2025
   read: IOPS=12, BW=500MiB/s (524MB/s)(40.0MiB/80msec)
    slat (nsec): min=5793.3k, max=5793.3k, avg=5793253.00, stdev= 0.00
    clat (nsec): min=69180k, max=69180k, avg=69180195.00, stdev= 0.00
     lat (nsec): min=74986k, max=74986k, avg=74985988.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[68682],  5.00th=[68682], 10.00th=[68682], 20.00th=[68682],
     | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[68682],
     | 70.00th=[68682], 80.00th=[68682], 90.00th=[68682], 95.00th=[68682],
     | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.59%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=40.0MiB (41.9MB), run=80-80msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout