== sanity test 398c: run fio to test AIO ================= 14:35:13 (1744137313)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.110633 s, 379 MB/s
osc.lustre-OST0000-osc-ffff8800b63d8000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30968: Tue Apr  8 14:35:18 2025
  write: IOPS=713, BW=2853KiB/s (2922kB/s)(10.0MiB/3589msec)
    slat (usec): min=12, max=273, avg=55.82, stdev=30.91
    clat (usec): min=2423, max=47708, avg=22306.52, stdev=3754.76
     lat (usec): min=2458, max=47742, avg=22363.76, stdev=3756.04
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[15008], 10.00th=[19006], 20.00th=[20579],
     | 30.00th=[21365], 40.00th=[22152], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23462], 80.00th=[24249], 90.00th=[25822], 95.00th=[28181],
     | 99.00th=[32113], 99.50th=[33162], 99.90th=[42206], 99.95th=[45351],
     | 99.99th=[47449]
   bw (  KiB/s): min= 2616, max= 3424, per=25.20%, avg=2860.57, stdev=265.20, samples=7
   iops        : min=  654, max=  856, avg=715.14, stdev=66.30, samples=7
  lat (msec)   : 4=0.20%, 10=0.35%, 20=13.95%, 50=85.51%
  cpu          : usr=0.72%, sys=3.99%, ctx=2072, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30969: Tue Apr  8 14:35:18 2025
  write: IOPS=710, BW=2840KiB/s (2909kB/s)(10.0MiB/3605msec)
    slat (usec): min=12, max=400, avg=58.23, stdev=33.14
    clat (usec): min=4888, max=40738, avg=22377.31, stdev=3541.26
     lat (usec): min=4927, max=40778, avg=22437.03, stdev=3546.18
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[15401], 10.00th=[19006], 20.00th=[20579],
     | 30.00th=[21365], 40.00th=[22152], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23462], 80.00th=[24249], 90.00th=[25822], 95.00th=[28181],
     | 99.00th=[32375], 99.50th=[32637], 99.90th=[38536], 99.95th=[38536],
     | 99.99th=[40633]
   bw (  KiB/s): min= 2624, max= 3328, per=25.05%, avg=2843.43, stdev=229.64, samples=7
   iops        : min=  656, max=  832, avg=710.86, stdev=57.41, samples=7
  lat (msec)   : 10=0.20%, 20=13.87%, 50=85.94%
  cpu          : usr=0.89%, sys=4.05%, ctx=2270, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30970: Tue Apr  8 14:35:18 2025
  write: IOPS=709, BW=2838KiB/s (2906kB/s)(10.0MiB/3608msec)
    slat (usec): min=12, max=413, avg=53.53, stdev=31.04
    clat (usec): min=8307, max=33128, avg=22379.17, stdev=3504.29
     lat (usec): min=8369, max=33367, avg=22434.07, stdev=3507.88
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[15270], 10.00th=[19268], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22152], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23462], 80.00th=[24249], 90.00th=[25822], 95.00th=[28181],
     | 99.00th=[31589], 99.50th=[32375], 99.90th=[32900], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 2664, max= 3328, per=24.96%, avg=2833.14, stdev=230.36, samples=7
   iops        : min=  666, max=  832, avg=708.29, stdev=57.59, samples=7
  lat (msec)   : 10=0.66%, 20=12.50%, 50=86.84%
  cpu          : usr=0.78%, sys=3.74%, ctx=1780, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30971: Tue Apr  8 14:35:18 2025
  write: IOPS=710, BW=2840KiB/s (2909kB/s)(10.0MiB/3605msec)
    slat (usec): min=12, max=516, avg=56.54, stdev=33.80
    clat (usec): min=2539, max=43180, avg=22403.88, stdev=3441.30
     lat (usec): min=2618, max=43218, avg=22461.79, stdev=3443.96
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[15401], 10.00th=[19006], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22152], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23462], 80.00th=[24249], 90.00th=[25822], 95.00th=[27919],
     | 99.00th=[32113], 99.50th=[32900], 99.90th=[35914], 99.95th=[36439],
     | 99.99th=[43254]
   bw (  KiB/s): min= 2616, max= 3328, per=25.05%, avg=2843.43, stdev=230.56, samples=7
   iops        : min=  654, max=  832, avg=710.86, stdev=57.64, samples=7
  lat (msec)   : 4=0.04%, 10=0.04%, 20=13.87%, 50=86.05%
  cpu          : usr=1.05%, sys=3.66%, ctx=2134, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.1MiB/s (11.6MB/s), 2838KiB/s-2853KiB/s (2906kB/s-2922kB/s), io=40.0MiB (41.9MB), run=3589-3608msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30983: Tue Apr  8 14:35:22 2025
   read: IOPS=410, BW=1642KiB/s (1682kB/s)(5048KiB/3074msec)
    slat (usec): min=712, max=2810, avg=1224.48, stdev=246.02
    clat (usec): min=2639, max=28131, avg=18111.54, stdev=2776.35
     lat (usec): min=4003, max=29341, avg=19338.07, stdev=2816.24
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13960], 10.00th=[14877], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[18220], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[20317], 90.00th=[21627], 95.00th=[22414],
     | 99.00th=[25035], 99.50th=[26084], 99.90th=[27657], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1560, max= 1704, per=24.97%, avg=1630.00, stdev=56.55, samples=6
   iops        : min=  390, max=  426, avg=407.50, stdev=14.14, samples=6
  write: IOPS=422, BW=1689KiB/s (1730kB/s)(5192KiB/3074msec)
    slat (usec): min=13, max=370, avg=49.55, stdev=25.11
    clat (usec): min=2589, max=27763, avg=18792.22, stdev=2354.62
     lat (usec): min=2630, max=27804, avg=18843.34, stdev=2356.64
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[15401], 10.00th=[16057], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[25035], 99.50th=[25560], 99.90th=[26346], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1496, max= 1816, per=24.63%, avg=1674.00, stdev=113.81, samples=6
   iops        : min=  374, max=  454, avg=418.50, stdev=28.45, samples=6
  lat (msec)   : 4=0.08%, 10=0.35%, 20=73.59%, 50=25.98%
  cpu          : usr=0.68%, sys=7.09%, ctx=2998, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30984: Tue Apr  8 14:35:22 2025
   read: IOPS=407, BW=1631KiB/s (1670kB/s)(4976KiB/3051msec)
    slat (usec): min=18, max=2579, avg=1218.61, stdev=251.44
    clat (usec): min=2727, max=34757, avg=18072.53, stdev=2975.65
     lat (usec): min=3685, max=36165, avg=19293.10, stdev=3010.65
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13829], 10.00th=[14615], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19268], 80.00th=[20317], 90.00th=[21365], 95.00th=[22938],
     | 99.00th=[26084], 99.50th=[28181], 99.90th=[32375], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1576, max= 1716, per=24.89%, avg=1624.67, stdev=57.15, samples=6
   iops        : min=  394, max=  429, avg=406.17, stdev=14.29, samples=6
  write: IOPS=431, BW=1725KiB/s (1767kB/s)(5264KiB/3051msec)
    slat (usec): min=13, max=1411, avg=51.40, stdev=47.28
    clat (usec): min=4959, max=34694, avg=18709.35, stdev=2585.44
     lat (usec): min=5001, max=34749, avg=18762.08, stdev=2585.28
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[15008], 10.00th=[16057], 20.00th=[16909],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[25035], 99.50th=[26084], 99.90th=[31065], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1608, max= 1824, per=25.44%, avg=1728.67, stdev=76.83, samples=6
   iops        : min=  402, max=  456, avg=432.17, stdev=19.21, samples=6
  lat (msec)   : 4=0.12%, 10=0.59%, 20=74.57%, 50=24.73%
  cpu          : usr=0.92%, sys=6.79%, ctx=2996, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30985: Tue Apr  8 14:35:22 2025
   read: IOPS=405, BW=1623KiB/s (1662kB/s)(4984KiB/3070msec)
    slat (usec): min=659, max=2864, avg=1218.80, stdev=249.34
    clat (usec): min=11187, max=29431, avg=18202.43, stdev=2762.06
     lat (usec): min=12453, max=30748, avg=19423.04, stdev=2799.22
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13960], 10.00th=[14746], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[18220], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[20317], 90.00th=[21365], 95.00th=[22676],
     | 99.00th=[27132], 99.50th=[27919], 99.90th=[28967], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1512, max= 1784, per=24.94%, avg=1628.00, stdev=94.08, samples=6
   iops        : min=  378, max=  446, avg=407.00, stdev=23.52, samples=6
  write: IOPS=428, BW=1712KiB/s (1753kB/s)(5256KiB/3070msec)
    slat (usec): min=14, max=974, avg=50.54, stdev=35.38
    clat (usec): min=3455, max=34654, avg=18646.46, stdev=2335.10
     lat (usec): min=3499, max=34690, avg=18698.20, stdev=2335.76
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[15139], 10.00th=[15926], 20.00th=[16909],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18744], 60.00th=[19006],
     | 70.00th=[19530], 80.00th=[20317], 90.00th=[21365], 95.00th=[22152],
     | 99.00th=[25035], 99.50th=[26084], 99.90th=[32900], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1600, max= 1944, per=25.07%, avg=1704.00, stdev=135.01, samples=6
   iops        : min=  400, max=  486, avg=426.00, stdev=33.75, samples=6
  lat (msec)   : 4=0.04%, 20=75.98%, 50=23.98%
  cpu          : usr=0.55%, sys=7.01%, ctx=3001, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30986: Tue Apr  8 14:35:22 2025
   read: IOPS=412, BW=1648KiB/s (1688kB/s)(5060KiB/3070msec)
    slat (usec): min=705, max=2802, avg=1220.36, stdev=245.02
    clat (usec): min=5843, max=30370, avg=18078.26, stdev=2854.87
     lat (usec): min=7050, max=31442, avg=19300.38, stdev=2886.23
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13960], 10.00th=[14615], 20.00th=[15664],
     | 30.00th=[16450], 40.00th=[17171], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19268], 80.00th=[20317], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[25822], 99.50th=[27395], 99.90th=[29754], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1608, max= 1704, per=25.26%, avg=1648.67, stdev=33.41, samples=6
   iops        : min=  402, max=  426, avg=412.17, stdev= 8.35, samples=6
  write: IOPS=421, BW=1687KiB/s (1728kB/s)(5180KiB/3070msec)
    slat (usec): min=13, max=370, avg=49.12, stdev=25.39
    clat (usec): min=7071, max=38670, avg=18832.45, stdev=2454.68
     lat (usec): min=7111, max=38720, avg=18882.80, stdev=2454.41
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15270], 10.00th=[16057], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[25822], 99.50th=[26870], 99.90th=[35390], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 1552, max= 1776, per=24.75%, avg=1682.00, stdev=88.50, samples=6
   iops        : min=  388, max=  444, avg=420.50, stdev=22.12, samples=6
  lat (msec)   : 10=0.23%, 20=73.95%, 50=25.82%
  cpu          : usr=0.62%, sys=7.07%, ctx=2956, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6528KiB/s (6685kB/s), 1623KiB/s-1648KiB/s (1662kB/s-1688kB/s), io=19.6MiB (20.5MB), run=3051-3074msec
  WRITE: bw=6796KiB/s (6959kB/s), 1687KiB/s-1725KiB/s (1728kB/s-1767kB/s), io=20.4MiB (21.4MB), run=3051-3074msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30993: Tue Apr  8 14:35:23 2025
   read: IOPS=13, BW=533MiB/s (559MB/s)(40.0MiB/75msec)
    slat (nsec): min=5560.1k, max=5560.1k, avg=5560093.00, stdev= 0.00
    clat (nsec): min=65716k, max=65716k, avg=65716000.00, stdev= 0.00
     lat (nsec): min=71290k, max=71290k, avg=71290170.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[65799],  5.00th=[65799], 10.00th=[65799], 20.00th=[65799],
     | 30.00th=[65799], 40.00th=[65799], 50.00th=[65799], 60.00th=[65799],
     | 70.00th=[65799], 80.00th=[65799], 90.00th=[65799], 95.00th=[65799],
     | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799],
     | 99.99th=[65799]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.67%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=533MiB/s (559MB/s), 533MiB/s-533MiB/s (559MB/s-559MB/s), io=40.0MiB (41.9MB), run=75-75msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout