== sanity test 398c: run fio to test AIO ================= 04:23:43 (1724315023)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.085141 s, 493 MB/s
osc.lustre-OST0000-osc-ffff8800b6788000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=5457: Thu Aug 22 04:23:47 2024
  write: IOPS=736, BW=2948KiB/s (3018kB/s)(10.0MiB/3474msec)
    slat (usec): min=14, max=350, avg=47.78, stdev=24.37
    clat (usec): min=2871, max=38819, avg=21594.80, stdev=3426.38
     lat (usec): min=2933, max=38874, avg=21643.78, stdev=3426.35
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[16057], 10.00th=[17433], 20.00th=[19530],
     | 30.00th=[20317], 40.00th=[21103], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[23200], 80.00th=[23725], 90.00th=[25035], 95.00th=[26084],
     | 99.00th=[31327], 99.50th=[32375], 99.90th=[38536], 99.95th=[38536],
     | 99.99th=[39060]
   bw (  KiB/s): min= 2810, max= 3512, per=25.22%, avg=2961.67, stdev=273.30, samples=6
   iops        : min=  702, max=  878, avg=740.33, stdev=68.38, samples=6
  lat (msec)   : 4=0.12%, 10=0.59%, 20=24.10%, 50=75.20%
  cpu          : usr=0.69%, sys=4.09%, ctx=1658, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=5458: Thu Aug 22 04:23:47 2024
  write: IOPS=733, BW=2936KiB/s (3006kB/s)(10.0MiB/3488msec)
    slat (usec): min=14, max=269, avg=39.83, stdev=21.62
    clat (usec): min=3437, max=32868, avg=21631.36, stdev=3091.15
     lat (usec): min=3470, max=32938, avg=21672.31, stdev=3092.90
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[16188], 10.00th=[17433], 20.00th=[19530],
     | 30.00th=[20317], 40.00th=[21103], 50.00th=[21890], 60.00th=[22676],
     | 70.00th=[23200], 80.00th=[23725], 90.00th=[25035], 95.00th=[26084],
     | 99.00th=[30540], 99.50th=[31327], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[32900]
   bw (  KiB/s): min= 2810, max= 3328, per=25.07%, avg=2944.33, stdev=194.70, samples=6
   iops        : min=  702, max=  832, avg=736.00, stdev=48.74, samples=6
  lat (msec)   : 4=0.04%, 10=0.04%, 20=23.32%, 50=76.60%
  cpu          : usr=0.54%, sys=3.50%, ctx=836, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=5459: Thu Aug 22 04:23:47 2024
  write: IOPS=734, BW=2938KiB/s (3009kB/s)(10.0MiB/3485msec)
    slat (usec): min=14, max=268, avg=45.40, stdev=22.16
    clat (usec): min=8767, max=35525, avg=21633.70, stdev=3134.55
     lat (usec): min=8834, max=35568, avg=21680.28, stdev=3136.40
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[16188], 10.00th=[17695], 20.00th=[19530],
     | 30.00th=[20579], 40.00th=[21365], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[23200], 80.00th=[23725], 90.00th=[25035], 95.00th=[25822],
     | 99.00th=[30540], 99.50th=[31327], 99.90th=[32900], 99.95th=[32900],
     | 99.99th=[35390]
   bw (  KiB/s): min= 2808, max= 3440, per=25.10%, avg=2948.00, stdev=243.43, samples=6
   iops        : min=  702, max=  860, avg=737.00, stdev=60.86, samples=6
  lat (msec)   : 10=0.62%, 20=23.24%, 50=76.13%
  cpu          : usr=0.69%, sys=3.82%, ctx=1399, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=5460: Thu Aug 22 04:23:47 2024
  write: IOPS=734, BW=2937KiB/s (3008kB/s)(10.0MiB/3486msec)
    slat (usec): min=14, max=243, avg=48.14, stdev=23.83
    clat (usec): min=2309, max=43688, avg=21652.14, stdev=3314.21
     lat (usec): min=2379, max=43716, avg=21701.55, stdev=3315.12
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[16057], 10.00th=[17433], 20.00th=[19530],
     | 30.00th=[20579], 40.00th=[21103], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[23200], 80.00th=[23725], 90.00th=[25035], 95.00th=[26084],
     | 99.00th=[30802], 99.50th=[32113], 99.90th=[40633], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 2802, max= 3408, per=25.14%, avg=2952.33, stdev=227.02, samples=6
   iops        : min=  700, max=  852, avg=738.00, stdev=56.82, samples=6
  lat (msec)   : 4=0.12%, 10=0.08%, 20=24.14%, 50=75.66%
  cpu          : usr=0.69%, sys=4.19%, ctx=1616, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.5MiB/s (12.0MB/s), 2936KiB/s-2948KiB/s (3006kB/s-3018kB/s), io=40.0MiB (41.9MB), run=3474-3488msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=5472: Thu Aug 22 04:23:51 2024
   read: IOPS=452, BW=1811KiB/s (1855kB/s)(5048KiB/2787msec)
    slat (usec): min=673, max=2121, avg=1140.99, stdev=221.54
    clat (usec): min=1901, max=29810, avg=16285.15, stdev=2269.41
     lat (usec): min=2785, max=30709, avg=17427.91, stdev=2282.78
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13042], 10.00th=[13698], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15533], 50.00th=[16057], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19006], 95.00th=[20055],
     | 99.00th=[21627], 99.50th=[22676], 99.90th=[28967], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1712, max= 1840, per=24.88%, avg=1790.40, stdev=47.46, samples=5
   iops        : min=  428, max=  460, avg=447.60, stdev=11.87, samples=5
  write: IOPS=465, BW=1863KiB/s (1908kB/s)(5192KiB/2787msec)
    slat (usec): min=15, max=339, avg=45.44, stdev=24.03
    clat (usec): min=11436, max=28524, avg=17094.97, stdev=1840.15
     lat (usec): min=11476, max=28583, avg=17141.57, stdev=1839.59
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14484], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[22414], 99.50th=[22938], 99.90th=[25297], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1816, max= 1928, per=25.03%, avg=1875.20, stdev=41.41, samples=5
   iops        : min=  454, max=  482, avg=468.80, stdev=10.35, samples=5
  lat (msec)   : 2=0.04%, 20=94.45%, 50=5.51%
  cpu          : usr=1.15%, sys=7.14%, ctx=2631, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=5473: Thu Aug 22 04:23:51 2024
   read: IOPS=446, BW=1787KiB/s (1830kB/s)(4976KiB/2784msec)
    slat (usec): min=646, max=4463, avg=1137.35, stdev=239.89
    clat (usec): min=2596, max=31297, avg=16460.95, stdev=2342.58
     lat (usec): min=4438, max=32657, avg=17600.05, stdev=2352.35
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13042], 10.00th=[13698], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[22152], 99.50th=[23462], 99.90th=[30016], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1688, max= 1896, per=24.72%, avg=1779.20, stdev=88.65, samples=5
   iops        : min=  422, max=  474, avg=444.80, stdev=22.16, samples=5
  write: IOPS=472, BW=1891KiB/s (1936kB/s)(5264KiB/2784msec)
    slat (usec): min=14, max=600, avg=46.35, stdev=28.65
    clat (usec): min=4350, max=27519, avg=17005.18, stdev=1812.88
     lat (usec): min=4410, max=27548, avg=17052.72, stdev=1811.40
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14222], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21627], 99.50th=[22152], 99.90th=[25560], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1800, max= 1984, per=25.45%, avg=1907.20, stdev=73.45, samples=5
   iops        : min=  450, max=  496, avg=476.80, stdev=18.36, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=94.69%, 50=5.20%
  cpu          : usr=1.04%, sys=7.15%, ctx=2578, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=5474: Thu Aug 22 04:23:51 2024
   read: IOPS=449, BW=1797KiB/s (1840kB/s)(4984KiB/2774msec)
    slat (usec): min=24, max=2796, avg=1130.49, stdev=247.28
    clat (usec): min=1413, max=31273, avg=16281.74, stdev=2423.04
     lat (usec): min=1442, max=32607, avg=17413.97, stdev=2463.19
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12780], 10.00th=[13435], 20.00th=[14484],
     | 30.00th=[15008], 40.00th=[15533], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21890], 99.50th=[22938], 99.90th=[30016], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1728, max= 1904, per=25.10%, avg=1806.40, stdev=77.23, samples=5
   iops        : min=  432, max=  476, avg=451.60, stdev=19.31, samples=5
  write: IOPS=473, BW=1895KiB/s (1940kB/s)(5256KiB/2774msec)
    slat (usec): min=15, max=1158, avg=46.84, stdev=38.14
    clat (usec): min=3934, max=35015, avg=17132.37, stdev=2276.99
     lat (usec): min=3982, max=35039, avg=17180.55, stdev=2277.18
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14222], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16712], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19268], 95.00th=[20317],
     | 99.00th=[22414], 99.50th=[27657], 99.90th=[34866], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1856, max= 1944, per=25.24%, avg=1891.20, stdev=41.80, samples=5
   iops        : min=  464, max=  486, avg=472.80, stdev=10.45, samples=5
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.31%, 20=94.10%, 50=5.47%
  cpu          : usr=0.69%, sys=7.50%, ctx=2602, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=5475: Thu Aug 22 04:23:51 2024
   read: IOPS=453, BW=1815KiB/s (1858kB/s)(5060KiB/2788msec)
    slat (usec): min=598, max=4503, avg=1135.88, stdev=242.42
    clat (usec): min=3998, max=24677, avg=16328.75, stdev=2313.85
     lat (usec): min=5189, max=26137, avg=17466.35, stdev=2335.40
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12780], 10.00th=[13566], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[22414], 99.50th=[23462], 99.90th=[24249], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 1752, max= 1872, per=25.23%, avg=1816.00, stdev=49.64, samples=5
   iops        : min=  438, max=  468, avg=454.00, stdev=12.41, samples=5
  write: IOPS=464, BW=1858KiB/s (1903kB/s)(5180KiB/2788msec)
    slat (usec): min=14, max=440, avg=47.34, stdev=26.82
    clat (usec): min=5195, max=23405, avg=17031.14, stdev=1970.84
     lat (usec): min=5235, max=23439, avg=17079.63, stdev=1970.06
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[13960], 10.00th=[14615], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[21627], 99.50th=[22414], 99.90th=[23462], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 1784, max= 1912, per=24.58%, avg=1841.60, stdev=58.35, samples=5
   iops        : min=  446, max=  478, avg=460.40, stdev=14.59, samples=5
  lat (msec)   : 4=0.04%, 10=0.31%, 20=93.48%, 50=6.17%
  cpu          : usr=0.97%, sys=7.28%, ctx=2691, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7198KiB/s (7371kB/s), 1787KiB/s-1815KiB/s (1830kB/s-1858kB/s), io=19.6MiB (20.5MB), run=2774-2788msec
  WRITE: bw=7494KiB/s (7673kB/s), 1858KiB/s-1895KiB/s (1903kB/s-1940kB/s), io=20.4MiB (21.4MB), run=2774-2788msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=5481: Thu Aug 22 04:23:52 2024
   read: IOPS=10, BW=430MiB/s (451MB/s)(40.0MiB/93msec)
    slat (nsec): min=9100.2k, max=9100.2k, avg=9100162.00, stdev= 0.00
    clat (nsec): min=79161k, max=79161k, avg=79161021.00, stdev= 0.00
     lat (nsec): min=88370k, max=88370k, avg=88370102.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[79168],  5.00th=[79168], 10.00th=[79168], 20.00th=[79168],
     | 30.00th=[79168], 40.00th=[79168], 50.00th=[79168], 60.00th=[79168],
     | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168],
     | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.78%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=430MiB/s (451MB/s), 430MiB/s-430MiB/s (451MB/s-451MB/s), io=40.0MiB (41.9MB), run=93-93msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout