== sanity test 398c: run fio to test AIO ================= 03:55:02 (1732438502)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.102724 s, 408 MB/s
osc.lustre-OST0000-osc-ffff8800aa9de000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=15344: Sun Nov 24 03:55:07 2024
  write: IOPS=649, BW=2596KiB/s (2659kB/s)(10.0MiB/3944msec)
    slat (usec): min=12, max=292, avg=50.07, stdev=31.37
    clat (usec): min=4300, max=49937, avg=24487.09, stdev=4808.70
     lat (usec): min=4314, max=49955, avg=24538.34, stdev=4813.43
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15926], 10.00th=[18744], 20.00th=[21890],
     | 30.00th=[22676], 40.00th=[23200], 50.00th=[23987], 60.00th=[24773],
     | 70.00th=[25560], 80.00th=[27395], 90.00th=[31589], 95.00th=[33817],
     | 99.00th=[36963], 99.50th=[38536], 99.90th=[45351], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min= 2347, max= 3296, per=25.21%, avg=2617.57, stdev=342.53, samples=7
   iops        : min=  586, max=  824, avg=654.29, stdev=85.73, samples=7
  lat (msec)   : 10=0.16%, 20=11.17%, 50=88.67%
  cpu          : usr=0.56%, sys=3.73%, ctx=1564, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=15345: Sun Nov 24 03:55:07 2024
  write: IOPS=648, BW=2596KiB/s (2658kB/s)(10.0MiB/3945msec)
    slat (usec): min=12, max=1982, avg=46.31, stdev=51.55
    clat (usec): min=3487, max=40616, avg=24438.91, stdev=4650.15
     lat (usec): min=3545, max=40765, avg=24486.41, stdev=4654.15
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[16319], 10.00th=[18482], 20.00th=[21890],
     | 30.00th=[22676], 40.00th=[23200], 50.00th=[23987], 60.00th=[24773],
     | 70.00th=[25822], 80.00th=[27395], 90.00th=[31065], 95.00th=[33424],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 2328, max= 3208, per=25.04%, avg=2600.00, stdev=310.90, samples=7
   iops        : min=  582, max=  802, avg=650.00, stdev=77.73, samples=7
  lat (msec)   : 4=0.04%, 10=0.12%, 20=12.73%, 50=87.11%
  cpu          : usr=0.76%, sys=3.32%, ctx=1196, majf=0, minf=34
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=15346: Sun Nov 24 03:55:07 2024
  write: IOPS=650, BW=2602KiB/s (2664kB/s)(10.0MiB/3936msec)
    slat (usec): min=13, max=1877, avg=52.75, stdev=54.63
    clat (usec): min=3855, max=40597, avg=24403.17, stdev=4788.42
     lat (usec): min=3920, max=40651, avg=24457.24, stdev=4790.69
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[16057], 10.00th=[18744], 20.00th=[21890],
     | 30.00th=[22676], 40.00th=[23200], 50.00th=[23987], 60.00th=[24773],
     | 70.00th=[25560], 80.00th=[27395], 90.00th=[31327], 95.00th=[33162],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 2312, max= 3336, per=25.20%, avg=2616.00, stdev=355.95, samples=7
   iops        : min=  578, max=  834, avg=654.00, stdev=88.99, samples=7
  lat (msec)   : 4=0.04%, 10=0.66%, 20=11.13%, 50=88.16%
  cpu          : usr=0.58%, sys=3.91%, ctx=1732, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=15347: Sun Nov 24 03:55:07 2024
  write: IOPS=650, BW=2604KiB/s (2666kB/s)(10.0MiB/3933msec)
    slat (usec): min=12, max=411, avg=53.91, stdev=34.35
    clat (usec): min=7812, max=46671, avg=24430.26, stdev=4883.45
     lat (usec): min=7854, max=46727, avg=24485.35, stdev=4883.30
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[15795], 10.00th=[18744], 20.00th=[21890],
     | 30.00th=[22676], 40.00th=[23200], 50.00th=[23987], 60.00th=[24773],
     | 70.00th=[25822], 80.00th=[27395], 90.00th=[31589], 95.00th=[33424],
     | 99.00th=[36963], 99.50th=[39060], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46924]
   bw (  KiB/s): min= 2312, max= 3360, per=25.27%, avg=2624.00, stdev=364.39, samples=7
   iops        : min=  578, max=  840, avg=656.00, stdev=91.10, samples=7
  lat (msec)   : 10=0.35%, 20=11.56%, 50=88.09%
  cpu          : usr=0.86%, sys=3.61%, ctx=1774, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.1MiB/s (10.6MB/s), 2596KiB/s-2604KiB/s (2658kB/s-2666kB/s), io=40.0MiB (41.9MB), run=3933-3945msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=15360: Sun Nov 24 03:55:11 2024
   read: IOPS=408, BW=1635KiB/s (1674kB/s)(5048KiB/3087msec)
    slat (usec): min=17, max=2622, avg=1278.58, stdev=319.92
    clat (usec): min=3000, max=32026, avg=18272.66, stdev=3169.76
     lat (usec): min=3037, max=33172, avg=19553.24, stdev=3286.74
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13829], 10.00th=[14746], 20.00th=[15795],
     | 30.00th=[16712], 40.00th=[17433], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[20579], 90.00th=[22152], 95.00th=[23987],
     | 99.00th=[27132], 99.50th=[28181], 99.90th=[30540], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1512, max= 1872, per=25.31%, avg=1632.67, stdev=137.51, samples=6
   iops        : min=  378, max=  468, avg=408.17, stdev=34.38, samples=6
  write: IOPS=420, BW=1682KiB/s (1722kB/s)(5192KiB/3087msec)
    slat (usec): min=13, max=1593, avg=56.01, stdev=59.57
    clat (usec): min=2289, max=30550, avg=18879.15, stdev=2840.31
     lat (usec): min=2346, max=30604, avg=18936.33, stdev=2844.76
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14877], 10.00th=[15795], 20.00th=[16909],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[22676], 95.00th=[23987],
     | 99.00th=[27657], 99.50th=[28705], 99.90th=[29492], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1588, max= 1768, per=25.05%, avg=1682.00, stdev=67.21, samples=6
   iops        : min=  397, max=  442, avg=420.50, stdev=16.80, samples=6
  lat (msec)   : 4=0.16%, 10=0.31%, 20=71.80%, 50=27.73%
  cpu          : usr=0.75%, sys=7.74%, ctx=2909, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=15361: Sun Nov 24 03:55:11 2024
   read: IOPS=399, BW=1599KiB/s (1638kB/s)(4976KiB/3111msec)
    slat (usec): min=710, max=2555, avg=1280.87, stdev=282.26
    clat (usec): min=4933, max=29520, avg=18272.16, stdev=3037.14
     lat (usec): min=6195, max=31527, avg=19555.07, stdev=3118.28
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13698], 10.00th=[14484], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[22152], 95.00th=[23462],
     | 99.00th=[26870], 99.50th=[28181], 99.90th=[28705], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1456, max= 1792, per=24.65%, avg=1590.00, stdev=118.11, samples=6
   iops        : min=  364, max=  448, avg=397.50, stdev=29.53, samples=6
  write: IOPS=423, BW=1692KiB/s (1733kB/s)(5264KiB/3111msec)
    slat (usec): min=14, max=415, avg=53.90, stdev=32.01
    clat (usec): min=9988, max=28385, avg=18942.68, stdev=2624.26
     lat (usec): min=10019, max=28443, avg=18997.71, stdev=2628.86
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15008], 10.00th=[15926], 20.00th=[16909],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[21103], 90.00th=[22414], 95.00th=[23725],
     | 99.00th=[26084], 99.50th=[26870], 99.90th=[28181], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1544, max= 1872, per=25.11%, avg=1686.00, stdev=111.53, samples=6
   iops        : min=  386, max=  468, avg=421.50, stdev=27.88, samples=6
  lat (msec)   : 10=0.12%, 20=71.29%, 50=28.59%
  cpu          : usr=0.74%, sys=7.68%, ctx=2974, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=15362: Sun Nov 24 03:55:11 2024
   read: IOPS=401, BW=1607KiB/s (1645kB/s)(4984KiB/3102msec)
    slat (usec): min=749, max=3665, avg=1297.37, stdev=306.55
    clat (usec): min=9928, max=30868, avg=18204.48, stdev=2891.87
     lat (usec): min=11382, max=32662, avg=19503.94, stdev=2985.05
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13960], 10.00th=[14746], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19268], 80.00th=[20317], 90.00th=[22152], 95.00th=[23462],
     | 99.00th=[26608], 99.50th=[27657], 99.90th=[28967], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1448, max= 1848, per=24.97%, avg=1610.67, stdev=148.78, samples=6
   iops        : min=  362, max=  462, avg=402.67, stdev=37.19, samples=6
  write: IOPS=423, BW=1694KiB/s (1735kB/s)(5256KiB/3102msec)
    slat (usec): min=12, max=318, avg=54.13, stdev=29.71
    clat (usec): min=12450, max=31023, avg=18955.40, stdev=2738.47
     lat (usec): min=12516, max=31343, avg=19010.67, stdev=2741.97
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15139], 10.00th=[15926], 20.00th=[16712],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[21103], 90.00th=[22414], 95.00th=[23987],
     | 99.00th=[26870], 99.50th=[28443], 99.90th=[31065], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1616, max= 1832, per=25.08%, avg=1684.00, stdev=79.48, samples=6
   iops        : min=  404, max=  458, avg=421.00, stdev=19.87, samples=6
  lat (msec)   : 10=0.04%, 20=73.40%, 50=26.56%
  cpu          : usr=0.71%, sys=7.80%, ctx=2930, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=15363: Sun Nov 24 03:55:11 2024
   read: IOPS=406, BW=1627KiB/s (1666kB/s)(5060KiB/3110msec)
    slat (usec): min=666, max=2904, avg=1286.81, stdev=299.83
    clat (usec): min=5905, max=29308, avg=18298.33, stdev=3042.07
     lat (usec): min=7805, max=30826, avg=19587.05, stdev=3123.43
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13829], 10.00th=[14615], 20.00th=[15664],
     | 30.00th=[16450], 40.00th=[17171], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[20579], 90.00th=[22414], 95.00th=[23725],
     | 99.00th=[27132], 99.50th=[28181], 99.90th=[28967], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1496, max= 1792, per=25.19%, avg=1624.67, stdev=104.01, samples=6
   iops        : min=  374, max=  448, avg=406.17, stdev=26.00, samples=6
  write: IOPS=416, BW=1666KiB/s (1706kB/s)(5180KiB/3110msec)
    slat (usec): min=13, max=1604, avg=58.01, stdev=65.70
    clat (usec): min=4738, max=34969, avg=18910.40, stdev=2749.58
     lat (usec): min=4809, max=35064, avg=18969.58, stdev=2755.97
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15008], 10.00th=[15795], 20.00th=[16712],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[22414], 95.00th=[23725],
     | 99.00th=[26870], 99.50th=[27395], 99.90th=[34866], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1568, max= 1784, per=24.73%, avg=1660.67, stdev=99.81, samples=6
   iops        : min=  392, max=  446, avg=415.17, stdev=24.95, samples=6
  lat (msec)   : 10=0.08%, 20=72.38%, 50=27.54%
  cpu          : usr=1.03%, sys=7.43%, ctx=3007, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6451KiB/s (6605kB/s), 1599KiB/s-1635KiB/s (1638kB/s-1674kB/s), io=19.6MiB (20.5MB), run=3087-3111msec
  WRITE: bw=6716KiB/s (6877kB/s), 1666KiB/s-1694KiB/s (1706kB/s-1735kB/s), io=20.4MiB (21.4MB), run=3087-3111msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=15369: Sun Nov 24 03:55:12 2024
   read: IOPS=10, BW=404MiB/s (424MB/s)(40.0MiB/99msec)
    slat (nsec): min=5274.1k, max=5274.1k, avg=5274091.00, stdev= 0.00
    clat (nsec): min=89203k, max=89203k, avg=89203477.00, stdev= 0.00
     lat (nsec): min=94487k, max=94487k, avg=94487169.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[89654],  5.00th=[89654], 10.00th=[89654], 20.00th=[89654],
     | 30.00th=[89654], 40.00th=[89654], 50.00th=[89654], 60.00th=[89654],
     | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654],
     | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.10%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=40.0MiB (41.9MB), run=99-99msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout