== sanity test 398c: run fio to test AIO ================= 23:41:49 (1708836109)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.081414 s, 515 MB/s
osc.lustre-OST0000-osc-ffff8800ab7f5000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=9170: Sat Feb 24 23:41:53 2024
  write: IOPS=750, BW=3000KiB/s (3072kB/s)(10.0MiB/3413msec)
    slat (usec): min=12, max=285, avg=45.76, stdev=25.13
    clat (usec): min=4186, max=34153, avg=21173.61, stdev=2880.81
     lat (usec): min=4301, max=34230, avg=21220.77, stdev=2880.57
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[16581], 10.00th=[17957], 20.00th=[19792],
     | 30.00th=[20317], 40.00th=[20841], 50.00th=[21365], 60.00th=[21890],
     | 70.00th=[22414], 80.00th=[22938], 90.00th=[24249], 95.00th=[25035],
     | 99.00th=[28967], 99.50th=[32637], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[34341]
   bw (  KiB/s): min= 2688, max= 3353, per=25.02%, avg=3001.50, stdev=221.00, samples=6
   iops        : min=  672, max=  838, avg=750.33, stdev=55.17, samples=6
  lat (msec)   : 10=0.08%, 20=26.02%, 50=73.91%
  cpu          : usr=0.76%, sys=4.16%, ctx=1346, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9171: Sat Feb 24 23:41:53 2024
  write: IOPS=754, BW=3018KiB/s (3090kB/s)(10.0MiB/3393msec)
    slat (usec): min=12, max=319, avg=46.37, stdev=25.85
    clat (usec): min=4500, max=36544, avg=21057.06, stdev=3141.70
     lat (usec): min=4552, max=36593, avg=21104.85, stdev=3142.07
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[16319], 10.00th=[17695], 20.00th=[19530],
     | 30.00th=[20317], 40.00th=[20841], 50.00th=[21365], 60.00th=[21890],
     | 70.00th=[22414], 80.00th=[22938], 90.00th=[24249], 95.00th=[25035],
     | 99.00th=[29492], 99.50th=[32375], 99.90th=[36439], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 2688, max= 3440, per=25.24%, avg=3028.00, stdev=254.53, samples=6
   iops        : min=  672, max=  860, avg=757.00, stdev=63.63, samples=6
  lat (msec)   : 10=0.94%, 20=25.27%, 50=73.79%
  cpu          : usr=0.74%, sys=4.22%, ctx=1473, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9172: Sat Feb 24 23:41:53 2024
  write: IOPS=750, BW=3003KiB/s (3075kB/s)(10.0MiB/3410msec)
    slat (usec): min=13, max=507, avg=49.87, stdev=29.56
    clat (usec): min=5160, max=37873, avg=21146.19, stdev=2876.28
     lat (usec): min=5220, max=37897, avg=21197.37, stdev=2875.28
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[16581], 10.00th=[17957], 20.00th=[19530],
     | 30.00th=[20317], 40.00th=[20841], 50.00th=[21365], 60.00th=[21890],
     | 70.00th=[22414], 80.00th=[22938], 90.00th=[23987], 95.00th=[25035],
     | 99.00th=[28705], 99.50th=[33424], 99.90th=[34341], 99.95th=[36439],
     | 99.99th=[38011]
   bw (  KiB/s): min= 2696, max= 3337, per=25.09%, avg=3009.50, stdev=215.11, samples=6
   iops        : min=  674, max=  834, avg=752.33, stdev=53.70, samples=6
  lat (msec)   : 10=0.12%, 20=27.15%, 50=72.73%
  cpu          : usr=0.67%, sys=4.52%, ctx=1706, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9173: Sat Feb 24 23:41:53 2024
  write: IOPS=749, BW=2999KiB/s (3071kB/s)(10.0MiB/3414msec)
    slat (usec): min=12, max=274, avg=44.66, stdev=24.76
    clat (usec): min=4722, max=36554, avg=21179.38, stdev=2853.72
     lat (usec): min=4773, max=36662, avg=21225.38, stdev=2856.20
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[16712], 10.00th=[17957], 20.00th=[19530],
     | 30.00th=[20317], 40.00th=[20841], 50.00th=[21365], 60.00th=[21890],
     | 70.00th=[22414], 80.00th=[22938], 90.00th=[24249], 95.00th=[25035],
     | 99.00th=[28181], 99.50th=[33424], 99.90th=[34341], 99.95th=[34866],
     | 99.99th=[36439]
   bw (  KiB/s): min= 2688, max= 3321, per=25.03%, avg=3002.83, stdev=208.59, samples=6
   iops        : min=  672, max=  830, avg=750.67, stdev=52.07, samples=6
  lat (msec)   : 10=0.04%, 20=27.07%, 50=72.89%
  cpu          : usr=0.85%, sys=3.98%, ctx=1253, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.7MiB/s (12.3MB/s), 2999KiB/s-3018KiB/s (3071kB/s-3090kB/s), io=40.0MiB (41.9MB), run=3393-3414msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=9186: Sat Feb 24 23:41:57 2024
   read: IOPS=412, BW=1652KiB/s (1691kB/s)(5048KiB/3056msec)
    slat (usec): min=743, max=2415, avg=1227.66, stdev=245.00
    clat (usec): min=1617, max=28776, avg=17955.78, stdev=2630.38
     lat (usec): min=3038, max=29765, avg=19185.31, stdev=2649.72
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[14091], 10.00th=[14877], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[20055], 90.00th=[21103], 95.00th=[22152],
     | 99.00th=[24511], 99.50th=[25560], 99.90th=[28443], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1560, max= 1844, per=25.07%, avg=1644.67, stdev=104.32, samples=6
   iops        : min=  390, max=  461, avg=411.17, stdev=26.08, samples=6
  write: IOPS=424, BW=1699KiB/s (1740kB/s)(5192KiB/3056msec)
    slat (usec): min=13, max=447, avg=44.32, stdev=24.34
    clat (usec): min=11411, max=28072, avg=18766.16, stdev=2145.60
     lat (usec): min=11453, max=28091, avg=18811.65, stdev=2143.95
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15401], 10.00th=[16057], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[19792], 80.00th=[20317], 90.00th=[21365], 95.00th=[22414],
     | 99.00th=[25035], 99.50th=[26608], 99.90th=[27919], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1592, max= 1792, per=24.85%, avg=1696.67, stdev=66.76, samples=6
   iops        : min=  398, max=  448, avg=424.17, stdev=16.69, samples=6
  lat (msec)   : 2=0.04%, 4=0.08%, 20=76.80%, 50=23.09%
  cpu          : usr=0.65%, sys=7.10%, ctx=2535, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9187: Sat Feb 24 23:41:57 2024
   read: IOPS=407, BW=1631KiB/s (1670kB/s)(4976KiB/3051msec)
    slat (usec): min=748, max=2524, avg=1213.04, stdev=246.40
    clat (usec): min=10834, max=27994, avg=18012.85, stdev=2574.61
     lat (usec): min=12231, max=29621, avg=19227.80, stdev=2592.42
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[14091], 10.00th=[14746], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[20317], 90.00th=[21627], 95.00th=[22152],
     | 99.00th=[24511], 99.50th=[25560], 99.90th=[26608], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1520, max= 1740, per=24.77%, avg=1624.67, stdev=81.13, samples=6
   iops        : min=  380, max=  435, avg=406.17, stdev=20.28, samples=6
  write: IOPS=431, BW=1725KiB/s (1767kB/s)(5264KiB/3051msec)
    slat (usec): min=14, max=154, avg=43.92, stdev=17.88
    clat (usec): min=12495, max=27956, avg=18715.92, stdev=2049.75
     lat (usec): min=12586, max=27986, avg=18761.11, stdev=2049.20
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15401], 10.00th=[16188], 20.00th=[16909],
     | 30.00th=[17433], 40.00th=[18220], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21365], 95.00th=[22152],
     | 99.00th=[23462], 99.50th=[23987], 99.90th=[27395], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1640, max= 1864, per=25.31%, avg=1728.67, stdev=91.65, samples=6
   iops        : min=  410, max=  466, avg=432.17, stdev=22.91, samples=6
  lat (msec)   : 20=75.74%, 50=24.26%
  cpu          : usr=0.79%, sys=6.85%, ctx=2425, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9188: Sat Feb 24 23:41:57 2024
   read: IOPS=409, BW=1639KiB/s (1679kB/s)(4984KiB/3040msec)
    slat (usec): min=15, max=2421, avg=1214.15, stdev=262.62
    clat (usec): min=1488, max=27138, avg=17954.38, stdev=2740.37
     lat (usec): min=1540, max=28427, avg=19170.52, stdev=2783.23
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13960], 10.00th=[14746], 20.00th=[15664],
     | 30.00th=[16450], 40.00th=[17171], 50.00th=[17695], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[20317], 90.00th=[21627], 95.00th=[22414],
     | 99.00th=[24511], 99.50th=[25035], 99.90th=[27132], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1424, max= 1760, per=25.03%, avg=1642.00, stdev=122.37, samples=6
   iops        : min=  356, max=  440, avg=410.50, stdev=30.59, samples=6
  write: IOPS=432, BW=1729KiB/s (1770kB/s)(5256KiB/3040msec)
    slat (usec): min=14, max=1711, avg=46.39, stdev=52.44
    clat (usec): min=3332, max=35299, avg=18704.25, stdev=2368.66
     lat (usec): min=3375, max=35382, avg=18751.89, stdev=2367.42
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[15139], 10.00th=[15926], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21365], 95.00th=[22152],
     | 99.00th=[23987], 99.50th=[25035], 99.90th=[34866], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1560, max= 1848, per=25.24%, avg=1723.33, stdev=119.85, samples=6
   iops        : min=  390, max=  462, avg=430.83, stdev=29.96, samples=6
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.16%, 20=74.69%, 50=25.04%
  cpu          : usr=0.95%, sys=6.68%, ctx=2434, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9189: Sat Feb 24 23:41:57 2024
   read: IOPS=413, BW=1654KiB/s (1694kB/s)(5060KiB/3059msec)
    slat (usec): min=714, max=2265, avg=1217.21, stdev=252.30
    clat (usec): min=3091, max=26614, avg=18095.36, stdev=2675.80
     lat (usec): min=4426, max=28151, avg=19314.49, stdev=2692.88
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14091], 10.00th=[14877], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[20317], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[24773], 99.50th=[25297], 99.90th=[26346], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min= 1584, max= 1804, per=25.19%, avg=1652.67, stdev=79.86, samples=6
   iops        : min=  396, max=  451, avg=413.17, stdev=19.96, samples=6
  write: IOPS=423, BW=1693KiB/s (1734kB/s)(5180KiB/3059msec)
    slat (usec): min=13, max=541, avg=43.12, stdev=23.27
    clat (usec): min=12549, max=28999, avg=18724.62, stdev=2147.39
     lat (usec): min=12583, max=29055, avg=18768.97, stdev=2146.55
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15401], 10.00th=[15926], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21365], 95.00th=[22414],
     | 99.00th=[24249], 99.50th=[24773], 99.90th=[27132], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1576, max= 1768, per=24.69%, avg=1686.00, stdev=80.31, samples=6
   iops        : min=  394, max=  442, avg=421.50, stdev=20.08, samples=6
  lat (msec)   : 4=0.08%, 10=0.04%, 20=74.88%, 50=25.00%
  cpu          : usr=0.92%, sys=6.64%, ctx=2428, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6560KiB/s (6718kB/s), 1631KiB/s-1654KiB/s (1670kB/s-1694kB/s), io=19.6MiB (20.5MB), run=3040-3059msec
  WRITE: bw=6830KiB/s (6994kB/s), 1693KiB/s-1729KiB/s (1734kB/s-1770kB/s), io=20.4MiB (21.4MB), run=3040-3059msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=9195: Sat Feb 24 23:41:59 2024
   read: IOPS=9, BW=392MiB/s (411MB/s)(40.0MiB/102msec)
    slat (nsec): min=5742.2k, max=5742.2k, avg=5742202.00, stdev= 0.00
    clat (nsec): min=90750k, max=90750k, avg=90749856.00, stdev= 0.00
     lat (nsec): min=96502k, max=96502k, avg=96501775.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[90702],  5.00th=[90702], 10.00th=[90702], 20.00th=[90702],
     | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702],
     | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702],
     | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.93%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=392MiB/s (411MB/s), 392MiB/s-392MiB/s (411MB/s-411MB/s), io=40.0MiB (41.9MB), run=102-102msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout