== sanity test 398c: run fio to test AIO ================= 19:23:09 (1712272989)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0829772 s, 505 MB/s
osc.lustre-OST0000-osc-ffff88008e5a5000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=16474: Thu Apr  4 19:23:14 2024
  write: IOPS=715, BW=2862KiB/s (2931kB/s)(10.0MiB/3578msec)
    slat (usec): min=13, max=441, avg=59.19, stdev=37.14
    clat (usec): min=5942, max=39288, avg=22225.50, stdev=3355.18
     lat (usec): min=6017, max=39391, avg=22286.05, stdev=3357.58
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[16188], 10.00th=[18220], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23725], 80.00th=[24511], 90.00th=[25822], 95.00th=[26870],
     | 99.00th=[32375], 99.50th=[33817], 99.90th=[38011], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 2688, max= 3336, per=25.02%, avg=2859.71, stdev=223.85, samples=7
   iops        : min=  672, max=  834, avg=714.86, stdev=55.94, samples=7
  lat (msec)   : 10=0.08%, 20=19.38%, 50=80.55%
  cpu          : usr=0.78%, sys=4.53%, ctx=1886, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16475: Thu Apr  4 19:23:14 2024
  write: IOPS=716, BW=2866KiB/s (2935kB/s)(10.0MiB/3573msec)
    slat (usec): min=11, max=410, avg=55.14, stdev=37.88
    clat (usec): min=3328, max=46453, avg=22200.96, stdev=3558.62
     lat (usec): min=3384, max=46486, avg=22257.33, stdev=3560.73
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[15664], 10.00th=[17957], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21890], 50.00th=[22414], 60.00th=[23200],
     | 70.00th=[23725], 80.00th=[24511], 90.00th=[25822], 95.00th=[26870],
     | 99.00th=[33424], 99.50th=[33817], 99.90th=[38536], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min= 2696, max= 3416, per=25.08%, avg=2866.29, stdev=251.19, samples=7
   iops        : min=  674, max=  854, avg=716.57, stdev=62.80, samples=7
  lat (msec)   : 4=0.04%, 10=0.47%, 20=19.22%, 50=80.27%
  cpu          : usr=0.78%, sys=4.26%, ctx=1506, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16476: Thu Apr  4 19:23:14 2024
  write: IOPS=714, BW=2857KiB/s (2926kB/s)(10.0MiB/3584msec)
    slat (usec): min=13, max=486, avg=57.26, stdev=39.04
    clat (usec): min=5915, max=39421, avg=22260.93, stdev=3411.87
     lat (usec): min=5974, max=39470, avg=22319.54, stdev=3413.31
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[16057], 10.00th=[18220], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22414], 60.00th=[23200],
     | 70.00th=[23725], 80.00th=[24511], 90.00th=[25822], 95.00th=[26870],
     | 99.00th=[32900], 99.50th=[36963], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 2712, max= 3328, per=25.03%, avg=2860.57, stdev=218.01, samples=7
   iops        : min=  678, max=  832, avg=715.14, stdev=54.50, samples=7
  lat (msec)   : 10=0.16%, 20=19.41%, 50=80.43%
  cpu          : usr=0.89%, sys=4.21%, ctx=1772, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16477: Thu Apr  4 19:23:14 2024
  write: IOPS=714, BW=2859KiB/s (2927kB/s)(10.0MiB/3582msec)
    slat (usec): min=11, max=339, avg=58.41, stdev=38.22
    clat (usec): min=10540, max=37893, avg=22244.67, stdev=3366.72
     lat (usec): min=10576, max=37920, avg=22304.41, stdev=3370.78
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[16057], 10.00th=[18220], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22676], 60.00th=[22938],
     | 70.00th=[23725], 80.00th=[24511], 90.00th=[25822], 95.00th=[26870],
     | 99.00th=[32900], 99.50th=[35390], 99.90th=[38011], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 2688, max= 3328, per=25.00%, avg=2857.14, stdev=220.86, samples=7
   iops        : min=  672, max=  832, avg=714.29, stdev=55.21, samples=7
  lat (msec)   : 20=19.14%, 50=80.86%
  cpu          : usr=1.09%, sys=4.08%, ctx=1852, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.2MiB/s (11.7MB/s), 2857KiB/s-2866KiB/s (2926kB/s-2935kB/s), io=40.0MiB (41.9MB), run=3573-3584msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16490: Thu Apr  4 19:23:18 2024
   read: IOPS=405, BW=1621KiB/s (1660kB/s)(5048KiB/3114msec)
    slat (usec): min=20, max=3320, avg=1210.35, stdev=290.66
    clat (usec): min=1551, max=42286, avg=18376.30, stdev=3098.70
     lat (usec): min=2084, max=44333, avg=19588.52, stdev=3165.74
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14222], 10.00th=[15139], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17695], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[26084], 99.50th=[27395], 99.90th=[40633], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1556, max= 1712, per=25.44%, avg=1631.33, stdev=53.45, samples=6
   iops        : min=  389, max=  428, avg=407.83, stdev=13.36, samples=6
  write: IOPS=416, BW=1667KiB/s (1707kB/s)(5192KiB/3114msec)
    slat (usec): min=14, max=442, avg=47.68, stdev=26.79
    clat (usec): min=3467, max=34335, avg=19199.83, stdev=2424.20
     lat (usec): min=3508, max=34418, avg=19248.69, stdev=2424.36
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15795], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20317], 80.00th=[20841], 90.00th=[22152], 95.00th=[23200],
     | 99.00th=[25560], 99.50th=[26870], 99.90th=[31065], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1536, max= 1752, per=25.13%, avg=1678.00, stdev=82.04, samples=6
   iops        : min=  384, max=  438, avg=419.50, stdev=20.51, samples=6
  lat (msec)   : 2=0.04%, 4=0.20%, 10=0.35%, 20=69.41%, 50=30.00%
  cpu          : usr=1.06%, sys=6.65%, ctx=2371, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16491: Thu Apr  4 19:23:18 2024
   read: IOPS=397, BW=1591KiB/s (1629kB/s)(4976KiB/3127msec)
    slat (usec): min=729, max=3387, avg=1224.25, stdev=259.78
    clat (usec): min=2958, max=36303, avg=18533.26, stdev=2705.23
     lat (usec): min=4876, max=37523, avg=19759.40, stdev=2730.52
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14484], 10.00th=[15270], 20.00th=[16319],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21890], 95.00th=[23200],
     | 99.00th=[25560], 99.50th=[26346], 99.90th=[30278], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 1472, max= 1696, per=24.85%, avg=1593.33, stdev=84.27, samples=6
   iops        : min=  368, max=  424, avg=398.33, stdev=21.07, samples=6
  write: IOPS=420, BW=1683KiB/s (1724kB/s)(5264KiB/3127msec)
    slat (usec): min=13, max=1160, avg=47.18, stdev=39.44
    clat (usec): min=11108, max=33261, avg=19126.05, stdev=2205.85
     lat (usec): min=11147, max=33343, avg=19174.34, stdev=2205.29
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[15926], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18482], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20579], 90.00th=[21890], 95.00th=[23200],
     | 99.00th=[25297], 99.50th=[26608], 99.90th=[28705], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 1616, max= 1824, per=25.38%, avg=1694.67, stdev=76.30, samples=6
   iops        : min=  404, max=  456, avg=423.67, stdev=19.08, samples=6
  lat (msec)   : 4=0.04%, 10=0.04%, 20=70.82%, 50=29.10%
  cpu          : usr=0.86%, sys=6.75%, ctx=2309, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16492: Thu Apr  4 19:23:18 2024
   read: IOPS=398, BW=1593KiB/s (1631kB/s)(4984KiB/3129msec)
    slat (usec): min=665, max=3292, avg=1216.09, stdev=259.31
    clat (usec): min=8174, max=29014, avg=18445.39, stdev=2672.87
     lat (usec): min=9533, max=31171, avg=19663.20, stdev=2708.55
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14615], 10.00th=[15270], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17695], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21890], 95.00th=[22938],
     | 99.00th=[25560], 99.50th=[26870], 99.90th=[28967], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1448, max= 1744, per=25.00%, avg=1603.33, stdev=100.13, samples=6
   iops        : min=  362, max=  436, avg=400.83, stdev=25.03, samples=6
  write: IOPS=419, BW=1680KiB/s (1720kB/s)(5256KiB/3129msec)
    slat (usec): min=12, max=1068, avg=47.90, stdev=38.10
    clat (usec): min=9765, max=28325, avg=19210.28, stdev=2292.13
     lat (usec): min=9810, max=28365, avg=19259.30, stdev=2292.09
    clat percentiles (usec):
     |  1.00th=[14484],  5.00th=[15795], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18482], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[22152], 95.00th=[23200],
     | 99.00th=[25822], 99.50th=[27919], 99.90th=[27919], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1560, max= 1744, per=25.13%, avg=1678.00, stdev=70.19, samples=6
   iops        : min=  390, max=  436, avg=419.50, stdev=17.55, samples=6
  lat (msec)   : 10=0.16%, 20=70.04%, 50=29.80%
  cpu          : usr=1.12%, sys=6.43%, ctx=2317, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16493: Thu Apr  4 19:23:18 2024
   read: IOPS=404, BW=1618KiB/s (1656kB/s)(5060KiB/3128msec)
    slat (usec): min=708, max=3374, avg=1225.26, stdev=273.23
    clat (usec): min=3148, max=28715, avg=18441.19, stdev=2651.61
     lat (usec): min=4244, max=29850, avg=19668.38, stdev=2689.06
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14484], 10.00th=[15270], 20.00th=[16057],
     | 30.00th=[16909], 40.00th=[17695], 50.00th=[18482], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[20579], 90.00th=[21627], 95.00th=[22938],
     | 99.00th=[25560], 99.50th=[26084], 99.90th=[27657], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1568, max= 1688, per=25.33%, avg=1624.67, stdev=48.29, samples=6
   iops        : min=  392, max=  422, avg=406.17, stdev=12.07, samples=6
  write: IOPS=414, BW=1656KiB/s (1696kB/s)(5180KiB/3128msec)
    slat (usec): min=12, max=1132, avg=47.75, stdev=44.28
    clat (usec): min=12029, max=30193, avg=19181.98, stdev=2212.54
     lat (usec): min=12069, max=30231, avg=19230.93, stdev=2212.36
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[16057], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18482], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[21890], 95.00th=[22938],
     | 99.00th=[25035], 99.50th=[25822], 99.90th=[30278], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1544, max= 1792, per=24.88%, avg=1660.67, stdev=93.45, samples=6
   iops        : min=  386, max=  448, avg=415.17, stdev=23.36, samples=6
  lat (msec)   : 4=0.04%, 10=0.08%, 20=69.84%, 50=30.04%
  cpu          : usr=1.44%, sys=6.14%, ctx=2380, majf=0, minf=35
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6414KiB/s (6567kB/s), 1591KiB/s-1621KiB/s (1629kB/s-1660kB/s), io=19.6MiB (20.5MB), run=3114-3129msec
  WRITE: bw=6677KiB/s (6837kB/s), 1656KiB/s-1683KiB/s (1696kB/s-1724kB/s), io=20.4MiB (21.4MB), run=3114-3129msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=16499: Thu Apr  4 19:23:19 2024
   read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec)
    slat (nsec): min=7872.5k, max=7872.5k, avg=7872510.00, stdev= 0.00
    clat (nsec): min=78389k, max=78389k, avg=78389230.00, stdev= 0.00
     lat (nsec): min=86286k, max=86286k, avg=86286428.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[78119],  5.00th=[78119], 10.00th=[78119], 20.00th=[78119],
     | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[78119],
     | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119],
     | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.11%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout