== sanity test 398c: run fio to test AIO ================= 06:55:13 (1731758113)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0936183 s, 448 MB/s
osc.lustre-OST0000-osc-ffff88012a441000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=13145: Sat Nov 16 06:55:18 2024
  write: IOPS=733, BW=2935KiB/s (3005kB/s)(10.0MiB/3489msec)
    slat (usec): min=14, max=335, avg=48.28, stdev=29.10
    clat (usec): min=2468, max=41962, avg=21668.65, stdev=2897.32
     lat (usec): min=2535, max=41997, avg=21718.21, stdev=2898.80
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[17695], 10.00th=[18744], 20.00th=[19792],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[24249], 95.00th=[25035],
     | 99.00th=[30278], 99.50th=[32375], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2824, max= 3249, per=25.21%, avg=2948.17, stdev=176.40, samples=6
   iops        : min=  706, max=  812, avg=737.00, stdev=44.01, samples=6
  lat (msec)   : 4=0.08%, 10=0.59%, 20=19.92%, 50=79.41%
  cpu          : usr=0.75%, sys=3.64%, ctx=1817, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13146: Sat Nov 16 06:55:18 2024
  write: IOPS=732, BW=2930KiB/s (3000kB/s)(10.0MiB/3495msec)
    slat (usec): min=13, max=352, avg=48.73, stdev=30.94
    clat (usec): min=4920, max=43378, avg=21675.91, stdev=2587.01
     lat (usec): min=4986, max=43397, avg=21725.90, stdev=2591.03
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[17695], 10.00th=[18482], 20.00th=[19792],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[24249], 95.00th=[24773],
     | 99.00th=[28443], 99.50th=[30540], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[43254]
   bw (  KiB/s): min= 2832, max= 3193, per=25.10%, avg=2936.17, stdev=159.28, samples=6
   iops        : min=  708, max=  798, avg=734.00, stdev=39.74, samples=6
  lat (msec)   : 10=0.08%, 20=20.66%, 50=79.26%
  cpu          : usr=0.74%, sys=3.69%, ctx=1697, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13147: Sat Nov 16 06:55:18 2024
  write: IOPS=731, BW=2926KiB/s (2996kB/s)(10.0MiB/3500msec)
    slat (usec): min=14, max=319, avg=51.86, stdev=31.98
    clat (usec): min=3109, max=35925, avg=21724.91, stdev=2458.11
     lat (usec): min=3185, max=35989, avg=21778.20, stdev=2456.39
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[17695], 10.00th=[18482], 20.00th=[19792],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[24249], 95.00th=[24773],
     | 99.00th=[28705], 99.50th=[32113], 99.90th=[33424], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2824, max= 3169, per=25.06%, avg=2930.83, stdev=150.21, samples=6
   iops        : min=  706, max=  792, avg=732.67, stdev=37.47, samples=6
  lat (msec)   : 4=0.04%, 20=20.66%, 50=79.30%
  cpu          : usr=0.80%, sys=4.09%, ctx=1825, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13148: Sat Nov 16 06:55:18 2024
  write: IOPS=731, BW=2924KiB/s (2994kB/s)(10.0MiB/3502msec)
    slat (usec): min=13, max=503, avg=49.86, stdev=31.79
    clat (usec): min=2393, max=32366, avg=21705.13, stdev=2408.73
     lat (usec): min=2517, max=32400, avg=21756.21, stdev=2410.31
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[17433], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[24249], 95.00th=[24773],
     | 99.00th=[27919], 99.50th=[30540], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 2816, max= 3080, per=24.86%, avg=2907.71, stdev=116.16, samples=7
   iops        : min=  704, max=  770, avg=726.86, stdev=29.09, samples=7
  lat (msec)   : 4=0.04%, 10=0.20%, 20=18.95%, 50=80.82%
  cpu          : usr=0.57%, sys=3.97%, ctx=1779, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.4MiB/s (11.0MB/s), 2924KiB/s-2935KiB/s (2994kB/s-3005kB/s), io=40.0MiB (41.9MB), run=3489-3502msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=13160: Sat Nov 16 06:55:22 2024
   read: IOPS=420, BW=1683KiB/s (1724kB/s)(5048KiB/2999msec)
    slat (usec): min=691, max=3119, avg=1206.37, stdev=247.68
    clat (usec): min=8697, max=33628, avg=17740.41, stdev=2876.83
     lat (usec): min=9894, max=34932, avg=18948.64, stdev=2912.22
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13435], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[21365], 95.00th=[22938],
     | 99.00th=[24773], 99.50th=[25560], 99.90th=[32637], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1576, max= 1728, per=24.86%, avg=1660.80, stdev=59.22, samples=5
   iops        : min=  394, max=  432, avg=415.20, stdev=14.81, samples=5
  write: IOPS=432, BW=1731KiB/s (1773kB/s)(5192KiB/2999msec)
    slat (usec): min=15, max=718, avg=48.92, stdev=33.22
    clat (usec): min=4034, max=37183, avg=18232.13, stdev=2559.20
     lat (usec): min=4097, max=37237, avg=18282.13, stdev=2558.79
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14353], 10.00th=[15139], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[18220], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[20317], 90.00th=[21365], 95.00th=[22414],
     | 99.00th=[24511], 99.50th=[26084], 99.90th=[33817], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1656, max= 1872, per=25.24%, avg=1755.20, stdev=78.91, samples=5
   iops        : min=  414, max=  468, avg=438.80, stdev=19.73, samples=5
  lat (msec)   : 10=0.12%, 20=77.89%, 50=21.99%
  cpu          : usr=0.77%, sys=7.10%, ctx=2683, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13161: Sat Nov 16 06:55:22 2024
   read: IOPS=414, BW=1658KiB/s (1697kB/s)(4976KiB/3002msec)
    slat (usec): min=657, max=2556, avg=1199.39, stdev=237.89
    clat (usec): min=4248, max=29837, avg=17708.99, stdev=2844.46
     lat (usec): min=5475, max=31307, avg=18910.41, stdev=2884.95
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13435], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[24773], 99.50th=[25035], 99.90th=[28705], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1568, max= 1704, per=24.63%, avg=1645.33, stdev=49.49, samples=6
   iops        : min=  392, max=  426, avg=411.33, stdev=12.37, samples=6
  write: IOPS=438, BW=1753KiB/s (1796kB/s)(5264KiB/3002msec)
    slat (usec): min=13, max=214, avg=47.68, stdev=22.94
    clat (usec): min=12125, max=29787, avg=18335.42, stdev=2457.67
     lat (usec): min=12163, max=29830, avg=18384.26, stdev=2458.36
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14484], 10.00th=[15139], 20.00th=[16188],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19530], 80.00th=[20317], 90.00th=[21365], 95.00th=[22676],
     | 99.00th=[24249], 99.50th=[24773], 99.90th=[27657], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1632, max= 1936, per=25.14%, avg=1748.00, stdev=108.25, samples=6
   iops        : min=  408, max=  484, avg=437.00, stdev=27.06, samples=6
  lat (msec)   : 10=0.08%, 20=77.03%, 50=22.89%
  cpu          : usr=0.90%, sys=6.96%, ctx=2712, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13162: Sat Nov 16 06:55:22 2024
   read: IOPS=414, BW=1659KiB/s (1699kB/s)(4984KiB/3004msec)
    slat (usec): min=732, max=3005, avg=1208.01, stdev=240.55
    clat (usec): min=2636, max=30548, avg=17603.47, stdev=2922.22
     lat (usec): min=3875, max=31529, avg=18813.42, stdev=2969.61
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13304], 10.00th=[14222], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16712], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[24773], 99.50th=[26084], 99.90th=[27657], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1392, max= 1808, per=24.75%, avg=1653.33, stdev=149.74, samples=6
   iops        : min=  348, max=  452, avg=413.33, stdev=37.44, samples=6
  write: IOPS=437, BW=1750KiB/s (1792kB/s)(5256KiB/3004msec)
    slat (usec): min=14, max=1187, avg=48.72, stdev=46.24
    clat (usec): min=11666, max=30445, avg=18386.39, stdev=2502.81
     lat (usec): min=11766, max=30462, avg=18436.24, stdev=2499.11
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14615], 10.00th=[15270], 20.00th=[16319],
     | 30.00th=[16909], 40.00th=[17695], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19530], 80.00th=[20317], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[24773], 99.50th=[25560], 99.90th=[30540], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1604, max= 1856, per=24.98%, avg=1737.33, stdev=94.54, samples=6
   iops        : min=  401, max=  464, avg=434.33, stdev=23.64, samples=6
  lat (msec)   : 4=0.08%, 10=0.08%, 20=77.07%, 50=22.77%
  cpu          : usr=1.03%, sys=6.86%, ctx=2653, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13163: Sat Nov 16 06:55:22 2024
   read: IOPS=424, BW=1699KiB/s (1739kB/s)(5060KiB/2979msec)
    slat (usec): min=13, max=2622, avg=1192.82, stdev=254.56
    clat (usec): min=1659, max=28396, avg=17500.05, stdev=2918.56
     lat (usec): min=1673, max=29609, avg=18694.89, stdev=2982.74
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13304], 10.00th=[14091], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[21365], 95.00th=[22676],
     | 99.00th=[24249], 99.50th=[24773], 99.90th=[26084], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1560, max= 1856, per=25.39%, avg=1696.00, stdev=121.06, samples=5
   iops        : min=  390, max=  464, avg=424.00, stdev=30.27, samples=5
  write: IOPS=434, BW=1739KiB/s (1781kB/s)(5180KiB/2979msec)
    slat (usec): min=13, max=1245, avg=48.57, stdev=43.45
    clat (usec): min=5091, max=34897, avg=18401.84, stdev=2586.71
     lat (usec): min=5120, max=34949, avg=18451.51, stdev=2585.83
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14353], 10.00th=[15139], 20.00th=[16188],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18482], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[20317], 90.00th=[21627], 95.00th=[22676],
     | 99.00th=[24511], 99.50th=[25297], 99.90th=[29754], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1656, max= 1944, per=25.15%, avg=1748.80, stdev=112.37, samples=5
   iops        : min=  414, max=  486, avg=437.20, stdev=28.09, samples=5
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.27%, 20=76.84%, 50=22.77%
  cpu          : usr=1.18%, sys=6.88%, ctx=2721, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6680KiB/s (6841kB/s), 1658KiB/s-1699KiB/s (1697kB/s-1739kB/s), io=19.6MiB (20.5MB), run=2979-3004msec
  WRITE: bw=6955KiB/s (7122kB/s), 1731KiB/s-1753KiB/s (1773kB/s-1796kB/s), io=20.4MiB (21.4MB), run=2979-3004msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=13170: Sat Nov 16 06:55:23 2024
   read: IOPS=14, BW=597MiB/s (626MB/s)(40.0MiB/67msec)
    slat (nsec): min=5664.7k, max=5664.7k, avg=5664723.00, stdev= 0.00
    clat (nsec): min=56000k, max=56000k, avg=55999783.00, stdev= 0.00
     lat (nsec): min=61676k, max=61676k, avg=61676267.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[55837],  5.00th=[55837], 10.00th=[55837], 20.00th=[55837],
     | 30.00th=[55837], 40.00th=[55837], 50.00th=[55837], 60.00th=[55837],
     | 70.00th=[55837], 80.00th=[55837], 90.00th=[55837], 95.00th=[55837],
     | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837],
     | 99.99th=[55837]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.09%, ctx=6, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=597MiB/s (626MB/s), 597MiB/s-597MiB/s (626MB/s-626MB/s), io=40.0MiB (41.9MB), run=67-67msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout