== sanity test 398c: run fio to test AIO ================= 07:09:47 (1726312187)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.137934 s, 304 MB/s
osc.lustre-OST0000-osc-ffff8800b6ed9800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=10476: Sat Sep 14 07:09:56 2024
  write: IOPS=381, BW=1527KiB/s (1564kB/s)(10.0MiB/6704msec)
    slat (usec): min=15, max=599, avg=91.31, stdev=51.10
    clat (usec): min=4733, max=98472, avg=41607.83, stdev=7449.37
     lat (usec): min=4850, max=98515, avg=41701.42, stdev=7456.30
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[22676], 10.00th=[33817], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[45351], 80.00th=[45876], 90.00th=[47449], 95.00th=[47973],
     | 99.00th=[50594], 99.50th=[58459], 99.90th=[93848], 99.95th=[93848],
     | 99.99th=[98042]
   bw (  KiB/s): min= 1360, max= 2219, per=25.23%, avg=1534.08, stdev=215.05, samples=13
   iops        : min=  340, max=  554, avg=383.46, stdev=53.56, samples=13
  lat (msec)   : 10=0.16%, 20=1.60%, 50=96.68%, 100=1.56%
  cpu          : usr=0.66%, sys=3.66%, ctx=2253, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10477: Sat Sep 14 07:09:56 2024
  write: IOPS=380, BW=1520KiB/s (1557kB/s)(10.0MiB/6735msec)
    slat (usec): min=14, max=367, avg=85.19, stdev=47.99
    clat (usec): min=9498, max=84572, avg=41789.93, stdev=6775.62
     lat (usec): min=9661, max=84715, avg=41877.27, stdev=6785.30
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[23200], 10.00th=[35914], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[42206], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47449], 95.00th=[47973],
     | 99.00th=[50594], 99.50th=[51643], 99.90th=[74974], 99.95th=[80217],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1376, max= 2195, per=25.10%, avg=1526.15, stdev=209.39, samples=13
   iops        : min=  344, max=  548, avg=381.46, stdev=52.15, samples=13
  lat (msec)   : 10=0.04%, 20=1.05%, 50=97.70%, 100=1.21%
  cpu          : usr=0.53%, sys=3.59%, ctx=1923, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10478: Sat Sep 14 07:09:56 2024
  write: IOPS=381, BW=1526KiB/s (1563kB/s)(10.0MiB/6709msec)
    slat (usec): min=12, max=606, avg=76.89, stdev=46.13
    clat (usec): min=6431, max=88908, avg=41658.30, stdev=6785.98
     lat (usec): min=6554, max=89144, avg=41737.34, stdev=6794.75
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[22414], 10.00th=[34866], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[41681], 50.00th=[43254], 60.00th=[44303],
     | 70.00th=[45351], 80.00th=[45876], 90.00th=[47449], 95.00th=[47973],
     | 99.00th=[49546], 99.50th=[50594], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[88605]
   bw (  KiB/s): min= 1312, max= 2267, per=25.15%, avg=1529.15, stdev=232.02, samples=13
   iops        : min=  328, max=  566, avg=382.23, stdev=57.81, samples=13
  lat (msec)   : 10=0.04%, 20=1.56%, 50=97.73%, 100=0.66%
  cpu          : usr=0.55%, sys=3.21%, ctx=1453, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10479: Sat Sep 14 07:09:56 2024
  write: IOPS=380, BW=1520KiB/s (1557kB/s)(10.0MiB/6735msec)
    slat (usec): min=11, max=502, avg=86.93, stdev=48.61
    clat (usec): min=17964, max=84472, avg=41751.90, stdev=6696.45
     lat (usec): min=18026, max=84634, avg=41840.93, stdev=6708.29
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[22938], 10.00th=[35914], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[42206], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47449], 95.00th=[47973],
     | 99.00th=[49546], 99.50th=[50594], 99.90th=[80217], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1376, max= 2211, per=25.10%, avg=1526.15, stdev=213.80, samples=13
   iops        : min=  344, max=  552, avg=381.46, stdev=53.26, samples=13
  lat (msec)   : 20=1.13%, 50=98.05%, 100=0.82%
  cpu          : usr=0.77%, sys=3.28%, ctx=1953, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6082KiB/s (6228kB/s), 1520KiB/s-1527KiB/s (1557kB/s-1564kB/s), io=40.0MiB (41.9MB), run=6704-6735msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=10495: Sat Sep 14 07:09:59 2024
   read: IOPS=449, BW=1798KiB/s (1842kB/s)(5048KiB/2807msec)
    slat (usec): min=649, max=3361, avg=1201.61, stdev=341.36
    clat (usec): min=3591, max=36911, avg=16342.22, stdev=3626.72
     lat (usec): min=4790, max=38755, avg=17546.16, stdev=3804.25
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12387], 10.00th=[13042], 20.00th=[13960],
     | 30.00th=[14615], 40.00th=[15270], 50.00th=[15795], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19268], 95.00th=[22414],
     | 99.00th=[30802], 99.50th=[32637], 99.90th=[36439], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1256, max= 1952, per=24.61%, avg=1758.40, stdev=284.56, samples=5
   iops        : min=  314, max=  488, avg=439.60, stdev=71.14, samples=5
  write: IOPS=462, BW=1850KiB/s (1894kB/s)(5192KiB/2807msec)
    slat (usec): min=12, max=365, avg=55.47, stdev=33.20
    clat (usec): min=10176, max=38216, avg=16971.81, stdev=3675.30
     lat (usec): min=10426, max=38328, avg=17028.54, stdev=3678.69
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13304], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[19530], 95.00th=[23725],
     | 99.00th=[34341], 99.50th=[35390], 99.90th=[38011], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1272, max= 2040, per=24.84%, avg=1848.00, stdev=324.22, samples=5
   iops        : min=  318, max=  510, avg=462.00, stdev=81.06, samples=5
  lat (msec)   : 4=0.04%, 10=0.23%, 20=91.88%, 50=7.85%
  cpu          : usr=1.43%, sys=9.09%, ctx=2567, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10496: Sat Sep 14 07:09:59 2024
   read: IOPS=445, BW=1780KiB/s (1823kB/s)(4976KiB/2795msec)
    slat (usec): min=648, max=4720, avg=1203.38, stdev=360.63
    clat (usec): min=2165, max=41568, avg=16377.37, stdev=3881.91
     lat (usec): min=3218, max=43505, avg=17583.26, stdev=4053.15
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11994], 10.00th=[12911], 20.00th=[13829],
     | 30.00th=[14615], 40.00th=[15270], 50.00th=[15795], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19530], 95.00th=[23200],
     | 99.00th=[32113], 99.50th=[33817], 99.90th=[38011], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1160, max= 2024, per=24.58%, avg=1756.80, stdev=343.56, samples=5
   iops        : min=  290, max=  506, avg=439.20, stdev=85.89, samples=5
  write: IOPS=470, BW=1883KiB/s (1929kB/s)(5264KiB/2795msec)
    slat (usec): min=14, max=273, avg=55.29, stdev=32.25
    clat (usec): min=2435, max=37845, avg=16956.63, stdev=3737.62
     lat (usec): min=2574, max=37926, avg=17013.22, stdev=3743.15
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13435], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[19530], 95.00th=[24773],
     | 99.00th=[34866], 99.50th=[35390], 99.90th=[36963], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1448, max= 2056, per=25.31%, avg=1883.20, stdev=248.94, samples=5
   iops        : min=  362, max=  514, avg=470.80, stdev=62.24, samples=5
  lat (msec)   : 4=0.12%, 10=0.27%, 20=90.94%, 50=8.67%
  cpu          : usr=1.04%, sys=9.31%, ctx=2490, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10497: Sat Sep 14 07:09:59 2024
   read: IOPS=447, BW=1790KiB/s (1833kB/s)(4984KiB/2785msec)
    slat (usec): min=24, max=3700, avg=1211.27, stdev=364.97
    clat (usec): min=5365, max=37521, avg=16362.79, stdev=3630.14
     lat (usec): min=5391, max=41224, avg=17576.68, stdev=3799.43
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12387], 10.00th=[13042], 20.00th=[13960],
     | 30.00th=[14615], 40.00th=[15270], 50.00th=[15795], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19530], 95.00th=[22414],
     | 99.00th=[31327], 99.50th=[33817], 99.90th=[36439], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1368, max= 2016, per=24.85%, avg=1776.00, stdev=242.19, samples=5
   iops        : min=  342, max=  504, avg=444.00, stdev=60.55, samples=5
  write: IOPS=471, BW=1887KiB/s (1933kB/s)(5256KiB/2785msec)
    slat (usec): min=13, max=519, avg=59.66, stdev=39.76
    clat (usec): min=10354, max=40996, avg=17071.31, stdev=3944.96
     lat (usec): min=10380, max=41031, avg=17132.46, stdev=3947.96
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13304], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16188], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[19268], 95.00th=[26608],
     | 99.00th=[34341], 99.50th=[36439], 99.90th=[39060], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 1368, max= 2088, per=25.12%, avg=1868.80, stdev=293.92, samples=5
   iops        : min=  342, max=  522, avg=467.20, stdev=73.48, samples=5
  lat (msec)   : 10=0.23%, 20=91.76%, 50=8.01%
  cpu          : usr=1.01%, sys=9.66%, ctx=2481, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10498: Sat Sep 14 07:09:59 2024
   read: IOPS=450, BW=1802KiB/s (1845kB/s)(5060KiB/2808msec)
    slat (usec): min=668, max=3003, avg=1197.83, stdev=329.94
    clat (usec): min=3077, max=37233, avg=16351.67, stdev=3555.19
     lat (usec): min=4508, max=40347, avg=17551.95, stdev=3721.85
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12125], 10.00th=[13042], 20.00th=[13960],
     | 30.00th=[14746], 40.00th=[15401], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19530], 95.00th=[21627],
     | 99.00th=[30802], 99.50th=[32637], 99.90th=[35914], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1240, max= 1944, per=24.79%, avg=1771.20, stdev=299.26, samples=5
   iops        : min=  310, max=  486, avg=442.80, stdev=74.81, samples=5
  write: IOPS=461, BW=1845KiB/s (1889kB/s)(5180KiB/2808msec)
    slat (usec): min=12, max=258, avg=55.71, stdev=32.37
    clat (usec): min=2022, max=36421, avg=16949.46, stdev=3778.09
     lat (usec): min=2080, max=36495, avg=17006.66, stdev=3784.21
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13304], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[19268], 95.00th=[24249],
     | 99.00th=[33162], 99.50th=[34341], 99.90th=[35390], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 1264, max= 1984, per=24.39%, avg=1814.40, stdev=311.20, samples=5
   iops        : min=  316, max=  496, avg=453.60, stdev=77.80, samples=5
  lat (msec)   : 4=0.08%, 10=0.55%, 20=91.80%, 50=7.58%
  cpu          : usr=0.89%, sys=9.37%, ctx=2652, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7147KiB/s (7318kB/s), 1780KiB/s-1802KiB/s (1823kB/s-1845kB/s), io=19.6MiB (20.5MB), run=2785-2808msec
  WRITE: bw=7440KiB/s (7619kB/s), 1845KiB/s-1887KiB/s (1889kB/s-1933kB/s), io=20.4MiB (21.4MB), run=2785-2808msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=10504: Sat Sep 14 07:10:01 2024
   read: IOPS=12, BW=488MiB/s (512MB/s)(40.0MiB/82msec)
    slat (nsec): min=8449.2k, max=8449.2k, avg=8449193.00, stdev= 0.00
    clat (nsec): min=68807k, max=68807k, avg=68806794.00, stdev= 0.00
     lat (nsec): min=77271k, max=77271k, avg=77270704.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[68682],  5.00th=[68682], 10.00th=[68682], 20.00th=[68682],
     | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[68682],
     | 70.00th=[68682], 80.00th=[68682], 90.00th=[68682], 95.00th=[68682],
     | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.11%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=488MiB/s (512MB/s), 488MiB/s-488MiB/s (512MB/s-512MB/s), io=40.0MiB (41.9MB), run=82-82msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout