== sanity test 398c: run fio to test AIO ================= 17:16:54 (1730582214)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.150665 s, 278 MB/s
osc.lustre-OST0000-osc-ffff8800a9688000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=17680: Sat Nov  2 17:17:02 2024
  write: IOPS=394, BW=1577KiB/s (1615kB/s)(10.0MiB/6494msec)
    slat (usec): min=18, max=510, avg=79.59, stdev=45.33
    clat (usec): min=6175, max=77467, avg=40290.04, stdev=5317.13
     lat (usec): min=6280, max=77534, avg=40371.59, stdev=5327.69
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[31065], 10.00th=[34866], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[39584], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[45351], 95.00th=[46400],
     | 99.00th=[48497], 99.50th=[52167], 99.90th=[72877], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1408, max= 1792, per=25.09%, avg=1582.33, stdev=121.75, samples=12
   iops        : min=  352, max=  448, avg=395.58, stdev=30.44, samples=12
  lat (msec)   : 10=0.04%, 20=1.09%, 50=98.24%, 100=0.62%
  cpu          : usr=0.63%, sys=3.70%, ctx=1341, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17681: Sat Nov  2 17:17:02 2024
  write: IOPS=395, BW=1583KiB/s (1621kB/s)(10.0MiB/6468msec)
    slat (usec): min=18, max=384, avg=83.55, stdev=44.17
    clat (usec): min=6317, max=88122, avg=40198.71, stdev=5786.77
     lat (usec): min=6437, max=88191, avg=40284.46, stdev=5793.81
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[30802], 10.00th=[34866], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[39584], 50.00th=[41157], 60.00th=[42206],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[45351], 95.00th=[46400],
     | 99.00th=[49021], 99.50th=[50594], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[88605]
   bw (  KiB/s): min= 1408, max= 1816, per=25.23%, avg=1591.33, stdev=124.33, samples=12
   iops        : min=  352, max=  454, avg=397.83, stdev=31.08, samples=12
  lat (msec)   : 10=0.12%, 20=0.90%, 50=98.44%, 100=0.55%
  cpu          : usr=0.70%, sys=3.82%, ctx=1550, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17682: Sat Nov  2 17:17:02 2024
  write: IOPS=397, BW=1588KiB/s (1626kB/s)(10.0MiB/6448msec)
    slat (usec): min=19, max=478, avg=92.95, stdev=44.25
    clat (usec): min=4391, max=86782, avg=40086.49, stdev=6196.56
     lat (usec): min=4561, max=86935, avg=40181.63, stdev=6200.47
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[29754], 10.00th=[34341], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[39584], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[45351], 95.00th=[46400],
     | 99.00th=[48497], 99.50th=[62653], 99.90th=[83362], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1408, max= 1880, per=25.35%, avg=1599.08, stdev=131.97, samples=12
   iops        : min=  352, max=  470, avg=399.75, stdev=33.02, samples=12
  lat (msec)   : 10=0.23%, 20=1.02%, 50=98.20%, 100=0.55%
  cpu          : usr=0.76%, sys=4.51%, ctx=1892, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17683: Sat Nov  2 17:17:02 2024
  write: IOPS=394, BW=1578KiB/s (1615kB/s)(10.0MiB/6491msec)
    slat (usec): min=14, max=511, avg=74.60, stdev=45.08
    clat (usec): min=5001, max=87984, avg=40310.22, stdev=5791.95
     lat (usec): min=5131, max=88180, avg=40386.84, stdev=5797.79
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[30278], 10.00th=[34866], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[39584], 50.00th=[41157], 60.00th=[42206],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[45351], 95.00th=[46400],
     | 99.00th=[50070], 99.50th=[50594], 99.90th=[87557], 99.95th=[87557],
     | 99.99th=[87557]
   bw (  KiB/s): min= 1408, max= 1792, per=25.13%, avg=1584.67, stdev=106.16, samples=12
   iops        : min=  352, max=  448, avg=396.17, stdev=26.54, samples=12
  lat (msec)   : 10=0.12%, 20=1.13%, 50=98.09%, 100=0.66%
  cpu          : usr=0.68%, sys=3.31%, ctx=1087, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6307KiB/s (6459kB/s), 1577KiB/s-1588KiB/s (1615kB/s-1626kB/s), io=40.0MiB (41.9MB), run=6448-6494msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=17698: Sat Nov  2 17:17:06 2024
   read: IOPS=471, BW=1888KiB/s (1933kB/s)(5048KiB/2674msec)
    slat (usec): min=592, max=3845, avg=1100.94, stdev=374.56
    clat (usec): min=8010, max=41816, avg=15676.87, stdev=4263.85
     lat (usec): min=9019, max=44307, avg=16779.67, stdev=4520.23
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[11469], 10.00th=[12125], 20.00th=[12911],
     | 30.00th=[13566], 40.00th=[14222], 50.00th=[14877], 60.00th=[15401],
     | 70.00th=[16057], 80.00th=[16909], 90.00th=[19268], 95.00th=[26084],
     | 99.00th=[33162], 99.50th=[35390], 99.90th=[41157], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1192, max= 2080, per=24.77%, avg=1858.40, stdev=375.61, samples=5
   iops        : min=  298, max=  520, avg=464.60, stdev=93.90, samples=5
  write: IOPS=485, BW=1942KiB/s (1988kB/s)(5192KiB/2674msec)
    slat (usec): min=13, max=937, avg=43.46, stdev=35.27
    clat (usec): min=9563, max=37426, avg=16285.29, stdev=4376.35
     lat (usec): min=9600, max=37489, avg=16329.79, stdev=4384.71
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[12649], 10.00th=[13304], 20.00th=[13960],
     | 30.00th=[14353], 40.00th=[14877], 50.00th=[15270], 60.00th=[15664],
     | 70.00th=[16188], 80.00th=[16909], 90.00th=[19006], 95.00th=[27395],
     | 99.00th=[35390], 99.50th=[36963], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1144, max= 2224, per=24.77%, avg=1935.00, stdev=450.91, samples=5
   iops        : min=  286, max=  556, avg=483.60, stdev=112.69, samples=5
  lat (msec)   : 10=0.31%, 20=90.86%, 50=8.83%
  cpu          : usr=1.05%, sys=7.41%, ctx=2452, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17699: Sat Nov  2 17:17:06 2024
   read: IOPS=469, BW=1876KiB/s (1921kB/s)(4976KiB/2652msec)
    slat (usec): min=35, max=3828, avg=1096.07, stdev=380.43
    clat (usec): min=1853, max=42207, avg=15617.20, stdev=4356.99
     lat (usec): min=1899, max=44984, avg=16715.12, stdev=4618.53
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[11469], 10.00th=[12125], 20.00th=[12911],
     | 30.00th=[13566], 40.00th=[14222], 50.00th=[14877], 60.00th=[15401],
     | 70.00th=[16057], 80.00th=[16712], 90.00th=[18744], 95.00th=[25822],
     | 99.00th=[34866], 99.50th=[36963], 99.90th=[40109], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1120, max= 2184, per=24.65%, avg=1849.60, stdev=420.61, samples=5
   iops        : min=  280, max=  546, avg=462.40, stdev=105.15, samples=5
  write: IOPS=496, BW=1985KiB/s (2033kB/s)(5264KiB/2652msec)
    slat (usec): min=12, max=616, avg=43.59, stdev=29.37
    clat (usec): min=6619, max=42513, avg=16315.46, stdev=4394.47
     lat (usec): min=6665, max=42623, avg=16360.05, stdev=4403.73
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12518], 10.00th=[13173], 20.00th=[13960],
     | 30.00th=[14484], 40.00th=[14877], 50.00th=[15270], 60.00th=[15795],
     | 70.00th=[16319], 80.00th=[16909], 90.00th=[19006], 95.00th=[27657],
     | 99.00th=[34341], 99.50th=[36963], 99.90th=[40633], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1408, max= 2216, per=25.52%, avg=1993.60, stdev=330.74, samples=5
   iops        : min=  352, max=  554, avg=498.40, stdev=82.68, samples=5
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.39%, 20=90.98%, 50=8.48%
  cpu          : usr=0.83%, sys=7.62%, ctx=2391, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17700: Sat Nov  2 17:17:06 2024
   read: IOPS=465, BW=1864KiB/s (1909kB/s)(4984KiB/2674msec)
    slat (usec): min=582, max=3670, avg=1098.79, stdev=361.61
    clat (usec): min=9485, max=38792, avg=15556.13, stdev=4053.91
     lat (usec): min=10429, max=41226, avg=16656.72, stdev=4299.07
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[11600], 10.00th=[12125], 20.00th=[12911],
     | 30.00th=[13698], 40.00th=[14222], 50.00th=[14746], 60.00th=[15270],
     | 70.00th=[15926], 80.00th=[16712], 90.00th=[18482], 95.00th=[25822],
     | 99.00th=[32637], 99.50th=[33817], 99.90th=[36439], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 1192, max= 2152, per=24.70%, avg=1853.60, stdev=379.91, samples=5
   iops        : min=  298, max=  538, avg=463.40, stdev=94.98, samples=5
  write: IOPS=491, BW=1966KiB/s (2013kB/s)(5256KiB/2674msec)
    slat (usec): min=13, max=346, avg=42.69, stdev=23.94
    clat (usec): min=9944, max=39075, avg=16395.19, stdev=4472.49
     lat (usec): min=9990, max=39110, avg=16438.94, stdev=4483.11
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[12649], 10.00th=[13173], 20.00th=[13960],
     | 30.00th=[14484], 40.00th=[15008], 50.00th=[15401], 60.00th=[15795],
     | 70.00th=[16188], 80.00th=[16909], 90.00th=[18744], 95.00th=[28181],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1144, max= 2264, per=24.87%, avg=1943.00, stdev=453.23, samples=5
   iops        : min=  286, max=  566, avg=485.60, stdev=113.25, samples=5
  lat (msec)   : 10=0.12%, 20=91.45%, 50=8.44%
  cpu          : usr=0.86%, sys=7.52%, ctx=2416, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17701: Sat Nov  2 17:17:06 2024
   read: IOPS=473, BW=1894KiB/s (1940kB/s)(5060KiB/2671msec)
    slat (usec): min=596, max=3663, avg=1099.50, stdev=379.16
    clat (usec): min=9363, max=37266, avg=15695.62, stdev=4158.86
     lat (usec): min=10129, max=39097, avg=16797.05, stdev=4420.66
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[11600], 10.00th=[12256], 20.00th=[13173],
     | 30.00th=[13698], 40.00th=[14353], 50.00th=[14877], 60.00th=[15270],
     | 70.00th=[16057], 80.00th=[16712], 90.00th=[18744], 95.00th=[26084],
     | 99.00th=[33817], 99.50th=[35914], 99.90th=[36963], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1200, max= 2096, per=25.05%, avg=1880.00, stdev=381.16, samples=5
   iops        : min=  300, max=  524, avg=470.00, stdev=95.29, samples=5
  write: IOPS=484, BW=1939KiB/s (1986kB/s)(5180KiB/2671msec)
    slat (usec): min=12, max=278, avg=42.15, stdev=23.99
    clat (usec): min=8182, max=38380, avg=16346.45, stdev=4277.59
     lat (usec): min=8216, max=38473, avg=16389.52, stdev=4288.77
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12780], 10.00th=[13304], 20.00th=[13960],
     | 30.00th=[14484], 40.00th=[14877], 50.00th=[15270], 60.00th=[15664],
     | 70.00th=[16188], 80.00th=[17171], 90.00th=[19268], 95.00th=[28181],
     | 99.00th=[33817], 99.50th=[35390], 99.90th=[37487], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 1216, max= 2216, per=24.62%, avg=1923.20, stdev=406.22, samples=5
   iops        : min=  304, max=  554, avg=480.80, stdev=101.55, samples=5
  lat (msec)   : 10=0.16%, 20=91.21%, 50=8.63%
  cpu          : usr=1.12%, sys=7.30%, ctx=2435, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7505KiB/s (7685kB/s), 1864KiB/s-1894KiB/s (1909kB/s-1940kB/s), io=19.6MiB (20.5MB), run=2652-2674msec
  WRITE: bw=7813KiB/s (8001kB/s), 1939KiB/s-1985KiB/s (1986kB/s-2033kB/s), io=20.4MiB (21.4MB), run=2652-2674msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=17707: Sat Nov  2 17:17:08 2024
   read: IOPS=6, BW=247MiB/s (259MB/s)(40.0MiB/162msec)
    slat (nsec): min=5489.7k, max=5489.7k, avg=5489711.00, stdev= 0.00
    clat (nsec): min=149573k, max=149573k, avg=149573100.00, stdev= 0.00
     lat (nsec): min=155072k, max=155072k, avg=155072043.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  150],  5.00th=[  150], 10.00th=[  150], 20.00th=[  150],
     | 30.00th=[  150], 40.00th=[  150], 50.00th=[  150], 60.00th=[  150],
     | 70.00th=[  150], 80.00th=[  150], 90.00th=[  150], 95.00th=[  150],
     | 99.00th=[  150], 99.50th=[  150], 99.90th=[  150], 99.95th=[  150],
     | 99.99th=[  150]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.11%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=40.0MiB (41.9MB), run=162-162msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout