== sanity test 398c: run fio to test AIO ================= 16:05:02 (1734123902)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.141129 s, 297 MB/s
osc.lustre-OST0000-osc-ffff8800b5ba6800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=21694: Fri Dec 13 16:05:10 2024
  write: IOPS=390, BW=1562KiB/s (1599kB/s)(10.0MiB/6556msec)
    slat (usec): min=14, max=527, avg=71.12, stdev=44.17
    clat (usec): min=20028, max=64719, avg=40700.12, stdev=5650.12
     lat (usec): min=20116, max=64796, avg=40772.84, stdev=5655.78
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[29230], 10.00th=[33424], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[45876], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[50594], 99.50th=[51643], 99.90th=[52691], 99.95th=[59507],
     | 99.99th=[64750]
   bw (  KiB/s): min= 1312, max= 1824, per=24.93%, avg=1557.54, stdev=153.93, samples=13
   iops        : min=  328, max=  456, avg=389.38, stdev=38.48, samples=13
  lat (msec)   : 50=98.79%, 100=1.21%
  cpu          : usr=0.43%, sys=3.36%, ctx=898, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21695: Fri Dec 13 16:05:10 2024
  write: IOPS=390, BW=1563KiB/s (1601kB/s)(10.0MiB/6551msec)
    slat (usec): min=13, max=665, avg=73.78, stdev=42.95
    clat (usec): min=10666, max=86085, avg=40696.93, stdev=5806.32
     lat (usec): min=10783, max=86282, avg=40772.39, stdev=5810.45
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[30016], 10.00th=[32900], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44303], 80.00th=[45876], 90.00th=[46924], 95.00th=[47973],
     | 99.00th=[49546], 99.50th=[50070], 99.90th=[60031], 99.95th=[77071],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1400, max= 1920, per=24.95%, avg=1558.77, stdev=176.48, samples=13
   iops        : min=  350, max=  480, avg=389.69, stdev=44.12, samples=13
  lat (msec)   : 20=0.04%, 50=99.34%, 100=0.62%
  cpu          : usr=0.50%, sys=3.42%, ctx=1014, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21696: Fri Dec 13 16:05:10 2024
  write: IOPS=390, BW=1563KiB/s (1601kB/s)(10.0MiB/6551msec)
    slat (usec): min=14, max=324, avg=78.44, stdev=43.02
    clat (usec): min=6653, max=80671, avg=40728.29, stdev=5740.53
     lat (usec): min=6742, max=80767, avg=40808.56, stdev=5744.24
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[30540], 10.00th=[32900], 20.00th=[35914],
     | 30.00th=[37487], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44303], 80.00th=[45876], 90.00th=[46924], 95.00th=[47973],
     | 99.00th=[50070], 99.50th=[51119], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[80217]
   bw (  KiB/s): min= 1392, max= 1920, per=24.96%, avg=1559.38, stdev=174.74, samples=13
   iops        : min=  348, max=  480, avg=389.85, stdev=43.68, samples=13
  lat (msec)   : 10=0.04%, 50=98.75%, 100=1.21%
  cpu          : usr=0.40%, sys=3.95%, ctx=1246, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21697: Fri Dec 13 16:05:10 2024
  write: IOPS=392, BW=1569KiB/s (1606kB/s)(10.0MiB/6528msec)
    slat (usec): min=15, max=305, avg=84.46, stdev=42.68
    clat (usec): min=5127, max=96352, avg=40583.10, stdev=6579.08
     lat (usec): min=5246, max=96406, avg=40669.32, stdev=6581.40
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[29230], 10.00th=[32900], 20.00th=[35390],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[45351], 90.00th=[46924], 95.00th=[47973],
     | 99.00th=[50594], 99.50th=[56886], 99.90th=[85459], 99.95th=[91751],
     | 99.99th=[95945]
   bw (  KiB/s): min= 1400, max= 2008, per=25.07%, avg=1566.15, stdev=183.37, samples=13
   iops        : min=  350, max=  502, avg=391.54, stdev=45.84, samples=13
  lat (msec)   : 10=0.16%, 20=0.23%, 50=98.24%, 100=1.37%
  cpu          : usr=0.46%, sys=4.29%, ctx=1436, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6248KiB/s (6398kB/s), 1562KiB/s-1569KiB/s (1599kB/s-1606kB/s), io=40.0MiB (41.9MB), run=6528-6556msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=21712: Fri Dec 13 16:05:15 2024
   read: IOPS=316, BW=1265KiB/s (1296kB/s)(5048KiB/3989msec)
    slat (usec): min=741, max=3027, avg=1577.93, stdev=362.16
    clat (usec): min=2536, max=42982, avg=23326.40, stdev=4769.18
     lat (usec): min=3771, max=44962, avg=24907.52, stdev=4938.76
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[17171], 10.00th=[18220], 20.00th=[19792],
     | 30.00th=[20841], 40.00th=[21890], 50.00th=[22676], 60.00th=[23725],
     | 70.00th=[25035], 80.00th=[26608], 90.00th=[29492], 95.00th=[32375],
     | 99.00th=[37487], 99.50th=[39060], 99.90th=[42730], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=  840, max= 1504, per=24.38%, avg=1226.29, stdev=231.20, samples=7
   iops        : min=  210, max=  376, avg=306.57, stdev=57.80, samples=7
  write: IOPS=325, BW=1302KiB/s (1333kB/s)(5192KiB/3989msec)
    slat (usec): min=14, max=711, avg=72.88, stdev=47.95
    clat (usec): min=5290, max=41386, avg=24347.76, stdev=4532.16
     lat (usec): min=5377, max=41477, avg=24422.21, stdev=4535.56
    clat percentiles (usec):
     |  1.00th=[16319],  5.00th=[18482], 10.00th=[19792], 20.00th=[20841],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23462], 60.00th=[24249],
     | 70.00th=[25822], 80.00th=[27919], 90.00th=[30802], 95.00th=[32637],
     | 99.00th=[38011], 99.50th=[39060], 99.90th=[41157], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min=  880, max= 1536, per=24.44%, avg=1280.00, stdev=211.86, samples=7
   iops        : min=  220, max=  384, avg=320.00, stdev=52.97, samples=7
  lat (msec)   : 4=0.12%, 10=0.27%, 20=16.72%, 50=82.89%
  cpu          : usr=0.93%, sys=9.00%, ctx=2248, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21713: Fri Dec 13 16:05:15 2024
   read: IOPS=312, BW=1248KiB/s (1278kB/s)(4976KiB/3987msec)
    slat (usec): min=849, max=3355, avg=1582.74, stdev=368.04
    clat (usec): min=3583, max=40937, avg=23419.72, stdev=4551.18
     lat (usec): min=5030, max=43051, avg=25005.54, stdev=4723.05
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[17433], 10.00th=[18220], 20.00th=[19792],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22676], 60.00th=[23725],
     | 70.00th=[25035], 80.00th=[26346], 90.00th=[29492], 95.00th=[32113],
     | 99.00th=[37487], 99.50th=[39060], 99.90th=[40633], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min=  854, max= 1477, per=24.07%, avg=1210.71, stdev=225.02, samples=7
   iops        : min=  213, max=  369, avg=302.57, stdev=56.34, samples=7
  write: IOPS=330, BW=1320KiB/s (1352kB/s)(5264KiB/3987msec)
    slat (usec): min=13, max=1362, avg=77.68, stdev=63.26
    clat (usec): min=3515, max=40948, avg=24397.94, stdev=4461.04
     lat (usec): min=3563, max=41005, avg=24477.13, stdev=4460.46
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[18744], 10.00th=[20055], 20.00th=[21103],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23462], 60.00th=[24511],
     | 70.00th=[25822], 80.00th=[27919], 90.00th=[30540], 95.00th=[33162],
     | 99.00th=[36963], 99.50th=[38536], 99.90th=[40633], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min=  958, max= 1448, per=24.95%, avg=1306.71, stdev=183.93, samples=7
   iops        : min=  239, max=  362, avg=326.57, stdev=46.12, samples=7
  lat (msec)   : 4=0.08%, 10=0.23%, 20=15.74%, 50=83.95%
  cpu          : usr=1.20%, sys=8.83%, ctx=2264, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21714: Fri Dec 13 16:05:15 2024
   read: IOPS=312, BW=1251KiB/s (1281kB/s)(4984KiB/3985msec)
    slat (usec): min=848, max=3299, avg=1602.38, stdev=375.95
    clat (usec): min=11977, max=39606, avg=23506.47, stdev=4628.28
     lat (usec): min=14501, max=42030, avg=25111.99, stdev=4807.57
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[17171], 10.00th=[18220], 20.00th=[19530],
     | 30.00th=[20841], 40.00th=[21890], 50.00th=[22938], 60.00th=[23987],
     | 70.00th=[25297], 80.00th=[26870], 90.00th=[29754], 95.00th=[32375],
     | 99.00th=[36963], 99.50th=[38536], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min=  880, max= 1448, per=24.24%, avg=1219.43, stdev=189.42, samples=7
   iops        : min=  220, max=  362, avg=304.86, stdev=47.35, samples=7
  write: IOPS=329, BW=1319KiB/s (1351kB/s)(5256KiB/3985msec)
    slat (usec): min=13, max=802, avg=77.10, stdev=56.19
    clat (usec): min=2842, max=44074, avg=24290.29, stdev=4502.53
     lat (usec): min=2885, max=44110, avg=24369.07, stdev=4501.25
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[19006], 10.00th=[19792], 20.00th=[20841],
     | 30.00th=[21627], 40.00th=[22414], 50.00th=[23200], 60.00th=[24249],
     | 70.00th=[25560], 80.00th=[27657], 90.00th=[30540], 95.00th=[33162],
     | 99.00th=[39060], 99.50th=[41157], 99.90th=[43254], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min=  928, max= 1520, per=24.83%, avg=1300.57, stdev=240.88, samples=7
   iops        : min=  232, max=  380, avg=325.14, stdev=60.22, samples=7
  lat (msec)   : 4=0.04%, 20=17.46%, 50=82.50%
  cpu          : usr=1.33%, sys=8.58%, ctx=2286, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21715: Fri Dec 13 16:05:15 2024
   read: IOPS=318, BW=1273KiB/s (1304kB/s)(5060KiB/3974msec)
    slat (usec): min=34, max=4446, avg=1586.41, stdev=382.96
    clat (usec): min=2234, max=46178, avg=23546.56, stdev=4808.43
     lat (usec): min=2338, max=47934, avg=25136.22, stdev=4991.01
    clat percentiles (usec):
     |  1.00th=[15008],  5.00th=[17433], 10.00th=[18482], 20.00th=[19792],
     | 30.00th=[20841], 40.00th=[21890], 50.00th=[22938], 60.00th=[24249],
     | 70.00th=[25297], 80.00th=[26870], 90.00th=[29754], 95.00th=[32637],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[44303], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min=  976, max= 1392, per=24.90%, avg=1252.57, stdev=167.42, samples=7
   iops        : min=  244, max=  348, avg=313.14, stdev=41.85, samples=7
  write: IOPS=325, BW=1303KiB/s (1335kB/s)(5180KiB/3974msec)
    slat (usec): min=13, max=774, avg=81.42, stdev=61.79
    clat (usec): min=7171, max=43505, avg=24319.80, stdev=4667.27
     lat (usec): min=7252, max=43564, avg=24402.99, stdev=4665.78
    clat percentiles (usec):
     |  1.00th=[16319],  5.00th=[18744], 10.00th=[19530], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22414], 50.00th=[23200], 60.00th=[24511],
     | 70.00th=[25822], 80.00th=[27657], 90.00th=[30802], 95.00th=[33817],
     | 99.00th=[39060], 99.50th=[39584], 99.90th=[43254], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  992, max= 1560, per=24.53%, avg=1284.57, stdev=199.24, samples=7
   iops        : min=  248, max=  390, avg=321.14, stdev=49.81, samples=7
  lat (msec)   : 4=0.12%, 10=0.16%, 20=18.09%, 50=81.64%
  cpu          : usr=0.86%, sys=9.44%, ctx=2340, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5031KiB/s (5152kB/s), 1248KiB/s-1273KiB/s (1278kB/s-1304kB/s), io=19.6MiB (20.5MB), run=3974-3989msec
  WRITE: bw=5237KiB/s (5363kB/s), 1302KiB/s-1320KiB/s (1333kB/s-1352kB/s), io=20.4MiB (21.4MB), run=3974-3989msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=21722: Fri Dec 13 16:05:16 2024
   read: IOPS=8, BW=357MiB/s (374MB/s)(40.0MiB/112msec)
    slat (nsec): min=7007.6k, max=7007.6k, avg=7007576.00, stdev= 0.00
    clat (nsec): min=99102k, max=99102k, avg=99102198.00, stdev= 0.00
     lat (nsec): min=106119k, max=106119k, avg=106119177.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[99091],  5.00th=[99091], 10.00th=[99091], 20.00th=[99091],
     | 30.00th=[99091], 40.00th=[99091], 50.00th=[99091], 60.00th=[99091],
     | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091],
     | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091],
     | 99.99th=[99091]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.25%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=357MiB/s (374MB/s), 357MiB/s-357MiB/s (374MB/s-374MB/s), io=40.0MiB (41.9MB), run=112-112msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout