== sanity test 398c: run fio to test AIO ================= 14:00:09 (1714413609)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0900268 s, 466 MB/s
osc.lustre-OST0000-osc-ffff88007b7ec000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=10359: Mon Apr 29 14:00:14 2024
  write: IOPS=713, BW=2855KiB/s (2923kB/s)(10.0MiB/3587msec)
    slat (usec): min=11, max=980, avg=51.46, stdev=38.05
    clat (usec): min=2440, max=64984, avg=22278.41, stdev=5959.46
     lat (usec): min=2490, max=65029, avg=22331.08, stdev=5966.79
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[16450], 10.00th=[17433], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20579], 50.00th=[21365], 60.00th=[22152],
     | 70.00th=[22938], 80.00th=[23725], 90.00th=[27132], 95.00th=[34341],
     | 99.00th=[42206], 99.50th=[55313], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min= 2171, max= 3472, per=25.08%, avg=2850.71, stdev=441.62, samples=7
   iops        : min=  542, max=  868, avg=712.57, stdev=110.60, samples=7
  lat (msec)   : 4=0.23%, 10=0.66%, 20=33.55%, 50=64.92%, 100=0.62%
  cpu          : usr=0.50%, sys=4.16%, ctx=1770, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10360: Mon Apr 29 14:00:14 2024
  write: IOPS=710, BW=2843KiB/s (2911kB/s)(10.0MiB/3602msec)
    slat (usec): min=12, max=1023, avg=55.97, stdev=46.09
    clat (usec): min=4683, max=65190, avg=22362.74, stdev=5702.43
     lat (usec): min=4733, max=65434, avg=22420.08, stdev=5714.47
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[16909], 10.00th=[17695], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20579], 50.00th=[21365], 60.00th=[21890],
     | 70.00th=[22938], 80.00th=[23987], 90.00th=[26870], 95.00th=[34341],
     | 99.00th=[44827], 99.50th=[55313], 99.90th=[62653], 99.95th=[64750],
     | 99.99th=[65274]
   bw (  KiB/s): min= 2176, max= 3344, per=24.96%, avg=2836.57, stdev=416.23, samples=7
   iops        : min=  544, max=  836, avg=709.14, stdev=104.06, samples=7
  lat (msec)   : 10=0.08%, 20=31.52%, 50=67.77%, 100=0.62%
  cpu          : usr=0.67%, sys=4.30%, ctx=2074, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10361: Mon Apr 29 14:00:14 2024
  write: IOPS=710, BW=2842KiB/s (2910kB/s)(10.0MiB/3603msec)
    slat (usec): min=11, max=1089, avg=48.93, stdev=37.31
    clat (usec): min=2532, max=64866, avg=22378.90, stdev=5771.33
     lat (usec): min=2583, max=64939, avg=22429.00, stdev=5779.34
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[16909], 10.00th=[17695], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20579], 50.00th=[21365], 60.00th=[21890],
     | 70.00th=[22938], 80.00th=[24249], 90.00th=[26608], 95.00th=[33817],
     | 99.00th=[44827], 99.50th=[57410], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min= 2144, max= 3352, per=24.93%, avg=2833.14, stdev=423.76, samples=7
   iops        : min=  536, max=  838, avg=708.29, stdev=105.94, samples=7
  lat (msec)   : 4=0.04%, 10=0.08%, 20=31.80%, 50=67.46%, 100=0.62%
  cpu          : usr=0.44%, sys=3.83%, ctx=1512, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10362: Mon Apr 29 14:00:14 2024
  write: IOPS=710, BW=2841KiB/s (2909kB/s)(10.0MiB/3604msec)
    slat (usec): min=11, max=935, avg=51.94, stdev=42.18
    clat (usec): min=2287, max=65364, avg=22376.18, stdev=5779.98
     lat (usec): min=2508, max=65413, avg=22429.28, stdev=5789.46
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[16712], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20579], 50.00th=[21365], 60.00th=[21890],
     | 70.00th=[22676], 80.00th=[23987], 90.00th=[26870], 95.00th=[33817],
     | 99.00th=[44827], 99.50th=[55313], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[65274]
   bw (  KiB/s): min= 2123, max= 3336, per=24.91%, avg=2831.29, stdev=426.26, samples=7
   iops        : min=  530, max=  834, avg=707.71, stdev=106.77, samples=7
  lat (msec)   : 4=0.08%, 10=0.04%, 20=32.03%, 50=67.23%, 100=0.62%
  cpu          : usr=0.58%, sys=3.94%, ctx=1708, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.1MiB/s (11.6MB/s), 2841KiB/s-2855KiB/s (2909kB/s-2923kB/s), io=40.0MiB (41.9MB), run=3587-3604msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=10375: Mon Apr 29 14:00:18 2024
   read: IOPS=403, BW=1613KiB/s (1652kB/s)(5048KiB/3129msec)
    slat (usec): min=723, max=3603, avg=1298.65, stdev=324.40
    clat (usec): min=9673, max=34799, avg=18684.54, stdev=3762.30
     lat (usec): min=10734, max=37059, avg=19985.11, stdev=3891.83
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13566], 10.00th=[14484], 20.00th=[15664],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[17957], 60.00th=[19006],
     | 70.00th=[20055], 80.00th=[21365], 90.00th=[23725], 95.00th=[26084],
     | 99.00th=[30016], 99.50th=[33424], 99.90th=[34866], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1424, max= 1848, per=25.25%, avg=1619.33, stdev=167.28, samples=6
   iops        : min=  356, max=  462, avg=404.83, stdev=41.82, samples=6
  write: IOPS=414, BW=1659KiB/s (1699kB/s)(5192KiB/3129msec)
    slat (usec): min=13, max=993, avg=53.21, stdev=48.01
    clat (usec): min=4611, max=34545, avg=18849.99, stdev=3455.56
     lat (usec): min=4649, max=34615, avg=18904.58, stdev=3460.21
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14615], 10.00th=[15270], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[20055], 80.00th=[21103], 90.00th=[23200], 95.00th=[25297],
     | 99.00th=[31065], 99.50th=[32900], 99.90th=[34341], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1208, max= 1984, per=24.86%, avg=1659.33, stdev=273.25, samples=6
   iops        : min=  302, max=  496, avg=414.83, stdev=68.31, samples=6
  lat (msec)   : 10=0.12%, 20=70.31%, 50=29.57%
  cpu          : usr=1.05%, sys=6.87%, ctx=2827, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10376: Mon Apr 29 14:00:18 2024
   read: IOPS=400, BW=1600KiB/s (1638kB/s)(4976KiB/3110msec)
    slat (usec): min=15, max=3992, avg=1283.10, stdev=338.88
    clat (usec): min=2091, max=40085, avg=18597.36, stdev=4002.38
     lat (usec): min=2107, max=42255, avg=19882.36, stdev=4145.41
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13829], 10.00th=[14484], 20.00th=[15401],
     | 30.00th=[16188], 40.00th=[17171], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[20055], 80.00th=[21365], 90.00th=[23462], 95.00th=[26346],
     | 99.00th=[31589], 99.50th=[34341], 99.90th=[38536], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1304, max= 1920, per=24.90%, avg=1596.67, stdev=205.28, samples=6
   iops        : min=  326, max=  480, avg=399.17, stdev=51.32, samples=6
  write: IOPS=423, BW=1693KiB/s (1733kB/s)(5264KiB/3110msec)
    slat (usec): min=12, max=610, avg=51.50, stdev=29.53
    clat (usec): min=3728, max=38310, avg=18880.98, stdev=3696.43
     lat (usec): min=3792, max=38352, avg=18933.70, stdev=3700.14
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[14091], 10.00th=[15139], 20.00th=[16319],
     | 30.00th=[16909], 40.00th=[17695], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[21365], 90.00th=[23200], 95.00th=[25035],
     | 99.00th=[32900], 99.50th=[34341], 99.90th=[38011], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 1328, max= 1920, per=25.51%, avg=1703.33, stdev=251.62, samples=6
   iops        : min=  332, max=  480, avg=425.83, stdev=62.90, samples=6
  lat (msec)   : 4=0.16%, 10=0.23%, 20=69.18%, 50=30.43%
  cpu          : usr=0.61%, sys=7.24%, ctx=2862, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10377: Mon Apr 29 14:00:18 2024
   read: IOPS=398, BW=1594KiB/s (1633kB/s)(4984KiB/3126msec)
    slat (usec): min=731, max=3322, avg=1296.52, stdev=319.84
    clat (usec): min=10363, max=38022, avg=18620.87, stdev=3901.62
     lat (usec): min=11538, max=40180, avg=19919.16, stdev=4037.26
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13435], 10.00th=[14484], 20.00th=[15401],
     | 30.00th=[16319], 40.00th=[17171], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[21103], 90.00th=[23725], 95.00th=[26084],
     | 99.00th=[31327], 99.50th=[32900], 99.90th=[35914], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1288, max= 1872, per=25.02%, avg=1604.67, stdev=241.94, samples=6
   iops        : min=  322, max=  468, avg=401.17, stdev=60.49, samples=6
  write: IOPS=420, BW=1681KiB/s (1722kB/s)(5256KiB/3126msec)
    slat (usec): min=13, max=311, avg=51.44, stdev=26.37
    clat (usec): min=3548, max=37542, avg=18928.66, stdev=3462.18
     lat (usec): min=3580, max=37735, avg=18981.25, stdev=3466.34
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14746], 10.00th=[15401], 20.00th=[16319],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[18220], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[21365], 90.00th=[23200], 95.00th=[25035],
     | 99.00th=[31065], 99.50th=[33424], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1304, max= 2064, per=25.11%, avg=1676.67, stdev=246.97, samples=6
   iops        : min=  326, max=  516, avg=419.17, stdev=61.74, samples=6
  lat (msec)   : 4=0.04%, 10=0.04%, 20=70.00%, 50=29.92%
  cpu          : usr=0.61%, sys=7.26%, ctx=2842, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10378: Mon Apr 29 14:00:18 2024
   read: IOPS=404, BW=1619KiB/s (1658kB/s)(5060KiB/3126msec)
    slat (usec): min=693, max=3373, avg=1288.60, stdev=325.78
    clat (usec): min=4926, max=35930, avg=18461.19, stdev=3749.10
     lat (usec): min=5935, max=38055, avg=19751.71, stdev=3869.66
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13435], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[16188], 40.00th=[17171], 50.00th=[17957], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[21103], 90.00th=[22938], 95.00th=[25560],
     | 99.00th=[29754], 99.50th=[33162], 99.90th=[35390], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1368, max= 1824, per=25.30%, avg=1622.67, stdev=201.97, samples=6
   iops        : min=  342, max=  456, avg=405.67, stdev=50.49, samples=6
  write: IOPS=414, BW=1657KiB/s (1697kB/s)(5180KiB/3126msec)
    slat (usec): min=14, max=993, avg=54.20, stdev=42.21
    clat (usec): min=4597, max=44760, avg=19067.86, stdev=3646.50
     lat (usec): min=4737, max=44836, avg=19123.44, stdev=3651.54
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14484], 10.00th=[15139], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17695], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20317], 80.00th=[21627], 90.00th=[23462], 95.00th=[25822],
     | 99.00th=[31327], 99.50th=[32900], 99.90th=[36439], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1304, max= 2008, per=24.89%, avg=1661.33, stdev=240.30, samples=6
   iops        : min=  326, max=  502, avg=415.33, stdev=60.08, samples=6
  lat (msec)   : 10=0.08%, 20=69.06%, 50=30.86%
  cpu          : usr=1.31%, sys=6.75%, ctx=2872, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6414KiB/s (6567kB/s), 1594KiB/s-1619KiB/s (1633kB/s-1658kB/s), io=19.6MiB (20.5MB), run=3110-3129msec
  WRITE: bw=6677KiB/s (6837kB/s), 1657KiB/s-1693KiB/s (1697kB/s-1733kB/s), io=20.4MiB (21.4MB), run=3110-3129msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=10384: Mon Apr 29 14:00:20 2024
   read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec)
    slat (nsec): min=9574.1k, max=9574.1k, avg=9574119.00, stdev= 0.00
    clat (nsec): min=61723k, max=61723k, avg=61722656.00, stdev= 0.00
     lat (nsec): min=71325k, max=71325k, avg=71325418.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[61604],  5.00th=[61604], 10.00th=[61604], 20.00th=[61604],
     | 30.00th=[61604], 40.00th=[61604], 50.00th=[61604], 60.00th=[61604],
     | 70.00th=[61604], 80.00th=[61604], 90.00th=[61604], 95.00th=[61604],
     | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[61604]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=13.33%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout