== sanity test 398c: run fio to test AIO ================= 03:39:31 (1734683971)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0879418 s, 477 MB/s
osc.lustre-OST0000-osc-ffff88012b162800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18304: Fri Dec 20 03:39:35 2024
  write: IOPS=827, BW=3311KiB/s (3390kB/s)(10.0MiB/3093msec)
    slat (usec): min=12, max=341, avg=44.02, stdev=26.41
    clat (usec): min=2548, max=35758, avg=19230.92, stdev=2841.17
     lat (usec): min=2596, max=35791, avg=19275.98, stdev=2842.40
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[15139], 10.00th=[16057], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18744], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22676], 95.00th=[23462],
     | 99.00th=[24773], 99.50th=[25035], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[35914]
   bw (  KiB/s): min= 3032, max= 3664, per=25.02%, avg=3297.50, stdev=215.14, samples=6
   iops        : min=  758, max=  916, avg=824.33, stdev=53.81, samples=6
  lat (msec)   : 4=0.12%, 10=1.05%, 20=59.41%, 50=39.41%
  cpu          : usr=0.45%, sys=4.27%, ctx=1524, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18305: Fri Dec 20 03:39:35 2024
  write: IOPS=823, BW=3295KiB/s (3374kB/s)(10.0MiB/3108msec)
    slat (usec): min=12, max=283, avg=42.73, stdev=23.03
    clat (usec): min=2166, max=27758, avg=19251.22, stdev=2629.67
     lat (usec): min=2277, max=27779, avg=19294.99, stdev=2631.35
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[15139], 10.00th=[16057], 20.00th=[17171],
     | 30.00th=[17957], 40.00th=[18744], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22414], 95.00th=[23462],
     | 99.00th=[24511], 99.50th=[25035], 99.90th=[26084], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 3032, max= 3456, per=24.80%, avg=3268.17, stdev=171.42, samples=6
   iops        : min=  758, max=  864, avg=817.00, stdev=42.87, samples=6
  lat (msec)   : 4=0.04%, 10=0.55%, 20=59.88%, 50=39.53%
  cpu          : usr=0.64%, sys=4.28%, ctx=1360, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18306: Fri Dec 20 03:39:35 2024
  write: IOPS=824, BW=3297KiB/s (3376kB/s)(10.0MiB/3106msec)
    slat (usec): min=11, max=379, avg=43.88, stdev=24.83
    clat (usec): min=2467, max=35710, avg=19300.61, stdev=2679.36
     lat (usec): min=2524, max=35905, avg=19345.58, stdev=2681.14
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[15139], 10.00th=[16188], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22414], 95.00th=[23200],
     | 99.00th=[24773], 99.50th=[25035], 99.90th=[35390], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 3056, max= 3576, per=24.96%, avg=3289.50, stdev=182.14, samples=6
   iops        : min=  764, max=  894, avg=822.33, stdev=45.56, samples=6
  lat (msec)   : 4=0.08%, 10=0.08%, 20=60.08%, 50=39.77%
  cpu          : usr=0.55%, sys=4.19%, ctx=1446, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18307: Fri Dec 20 03:39:35 2024
  write: IOPS=824, BW=3297KiB/s (3376kB/s)(10.0MiB/3106msec)
    slat (usec): min=11, max=777, avg=44.36, stdev=29.44
    clat (usec): min=7569, max=35861, avg=19274.39, stdev=2644.77
     lat (usec): min=7653, max=35884, avg=19319.85, stdev=2647.24
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[15270], 10.00th=[16188], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22414], 95.00th=[23200],
     | 99.00th=[24773], 99.50th=[25035], 99.90th=[33817], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 3048, max= 3512, per=24.94%, avg=3286.67, stdev=168.37, samples=6
   iops        : min=  762, max=  878, avg=821.67, stdev=42.09, samples=6
  lat (msec)   : 10=0.66%, 20=60.08%, 50=39.26%
  cpu          : usr=0.93%, sys=3.86%, ctx=1595, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.9MiB/s (13.5MB/s), 3295KiB/s-3311KiB/s (3374kB/s-3390kB/s), io=40.0MiB (41.9MB), run=3093-3108msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18319: Fri Dec 20 03:39:39 2024
   read: IOPS=501, BW=2008KiB/s (2056kB/s)(5048KiB/2514msec)
    slat (usec): min=605, max=2233, avg=1026.21, stdev=226.45
    clat (usec): min=1915, max=24848, avg=14823.67, stdev=2140.67
     lat (usec): min=2678, max=25726, avg=15851.54, stdev=2173.93
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11731], 10.00th=[12256], 20.00th=[12911],
     | 30.00th=[13698], 40.00th=[14222], 50.00th=[14877], 60.00th=[15401],
     | 70.00th=[15926], 80.00th=[16581], 90.00th=[17433], 95.00th=[18220],
     | 99.00th=[20317], 99.50th=[21103], 99.90th=[24249], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 1832, max= 2216, per=25.12%, avg=2003.20, stdev=156.32, samples=5
   iops        : min=  458, max=  554, avg=500.80, stdev=39.08, samples=5
  write: IOPS=516, BW=2065KiB/s (2115kB/s)(5192KiB/2514msec)
    slat (usec): min=11, max=274, avg=41.83, stdev=22.47
    clat (usec): min=9755, max=23061, avg=15323.74, stdev=1746.52
     lat (usec): min=9782, max=23095, avg=15366.64, stdev=1748.84
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12518], 10.00th=[13173], 20.00th=[13960],
     | 30.00th=[14353], 40.00th=[14877], 50.00th=[15270], 60.00th=[15664],
     | 70.00th=[16188], 80.00th=[16712], 90.00th=[17433], 95.00th=[18482],
     | 99.00th=[19792], 99.50th=[20317], 99.90th=[21103], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 1992, max= 2128, per=24.84%, avg=2062.40, stdev=52.28, samples=5
   iops        : min=  498, max=  532, avg=515.60, stdev=13.07, samples=5
  lat (msec)   : 2=0.04%, 10=0.27%, 20=98.79%, 50=0.90%
  cpu          : usr=0.76%, sys=7.04%, ctx=2832, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18320: Fri Dec 20 03:39:39 2024
   read: IOPS=495, BW=1982KiB/s (2029kB/s)(4976KiB/2511msec)
    slat (usec): min=622, max=2459, avg=1022.83, stdev=227.81
    clat (usec): min=3642, max=28290, avg=14899.24, stdev=2283.37
     lat (usec): min=4988, max=29334, avg=15923.67, stdev=2310.58
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[11469], 10.00th=[12125], 20.00th=[12911],
     | 30.00th=[13698], 40.00th=[14353], 50.00th=[14877], 60.00th=[15401],
     | 70.00th=[15926], 80.00th=[16712], 90.00th=[17433], 95.00th=[18220],
     | 99.00th=[21103], 99.50th=[22676], 99.90th=[27395], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1848, max= 2144, per=24.75%, avg=1974.40, stdev=142.82, samples=5
   iops        : min=  462, max=  536, avg=493.60, stdev=35.70, samples=5
  write: IOPS=524, BW=2096KiB/s (2147kB/s)(5264KiB/2511msec)
    slat (usec): min=13, max=879, avg=41.35, stdev=30.09
    clat (usec): min=8447, max=28224, avg=15284.71, stdev=1875.50
     lat (usec): min=8481, max=28282, avg=15327.12, stdev=1876.62
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[12387], 10.00th=[13042], 20.00th=[13829],
     | 30.00th=[14353], 40.00th=[14877], 50.00th=[15270], 60.00th=[15664],
     | 70.00th=[16188], 80.00th=[16712], 90.00th=[17695], 95.00th=[18220],
     | 99.00th=[19792], 99.50th=[20579], 99.90th=[26608], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 2024, max= 2168, per=25.26%, avg=2097.60, stdev=55.25, samples=5
   iops        : min=  506, max=  542, avg=524.40, stdev=13.81, samples=5
  lat (msec)   : 4=0.04%, 10=0.39%, 20=98.12%, 50=1.45%
  cpu          : usr=0.72%, sys=6.97%, ctx=2884, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18321: Fri Dec 20 03:39:39 2024
   read: IOPS=499, BW=1998KiB/s (2046kB/s)(4984KiB/2495msec)
    slat (usec): min=10, max=2407, avg=1009.63, stdev=232.39
    clat (usec): min=2039, max=26307, avg=14807.77, stdev=2326.55
     lat (usec): min=2135, max=27468, avg=15819.00, stdev=2365.70
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[11469], 10.00th=[11994], 20.00th=[12911],
     | 30.00th=[13698], 40.00th=[14222], 50.00th=[14877], 60.00th=[15270],
     | 70.00th=[15926], 80.00th=[16712], 90.00th=[17433], 95.00th=[18482],
     | 99.00th=[20317], 99.50th=[20841], 99.90th=[25297], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1920, max= 2072, per=24.97%, avg=1992.00, stdev=79.20, samples=4
   iops        : min=  480, max=  518, avg=498.00, stdev=19.80, samples=4
  write: IOPS=526, BW=2107KiB/s (2157kB/s)(5256KiB/2495msec)
    slat (usec): min=11, max=971, avg=41.89, stdev=33.49
    clat (usec): min=4161, max=28639, avg=15262.59, stdev=2082.72
     lat (usec): min=4278, max=28704, avg=15305.54, stdev=2082.50
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12387], 10.00th=[13042], 20.00th=[13698],
     | 30.00th=[14222], 40.00th=[14615], 50.00th=[15139], 60.00th=[15664],
     | 70.00th=[16057], 80.00th=[16909], 90.00th=[17695], 95.00th=[18482],
     | 99.00th=[20055], 99.50th=[22676], 99.90th=[28705], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1960, max= 2192, per=25.00%, avg=2076.00, stdev=113.23, samples=4
   iops        : min=  490, max=  548, avg=519.00, stdev=28.31, samples=4
  lat (msec)   : 4=0.23%, 10=0.66%, 20=97.97%, 50=1.13%
  cpu          : usr=1.16%, sys=6.58%, ctx=2703, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18322: Fri Dec 20 03:39:39 2024
   read: IOPS=502, BW=2011KiB/s (2059kB/s)(5060KiB/2516msec)
    slat (usec): min=575, max=2165, avg=1011.78, stdev=213.95
    clat (usec): min=2210, max=23263, avg=14840.89, stdev=2106.28
     lat (usec): min=3028, max=24056, avg=15854.20, stdev=2135.65
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[11600], 10.00th=[12256], 20.00th=[13042],
     | 30.00th=[13698], 40.00th=[14353], 50.00th=[14877], 60.00th=[15270],
     | 70.00th=[15926], 80.00th=[16581], 90.00th=[17433], 95.00th=[18220],
     | 99.00th=[20055], 99.50th=[21103], 99.90th=[22938], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 1856, max= 2144, per=25.18%, avg=2008.00, stdev=140.40, samples=5
   iops        : min=  464, max=  536, avg=502.00, stdev=35.10, samples=5
  write: IOPS=514, BW=2059KiB/s (2108kB/s)(5180KiB/2516msec)
    slat (usec): min=12, max=959, avg=43.28, stdev=36.79
    clat (usec): min=10080, max=20924, avg=15308.97, stdev=1739.65
     lat (usec): min=10118, max=20984, avg=15353.43, stdev=1739.77
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12518], 10.00th=[13173], 20.00th=[13829],
     | 30.00th=[14353], 40.00th=[14877], 50.00th=[15270], 60.00th=[15664],
     | 70.00th=[16188], 80.00th=[16712], 90.00th=[17433], 95.00th=[18482],
     | 99.00th=[20055], 99.50th=[20317], 99.90th=[20841], 99.95th=[20841],
     | 99.99th=[20841]
   bw (  KiB/s): min= 1960, max= 2104, per=24.65%, avg=2046.40, stdev=55.25, samples=5
   iops        : min=  490, max=  526, avg=511.60, stdev=13.81, samples=5
  lat (msec)   : 4=0.08%, 10=0.12%, 20=98.87%, 50=0.94%
  cpu          : usr=0.72%, sys=6.92%, ctx=2954, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7976KiB/s (8168kB/s), 1982KiB/s-2011KiB/s (2029kB/s-2059kB/s), io=19.6MiB (20.5MB), run=2495-2516msec
  WRITE: bw=8304KiB/s (8503kB/s), 2059KiB/s-2107KiB/s (2108kB/s-2157kB/s), io=20.4MiB (21.4MB), run=2495-2516msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18328: Fri Dec 20 03:39:40 2024
   read: IOPS=14, BW=563MiB/s (591MB/s)(40.0MiB/71msec)
    slat (nsec): min=6675.2k, max=6675.2k, avg=6675193.00, stdev= 0.00
    clat (nsec): min=59508k, max=59508k, avg=59508109.00, stdev= 0.00
     lat (nsec): min=66214k, max=66214k, avg=66213880.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[59507],  5.00th=[59507], 10.00th=[59507], 20.00th=[59507],
     | 30.00th=[59507], 40.00th=[59507], 50.00th=[59507], 60.00th=[59507],
     | 70.00th=[59507], 80.00th=[59507], 90.00th=[59507], 95.00th=[59507],
     | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.57%, ctx=10, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=563MiB/s (591MB/s), 563MiB/s-563MiB/s (591MB/s-591MB/s), io=40.0MiB (41.9MB), run=71-71msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout