== sanity test 398c: run fio to test AIO ================= 18:48:36 (1710110916)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0812905 s, 516 MB/s
osc.lustre-OST0000-osc-ffff8800ab33b000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=14250: Sun Mar 10 18:48:41 2024
  write: IOPS=772, BW=3089KiB/s (3163kB/s)(10.0MiB/3315msec)
    slat (usec): min=12, max=266, avg=47.56, stdev=26.27
    clat (usec): min=2090, max=42046, avg=20581.98, stdev=2601.16
     lat (usec): min=2138, max=42068, avg=20630.87, stdev=2603.43
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[16909], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[26346], 99.50th=[30540], 99.90th=[39584], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2944, max= 3272, per=25.20%, avg=3109.50, stdev=119.18, samples=6
   iops        : min=  736, max=  818, avg=777.33, stdev=29.79, samples=6
  lat (msec)   : 4=0.16%, 20=37.97%, 50=61.88%
  cpu          : usr=0.78%, sys=3.92%, ctx=1979, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14251: Sun Mar 10 18:48:41 2024
  write: IOPS=772, BW=3091KiB/s (3165kB/s)(10.0MiB/3313msec)
    slat (usec): min=12, max=174, avg=43.05, stdev=24.02
    clat (usec): min=2151, max=42093, avg=20583.26, stdev=2704.79
     lat (usec): min=2237, max=42177, avg=20627.60, stdev=2704.81
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[16909], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[20317], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[23200], 95.00th=[23725],
     | 99.00th=[26608], 99.50th=[33424], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2952, max= 3336, per=25.19%, avg=3108.00, stdev=137.61, samples=6
   iops        : min=  738, max=  834, avg=777.00, stdev=34.40, samples=6
  lat (msec)   : 4=0.08%, 10=0.47%, 20=35.82%, 50=63.63%
  cpu          : usr=0.75%, sys=3.50%, ctx=1610, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14252: Sun Mar 10 18:48:41 2024
  write: IOPS=771, BW=3085KiB/s (3159kB/s)(10.0MiB/3319msec)
    slat (usec): min=12, max=295, avg=45.63, stdev=25.39
    clat (usec): min=1844, max=37493, avg=20613.04, stdev=2414.27
     lat (usec): min=1945, max=37518, avg=20660.00, stdev=2414.54
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[17171], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[26346], 99.50th=[31589], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[37487]
   bw (  KiB/s): min= 2920, max= 3248, per=25.16%, avg=3104.00, stdev=109.34, samples=6
   iops        : min=  730, max=  812, avg=776.00, stdev=27.33, samples=6
  lat (msec)   : 2=0.04%, 10=0.04%, 20=36.60%, 50=63.32%
  cpu          : usr=0.84%, sys=3.62%, ctx=1821, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14253: Sun Mar 10 18:48:41 2024
  write: IOPS=771, BW=3084KiB/s (3158kB/s)(10.0MiB/3320msec)
    slat (usec): min=11, max=320, avg=46.67, stdev=26.56
    clat (usec): min=4785, max=33765, avg=20620.10, stdev=2362.54
     lat (usec): min=4820, max=33782, avg=20668.05, stdev=2362.95
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[17171], 10.00th=[18220], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[22938], 95.00th=[23725],
     | 99.00th=[26346], 99.50th=[29492], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 2928, max= 3224, per=25.10%, avg=3096.00, stdev=104.06, samples=6
   iops        : min=  732, max=  806, avg=774.00, stdev=26.02, samples=6
  lat (msec)   : 10=0.08%, 20=36.76%, 50=63.16%
  cpu          : usr=0.60%, sys=3.86%, ctx=1993, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.0MiB/s (12.6MB/s), 3084KiB/s-3091KiB/s (3158kB/s-3165kB/s), io=40.0MiB (41.9MB), run=3313-3320msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=14266: Sun Mar 10 18:48:44 2024
   read: IOPS=458, BW=1834KiB/s (1878kB/s)(5048KiB/2752msec)
    slat (usec): min=13, max=2711, avg=1130.52, stdev=248.74
    clat (usec): min=2747, max=29495, avg=16325.52, stdev=2244.93
     lat (usec): min=2815, max=30451, avg=17457.90, stdev=2259.48
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13042], 10.00th=[13566], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21627], 99.50th=[21890], 99.90th=[28705], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1732, max= 1864, per=25.13%, avg=1825.60, stdev=54.45, samples=5
   iops        : min=  433, max=  466, avg=456.40, stdev=13.61, samples=5
  write: IOPS=471, BW=1887KiB/s (1932kB/s)(5192KiB/2752msec)
    slat (usec): min=13, max=573, avg=43.17, stdev=24.63
    clat (usec): min=9565, max=28572, avg=16865.23, stdev=1733.37
     lat (usec): min=9579, max=28606, avg=16909.62, stdev=1733.37
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14222], 10.00th=[14746], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19530],
     | 99.00th=[21890], 99.50th=[23200], 99.90th=[26084], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1804, max= 1976, per=25.02%, avg=1892.80, stdev=62.57, samples=5
   iops        : min=  451, max=  494, avg=473.20, stdev=15.64, samples=5
  lat (msec)   : 4=0.08%, 10=0.04%, 20=95.62%, 50=4.26%
  cpu          : usr=0.87%, sys=7.20%, ctx=2552, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14267: Sun Mar 10 18:48:44 2024
   read: IOPS=450, BW=1802KiB/s (1845kB/s)(4976KiB/2762msec)
    slat (usec): min=13, max=2385, avg=1134.05, stdev=233.60
    clat (usec): min=2960, max=33980, avg=16405.83, stdev=2320.34
     lat (usec): min=3854, max=35028, avg=17541.74, stdev=2326.29
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13173], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21365], 99.50th=[23462], 99.90th=[32637], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1668, max= 1936, per=24.64%, avg=1790.40, stdev=100.82, samples=5
   iops        : min=  417, max=  484, avg=447.60, stdev=25.21, samples=5
  write: IOPS=476, BW=1906KiB/s (1952kB/s)(5264KiB/2762msec)
    slat (usec): min=12, max=1459, avg=44.15, stdev=56.68
    clat (usec): min=4751, max=29288, avg=16913.15, stdev=1786.12
     lat (usec): min=4794, max=29341, avg=16958.62, stdev=1784.57
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14222], 10.00th=[14746], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21627], 99.50th=[22152], 99.90th=[25822], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1816, max= 2016, per=25.36%, avg=1918.40, stdev=86.38, samples=5
   iops        : min=  454, max=  504, avg=479.60, stdev=21.59, samples=5
  lat (msec)   : 4=0.08%, 10=0.12%, 20=95.55%, 50=4.26%
  cpu          : usr=1.09%, sys=6.66%, ctx=2616, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14268: Sun Mar 10 18:48:44 2024
   read: IOPS=451, BW=1806KiB/s (1850kB/s)(4984KiB/2759msec)
    slat (usec): min=14, max=2357, avg=1133.33, stdev=236.11
    clat (usec): min=9253, max=32092, avg=16326.45, stdev=2270.91
     lat (usec): min=9681, max=33315, avg=17461.65, stdev=2288.88
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[12911], 10.00th=[13566], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16188], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[22938], 99.50th=[23725], 99.90th=[30016], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1748, max= 1936, per=24.91%, avg=1809.60, stdev=72.81, samples=5
   iops        : min=  437, max=  484, avg=452.40, stdev=18.20, samples=5
  write: IOPS=476, BW=1905KiB/s (1951kB/s)(5256KiB/2759msec)
    slat (usec): min=13, max=553, avg=42.28, stdev=24.64
    clat (usec): min=3324, max=32008, avg=16953.55, stdev=1943.62
     lat (usec): min=3360, max=32025, avg=16997.04, stdev=1942.71
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14353], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19006], 95.00th=[20055],
     | 99.00th=[21627], 99.50th=[23200], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1780, max= 1976, per=25.13%, avg=1900.80, stdev=83.21, samples=5
   iops        : min=  445, max=  494, avg=475.20, stdev=20.80, samples=5
  lat (msec)   : 4=0.04%, 10=0.20%, 20=95.04%, 50=4.73%
  cpu          : usr=0.91%, sys=6.93%, ctx=2612, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14269: Sun Mar 10 18:48:44 2024
   read: IOPS=458, BW=1832KiB/s (1876kB/s)(5060KiB/2762msec)
    slat (usec): min=13, max=2067, avg=1128.56, stdev=239.10
    clat (usec): min=2420, max=23505, avg=16352.24, stdev=2181.34
     lat (usec): min=3868, max=24454, avg=17482.69, stdev=2186.38
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13042], 10.00th=[13698], 20.00th=[14484],
     | 30.00th=[15008], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19268], 95.00th=[19792],
     | 99.00th=[21890], 99.50th=[22152], 99.90th=[23462], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 1760, max= 1896, per=25.25%, avg=1834.40, stdev=51.35, samples=5
   iops        : min=  440, max=  474, avg=458.60, stdev=12.84, samples=5
  write: IOPS=468, BW=1875KiB/s (1920kB/s)(5180KiB/2762msec)
    slat (usec): min=12, max=318, avg=42.27, stdev=22.53
    clat (usec): min=7712, max=27304, avg=16964.19, stdev=1799.51
     lat (usec): min=7999, max=27358, avg=17007.84, stdev=1797.43
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14353], 10.00th=[14746], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21890], 99.50th=[22152], 99.90th=[27395], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1788, max= 2024, per=24.70%, avg=1868.00, stdev=97.57, samples=5
   iops        : min=  447, max=  506, avg=467.00, stdev=24.39, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=95.08%, 50=4.80%
  cpu          : usr=0.98%, sys=6.85%, ctx=2598, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7266KiB/s (7440kB/s), 1802KiB/s-1834KiB/s (1845kB/s-1878kB/s), io=19.6MiB (20.5MB), run=2752-2762msec
  WRITE: bw=7564KiB/s (7746kB/s), 1875KiB/s-1906KiB/s (1920kB/s-1952kB/s), io=20.4MiB (21.4MB), run=2752-2762msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=14274: Sun Mar 10 18:48:46 2024
   read: IOPS=14, BW=580MiB/s (608MB/s)(40.0MiB/69msec)
    slat (nsec): min=7442.2k, max=7442.2k, avg=7442196.00, stdev= 0.00
    clat (nsec): min=60293k, max=60293k, avg=60292948.00, stdev= 0.00
     lat (nsec): min=67755k, max=67755k, avg=67754681.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[60031],  5.00th=[60031], 10.00th=[60031], 20.00th=[60031],
     | 30.00th=[60031], 40.00th=[60031], 50.00th=[60031], 60.00th=[60031],
     | 70.00th=[60031], 80.00th=[60031], 90.00th=[60031], 95.00th=[60031],
     | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.29%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-608MB/s), io=40.0MiB (41.9MB), run=69-69msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout