== sanity test 398c: run fio to test AIO ================= 01:26:04 (1735107964)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0959965 s, 437 MB/s
osc.lustre-OST0000-osc-ffff8800b6465800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=26632: Wed Dec 25 01:26:09 2024
  write: IOPS=655, BW=2624KiB/s (2687kB/s)(10.0MiB/3903msec)
    slat (usec): min=12, max=412, avg=53.23, stdev=35.28
    clat (usec): min=10498, max=47080, avg=24202.93, stdev=7117.89
     lat (usec): min=10550, max=47179, avg=24257.67, stdev=7130.67
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[18220], 10.00th=[18744], 20.00th=[19792],
     | 30.00th=[20579], 40.00th=[21103], 50.00th=[21627], 60.00th=[22414],
     | 70.00th=[23462], 80.00th=[25822], 90.00th=[38536], 95.00th=[41681],
     | 99.00th=[45351], 99.50th=[45351], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1560, max= 3072, per=24.50%, avg=2570.86, stdev=594.69, samples=7
   iops        : min=  390, max=  768, avg=642.71, stdev=148.67, samples=7
  lat (msec)   : 20=22.07%, 50=77.93%
  cpu          : usr=0.85%, sys=3.56%, ctx=1580, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=26633: Wed Dec 25 01:26:09 2024
  write: IOPS=657, BW=2630KiB/s (2693kB/s)(10.0MiB/3894msec)
    slat (usec): min=12, max=540, avg=55.34, stdev=37.30
    clat (usec): min=5216, max=49805, avg=24197.48, stdev=7187.45
     lat (usec): min=5239, max=49836, avg=24254.22, stdev=7200.76
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[17957], 10.00th=[18744], 20.00th=[19792],
     | 30.00th=[20579], 40.00th=[21103], 50.00th=[21627], 60.00th=[22414],
     | 70.00th=[23462], 80.00th=[25822], 90.00th=[38536], 95.00th=[41681],
     | 99.00th=[45351], 99.50th=[45876], 99.90th=[49546], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min= 1552, max= 3072, per=24.67%, avg=2589.14, stdev=604.33, samples=7
   iops        : min=  388, max=  768, avg=647.29, stdev=151.08, samples=7
  lat (msec)   : 10=0.27%, 20=21.95%, 50=77.77%
  cpu          : usr=0.31%, sys=4.16%, ctx=1644, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=26634: Wed Dec 25 01:26:09 2024
  write: IOPS=655, BW=2624KiB/s (2687kB/s)(10.0MiB/3903msec)
    slat (usec): min=11, max=585, avg=53.39, stdev=34.04
    clat (usec): min=4096, max=48417, avg=24173.88, stdev=7092.79
     lat (usec): min=4159, max=48665, avg=24228.67, stdev=7102.74
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[17957], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[20579], 40.00th=[21103], 50.00th=[21627], 60.00th=[22152],
     | 70.00th=[23462], 80.00th=[25822], 90.00th=[38011], 95.00th=[42206],
     | 99.00th=[44827], 99.50th=[45876], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[48497]
   bw (  KiB/s): min= 1552, max= 3072, per=24.45%, avg=2566.14, stdev=589.38, samples=7
   iops        : min=  388, max=  768, avg=641.43, stdev=147.46, samples=7
  lat (msec)   : 10=0.08%, 20=20.43%, 50=79.49%
  cpu          : usr=0.72%, sys=3.79%, ctx=1552, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=26635: Wed Dec 25 01:26:09 2024
  write: IOPS=657, BW=2628KiB/s (2691kB/s)(10.0MiB/3896msec)
    slat (usec): min=13, max=601, avg=55.25, stdev=35.44
    clat (usec): min=2957, max=49811, avg=24190.60, stdev=7156.13
     lat (usec): min=3024, max=49851, avg=24247.17, stdev=7168.74
    clat percentiles (usec):
     |  1.00th=[16581],  5.00th=[17957], 10.00th=[18744], 20.00th=[19792],
     | 30.00th=[20579], 40.00th=[21103], 50.00th=[21627], 60.00th=[22414],
     | 70.00th=[23462], 80.00th=[25822], 90.00th=[38011], 95.00th=[41681],
     | 99.00th=[45351], 99.50th=[46400], 99.90th=[49546], 99.95th=[49546],
     | 99.99th=[50070]
   bw (  KiB/s): min= 1568, max= 3096, per=24.61%, avg=2582.29, stdev=609.73, samples=7
   iops        : min=  392, max=  774, avg=645.57, stdev=152.43, samples=7
  lat (msec)   : 4=0.08%, 10=0.04%, 20=21.56%, 50=78.32%
  cpu          : usr=0.64%, sys=4.06%, ctx=1705, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.2MiB/s (10.7MB/s), 2624KiB/s-2630KiB/s (2687kB/s-2693kB/s), io=40.0MiB (41.9MB), run=3894-3903msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=26647: Wed Dec 25 01:26:13 2024
   read: IOPS=453, BW=1816KiB/s (1859kB/s)(5048KiB/2780msec)
    slat (usec): min=632, max=2400, avg=1126.96, stdev=225.26
    clat (usec): min=10015, max=24835, avg=16377.79, stdev=2218.44
     lat (usec): min=11189, max=26062, avg=17506.56, stdev=2221.33
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12780], 10.00th=[13566], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21365], 99.50th=[22414], 99.90th=[23725], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 1696, max= 1832, per=24.75%, avg=1786.40, stdev=55.54, samples=5
   iops        : min=  424, max=  458, avg=446.60, stdev=13.89, samples=5
  write: IOPS=466, BW=1868KiB/s (1912kB/s)(5192KiB/2780msec)
    slat (usec): min=12, max=319, avg=45.10, stdev=21.68
    clat (usec): min=10764, max=23642, avg=16884.22, stdev=1762.76
     lat (usec): min=10805, max=23687, avg=16930.58, stdev=1762.46
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[14091], 10.00th=[14746], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[19792],
     | 99.00th=[21627], 99.50th=[22152], 99.90th=[22938], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 1832, max= 1916, per=24.88%, avg=1869.60, stdev=34.59, samples=5
   iops        : min=  458, max=  479, avg=467.40, stdev= 8.65, samples=5
  lat (msec)   : 20=95.47%, 50=4.53%
  cpu          : usr=0.58%, sys=7.16%, ctx=2901, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26648: Wed Dec 25 01:26:13 2024
   read: IOPS=451, BW=1806KiB/s (1850kB/s)(4976KiB/2755msec)
    slat (usec): min=24, max=3879, avg=1132.48, stdev=252.17
    clat (usec): min=2353, max=27629, avg=16368.49, stdev=2395.07
     lat (usec): min=2702, max=28602, avg=17502.83, stdev=2399.60
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12780], 10.00th=[13566], 20.00th=[14222],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21890], 99.50th=[22414], 99.90th=[25560], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1724, max= 1880, per=24.73%, avg=1784.80, stdev=67.91, samples=5
   iops        : min=  431, max=  470, avg=446.20, stdev=16.98, samples=5
  write: IOPS=477, BW=1911KiB/s (1957kB/s)(5264KiB/2755msec)
    slat (usec): min=13, max=1400, avg=48.25, stdev=49.31
    clat (usec): min=2998, max=26883, avg=16796.67, stdev=1897.92
     lat (usec): min=3067, max=26930, avg=16846.18, stdev=1896.49
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13829], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21365], 99.50th=[21890], 99.90th=[23200], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1792, max= 1992, per=25.45%, avg=1912.80, stdev=80.52, samples=5
   iops        : min=  448, max=  498, avg=478.20, stdev=20.13, samples=5
  lat (msec)   : 4=0.16%, 10=0.23%, 20=94.73%, 50=4.88%
  cpu          : usr=0.94%, sys=7.01%, ctx=2931, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26649: Wed Dec 25 01:26:13 2024
   read: IOPS=448, BW=1795KiB/s (1838kB/s)(4984KiB/2777msec)
    slat (usec): min=703, max=2306, avg=1141.77, stdev=216.29
    clat (usec): min=3401, max=23583, avg=16395.03, stdev=2226.36
     lat (usec): min=4663, max=24847, avg=17538.51, stdev=2224.68
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12911], 10.00th=[13566], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21365], 99.50th=[21627], 99.90th=[22414], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 1720, max= 1940, per=24.75%, avg=1786.40, stdev=88.16, samples=5
   iops        : min=  430, max=  485, avg=446.60, stdev=22.04, samples=5
  write: IOPS=473, BW=1893KiB/s (1938kB/s)(5256KiB/2777msec)
    slat (usec): min=12, max=223, avg=46.57, stdev=22.17
    clat (usec): min=6928, max=25454, avg=16812.73, stdev=1781.42
     lat (usec): min=6976, max=25489, avg=16860.49, stdev=1780.65
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[13960], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16909], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19530],
     | 99.00th=[21365], 99.50th=[22676], 99.90th=[25560], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 1760, max= 1976, per=24.94%, avg=1874.40, stdev=103.83, samples=5
   iops        : min=  440, max=  494, avg=468.60, stdev=25.96, samples=5
  lat (msec)   : 4=0.04%, 10=0.04%, 20=95.51%, 50=4.41%
  cpu          : usr=0.76%, sys=7.10%, ctx=2911, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26650: Wed Dec 25 01:26:13 2024
   read: IOPS=455, BW=1821KiB/s (1865kB/s)(5060KiB/2778msec)
    slat (usec): min=674, max=2522, avg=1136.38, stdev=224.06
    clat (usec): min=3046, max=25141, avg=16344.39, stdev=2185.04
     lat (usec): min=4252, max=26732, avg=17482.44, stdev=2191.20
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12780], 10.00th=[13566], 20.00th=[14484],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21365], 99.50th=[22152], 99.90th=[23987], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1712, max= 1880, per=25.18%, avg=1817.60, stdev=66.55, samples=5
   iops        : min=  428, max=  470, avg=454.40, stdev=16.64, samples=5
  write: IOPS=466, BW=1865KiB/s (1909kB/s)(5180KiB/2778msec)
    slat (usec): min=13, max=1828, avg=48.66, stdev=66.70
    clat (usec): min=7685, max=27998, avg=16819.91, stdev=1827.85
     lat (usec): min=7736, max=28049, avg=16869.78, stdev=1829.63
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14091], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21365], 99.50th=[22414], 99.90th=[26870], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1760, max= 1952, per=24.55%, avg=1844.80, stdev=75.39, samples=5
   iops        : min=  440, max=  488, avg=461.20, stdev=18.85, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=95.66%, 50=4.22%
  cpu          : usr=0.76%, sys=7.06%, ctx=2919, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7219KiB/s (7392kB/s), 1795KiB/s-1821KiB/s (1838kB/s-1865kB/s), io=19.6MiB (20.5MB), run=2755-2780msec
  WRITE: bw=7515KiB/s (7695kB/s), 1865KiB/s-1911KiB/s (1909kB/s-1957kB/s), io=20.4MiB (21.4MB), run=2755-2780msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=26656: Wed Dec 25 01:26:14 2024
   read: IOPS=12, BW=482MiB/s (505MB/s)(40.0MiB/83msec)
    slat (nsec): min=6904.8k, max=6904.8k, avg=6904759.00, stdev= 0.00
    clat (nsec): min=71842k, max=71842k, avg=71841834.00, stdev= 0.00
     lat (nsec): min=78760k, max=78760k, avg=78759863.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[71828],  5.00th=[71828], 10.00th=[71828], 20.00th=[71828],
     | 30.00th=[71828], 40.00th=[71828], 50.00th=[71828], 60.00th=[71828],
     | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828],
     | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[71828]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.76%, ctx=4, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=482MiB/s (505MB/s), 482MiB/s-482MiB/s (505MB/s-505MB/s), io=40.0MiB (41.9MB), run=83-83msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout