== sanity test 398c: run fio to test AIO ================= 18:13:47 (1718403227)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.160602 s, 261 MB/s
osc.lustre-OST0000-osc-ffff8800ab85e800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=14674: Fri Jun 14 18:13:53 2024
  write: IOPS=649, BW=2598KiB/s (2660kB/s)(10.0MiB/3942msec)
    slat (usec): min=11, max=490, avg=62.81, stdev=40.76
    clat (usec): min=2256, max=66206, avg=24529.47, stdev=8849.40
     lat (usec): min=2300, max=66234, avg=24594.07, stdev=8866.35
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12780], 10.00th=[13960], 20.00th=[15401],
     | 30.00th=[17957], 40.00th=[19792], 50.00th=[22152], 60.00th=[27919],
     | 70.00th=[32375], 80.00th=[34341], 90.00th=[35914], 95.00th=[36963],
     | 99.00th=[39060], 99.50th=[43254], 99.90th=[59507], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min= 1820, max= 3760, per=26.05%, avg=2689.86, stdev=849.56, samples=7
   iops        : min=  455, max=  940, avg=672.43, stdev=212.36, samples=7
  lat (msec)   : 4=0.08%, 10=0.31%, 20=41.45%, 50=57.89%, 100=0.27%
  cpu          : usr=0.69%, sys=4.69%, ctx=1512, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14675: Fri Jun 14 18:13:53 2024
  write: IOPS=645, BW=2583KiB/s (2645kB/s)(10.0MiB/3965msec)
    slat (usec): min=11, max=289, avg=59.01, stdev=37.48
    clat (usec): min=9001, max=59238, avg=24658.59, stdev=8735.65
     lat (usec): min=9051, max=59346, avg=24719.32, stdev=8750.47
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12911], 10.00th=[13829], 20.00th=[15401],
     | 30.00th=[18220], 40.00th=[19792], 50.00th=[22414], 60.00th=[28443],
     | 70.00th=[32637], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[42206], 99.50th=[42730], 99.90th=[47973], 99.95th=[55313],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1832, max= 3704, per=25.89%, avg=2672.71, stdev=830.99, samples=7
   iops        : min=  458, max=  926, avg=668.14, stdev=207.72, samples=7
  lat (msec)   : 10=0.04%, 20=41.80%, 50=58.09%, 100=0.08%
  cpu          : usr=0.86%, sys=4.19%, ctx=1301, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14676: Fri Jun 14 18:13:53 2024
  write: IOPS=645, BW=2581KiB/s (2643kB/s)(10.0MiB/3967msec)
    slat (usec): min=12, max=358, avg=63.88, stdev=41.02
    clat (usec): min=10574, max=64925, avg=24667.13, stdev=8763.00
     lat (usec): min=10597, max=64954, avg=24732.71, stdev=8781.18
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12780], 10.00th=[13960], 20.00th=[15401],
     | 30.00th=[18220], 40.00th=[19792], 50.00th=[22152], 60.00th=[28181],
     | 70.00th=[32375], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[42730], 99.50th=[42730], 99.90th=[50594], 99.95th=[50594],
     | 99.99th=[64750]
   bw (  KiB/s): min= 1856, max= 3672, per=25.88%, avg=2672.14, stdev=824.36, samples=7
   iops        : min=  464, max=  918, avg=668.00, stdev=206.06, samples=7
  lat (msec)   : 20=41.41%, 50=58.40%, 100=0.20%
  cpu          : usr=0.73%, sys=4.49%, ctx=1667, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=14677: Fri Jun 14 18:13:53 2024
  write: IOPS=645, BW=2583KiB/s (2645kB/s)(10.0MiB/3965msec)
    slat (usec): min=12, max=353, avg=68.12, stdev=44.50
    clat (usec): min=7182, max=62560, avg=24637.12, stdev=8804.25
     lat (usec): min=7282, max=62682, avg=24706.94, stdev=8825.58
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12780], 10.00th=[13960], 20.00th=[15533],
     | 30.00th=[17957], 40.00th=[19530], 50.00th=[22414], 60.00th=[27919],
     | 70.00th=[32375], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[41157], 99.50th=[41157], 99.90th=[62653], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1876, max= 3664, per=25.94%, avg=2678.43, stdev=819.53, samples=7
   iops        : min=  469, max=  916, avg=669.57, stdev=204.85, samples=7
  lat (msec)   : 10=0.12%, 20=41.88%, 50=57.73%, 100=0.27%
  cpu          : usr=0.91%, sys=4.57%, ctx=1921, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.1MiB/s (10.6MB/s), 2581KiB/s-2598KiB/s (2643kB/s-2660kB/s), io=40.0MiB (41.9MB), run=3942-3967msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=14690: Fri Jun 14 18:13:56 2024
   read: IOPS=653, BW=2613KiB/s (2676kB/s)(5048KiB/1932msec)
    slat (usec): min=413, max=1518, avg=740.23, stdev=135.50
    clat (usec): min=2982, max=17305, avg=11455.04, stdev=1655.00
     lat (usec): min=3690, max=18742, avg=12197.04, stdev=1682.58
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 8979], 10.00th=[ 9503], 20.00th=[10159],
     | 30.00th=[10552], 40.00th=[10945], 50.00th=[11338], 60.00th=[11731],
     | 70.00th=[12256], 80.00th=[12780], 90.00th=[13566], 95.00th=[14222],
     | 99.00th=[16057], 99.50th=[16450], 99.90th=[17171], 99.95th=[17433],
     | 99.99th=[17433]
   bw (  KiB/s): min= 2304, max= 2656, per=24.29%, avg=2522.67, stdev=190.89, samples=3
   iops        : min=  576, max=  664, avg=630.67, stdev=47.72, samples=3
  write: IOPS=671, BW=2687KiB/s (2752kB/s)(5192KiB/1932msec)
    slat (usec): min=12, max=820, avg=40.39, stdev=31.63
    clat (usec): min=6194, max=19117, avg=11744.14, stdev=1459.20
     lat (usec): min=6240, max=19144, avg=11785.63, stdev=1458.42
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10683],
     | 30.00th=[11076], 40.00th=[11338], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12256], 80.00th=[12780], 90.00th=[13435], 95.00th=[14091],
     | 99.00th=[16712], 99.50th=[17695], 99.90th=[18220], 99.95th=[19006],
     | 99.99th=[19006]
   bw (  KiB/s): min= 2512, max= 2752, per=24.56%, avg=2656.00, stdev=127.00, samples=3
   iops        : min=  628, max=  688, avg=664.00, stdev=31.75, samples=3
  lat (msec)   : 4=0.08%, 10=13.67%, 20=86.25%
  cpu          : usr=1.19%, sys=9.53%, ctx=2621, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14691: Fri Jun 14 18:13:56 2024
   read: IOPS=644, BW=2578KiB/s (2640kB/s)(4976KiB/1930msec)
    slat (usec): min=433, max=1673, avg=738.79, stdev=135.70
    clat (usec): min=2450, max=21642, avg=11450.41, stdev=1752.28
     lat (usec): min=3065, max=22159, avg=12191.05, stdev=1788.89
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 8979], 10.00th=[ 9503], 20.00th=[10159],
     | 30.00th=[10552], 40.00th=[10945], 50.00th=[11338], 60.00th=[11731],
     | 70.00th=[12256], 80.00th=[12780], 90.00th=[13566], 95.00th=[14222],
     | 99.00th=[16581], 99.50th=[18744], 99.90th=[20841], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 2320, max= 2624, per=23.98%, avg=2490.67, stdev=155.40, samples=3
   iops        : min=  580, max=  656, avg=622.67, stdev=38.85, samples=3
  write: IOPS=681, BW=2727KiB/s (2793kB/s)(5264KiB/1930msec)
    slat (usec): min=12, max=275, avg=39.79, stdev=21.16
    clat (usec): min=6331, max=22162, avg=11756.58, stdev=1579.00
     lat (usec): min=6368, max=22189, avg=11797.33, stdev=1579.29
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11731], 60.00th=[11994],
     | 70.00th=[12387], 80.00th=[12780], 90.00th=[13435], 95.00th=[14222],
     | 99.00th=[17171], 99.50th=[17957], 99.90th=[21627], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 2512, max= 2840, per=25.03%, avg=2706.67, stdev=172.39, samples=3
   iops        : min=  628, max=  710, avg=676.67, stdev=43.10, samples=3
  lat (msec)   : 4=0.04%, 10=13.95%, 20=85.86%, 50=0.16%
  cpu          : usr=1.30%, sys=9.18%, ctx=2666, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14692: Fri Jun 14 18:13:56 2024
   read: IOPS=646, BW=2586KiB/s (2648kB/s)(4984KiB/1927msec)
    slat (usec): min=34, max=1564, avg=733.83, stdev=145.37
    clat (usec): min=1670, max=19710, avg=11451.56, stdev=1720.91
     lat (usec): min=1721, max=20583, avg=12187.02, stdev=1760.85
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 8848], 10.00th=[ 9372], 20.00th=[10159],
     | 30.00th=[10552], 40.00th=[10945], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12256], 80.00th=[12649], 90.00th=[13435], 95.00th=[14222],
     | 99.00th=[15795], 99.50th=[16909], 99.90th=[19006], 99.95th=[19792],
     | 99.99th=[19792]
   bw (  KiB/s): min= 2488, max= 2640, per=24.57%, avg=2552.00, stdev=78.79, samples=3
   iops        : min=  622, max=  660, avg=638.00, stdev=19.70, samples=3
  write: IOPS=681, BW=2728KiB/s (2793kB/s)(5256KiB/1927msec)
    slat (usec): min=12, max=738, avg=41.66, stdev=32.39
    clat (usec): min=4707, max=22473, avg=11787.40, stdev=1625.86
     lat (usec): min=4756, max=22489, avg=11830.13, stdev=1625.43
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9372], 10.00th=[ 9896], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11731], 60.00th=[12125],
     | 70.00th=[12387], 80.00th=[12911], 90.00th=[13435], 95.00th=[14222],
     | 99.00th=[16581], 99.50th=[17957], 99.90th=[22414], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 2480, max= 2784, per=24.61%, avg=2661.33, stdev=160.27, samples=3
   iops        : min=  620, max=  696, avg=665.33, stdev=40.07, samples=3
  lat (msec)   : 2=0.08%, 4=0.08%, 10=13.91%, 20=85.78%, 50=0.16%
  cpu          : usr=1.09%, sys=9.45%, ctx=2674, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14693: Fri Jun 14 18:13:56 2024
   read: IOPS=654, BW=2619KiB/s (2682kB/s)(5060KiB/1932msec)
    slat (usec): min=430, max=1464, avg=743.79, stdev=137.57
    clat (usec): min=1500, max=17465, avg=11460.84, stdev=1716.70
     lat (usec): min=2097, max=18390, avg=12206.23, stdev=1745.96
    clat percentiles (usec):
     |  1.00th=[ 7832],  5.00th=[ 8848], 10.00th=[ 9503], 20.00th=[10028],
     | 30.00th=[10552], 40.00th=[10945], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12256], 80.00th=[12780], 90.00th=[13698], 95.00th=[14353],
     | 99.00th=[15795], 99.50th=[16581], 99.90th=[16909], 99.95th=[17433],
     | 99.99th=[17433]
   bw (  KiB/s): min= 2312, max= 2680, per=24.57%, avg=2552.00, stdev=208.00, samples=3
   iops        : min=  578, max=  670, avg=638.00, stdev=52.00, samples=3
  write: IOPS=670, BW=2681KiB/s (2746kB/s)(5180KiB/1932msec)
    slat (usec): min=12, max=283, avg=40.37, stdev=22.70
    clat (usec): min=2134, max=19101, avg=11754.78, stdev=1544.32
     lat (usec): min=2170, max=19124, avg=11796.23, stdev=1545.81
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9503], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11731], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[12911], 90.00th=[13566], 95.00th=[14091],
     | 99.00th=[16319], 99.50th=[17433], 99.90th=[19006], 99.95th=[19006],
     | 99.99th=[19006]
   bw (  KiB/s): min= 2504, max= 2704, per=24.34%, avg=2632.00, stdev=111.14, samples=3
   iops        : min=  626, max=  676, avg=658.00, stdev=27.78, samples=3
  lat (msec)   : 2=0.04%, 4=0.08%, 10=14.92%, 20=84.96%
  cpu          : usr=0.83%, sys=9.63%, ctx=2668, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=10.1MiB/s (10.6MB/s), 2578KiB/s-2619KiB/s (2640kB/s-2682kB/s), io=19.6MiB (20.5MB), run=1927-1932msec
  WRITE: bw=10.6MiB/s (11.1MB/s), 2681KiB/s-2728KiB/s (2746kB/s-2793kB/s), io=20.4MiB (21.4MB), run=1927-1932msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=14699: Fri Jun 14 18:13:57 2024
   read: IOPS=10, BW=426MiB/s (446MB/s)(40.0MiB/94msec)
    slat (nsec): min=7174.2k, max=7174.2k, avg=7174185.00, stdev= 0.00
    clat (nsec): min=82868k, max=82868k, avg=82867609.00, stdev= 0.00
     lat (nsec): min=90052k, max=90052k, avg=90052089.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[83362],  5.00th=[83362], 10.00th=[83362], 20.00th=[83362],
     | 30.00th=[83362], 40.00th=[83362], 50.00th=[83362], 60.00th=[83362],
     | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362],
     | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[83362]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.45%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=40.0MiB (41.9MB), run=94-94msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck