== sanity test 398c: run fio to test AIO ================= 19:13:45 (1717024425)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0772869 s, 543 MB/s
osc.lustre-OST0000-osc-ffff8800aa34d000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=20178: Wed May 29 19:14:19 2024
  write: IOPS=79, BW=319KiB/s (326kB/s)(10.0MiB/32135msec)
    slat (usec): min=11, max=786, avg=72.78, stdev=46.50
    clat (msec): min=13, max=394, avg=200.69, stdev=51.62
     lat (msec): min=13, max=394, avg=200.76, stdev=51.62
    clat percentiles (msec):
     |  1.00th=[   73],  5.00th=[  122], 10.00th=[  138], 20.00th=[  157],
     | 30.00th=[  171], 40.00th=[  186], 50.00th=[  201], 60.00th=[  213],
     | 70.00th=[  228], 80.00th=[  245], 90.00th=[  266], 95.00th=[  284],
     | 99.00th=[  330], 99.50th=[  347], 99.90th=[  376], 99.95th=[  384],
     | 99.99th=[  397]
   bw (  KiB/s): min=  184, max=  672, per=25.04%, avg=317.95, stdev=77.63, samples=64
   iops        : min=   46, max=  168, avg=79.36, stdev=19.43, samples=64
  lat (msec)   : 20=0.08%, 50=0.12%, 100=1.84%, 250=81.13%, 500=16.84%
  cpu          : usr=0.23%, sys=0.59%, ctx=2016, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20179: Wed May 29 19:14:19 2024
  write: IOPS=79, BW=318KiB/s (325kB/s)(10.0MiB/32228msec)
    slat (usec): min=9, max=547, avg=70.84, stdev=45.70
    clat (msec): min=32, max=394, avg=201.26, stdev=48.99
     lat (msec): min=32, max=394, avg=201.33, stdev=48.99
    clat percentiles (msec):
     |  1.00th=[  103],  5.00th=[  130], 10.00th=[  144], 20.00th=[  159],
     | 30.00th=[  174], 40.00th=[  186], 50.00th=[  201], 60.00th=[  213],
     | 70.00th=[  226], 80.00th=[  243], 90.00th=[  266], 95.00th=[  279],
     | 99.00th=[  330], 99.50th=[  347], 99.90th=[  397], 99.95th=[  397],
     | 99.99th=[  397]
   bw (  KiB/s): min=  184, max=  512, per=24.88%, avg=316.03, stdev=69.57, samples=64
   iops        : min=   46, max=  128, avg=78.94, stdev=17.40, samples=64
  lat (msec)   : 50=0.04%, 100=0.66%, 250=82.70%, 500=16.60%
  cpu          : usr=0.14%, sys=0.64%, ctx=2024, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20180: Wed May 29 19:14:19 2024
  write: IOPS=79, BW=318KiB/s (326kB/s)(10.0MiB/32176msec)
    slat (usec): min=10, max=837, avg=71.90, stdev=47.95
    clat (msec): min=59, max=394, avg=200.94, stdev=49.91
     lat (msec): min=59, max=394, avg=201.01, stdev=49.92
    clat percentiles (msec):
     |  1.00th=[  102],  5.00th=[  126], 10.00th=[  140], 20.00th=[  157],
     | 30.00th=[  174], 40.00th=[  186], 50.00th=[  201], 60.00th=[  211],
     | 70.00th=[  228], 80.00th=[  245], 90.00th=[  266], 95.00th=[  284],
     | 99.00th=[  330], 99.50th=[  338], 99.90th=[  372], 99.95th=[  380],
     | 99.99th=[  397]
   bw (  KiB/s): min=  168, max=  552, per=24.95%, avg=316.83, stdev=69.27, samples=64
   iops        : min=   42, max=  138, avg=79.08, stdev=17.34, samples=64
  lat (msec)   : 100=0.86%, 250=82.73%, 500=16.41%
  cpu          : usr=0.18%, sys=0.66%, ctx=1947, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20181: Wed May 29 19:14:19 2024
  write: IOPS=79, BW=318KiB/s (325kB/s)(10.0MiB/32248msec)
    slat (usec): min=10, max=367, avg=74.27, stdev=45.63
    clat (msec): min=27, max=394, avg=201.39, stdev=49.67
     lat (msec): min=27, max=394, avg=201.46, stdev=49.67
    clat percentiles (msec):
     |  1.00th=[  103],  5.00th=[  127], 10.00th=[  140], 20.00th=[  157],
     | 30.00th=[  176], 40.00th=[  186], 50.00th=[  201], 60.00th=[  213],
     | 70.00th=[  228], 80.00th=[  245], 90.00th=[  266], 95.00th=[  284],
     | 99.00th=[  330], 99.50th=[  347], 99.90th=[  397], 99.95th=[  397],
     | 99.99th=[  397]
   bw (  KiB/s): min=  184, max=  536, per=24.87%, avg=315.80, stdev=69.84, samples=64
   iops        : min=   46, max=  134, avg=78.89, stdev=17.45, samples=64
  lat (msec)   : 50=0.04%, 100=0.66%, 250=82.23%, 500=17.07%
  cpu          : usr=0.19%, sys=0.62%, ctx=2122, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1270KiB/s (1301kB/s), 318KiB/s-319KiB/s (325kB/s-326kB/s), io=40.0MiB (41.9MB), run=32135-32248msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=20216: Wed May 29 19:14:38 2024
   read: IOPS=70, BW=283KiB/s (289kB/s)(5048KiB/17867msec)
    slat (usec): min=497, max=2553, avg=882.57, stdev=261.95
    clat (msec): min=10, max=179, avg=74.77, stdev=15.50
     lat (msec): min=10, max=180, avg=75.65, stdev=15.54
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   54], 10.00th=[   58], 20.00th=[   63],
     | 30.00th=[   68], 40.00th=[   71], 50.00th=[   74], 60.00th=[   78],
     | 70.00th=[   82], 80.00th=[   86], 90.00th=[   93], 95.00th=[   99],
     | 99.00th=[  123], 99.50th=[  130], 99.90th=[  167], 99.95th=[  180],
     | 99.99th=[  180]
   bw (  KiB/s): min=  160, max=  384, per=25.15%, avg=282.17, stdev=55.94, samples=35
   iops        : min=   40, max=   96, avg=70.46, stdev=14.00, samples=35
  write: IOPS=72, BW=291KiB/s (298kB/s)(5192KiB/17867msec)
    slat (usec): min=11, max=241, avg=63.55, stdev=36.05
    clat (msec): min=51, max=285, avg=146.36, stdev=34.69
     lat (msec): min=51, max=285, avg=146.42, stdev=34.70
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[   95], 10.00th=[  103], 20.00th=[  115],
     | 30.00th=[  127], 40.00th=[  136], 50.00th=[  146], 60.00th=[  155],
     | 70.00th=[  163], 80.00th=[  176], 90.00th=[  192], 95.00th=[  209],
     | 99.00th=[  243], 99.50th=[  257], 99.90th=[  284], 99.95th=[  288],
     | 99.99th=[  288]
   bw (  KiB/s): min=  224, max=  360, per=24.84%, avg=290.17, stdev=36.89, samples=35
   iops        : min=   56, max=   90, avg=72.46, stdev= 9.19, samples=35
  lat (msec)   : 20=0.27%, 50=1.05%, 100=50.23%, 250=48.16%, 500=0.27%
  cpu          : usr=0.12%, sys=1.62%, ctx=3126, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20217: Wed May 29 19:14:38 2024
   read: IOPS=69, BW=278KiB/s (285kB/s)(4976KiB/17877msec)
    slat (usec): min=466, max=3637, avg=912.99, stdev=313.51
    clat (msec): min=10, max=134, avg=74.51, stdev=15.13
     lat (msec): min=10, max=135, avg=75.43, stdev=15.18
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   54], 10.00th=[   57], 20.00th=[   63],
     | 30.00th=[   67], 40.00th=[   71], 50.00th=[   74], 60.00th=[   79],
     | 70.00th=[   82], 80.00th=[   86], 90.00th=[   93], 95.00th=[  100],
     | 99.00th=[  117], 99.50th=[  125], 99.90th=[  136], 99.95th=[  136],
     | 99.99th=[  136]
   bw (  KiB/s): min=  168, max=  464, per=24.77%, avg=277.94, stdev=61.34, samples=35
   iops        : min=   42, max=  116, avg=69.49, stdev=15.33, samples=35
  write: IOPS=73, BW=294KiB/s (302kB/s)(5264KiB/17877msec)
    slat (usec): min=12, max=463, avg=61.98, stdev=38.93
    clat (msec): min=38, max=289, avg=145.80, stdev=35.09
     lat (msec): min=38, max=289, avg=145.86, stdev=35.09
    clat percentiles (msec):
     |  1.00th=[   81],  5.00th=[   91], 10.00th=[  103], 20.00th=[  115],
     | 30.00th=[  127], 40.00th=[  136], 50.00th=[  144], 60.00th=[  155],
     | 70.00th=[  163], 80.00th=[  176], 90.00th=[  192], 95.00th=[  211],
     | 99.00th=[  239], 99.50th=[  255], 99.90th=[  284], 99.95th=[  292],
     | 99.99th=[  292]
   bw (  KiB/s): min=  216, max=  392, per=25.05%, avg=292.57, stdev=46.25, samples=35
   iops        : min=   54, max=   98, avg=73.14, stdev=11.56, samples=35
  lat (msec)   : 20=0.20%, 50=1.52%, 100=49.18%, 250=48.75%, 500=0.35%
  cpu          : usr=0.20%, sys=1.50%, ctx=3255, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20218: Wed May 29 19:14:38 2024
   read: IOPS=69, BW=279KiB/s (286kB/s)(4984KiB/17857msec)
    slat (usec): min=22, max=3593, avg=898.77, stdev=295.91
    clat (usec): min=954, max=156926, avg=73961.13, stdev=16147.48
     lat (msec): min=2, max=158, avg=74.86, stdev=16.20
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   53], 10.00th=[   58], 20.00th=[   64],
     | 30.00th=[   67], 40.00th=[   70], 50.00th=[   73], 60.00th=[   78],
     | 70.00th=[   82], 80.00th=[   85], 90.00th=[   92], 95.00th=[  101],
     | 99.00th=[  116], 99.50th=[  124], 99.90th=[  157], 99.95th=[  157],
     | 99.99th=[  157]
   bw (  KiB/s): min=  184, max=  448, per=24.99%, avg=280.34, stdev=59.00, samples=35
   iops        : min=   46, max=  112, avg=70.00, stdev=14.72, samples=35
  write: IOPS=73, BW=294KiB/s (301kB/s)(5256KiB/17857msec)
    slat (usec): min=11, max=909, avg=63.92, stdev=50.91
    clat (msec): min=18, max=290, avg=146.23, stdev=34.83
     lat (msec): min=18, max=290, avg=146.30, stdev=34.83
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   93], 10.00th=[  104], 20.00th=[  117],
     | 30.00th=[  127], 40.00th=[  138], 50.00th=[  144], 60.00th=[  155],
     | 70.00th=[  165], 80.00th=[  176], 90.00th=[  188], 95.00th=[  205],
     | 99.00th=[  241], 99.50th=[  243], 99.90th=[  257], 99.95th=[  292],
     | 99.99th=[  292]
   bw (  KiB/s): min=  216, max=  360, per=25.10%, avg=293.14, stdev=41.09, samples=35
   iops        : min=   54, max=   90, avg=73.20, stdev=10.26, samples=35
  lat (usec)   : 1000=0.04%
  lat (msec)   : 4=0.12%, 10=0.04%, 20=0.43%, 50=1.52%, 100=48.20%
  lat (msec)   : 250=49.49%, 500=0.16%
  cpu          : usr=0.24%, sys=1.52%, ctx=3048, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20219: Wed May 29 19:14:38 2024
   read: IOPS=71, BW=284KiB/s (291kB/s)(5060KiB/17813msec)
    slat (usec): min=455, max=3658, avg=909.40, stdev=305.22
    clat (usec): min=1664, max=170912, avg=74807.08, stdev=15683.20
     lat (msec): min=2, max=171, avg=75.72, stdev=15.74
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   54], 10.00th=[   58], 20.00th=[   63],
     | 30.00th=[   68], 40.00th=[   71], 50.00th=[   74], 60.00th=[   79],
     | 70.00th=[   82], 80.00th=[   86], 90.00th=[   92], 95.00th=[   99],
     | 99.00th=[  122], 99.50th=[  134], 99.90th=[  171], 99.95th=[  171],
     | 99.99th=[  171]
   bw (  KiB/s): min=  160, max=  456, per=25.46%, avg=285.66, stdev=62.93, samples=35
   iops        : min=   40, max=  114, avg=71.37, stdev=15.76, samples=35
  write: IOPS=72, BW=291KiB/s (298kB/s)(5180KiB/17813msec)
    slat (usec): min=10, max=800, avg=65.47, stdev=45.38
    clat (msec): min=64, max=278, avg=145.81, stdev=34.71
     lat (msec): min=64, max=278, avg=145.87, stdev=34.71
    clat percentiles (msec):
     |  1.00th=[   80],  5.00th=[   94], 10.00th=[  104], 20.00th=[  115],
     | 30.00th=[  127], 40.00th=[  136], 50.00th=[  144], 60.00th=[  153],
     | 70.00th=[  163], 80.00th=[  174], 90.00th=[  190], 95.00th=[  209],
     | 99.00th=[  241], 99.50th=[  268], 99.90th=[  279], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  208, max=  400, per=24.91%, avg=290.91, stdev=38.64, samples=35
   iops        : min=   52, max=  100, avg=72.69, stdev= 9.64, samples=35
  lat (msec)   : 2=0.04%, 20=0.20%, 50=1.48%, 100=49.73%, 250=48.24%
  lat (msec)   : 500=0.31%
  cpu          : usr=0.27%, sys=1.48%, ctx=3344, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1123KiB/s (1150kB/s), 278KiB/s-284KiB/s (285kB/s-291kB/s), io=19.6MiB (20.5MB), run=17813-17877msec
  WRITE: bw=1169KiB/s (1197kB/s), 291KiB/s-294KiB/s (298kB/s-302kB/s), io=20.4MiB (21.4MB), run=17813-17877msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=20237: Wed May 29 19:14:39 2024
   read: IOPS=11, BW=465MiB/s (488MB/s)(40.0MiB/86msec)
    slat (nsec): min=7472.8k, max=7472.8k, avg=7472750.00, stdev= 0.00
    clat (nsec): min=75899k, max=75899k, avg=75899109.00, stdev= 0.00
     lat (nsec): min=83394k, max=83394k, avg=83393562.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[76022],  5.00th=[76022], 10.00th=[76022], 20.00th=[76022],
     | 30.00th=[76022], 40.00th=[76022], 50.00th=[76022], 60.00th=[76022],
     | 70.00th=[76022], 80.00th=[76022], 90.00th=[76022], 95.00th=[76022],
     | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.41%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=465MiB/s (488MB/s), 465MiB/s-465MiB/s (488MB/s-488MB/s), io=40.0MiB (41.9MB), run=86-86msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck