== sanity test 398c: run fio to test AIO ================= 16:29:49 (1748982589)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0896239 s, 468 MB/s
osc.lustre-OST0000-osc-ffff8800a799d000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=8292: Tue Jun  3 16:29:53 2025
  write: IOPS=967, BW=3870KiB/s (3963kB/s)(10.0MiB/2646msec)
    slat (usec): min=11, max=185, avg=35.76, stdev=18.03
    clat (usec): min=1728, max=43060, avg=16453.99, stdev=2729.79
     lat (usec): min=1759, max=43077, avg=16490.72, stdev=2729.49
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13173], 10.00th=[13829], 20.00th=[14877],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18744], 95.00th=[19268],
     | 99.00th=[28181], 99.50th=[34866], 99.90th=[41157], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 3736, max= 4080, per=25.04%, avg=3875.20, stdev=127.32, samples=5
   iops        : min=  934, max= 1020, avg=968.80, stdev=31.83, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.04%, 20=97.11%, 50=2.77%
  cpu          : usr=1.02%, sys=4.08%, ctx=1193, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8293: Tue Jun  3 16:29:53 2025
  write: IOPS=973, BW=3892KiB/s (3985kB/s)(10.0MiB/2631msec)
    slat (usec): min=11, max=195, avg=35.40, stdev=18.90
    clat (usec): min=1881, max=40295, avg=16374.73, stdev=2733.29
     lat (usec): min=1953, max=40380, avg=16411.07, stdev=2731.95
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[13042], 10.00th=[13698], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[25822], 99.50th=[33817], 99.90th=[38536], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 3720, max= 4104, per=25.23%, avg=3905.60, stdev=148.96, samples=5
   iops        : min=  930, max= 1026, avg=976.40, stdev=37.24, samples=5
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.35%, 20=95.43%, 50=4.10%
  cpu          : usr=1.03%, sys=4.07%, ctx=1152, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8294: Tue Jun  3 16:29:53 2025
  write: IOPS=969, BW=3877KiB/s (3970kB/s)(10.0MiB/2641msec)
    slat (usec): min=11, max=253, avg=36.53, stdev=19.50
    clat (usec): min=3650, max=42973, avg=16420.25, stdev=2753.45
     lat (usec): min=3684, max=43012, avg=16457.78, stdev=2753.10
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13304], 10.00th=[13960], 20.00th=[14877],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18744], 95.00th=[19268],
     | 99.00th=[28181], 99.50th=[33817], 99.90th=[42730], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 3736, max= 4104, per=25.10%, avg=3884.80, stdev=134.53, samples=5
   iops        : min=  934, max= 1026, avg=971.20, stdev=33.63, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=97.11%, 50=2.77%
  cpu          : usr=0.95%, sys=4.24%, ctx=1206, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8295: Tue Jun  3 16:29:53 2025
  write: IOPS=968, BW=3873KiB/s (3966kB/s)(10.0MiB/2644msec)
    slat (usec): min=11, max=272, avg=36.48, stdev=20.56
    clat (usec): min=3709, max=42966, avg=16444.53, stdev=2715.62
     lat (usec): min=3731, max=42984, avg=16481.94, stdev=2714.94
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13173], 10.00th=[13698], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[28181], 99.50th=[34866], 99.90th=[40109], 99.95th=[41157],
     | 99.99th=[42730]
   bw (  KiB/s): min= 3728, max= 4072, per=25.04%, avg=3875.20, stdev=126.95, samples=5
   iops        : min=  932, max= 1018, avg=968.80, stdev=31.74, samples=5
  lat (msec)   : 4=0.08%, 10=0.20%, 20=96.52%, 50=3.20%
  cpu          : usr=0.87%, sys=4.16%, ctx=1222, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=15.1MiB/s (15.9MB/s), 3870KiB/s-3892KiB/s (3963kB/s-3985kB/s), io=40.0MiB (41.9MB), run=2631-2646msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=8307: Tue Jun  3 16:29:56 2025
   read: IOPS=626, BW=2506KiB/s (2567kB/s)(5048KiB/2014msec)
    slat (usec): min=19, max=1253, avg=754.05, stdev=131.76
    clat (usec): min=5143, max=25563, avg=12041.20, stdev=1850.11
     lat (usec): min=5761, max=26287, avg=12796.93, stdev=1879.08
    clat percentiles (usec):
     |  1.00th=[ 8160],  5.00th=[ 9372], 10.00th=[ 9896], 20.00th=[10552],
     | 30.00th=[11076], 40.00th=[11469], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[14222], 95.00th=[15008],
     | 99.00th=[16319], 99.50th=[17695], 99.90th=[24773], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 2416, max= 2560, per=25.24%, avg=2506.00, stdev=64.46, samples=4
   iops        : min=  604, max=  640, avg=626.50, stdev=16.11, samples=4
  write: IOPS=644, BW=2578KiB/s (2640kB/s)(5192KiB/2014msec)
    slat (usec): min=11, max=657, avg=40.81, stdev=28.03
    clat (usec): min=1986, max=24730, avg=12277.59, stdev=1484.92
     lat (usec): min=2106, max=24760, avg=12319.56, stdev=1484.13
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[10290], 10.00th=[10814], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[11863], 50.00th=[12256], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13304], 90.00th=[13960], 95.00th=[14484],
     | 99.00th=[15664], 99.50th=[15926], 99.90th=[23987], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 2424, max= 2648, per=25.00%, avg=2584.00, stdev=107.73, samples=4
   iops        : min=  606, max=  662, avg=646.00, stdev=26.93, samples=4
  lat (msec)   : 2=0.04%, 4=0.04%, 10=7.15%, 20=92.42%, 50=0.35%
  cpu          : usr=0.89%, sys=9.14%, ctx=2854, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8308: Tue Jun  3 16:29:56 2025
   read: IOPS=615, BW=2463KiB/s (2522kB/s)(4976KiB/2020msec)
    slat (usec): min=465, max=1507, avg=754.29, stdev=119.66
    clat (usec): min=2852, max=25601, avg=12095.50, stdev=1781.60
     lat (usec): min=3439, max=26511, avg=12851.40, stdev=1797.34
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10683],
     | 30.00th=[11207], 40.00th=[11600], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12911], 80.00th=[13304], 90.00th=[14091], 95.00th=[14746],
     | 99.00th=[16188], 99.50th=[16581], 99.90th=[24773], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 2368, max= 2496, per=24.78%, avg=2460.00, stdev=61.45, samples=4
   iops        : min=  592, max=  624, avg=615.00, stdev=15.36, samples=4
  write: IOPS=651, BW=2606KiB/s (2668kB/s)(5264KiB/2020msec)
    slat (usec): min=11, max=346, avg=39.70, stdev=21.79
    clat (usec): min=3319, max=23126, avg=12263.49, stdev=1311.53
     lat (usec): min=3364, max=23153, avg=12304.21, stdev=1311.45
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[10290], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[11994], 50.00th=[12256], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13304], 90.00th=[13829], 95.00th=[14222],
     | 99.00th=[15401], 99.50th=[16188], 99.90th=[20579], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 2472, max= 2728, per=25.31%, avg=2616.00, stdev=106.33, samples=4
   iops        : min=  618, max=  682, avg=654.00, stdev=26.58, samples=4
  lat (msec)   : 4=0.08%, 10=6.09%, 20=93.52%, 50=0.31%
  cpu          : usr=1.49%, sys=8.42%, ctx=2910, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8309: Tue Jun  3 16:29:56 2025
   read: IOPS=617, BW=2471KiB/s (2530kB/s)(4984KiB/2017msec)
    slat (usec): min=13, max=1572, avg=754.09, stdev=136.02
    clat (usec): min=1048, max=24860, avg=12028.05, stdev=1891.54
     lat (usec): min=1063, max=25553, avg=12783.92, stdev=1922.38
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10552],
     | 30.00th=[11207], 40.00th=[11600], 50.00th=[12125], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[14353], 95.00th=[15008],
     | 99.00th=[16188], 99.50th=[16909], 99.90th=[19530], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 2344, max= 2544, per=24.92%, avg=2474.00, stdev=94.52, samples=4
   iops        : min=  586, max=  636, avg=618.50, stdev=23.63, samples=4
  write: IOPS=651, BW=2606KiB/s (2668kB/s)(5256KiB/2017msec)
    slat (usec): min=11, max=339, avg=39.07, stdev=21.10
    clat (usec): min=3423, max=24759, avg=12338.05, stdev=1463.84
     lat (usec): min=3436, max=24796, avg=12378.16, stdev=1462.58
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10421], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[11994], 50.00th=[12387], 60.00th=[12649],
     | 70.00th=[13042], 80.00th=[13435], 90.00th=[13960], 95.00th=[14484],
     | 99.00th=[15401], 99.50th=[15533], 99.90th=[22938], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 2544, max= 2632, per=25.15%, avg=2600.00, stdev=39.73, samples=4
   iops        : min=  636, max=  658, avg=650.00, stdev= 9.93, samples=4
  lat (msec)   : 2=0.16%, 4=0.08%, 10=7.23%, 20=92.38%, 50=0.16%
  cpu          : usr=1.34%, sys=8.68%, ctx=2603, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8310: Tue Jun  3 16:29:56 2025
   read: IOPS=625, BW=2504KiB/s (2564kB/s)(5060KiB/2021msec)
    slat (usec): min=420, max=1330, avg=755.34, stdev=115.46
    clat (usec): min=6532, max=17872, avg=12061.22, stdev=1600.78
     lat (usec): min=7266, max=18542, avg=12818.18, stdev=1621.04
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10683],
     | 30.00th=[11207], 40.00th=[11600], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12911], 80.00th=[13304], 90.00th=[14222], 95.00th=[14746],
     | 99.00th=[16057], 99.50th=[16319], 99.90th=[17695], 99.95th=[17957],
     | 99.99th=[17957]
   bw (  KiB/s): min= 2400, max= 2568, per=25.18%, avg=2500.00, stdev=72.88, samples=4
   iops        : min=  600, max=  642, avg=625.00, stdev=18.22, samples=4
  write: IOPS=640, BW=2563KiB/s (2625kB/s)(5180KiB/2021msec)
    slat (usec): min=11, max=653, avg=41.11, stdev=34.97
    clat (usec): min=7388, max=17782, avg=12291.04, stdev=1328.45
     lat (usec): min=7424, max=17838, avg=12333.27, stdev=1327.66
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[10290], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[11863], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[13960], 95.00th=[14484],
     | 99.00th=[15533], 99.50th=[16319], 99.90th=[17695], 99.95th=[17695],
     | 99.99th=[17695]
   bw (  KiB/s): min= 2488, max= 2600, per=24.75%, avg=2558.00, stdev=49.48, samples=4
   iops        : min=  622, max=  650, avg=639.50, stdev=12.37, samples=4
  lat (msec)   : 10=6.52%, 20=93.48%
  cpu          : usr=1.29%, sys=8.71%, ctx=2877, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9930KiB/s (10.2MB/s), 2463KiB/s-2506KiB/s (2522kB/s-2567kB/s), io=19.6MiB (20.5MB), run=2014-2021msec
  WRITE: bw=10.1MiB/s (10.6MB/s), 2563KiB/s-2606KiB/s (2625kB/s-2668kB/s), io=20.4MiB (21.4MB), run=2014-2021msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=8315: Tue Jun  3 16:29:58 2025
   read: IOPS=13, BW=548MiB/s (575MB/s)(40.0MiB/73msec)
    slat (nsec): min=5855.1k, max=5855.1k, avg=5855099.00, stdev= 0.00
    clat (nsec): min=64093k, max=64093k, avg=64092738.00, stdev= 0.00
     lat (nsec): min=69958k, max=69958k, avg=69958314.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[64226],  5.00th=[64226], 10.00th=[64226], 20.00th=[64226],
     | 30.00th=[64226], 40.00th=[64226], 50.00th=[64226], 60.00th=[64226],
     | 70.00th=[64226], 80.00th=[64226], 90.00th=[64226], 95.00th=[64226],
     | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[64226]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.33%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=548MiB/s (575MB/s), 548MiB/s-548MiB/s (575MB/s-575MB/s), io=40.0MiB (41.9MB), run=73-73msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck