== sanity test 398c: run fio to test AIO ================= 23:40:56 (1708836056)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.09439 s, 444 MB/s
osc.lustre-OST0000-osc-ffff88012a5eb800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=6290: Sat Feb 24 23:41:01 2024
  write: IOPS=727, BW=2912KiB/s (2981kB/s)(10.0MiB/3517msec)
    slat (usec): min=12, max=446, avg=49.72, stdev=26.87
    clat (usec): min=4712, max=46293, avg=21819.68, stdev=2577.46
     lat (usec): min=4758, max=46328, avg=21870.85, stdev=2577.19
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[17695], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21627], 50.00th=[22152], 60.00th=[22676],
     | 70.00th=[22938], 80.00th=[23200], 90.00th=[23987], 95.00th=[25035],
     | 99.00th=[28705], 99.50th=[31327], 99.90th=[44303], 99.95th=[44303],
     | 99.99th=[46400]
   bw (  KiB/s): min= 2768, max= 3073, per=24.96%, avg=2906.71, stdev=97.87, samples=7
   iops        : min=  692, max=  768, avg=726.57, stdev=24.43, samples=7
  lat (msec)   : 10=0.04%, 20=18.44%, 50=81.52%
  cpu          : usr=0.77%, sys=4.01%, ctx=1831, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6291: Sat Feb 24 23:41:01 2024
  write: IOPS=728, BW=2915KiB/s (2985kB/s)(10.0MiB/3513msec)
    slat (usec): min=12, max=279, avg=49.23, stdev=25.71
    clat (usec): min=2452, max=45362, avg=21796.89, stdev=2631.50
     lat (usec): min=2509, max=45378, avg=21847.53, stdev=2631.73
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[17433], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21627], 50.00th=[22152], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[23987], 95.00th=[25035],
     | 99.00th=[28967], 99.50th=[31327], 99.90th=[45351], 99.95th=[45351],
     | 99.99th=[45351]
   bw (  KiB/s): min= 2760, max= 3089, per=24.96%, avg=2906.71, stdev=104.56, samples=7
   iops        : min=  690, max=  772, avg=726.57, stdev=26.10, samples=7
  lat (msec)   : 4=0.08%, 20=18.95%, 50=80.98%
  cpu          : usr=0.83%, sys=3.99%, ctx=1821, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6292: Sat Feb 24 23:41:01 2024
  write: IOPS=732, BW=2931KiB/s (3001kB/s)(10.0MiB/3494msec)
    slat (usec): min=12, max=264, avg=50.13, stdev=25.33
    clat (usec): min=2915, max=43212, avg=21683.64, stdev=2778.40
     lat (usec): min=2973, max=43291, avg=21735.17, stdev=2779.56
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[16909], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21627], 50.00th=[22152], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[23987], 95.00th=[25035],
     | 99.00th=[29230], 99.50th=[31327], 99.90th=[40633], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 2832, max= 3169, per=25.31%, avg=2947.17, stdev=128.18, samples=6
   iops        : min=  708, max=  792, avg=736.67, stdev=32.02, samples=6
  lat (msec)   : 4=0.12%, 10=0.35%, 20=19.92%, 50=79.61%
  cpu          : usr=0.77%, sys=4.12%, ctx=1823, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6293: Sat Feb 24 23:41:01 2024
  write: IOPS=728, BW=2913KiB/s (2983kB/s)(10.0MiB/3515msec)
    slat (usec): min=12, max=332, avg=49.60, stdev=27.99
    clat (usec): min=11644, max=42727, avg=21809.41, stdev=2479.51
     lat (usec): min=11700, max=42788, avg=21860.49, stdev=2480.56
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[17695], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21627], 50.00th=[22152], 60.00th=[22676],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[23987], 95.00th=[25035],
     | 99.00th=[28967], 99.50th=[31327], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[42730]
   bw (  KiB/s): min= 2784, max= 3065, per=24.96%, avg=2906.71, stdev=90.94, samples=7
   iops        : min=  696, max=  766, avg=726.57, stdev=22.68, samples=7
  lat (msec)   : 20=18.98%, 50=81.02%
  cpu          : usr=0.97%, sys=3.76%, ctx=1783, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.4MiB/s (11.9MB/s), 2912KiB/s-2931KiB/s (2981kB/s-3001kB/s), io=40.0MiB (41.9MB), run=3494-3517msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=6306: Sat Feb 24 23:41:04 2024
   read: IOPS=415, BW=1663KiB/s (1703kB/s)(5048KiB/3035msec)
    slat (usec): min=713, max=2804, avg=1255.18, stdev=256.64
    clat (usec): min=3155, max=26779, avg=17927.46, stdev=2551.19
     lat (usec): min=4663, max=28639, avg=19184.53, stdev=2570.52
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13960], 10.00th=[14746], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[20055], 90.00th=[21103], 95.00th=[22152],
     | 99.00th=[23987], 99.50th=[24773], 99.90th=[26608], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1528, max= 1824, per=25.02%, avg=1654.00, stdev=102.63, samples=6
   iops        : min=  382, max=  456, avg=413.50, stdev=25.66, samples=6
  write: IOPS=427, BW=1711KiB/s (1752kB/s)(5192KiB/3035msec)
    slat (usec): min=11, max=1703, avg=50.01, stdev=79.88
    clat (usec): min=4175, max=26132, avg=18461.69, stdev=2098.50
     lat (usec): min=4242, max=26178, avg=18513.20, stdev=2100.50
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15270], 10.00th=[15926], 20.00th=[16712],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18482], 60.00th=[19006],
     | 70.00th=[19530], 80.00th=[20055], 90.00th=[21103], 95.00th=[21890],
     | 99.00th=[23200], 99.50th=[23987], 99.90th=[24773], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 1640, max= 1768, per=24.79%, avg=1706.00, stdev=45.38, samples=6
   iops        : min=  410, max=  442, avg=426.50, stdev=11.34, samples=6
  lat (msec)   : 4=0.04%, 10=0.23%, 20=78.48%, 50=21.25%
  cpu          : usr=0.92%, sys=7.09%, ctx=2369, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6307: Sat Feb 24 23:41:04 2024
   read: IOPS=411, BW=1644KiB/s (1684kB/s)(4976KiB/3026msec)
    slat (usec): min=707, max=2705, avg=1256.11, stdev=264.91
    clat (usec): min=2180, max=31405, avg=17919.41, stdev=2664.61
     lat (usec): min=3990, max=33028, avg=19177.68, stdev=2673.41
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13960], 10.00th=[14615], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[20055], 90.00th=[21103], 95.00th=[22414],
     | 99.00th=[24773], 99.50th=[26346], 99.90th=[30278], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1488, max= 1752, per=24.81%, avg=1640.67, stdev=107.58, samples=6
   iops        : min=  372, max=  438, avg=410.17, stdev=26.90, samples=6
  write: IOPS=434, BW=1740KiB/s (1781kB/s)(5264KiB/3026msec)
    slat (usec): min=13, max=305, avg=47.01, stdev=22.97
    clat (usec): min=5062, max=31302, avg=18425.63, stdev=2115.67
     lat (usec): min=5109, max=31397, avg=18474.00, stdev=2115.58
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15270], 10.00th=[15926], 20.00th=[16712],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[19530], 80.00th=[20055], 90.00th=[20841], 95.00th=[21890],
     | 99.00th=[23725], 99.50th=[25035], 99.90th=[28705], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1612, max= 1792, per=25.29%, avg=1740.67, stdev=66.47, samples=6
   iops        : min=  403, max=  448, avg=435.17, stdev=16.62, samples=6
  lat (msec)   : 4=0.04%, 10=0.16%, 20=79.38%, 50=20.43%
  cpu          : usr=1.09%, sys=6.94%, ctx=2297, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6308: Sat Feb 24 23:41:04 2024
   read: IOPS=415, BW=1660KiB/s (1700kB/s)(4984KiB/3002msec)
    slat (usec): min=32, max=2889, avg=1246.58, stdev=279.39
    clat (usec): min=2017, max=29447, avg=17800.85, stdev=2635.90
     lat (usec): min=2122, max=30522, avg=19049.64, stdev=2677.09
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13698], 10.00th=[14615], 20.00th=[15664],
     | 30.00th=[16319], 40.00th=[17171], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[20055], 90.00th=[21103], 95.00th=[22152],
     | 99.00th=[23462], 99.50th=[23725], 99.90th=[28443], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1552, max= 1712, per=25.06%, avg=1656.67, stdev=59.46, samples=6
   iops        : min=  388, max=  428, avg=414.17, stdev=14.86, samples=6
  write: IOPS=437, BW=1751KiB/s (1793kB/s)(5256KiB/3002msec)
    slat (usec): min=12, max=967, avg=48.01, stdev=36.87
    clat (usec): min=3800, max=35157, avg=18351.49, stdev=2362.30
     lat (usec): min=3982, max=35177, avg=18401.00, stdev=2361.31
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15139], 10.00th=[15795], 20.00th=[16581],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[18220], 60.00th=[18744],
     | 70.00th=[19268], 80.00th=[19792], 90.00th=[20841], 95.00th=[21627],
     | 99.00th=[24773], 99.50th=[28181], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1504, max= 1976, per=25.21%, avg=1735.33, stdev=150.74, samples=6
   iops        : min=  376, max=  494, avg=433.83, stdev=37.69, samples=6
  lat (msec)   : 4=0.12%, 10=0.23%, 20=80.59%, 50=19.06%
  cpu          : usr=1.10%, sys=7.10%, ctx=2228, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6309: Sat Feb 24 23:41:04 2024
   read: IOPS=417, BW=1668KiB/s (1708kB/s)(5060KiB/3033msec)
    slat (usec): min=729, max=2713, avg=1251.16, stdev=264.87
    clat (usec): min=5297, max=26898, avg=17859.52, stdev=2525.19
     lat (usec): min=6433, max=27879, avg=19112.64, stdev=2549.82
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14091], 10.00th=[14746], 20.00th=[15795],
     | 30.00th=[16450], 40.00th=[17171], 50.00th=[17695], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[21103], 95.00th=[22152],
     | 99.00th=[24249], 99.50th=[25035], 99.90th=[26346], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1568, max= 1792, per=25.22%, avg=1667.33, stdev=89.53, samples=6
   iops        : min=  392, max=  448, avg=416.83, stdev=22.38, samples=6
  write: IOPS=426, BW=1708KiB/s (1749kB/s)(5180KiB/3033msec)
    slat (usec): min=14, max=306, avg=46.63, stdev=21.70
    clat (usec): min=11419, max=30640, avg=18539.38, stdev=2081.98
     lat (usec): min=11462, max=30746, avg=18587.40, stdev=2082.68
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15270], 10.00th=[15926], 20.00th=[16909],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18482], 60.00th=[19006],
     | 70.00th=[19530], 80.00th=[20055], 90.00th=[21103], 95.00th=[21890],
     | 99.00th=[24249], 99.50th=[25297], 99.90th=[29754], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1600, max= 1768, per=24.73%, avg=1702.00, stdev=59.53, samples=6
   iops        : min=  400, max=  442, avg=425.50, stdev=14.88, samples=6
  lat (msec)   : 10=0.08%, 20=78.48%, 50=21.45%
  cpu          : usr=1.15%, sys=6.86%, ctx=2375, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6612KiB/s (6771kB/s), 1644KiB/s-1668KiB/s (1684kB/s-1708kB/s), io=19.6MiB (20.5MB), run=3002-3035msec
  WRITE: bw=6884KiB/s (7049kB/s), 1708KiB/s-1751KiB/s (1749kB/s-1793kB/s), io=20.4MiB (21.4MB), run=3002-3035msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=6315: Sat Feb 24 23:41:06 2024
   read: IOPS=9, BW=370MiB/s (388MB/s)(40.0MiB/108msec)
    slat (nsec): min=8201.7k, max=8201.7k, avg=8201745.00, stdev= 0.00
    clat (nsec): min=95291k, max=95291k, avg=95290560.00, stdev= 0.00
     lat (nsec): min=103514k, max=103514k, avg=103514118.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[94897],  5.00th=[94897], 10.00th=[94897], 20.00th=[94897],
     | 30.00th=[94897], 40.00th=[94897], 50.00th=[94897], 60.00th=[94897],
     | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897],
     | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897],
     | 99.99th=[94897]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.41%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=370MiB/s (388MB/s), 370MiB/s-370MiB/s (388MB/s-388MB/s), io=40.0MiB (41.9MB), run=108-108msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout