== sanity test 398c: run fio to test AIO ================= 07:46:35 (1744544795)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.106398 s, 394 MB/s
osc.lustre-OST0000-osc-ffff88007e458000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=797: Sun Apr 13 07:46:39 2025
  write: IOPS=766, BW=3068KiB/s (3141kB/s)(10.0MiB/3338msec)
    slat (usec): min=13, max=314, avg=50.45, stdev=27.62
    clat (usec): min=2495, max=43651, avg=20690.03, stdev=2521.71
     lat (usec): min=2542, max=43707, avg=20741.75, stdev=2522.25
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[16909], 10.00th=[17695], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21890], 80.00th=[22414], 90.00th=[23200], 95.00th=[24249],
     | 99.00th=[25560], 99.50th=[26084], 99.90th=[41157], 99.95th=[41157],
     | 99.99th=[43779]
   bw (  KiB/s): min= 2810, max= 3320, per=25.20%, avg=3080.33, stdev=195.27, samples=6
   iops        : min=  702, max=  830, avg=770.00, stdev=48.96, samples=6
  lat (msec)   : 4=0.08%, 10=0.27%, 20=31.41%, 50=68.24%
  cpu          : usr=0.66%, sys=4.29%, ctx=1839, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=798: Sun Apr 13 07:46:39 2025
  write: IOPS=763, BW=3056KiB/s (3129kB/s)(10.0MiB/3351msec)
    slat (usec): min=12, max=303, avg=47.07, stdev=26.81
    clat (usec): min=2987, max=28532, avg=20707.78, stdev=2134.76
     lat (usec): min=3087, max=28580, avg=20756.02, stdev=2134.40
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[17171], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20579], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[23200], 95.00th=[23987],
     | 99.00th=[25035], 99.50th=[25560], 99.90th=[27657], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 2816, max= 3200, per=25.00%, avg=3056.00, stdev=164.42, samples=6
   iops        : min=  704, max=  800, avg=764.00, stdev=41.10, samples=6
  lat (msec)   : 4=0.04%, 10=0.04%, 20=32.93%, 50=66.99%
  cpu          : usr=0.66%, sys=3.88%, ctx=1368, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=799: Sun Apr 13 07:46:39 2025
  write: IOPS=764, BW=3057KiB/s (3130kB/s)(10.0MiB/3350msec)
    slat (usec): min=12, max=524, avg=47.18, stdev=27.98
    clat (usec): min=4147, max=33305, avg=20732.43, stdev=2271.71
     lat (usec): min=4207, max=33374, avg=20780.78, stdev=2272.54
    clat percentiles (usec):
     |  1.00th=[15139],  5.00th=[17171], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[23200], 95.00th=[23987],
     | 99.00th=[26084], 99.50th=[28705], 99.90th=[33162], 99.95th=[33162],
     | 99.99th=[33424]
   bw (  KiB/s): min= 2816, max= 3208, per=25.11%, avg=3069.33, stdev=166.17, samples=6
   iops        : min=  704, max=  802, avg=767.33, stdev=41.54, samples=6
  lat (msec)   : 10=0.08%, 20=32.97%, 50=66.95%
  cpu          : usr=0.54%, sys=4.03%, ctx=1555, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=800: Sun Apr 13 07:46:39 2025
  write: IOPS=764, BW=3057KiB/s (3130kB/s)(10.0MiB/3350msec)
    slat (usec): min=12, max=428, avg=50.11, stdev=28.34
    clat (usec): min=14814, max=35601, avg=20755.22, stdev=2112.21
     lat (usec): min=14918, max=35658, avg=20806.57, stdev=2110.53
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[17171], 10.00th=[17695], 20.00th=[19268],
     | 30.00th=[19792], 40.00th=[20579], 50.00th=[20841], 60.00th=[21365],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[23200], 95.00th=[23987],
     | 99.00th=[25297], 99.50th=[26084], 99.90th=[33424], 99.95th=[33424],
     | 99.99th=[35390]
   bw (  KiB/s): min= 2810, max= 3208, per=25.03%, avg=3059.00, stdev=183.96, samples=6
   iops        : min=  702, max=  802, avg=764.67, stdev=46.12, samples=6
  lat (msec)   : 20=32.27%, 50=67.73%
  cpu          : usr=0.78%, sys=4.12%, ctx=1727, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.9MiB/s (12.5MB/s), 3056KiB/s-3068KiB/s (3129kB/s-3141kB/s), io=40.0MiB (41.9MB), run=3338-3351msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=824: Sun Apr 13 07:46:43 2025
   read: IOPS=460, BW=1843KiB/s (1887kB/s)(5048KiB/2739msec)
    slat (usec): min=13, max=2374, avg=1112.66, stdev=239.39
    clat (usec): min=2357, max=27134, avg=16237.72, stdev=2247.05
     lat (usec): min=2376, max=28204, avg=17352.06, stdev=2276.87
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[13042], 10.00th=[13698], 20.00th=[14484],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[18744], 95.00th=[19792],
     | 99.00th=[21103], 99.50th=[22152], 99.90th=[26346], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1776, max= 1864, per=25.16%, avg=1832.00, stdev=34.41, samples=5
   iops        : min=  444, max=  466, avg=458.00, stdev= 8.60, samples=5
  write: IOPS=473, BW=1896KiB/s (1941kB/s)(5192KiB/2739msec)
    slat (usec): min=13, max=555, avg=44.88, stdev=29.14
    clat (usec): min=4100, max=29231, avg=16759.26, stdev=1896.63
     lat (usec): min=4131, max=29300, avg=16805.16, stdev=1896.28
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[14091], 10.00th=[14615], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19530],
     | 99.00th=[21365], 99.50th=[21890], 99.90th=[27132], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1848, max= 1968, per=25.10%, avg=1902.40, stdev=45.04, samples=5
   iops        : min=  462, max=  492, avg=475.60, stdev=11.26, samples=5
  lat (msec)   : 4=0.16%, 10=0.47%, 20=95.66%, 50=3.71%
  cpu          : usr=0.84%, sys=7.45%, ctx=2237, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=826: Sun Apr 13 07:46:43 2025
   read: IOPS=451, BW=1806KiB/s (1850kB/s)(4976KiB/2755msec)
    slat (usec): min=711, max=2318, avg=1119.93, stdev=213.41
    clat (usec): min=1847, max=23932, avg=16240.74, stdev=2112.86
     lat (usec): min=2909, max=25723, avg=17362.32, stdev=2121.77
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12911], 10.00th=[13566], 20.00th=[14484],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16188], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19792],
     | 99.00th=[21365], 99.50th=[22152], 99.90th=[23462], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min= 1632, max= 1904, per=24.55%, avg=1787.20, stdev=114.21, samples=5
   iops        : min=  408, max=  476, avg=446.80, stdev=28.55, samples=5
  write: IOPS=477, BW=1911KiB/s (1957kB/s)(5264KiB/2755msec)
    slat (usec): min=14, max=237, avg=43.31, stdev=20.77
    clat (usec): min=3678, max=21661, avg=16748.21, stdev=1647.47
     lat (usec): min=3728, max=21701, avg=16792.63, stdev=1647.29
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14091], 10.00th=[14746], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[20579], 99.50th=[21103], 99.90th=[21627], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 1824, max= 2040, per=25.27%, avg=1915.20, stdev=93.05, samples=5
   iops        : min=  456, max=  510, avg=478.80, stdev=23.26, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.16%, 20=96.56%, 50=3.20%
  cpu          : usr=0.58%, sys=7.55%, ctx=2280, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=827: Sun Apr 13 07:46:43 2025
   read: IOPS=453, BW=1813KiB/s (1857kB/s)(4984KiB/2749msec)
    slat (usec): min=695, max=2326, avg=1119.13, stdev=215.09
    clat (usec): min=7764, max=22846, avg=16296.06, stdev=2113.29
     lat (usec): min=9258, max=24140, avg=17416.77, stdev=2114.90
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12911], 10.00th=[13566], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19530],
     | 99.00th=[20841], 99.50th=[21365], 99.90th=[22152], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 1768, max= 1920, per=24.88%, avg=1811.20, stdev=62.89, samples=5
   iops        : min=  442, max=  480, avg=452.80, stdev=15.72, samples=5
  write: IOPS=477, BW=1912KiB/s (1958kB/s)(5256KiB/2749msec)
    slat (usec): min=14, max=193, avg=43.52, stdev=19.26
    clat (usec): min=11553, max=26236, avg=16716.47, stdev=1659.42
     lat (usec): min=11626, max=26268, avg=16761.05, stdev=1659.36
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14222], 10.00th=[14746], 20.00th=[15533],
     | 30.00th=[15795], 40.00th=[16319], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18482], 95.00th=[19530],
     | 99.00th=[21103], 99.50th=[22414], 99.90th=[26084], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1768, max= 2000, per=25.10%, avg=1902.40, stdev=95.57, samples=5
   iops        : min=  442, max=  500, avg=475.60, stdev=23.89, samples=5
  lat (msec)   : 10=0.08%, 20=96.99%, 50=2.93%
  cpu          : usr=0.80%, sys=7.31%, ctx=2268, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=828: Sun Apr 13 07:46:43 2025
   read: IOPS=458, BW=1836KiB/s (1880kB/s)(5060KiB/2756msec)
    slat (usec): min=682, max=2426, avg=1123.01, stdev=220.09
    clat (usec): min=2410, max=22820, avg=16340.71, stdev=2059.85
     lat (usec): min=3699, max=23800, avg=17465.34, stdev=2055.42
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13173], 10.00th=[13829], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21103], 99.50th=[21365], 99.90th=[22414], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 1696, max= 1912, per=25.14%, avg=1830.40, stdev=84.17, samples=5
   iops        : min=  424, max=  478, avg=457.60, stdev=21.04, samples=5
  write: IOPS=469, BW=1880KiB/s (1925kB/s)(5180KiB/2756msec)
    slat (usec): min=14, max=526, avg=42.98, stdev=22.82
    clat (usec): min=9517, max=23638, avg=16681.52, stdev=1708.83
     lat (usec): min=9533, max=23686, avg=16725.69, stdev=1709.00
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13960], 10.00th=[14615], 20.00th=[15270],
     | 30.00th=[15795], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[20841], 99.50th=[21365], 99.90th=[22676], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 1744, max= 1968, per=24.55%, avg=1860.80, stdev=84.78, samples=5
   iops        : min=  436, max=  492, avg=465.20, stdev=21.19, samples=5
  lat (msec)   : 4=0.04%, 10=0.20%, 20=96.21%, 50=3.55%
  cpu          : usr=0.73%, sys=7.30%, ctx=2347, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7282KiB/s (7456kB/s), 1806KiB/s-1843KiB/s (1850kB/s-1887kB/s), io=19.6MiB (20.5MB), run=2739-2756msec
  WRITE: bw=7581KiB/s (7762kB/s), 1880KiB/s-1912KiB/s (1925kB/s-1958kB/s), io=20.4MiB (21.4MB), run=2739-2756msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=834: Sun Apr 13 07:46:44 2025
   read: IOPS=10, BW=435MiB/s (456MB/s)(40.0MiB/92msec)
    slat (nsec): min=6123.9k, max=6123.9k, avg=6123946.00, stdev= 0.00
    clat (nsec): min=81857k, max=81857k, avg=81857347.00, stdev= 0.00
     lat (nsec): min=87995k, max=87995k, avg=87995023.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[82314],  5.00th=[82314], 10.00th=[82314], 20.00th=[82314],
     | 30.00th=[82314], 40.00th=[82314], 50.00th=[82314], 60.00th=[82314],
     | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314],
     | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[82314]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.69%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=435MiB/s (456MB/s), 435MiB/s-435MiB/s (456MB/s-456MB/s), io=40.0MiB (41.9MB), run=92-92msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout