== sanity test 398c: run fio to test AIO ================= 18:47:03 (1741906023)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.092063 s, 456 MB/s
osc.lustre-OST0000-osc-ffff88007e0f1000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=23672: Thu Mar 13 18:47:11 2025
  write: IOPS=404, BW=1618KiB/s (1657kB/s)(10.0MiB/6330msec)
    slat (usec): min=12, max=1194, avg=81.42, stdev=54.81
    clat (usec): min=3490, max=81989, avg=39361.13, stdev=9464.58
     lat (usec): min=3545, max=82251, avg=39444.70, stdev=9475.51
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[21365], 10.00th=[22676], 20.00th=[29492],
     | 30.00th=[36963], 40.00th=[41157], 50.00th=[43779], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47449], 95.00th=[49546],
     | 99.00th=[51643], 99.50th=[53216], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1384, max= 2544, per=25.20%, avg=1628.67, stdev=347.01, samples=12
   iops        : min=  346, max=  636, avg=407.17, stdev=86.75, samples=12
  lat (msec)   : 4=0.04%, 10=0.08%, 20=1.29%, 50=95.16%, 100=3.44%
  cpu          : usr=0.55%, sys=4.00%, ctx=1593, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23673: Thu Mar 13 18:47:11 2025
  write: IOPS=404, BW=1617KiB/s (1656kB/s)(10.0MiB/6333msec)
    slat (usec): min=14, max=1134, avg=78.71, stdev=52.03
    clat (usec): min=8906, max=62850, avg=39365.18, stdev=9285.19
     lat (usec): min=8995, max=62977, avg=39445.89, stdev=9294.97
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[21627], 10.00th=[22676], 20.00th=[30016],
     | 30.00th=[36963], 40.00th=[41157], 50.00th=[43779], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47449], 95.00th=[49546],
     | 99.00th=[51643], 99.50th=[52167], 99.90th=[56886], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1320, max= 2512, per=25.12%, avg=1623.33, stdev=337.94, samples=12
   iops        : min=  330, max=  628, avg=405.83, stdev=84.49, samples=12
  lat (msec)   : 10=0.04%, 20=1.72%, 50=95.12%, 100=3.12%
  cpu          : usr=0.76%, sys=3.41%, ctx=1424, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23674: Thu Mar 13 18:47:11 2025
  write: IOPS=405, BW=1622KiB/s (1661kB/s)(10.0MiB/6312msec)
    slat (usec): min=13, max=667, avg=83.22, stdev=54.41
    clat (usec): min=3841, max=86462, avg=39252.31, stdev=9843.14
     lat (usec): min=3934, max=86516, avg=39337.74, stdev=9853.77
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[20841], 10.00th=[22414], 20.00th=[29492],
     | 30.00th=[36963], 40.00th=[41157], 50.00th=[43254], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47973], 95.00th=[49546],
     | 99.00th=[51643], 99.50th=[59507], 99.90th=[82314], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1360, max= 2536, per=25.32%, avg=1636.00, stdev=347.90, samples=12
   iops        : min=  340, max=  634, avg=409.00, stdev=86.98, samples=12
  lat (msec)   : 4=0.04%, 10=0.23%, 20=2.42%, 50=93.91%, 100=3.40%
  cpu          : usr=0.89%, sys=3.63%, ctx=1577, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23675: Thu Mar 13 18:47:11 2025
  write: IOPS=403, BW=1616KiB/s (1654kB/s)(10.0MiB/6338msec)
    slat (usec): min=12, max=566, avg=72.40, stdev=42.86
    clat (usec): min=15779, max=67244, avg=39373.77, stdev=9231.05
     lat (usec): min=15906, max=67331, avg=39448.09, stdev=9243.19
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[21627], 10.00th=[22676], 20.00th=[30540],
     | 30.00th=[36963], 40.00th=[41157], 50.00th=[43254], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47973], 95.00th=[49021],
     | 99.00th=[51643], 99.50th=[53216], 99.90th=[53740], 99.95th=[54264],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1296, max= 2480, per=25.11%, avg=1622.67, stdev=326.39, samples=12
   iops        : min=  324, max=  620, avg=405.67, stdev=81.60, samples=12
  lat (msec)   : 20=1.13%, 50=95.55%, 100=3.32%
  cpu          : usr=0.54%, sys=3.50%, ctx=1081, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6463KiB/s (6618kB/s), 1616KiB/s-1622KiB/s (1654kB/s-1661kB/s), io=40.0MiB (41.9MB), run=6312-6338msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=23689: Thu Mar 13 18:47:14 2025
   read: IOPS=460, BW=1842KiB/s (1886kB/s)(5048KiB/2741msec)
    slat (usec): min=663, max=1921, avg=1090.09, stdev=211.31
    clat (usec): min=9485, max=25028, avg=16135.66, stdev=2385.62
     lat (usec): min=10514, max=26108, avg=17227.53, stdev=2390.89
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12518], 10.00th=[13173], 20.00th=[13960],
     | 30.00th=[14746], 40.00th=[15401], 50.00th=[16057], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[19006], 95.00th=[20055],
     | 99.00th=[22676], 99.50th=[23725], 99.90th=[25035], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1736, max= 1988, per=25.03%, avg=1832.80, stdev=101.29, samples=5
   iops        : min=  434, max=  497, avg=458.20, stdev=25.32, samples=5
  write: IOPS=473, BW=1894KiB/s (1940kB/s)(5192KiB/2741msec)
    slat (usec): min=13, max=1160, avg=46.89, stdev=41.63
    clat (usec): min=9442, max=23540, avg=16772.14, stdev=1919.36
     lat (usec): min=9477, max=23564, avg=16820.12, stdev=1919.91
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13829], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21890], 99.50th=[22414], 99.90th=[23462], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 1848, max= 1996, per=24.97%, avg=1903.20, stdev=59.09, samples=5
   iops        : min=  462, max=  499, avg=475.80, stdev=14.77, samples=5
  lat (msec)   : 10=0.08%, 20=94.49%, 50=5.43%
  cpu          : usr=0.99%, sys=7.30%, ctx=2817, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23690: Thu Mar 13 18:47:14 2025
   read: IOPS=455, BW=1821KiB/s (1864kB/s)(4976KiB/2733msec)
    slat (usec): min=630, max=2156, avg=1097.46, stdev=220.75
    clat (usec): min=4418, max=33794, avg=16155.18, stdev=2443.53
     lat (usec): min=5691, max=34690, avg=17254.39, stdev=2450.43
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12518], 10.00th=[13173], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[16057], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21890], 99.50th=[22938], 99.90th=[25560], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1704, max= 1912, per=24.77%, avg=1813.60, stdev=101.57, samples=5
   iops        : min=  426, max=  478, avg=453.40, stdev=25.39, samples=5
  write: IOPS=481, BW=1926KiB/s (1972kB/s)(5264KiB/2733msec)
    slat (usec): min=12, max=1072, avg=44.00, stdev=35.00
    clat (usec): min=3921, max=30781, avg=16670.85, stdev=2075.48
     lat (usec): min=3981, max=30843, avg=16715.90, stdev=2076.08
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13566], 10.00th=[14353], 20.00th=[15008],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21627], 99.50th=[22414], 99.90th=[24773], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1848, max= 2091, per=25.49%, avg=1943.00, stdev=95.55, samples=5
   iops        : min=  462, max=  522, avg=485.60, stdev=23.60, samples=5
  lat (msec)   : 4=0.04%, 10=0.43%, 20=94.10%, 50=5.43%
  cpu          : usr=0.99%, sys=7.21%, ctx=2680, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23691: Thu Mar 13 18:47:14 2025
   read: IOPS=459, BW=1839KiB/s (1883kB/s)(4984KiB/2710msec)
    slat (usec): min=11, max=2102, avg=1098.07, stdev=242.06
    clat (usec): min=1885, max=30418, avg=15993.97, stdev=2533.21
     lat (usec): min=1907, max=31590, avg=17093.79, stdev=2574.39
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[12387], 10.00th=[13173], 20.00th=[13960],
     | 30.00th=[14746], 40.00th=[15401], 50.00th=[16057], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19006], 95.00th=[20055],
     | 99.00th=[22152], 99.50th=[22938], 99.90th=[24511], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1744, max= 1956, per=25.32%, avg=1853.60, stdev=85.40, samples=5
   iops        : min=  436, max=  489, avg=463.40, stdev=21.35, samples=5
  write: IOPS=484, BW=1939KiB/s (1986kB/s)(5256KiB/2710msec)
    slat (usec): min=12, max=1010, avg=48.09, stdev=44.57
    clat (usec): min=7351, max=32318, avg=16653.95, stdev=2164.79
     lat (usec): min=7387, max=32358, avg=16703.19, stdev=2163.30
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13698], 10.00th=[14353], 20.00th=[15139],
     | 30.00th=[15664], 40.00th=[16057], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21890], 99.50th=[24249], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1848, max= 2027, per=25.43%, avg=1938.20, stdev=81.09, samples=5
   iops        : min=  462, max=  506, avg=484.40, stdev=20.07, samples=5
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.62%, 20=94.88%, 50=4.38%
  cpu          : usr=1.26%, sys=7.05%, ctx=2889, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23692: Thu Mar 13 18:47:14 2025
   read: IOPS=461, BW=1846KiB/s (1890kB/s)(5060KiB/2741msec)
    slat (usec): min=617, max=2148, avg=1094.72, stdev=220.35
    clat (usec): min=8154, max=23716, avg=16160.07, stdev=2309.19
     lat (usec): min=9216, max=24814, avg=17256.63, stdev=2322.75
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12649], 10.00th=[13304], 20.00th=[14222],
     | 30.00th=[14877], 40.00th=[15533], 50.00th=[16057], 60.00th=[16712],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[21627], 99.50th=[22676], 99.90th=[23200], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 1752, max= 2003, per=25.21%, avg=1845.40, stdev=99.59, samples=5
   iops        : min=  438, max=  500, avg=461.20, stdev=24.60, samples=5
  write: IOPS=472, BW=1890KiB/s (1935kB/s)(5180KiB/2741msec)
    slat (usec): min=12, max=1633, avg=46.75, stdev=57.02
    clat (usec): min=9220, max=26017, avg=16715.15, stdev=1873.94
     lat (usec): min=9250, max=26059, avg=16762.88, stdev=1874.23
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13698], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792],
     | 99.00th=[21365], 99.50th=[21890], 99.90th=[25035], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 1792, max= 1948, per=24.72%, avg=1884.00, stdev=60.00, samples=5
   iops        : min=  448, max=  487, avg=471.00, stdev=15.00, samples=5
  lat (msec)   : 10=0.31%, 20=95.39%, 50=4.30%
  cpu          : usr=0.51%, sys=7.74%, ctx=2774, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7321KiB/s (7497kB/s), 1821KiB/s-1846KiB/s (1864kB/s-1890kB/s), io=19.6MiB (20.5MB), run=2710-2741msec
  WRITE: bw=7622KiB/s (7805kB/s), 1890KiB/s-1939KiB/s (1935kB/s-1986kB/s), io=20.4MiB (21.4MB), run=2710-2741msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=23698: Thu Mar 13 18:47:16 2025
   read: IOPS=8, BW=325MiB/s (341MB/s)(40.0MiB/123msec)
    slat (nsec): min=5293.3k, max=5293.3k, avg=5293302.00, stdev= 0.00
    clat (nsec): min=112502k, max=112502k, avg=112501641.00, stdev= 0.00
     lat (nsec): min=117804k, max=117804k, avg=117803658.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  113],  5.00th=[  113], 10.00th=[  113], 20.00th=[  113],
     | 30.00th=[  113], 40.00th=[  113], 50.00th=[  113], 60.00th=[  113],
     | 70.00th=[  113], 80.00th=[  113], 90.00th=[  113], 95.00th=[  113],
     | 99.00th=[  113], 99.50th=[  113], 99.90th=[  113], 99.95th=[  113],
     | 99.99th=[  113]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.10%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=40.0MiB (41.9MB), run=123-123msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout