== sanity test 398c: run fio to test AIO ================= 09:05:24 (1748869524)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0739448 s, 567 MB/s
osc.lustre-OST0000-osc-ffff88012d10f800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=9120: Mon Jun  2 09:05:29 2025
  write: IOPS=691, BW=2765KiB/s (2832kB/s)(10.0MiB/3703msec)
    slat (usec): min=12, max=373, avg=42.47, stdev=23.85
    clat (usec): min=7985, max=34025, avg=22953.89, stdev=2925.36
     lat (usec): min=8030, max=34045, avg=22997.52, stdev=2926.61
    clat percentiles (usec):
     |  1.00th=[16319],  5.00th=[18744], 10.00th=[19792], 20.00th=[20841],
     | 30.00th=[21365], 40.00th=[22152], 50.00th=[22938], 60.00th=[23462],
     | 70.00th=[24249], 80.00th=[25035], 90.00th=[26870], 95.00th=[27919],
     | 99.00th=[30540], 99.50th=[32637], 99.90th=[33162], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 2480, max= 2968, per=24.82%, avg=2745.14, stdev=209.10, samples=7
   iops        : min=  620, max=  742, avg=686.29, stdev=52.27, samples=7
  lat (msec)   : 10=0.04%, 20=12.19%, 50=87.77%
  cpu          : usr=0.68%, sys=3.27%, ctx=1312, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9121: Mon Jun  2 09:05:29 2025
  write: IOPS=691, BW=2767KiB/s (2833kB/s)(10.0MiB/3701msec)
    slat (usec): min=12, max=1669, avg=43.22, stdev=43.38
    clat (usec): min=4007, max=36102, avg=22988.23, stdev=2922.24
     lat (usec): min=4056, max=36159, avg=23032.49, stdev=2918.64
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[18744], 10.00th=[19530], 20.00th=[20579],
     | 30.00th=[21365], 40.00th=[22152], 50.00th=[22676], 60.00th=[23462],
     | 70.00th=[24249], 80.00th=[25297], 90.00th=[27132], 95.00th=[28181],
     | 99.00th=[31327], 99.50th=[31589], 99.90th=[31851], 99.95th=[32113],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2488, max= 3016, per=24.89%, avg=2753.14, stdev=213.99, samples=7
   iops        : min=  622, max=  754, avg=688.29, stdev=53.50, samples=7
  lat (msec)   : 10=0.08%, 20=12.81%, 50=87.11%
  cpu          : usr=0.78%, sys=3.38%, ctx=1219, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9122: Mon Jun  2 09:05:29 2025
  write: IOPS=693, BW=2773KiB/s (2839kB/s)(10.0MiB/3693msec)
    slat (usec): min=11, max=648, avg=44.95, stdev=29.37
    clat (usec): min=4912, max=44108, avg=22938.07, stdev=3190.34
     lat (usec): min=4966, max=44167, avg=22984.07, stdev=3192.60
    clat percentiles (usec):
     |  1.00th=[16319],  5.00th=[18744], 10.00th=[19530], 20.00th=[20579],
     | 30.00th=[21365], 40.00th=[21890], 50.00th=[22676], 60.00th=[23462],
     | 70.00th=[23987], 80.00th=[25035], 90.00th=[26870], 95.00th=[28181],
     | 99.00th=[31851], 99.50th=[32900], 99.90th=[43779], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min= 2464, max= 3024, per=25.06%, avg=2771.43, stdev=199.03, samples=7
   iops        : min=  616, max=  756, avg=692.86, stdev=49.76, samples=7
  lat (msec)   : 10=0.20%, 20=12.81%, 50=86.99%
  cpu          : usr=0.70%, sys=3.44%, ctx=1434, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9123: Mon Jun  2 09:05:29 2025
  write: IOPS=692, BW=2771KiB/s (2837kB/s)(10.0MiB/3696msec)
    slat (usec): min=13, max=1472, avg=45.35, stdev=37.88
    clat (usec): min=2401, max=42377, avg=22929.89, stdev=3077.50
     lat (usec): min=2448, max=42413, avg=22976.30, stdev=3076.69
    clat percentiles (usec):
     |  1.00th=[16319],  5.00th=[18744], 10.00th=[19530], 20.00th=[20579],
     | 30.00th=[21365], 40.00th=[21890], 50.00th=[22676], 60.00th=[23462],
     | 70.00th=[23987], 80.00th=[25035], 90.00th=[26870], 95.00th=[28181],
     | 99.00th=[31065], 99.50th=[32637], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2472, max= 3016, per=24.97%, avg=2762.29, stdev=205.41, samples=7
   iops        : min=  618, max=  754, avg=690.57, stdev=51.35, samples=7
  lat (msec)   : 4=0.12%, 10=0.08%, 20=12.93%, 50=86.88%
  cpu          : usr=0.62%, sys=3.63%, ctx=1554, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.8MiB/s (11.3MB/s), 2765KiB/s-2773KiB/s (2832kB/s-2839kB/s), io=40.0MiB (41.9MB), run=3693-3703msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=9136: Mon Jun  2 09:05:33 2025
   read: IOPS=440, BW=1762KiB/s (1804kB/s)(5048KiB/2865msec)
    slat (usec): min=12, max=2493, avg=1172.31, stdev=262.12
    clat (usec): min=1502, max=29952, avg=16931.45, stdev=2750.72
     lat (usec): min=1614, max=31039, avg=18105.49, stdev=2810.44
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[12911], 10.00th=[13829], 20.00th=[14877],
     | 30.00th=[15533], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20317], 95.00th=[21103],
     | 99.00th=[23462], 99.50th=[25035], 99.90th=[28443], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1632, max= 1856, per=24.99%, avg=1740.00, stdev=98.39, samples=5
   iops        : min=  408, max=  464, avg=435.00, stdev=24.60, samples=5
  write: IOPS=453, BW=1812KiB/s (1856kB/s)(5192KiB/2865msec)
    slat (usec): min=13, max=1209, avg=46.26, stdev=54.14
    clat (usec): min=3539, max=28525, avg=17578.47, stdev=2323.91
     lat (usec): min=3626, max=28553, avg=17625.83, stdev=2323.61
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14091], 10.00th=[14877], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19268], 90.00th=[20317], 95.00th=[21103],
     | 99.00th=[22938], 99.50th=[25035], 99.90th=[27657], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1688, max= 1896, per=25.11%, avg=1820.00, stdev=85.51, samples=5
   iops        : min=  422, max=  474, avg=455.00, stdev=21.38, samples=5
  lat (msec)   : 2=0.04%, 4=0.16%, 10=0.62%, 20=86.76%, 50=12.42%
  cpu          : usr=0.87%, sys=6.60%, ctx=2703, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9137: Mon Jun  2 09:05:33 2025
   read: IOPS=432, BW=1730KiB/s (1772kB/s)(4976KiB/2876msec)
    slat (usec): min=729, max=2468, avg=1178.17, stdev=237.28
    clat (usec): min=3473, max=32509, avg=16956.70, stdev=2538.65
     lat (usec): min=4423, max=33473, avg=18136.50, stdev=2557.65
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13173], 10.00th=[13960], 20.00th=[15008],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22938], 99.50th=[25297], 99.90th=[31327], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 1524, max= 1864, per=24.55%, avg=1709.60, stdev=152.93, samples=5
   iops        : min=  381, max=  466, avg=427.40, stdev=38.23, samples=5
  write: IOPS=457, BW=1830KiB/s (1874kB/s)(5264KiB/2876msec)
    slat (usec): min=13, max=1019, avg=44.36, stdev=34.61
    clat (usec): min=10240, max=31211, avg=17587.62, stdev=2125.56
     lat (usec): min=10456, max=31244, avg=17632.96, stdev=2124.63
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14484], 10.00th=[15139], 20.00th=[15795],
     | 30.00th=[16319], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19268], 90.00th=[20317], 95.00th=[21103],
     | 99.00th=[22938], 99.50th=[24249], 99.90th=[27395], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1772, max= 1920, per=25.42%, avg=1842.40, stdev=65.46, samples=5
   iops        : min=  443, max=  480, avg=460.60, stdev=16.36, samples=5
  lat (msec)   : 4=0.04%, 10=0.16%, 20=88.67%, 50=11.13%
  cpu          : usr=0.77%, sys=6.82%, ctx=2565, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9138: Mon Jun  2 09:05:33 2025
   read: IOPS=432, BW=1732KiB/s (1773kB/s)(4984KiB/2878msec)
    slat (usec): min=672, max=2500, avg=1180.95, stdev=243.61
    clat (usec): min=2449, max=30990, avg=17002.53, stdev=2497.64
     lat (usec): min=3675, max=31959, avg=18185.18, stdev=2520.58
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13173], 10.00th=[13960], 20.00th=[14877],
     | 30.00th=[15795], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[23462], 99.50th=[24773], 99.90th=[28181], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1592, max= 1816, per=24.59%, avg=1712.00, stdev=90.86, samples=5
   iops        : min=  398, max=  454, avg=428.00, stdev=22.72, samples=5
  write: IOPS=456, BW=1826KiB/s (1870kB/s)(5256KiB/2878msec)
    slat (usec): min=13, max=209, avg=43.47, stdev=22.13
    clat (usec): min=9824, max=30915, avg=17578.42, stdev=2182.65
     lat (usec): min=9942, max=30930, avg=17622.89, stdev=2182.52
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14353], 10.00th=[15008], 20.00th=[15795],
     | 30.00th=[16450], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20317], 95.00th=[21103],
     | 99.00th=[23725], 99.50th=[23987], 99.90th=[30802], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1576, max= 2144, per=25.21%, avg=1827.20, stdev=226.18, samples=5
   iops        : min=  394, max=  536, avg=456.80, stdev=56.54, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=88.40%, 50=11.48%
  cpu          : usr=0.90%, sys=6.43%, ctx=2608, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9139: Mon Jun  2 09:05:33 2025
   read: IOPS=438, BW=1756KiB/s (1798kB/s)(5060KiB/2882msec)
    slat (usec): min=661, max=2367, avg=1177.71, stdev=245.53
    clat (usec): min=2303, max=26440, avg=16909.70, stdev=2455.22
     lat (usec): min=3648, max=27460, avg=18089.24, stdev=2483.80
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12911], 10.00th=[13829], 20.00th=[14877],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19792], 95.00th=[21103],
     | 99.00th=[23725], 99.50th=[24249], 99.90th=[25560], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1600, max= 1840, per=24.79%, avg=1726.40, stdev=104.06, samples=5
   iops        : min=  400, max=  460, avg=431.60, stdev=26.02, samples=5
  write: IOPS=449, BW=1797KiB/s (1840kB/s)(5180KiB/2882msec)
    slat (usec): min=13, max=1187, avg=45.13, stdev=45.85
    clat (usec): min=10343, max=27622, avg=17641.92, stdev=2139.23
     lat (usec): min=10363, max=27681, avg=17688.13, stdev=2140.72
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14353], 10.00th=[15008], 20.00th=[15926],
     | 30.00th=[16450], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19268], 90.00th=[20317], 95.00th=[21103],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[26608], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1616, max= 1904, per=24.65%, avg=1787.20, stdev=128.20, samples=5
   iops        : min=  404, max=  476, avg=446.80, stdev=32.05, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=89.22%, 50=10.66%
  cpu          : usr=0.87%, sys=6.66%, ctx=2686, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6963KiB/s (7130kB/s), 1730KiB/s-1762KiB/s (1772kB/s-1804kB/s), io=19.6MiB (20.5MB), run=2865-2882msec
  WRITE: bw=7249KiB/s (7423kB/s), 1797KiB/s-1830KiB/s (1840kB/s-1874kB/s), io=20.4MiB (21.4MB), run=2865-2882msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=9145: Mon Jun  2 09:05:34 2025
   read: IOPS=13, BW=548MiB/s (575MB/s)(40.0MiB/73msec)
    slat (nsec): min=5458.9k, max=5458.9k, avg=5458853.00, stdev= 0.00
    clat (nsec): min=62864k, max=62864k, avg=62864057.00, stdev= 0.00
     lat (nsec): min=68332k, max=68332k, avg=68331510.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[62653],  5.00th=[62653], 10.00th=[62653], 20.00th=[62653],
     | 30.00th=[62653], 40.00th=[62653], 50.00th=[62653], 60.00th=[62653],
     | 70.00th=[62653], 80.00th=[62653], 90.00th=[62653], 95.00th=[62653],
     | 99.00th=[62653], 99.50th=[62653], 99.90th=[62653], 99.95th=[62653],
     | 99.99th=[62653]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.33%, ctx=3, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=548MiB/s (575MB/s), 548MiB/s-548MiB/s (575MB/s-575MB/s), io=40.0MiB (41.9MB), run=73-73msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout