== sanity test 398c: run fio to test AIO ================= 10:12:32 (1747404752)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.114871 s, 365 MB/s
osc.lustre-OST0000-osc-ffff880084a25000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=21674: Fri May 16 10:12:39 2025
  write: IOPS=461, BW=1847KiB/s (1891kB/s)(10.0MiB/5544msec)
    slat (usec): min=18, max=34528, avg=96.87, stdev=682.06
    clat (usec): min=8414, max=70627, avg=34396.61, stdev=4335.96
     lat (usec): min=8587, max=70728, avg=34495.87, stdev=4320.90
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27132], 10.00th=[28443], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[39060], 95.00th=[39584],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[63177], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min= 1644, max= 2056, per=25.00%, avg=1847.27, stdev=137.20, samples=11
   iops        : min=  411, max=  514, avg=461.82, stdev=34.30, samples=11
  lat (msec)   : 10=0.04%, 20=0.12%, 50=99.45%, 100=0.39%
  cpu          : usr=0.87%, sys=5.32%, ctx=1618, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21675: Fri May 16 10:12:39 2025
  write: IOPS=462, BW=1848KiB/s (1892kB/s)(10.0MiB/5541msec)
    slat (usec): min=17, max=534, avg=74.71, stdev=43.26
    clat (usec): min=23825, max=71309, avg=34499.26, stdev=4237.02
     lat (usec): min=23926, max=71456, avg=34576.19, stdev=4240.99
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[27395], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[39060], 95.00th=[39584],
     | 99.00th=[44827], 99.50th=[44827], 99.90th=[63701], 99.95th=[70779],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1656, max= 2048, per=25.00%, avg=1846.91, stdev=135.91, samples=11
   iops        : min=  414, max=  512, avg=461.73, stdev=33.98, samples=11
  lat (msec)   : 50=99.65%, 100=0.35%
  cpu          : usr=0.85%, sys=4.06%, ctx=1084, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21676: Fri May 16 10:12:39 2025
  write: IOPS=463, BW=1855KiB/s (1900kB/s)(10.0MiB/5519msec)
    slat (usec): min=14, max=497, avg=79.76, stdev=42.39
    clat (usec): min=6568, max=70960, avg=34364.95, stdev=4367.11
     lat (usec): min=6707, max=71061, avg=34446.83, stdev=4366.52
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27132], 10.00th=[28181], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[42206], 99.50th=[43779], 99.90th=[67634], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min= 1648, max= 2064, per=25.05%, avg=1850.55, stdev=153.34, samples=11
   iops        : min=  412, max=  516, avg=462.64, stdev=38.34, samples=11
  lat (msec)   : 10=0.08%, 20=0.16%, 50=99.57%, 100=0.20%
  cpu          : usr=0.87%, sys=4.46%, ctx=1295, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21677: Fri May 16 10:12:39 2025
  write: IOPS=462, BW=1849KiB/s (1894kB/s)(10.0MiB/5537msec)
    slat (usec): min=19, max=358, avg=77.59, stdev=43.87
    clat (usec): min=8108, max=74436, avg=34453.77, stdev=4234.33
     lat (usec): min=8317, max=74515, avg=34533.57, stdev=4234.38
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[27395], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33424], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[39060], 95.00th=[40109],
     | 99.00th=[42206], 99.50th=[42730], 99.90th=[67634], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1648, max= 2048, per=25.05%, avg=1850.55, stdev=125.63, samples=11
   iops        : min=  412, max=  512, avg=462.64, stdev=31.41, samples=11
  lat (msec)   : 10=0.04%, 50=99.73%, 100=0.23%
  cpu          : usr=0.89%, sys=4.06%, ctx=1235, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7388KiB/s (7565kB/s), 1847KiB/s-1855KiB/s (1891kB/s-1900kB/s), io=40.0MiB (41.9MB), run=5519-5544msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=21691: Fri May 16 10:12:42 2025
   read: IOPS=631, BW=2527KiB/s (2587kB/s)(5048KiB/1998msec)
    slat (usec): min=406, max=1523, avg=724.41, stdev=162.33
    clat (usec): min=1564, max=22616, avg=11825.74, stdev=2042.44
     lat (usec): min=2377, max=23705, avg=12551.75, stdev=2092.48
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11207], 50.00th=[11731], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13173], 90.00th=[14091], 95.00th=[15139],
     | 99.00th=[19530], 99.50th=[20841], 99.90th=[22414], 99.95th=[22676],
     | 99.99th=[22676]
   bw (  KiB/s): min= 2256, max= 2624, per=24.64%, avg=2474.67, stdev=193.55, samples=3
   iops        : min=  564, max=  656, avg=618.67, stdev=48.39, samples=3
  write: IOPS=649, BW=2599KiB/s (2661kB/s)(5192KiB/1998msec)
    slat (usec): min=12, max=917, avg=38.19, stdev=42.10
    clat (usec): min=8243, max=21746, avg=12198.38, stdev=1867.13
     lat (usec): min=8291, max=21759, avg=12237.61, stdev=1866.59
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9896], 10.00th=[10159], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11600], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12649], 80.00th=[13173], 90.00th=[14091], 95.00th=[15270],
     | 99.00th=[20055], 99.50th=[20579], 99.90th=[21627], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 2400, max= 2768, per=25.02%, avg=2616.00, stdev=192.17, samples=3
   iops        : min=  600, max=  692, avg=654.00, stdev=48.04, samples=3
  lat (msec)   : 2=0.04%, 10=11.25%, 20=87.81%, 50=0.90%
  cpu          : usr=1.00%, sys=9.01%, ctx=2504, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21692: Fri May 16 10:12:42 2025
   read: IOPS=623, BW=2494KiB/s (2554kB/s)(4976KiB/1995msec)
    slat (usec): min=410, max=1667, avg=723.09, stdev=161.17
    clat (usec): min=2683, max=25512, avg=11871.59, stdev=2123.88
     lat (usec): min=3341, max=26552, avg=12596.41, stdev=2173.82
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10290],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11600], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13173], 90.00th=[13960], 95.00th=[15008],
     | 99.00th=[21103], 99.50th=[22152], 99.90th=[23462], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 2264, max= 2584, per=24.40%, avg=2450.67, stdev=166.53, samples=3
   iops        : min=  566, max=  646, avg=612.67, stdev=41.63, samples=3
  write: IOPS=659, BW=2639KiB/s (2702kB/s)(5264KiB/1995msec)
    slat (usec): min=12, max=816, avg=36.33, stdev=28.99
    clat (usec): min=1959, max=23466, avg=12179.16, stdev=1905.53
     lat (usec): min=1995, max=23502, avg=12216.54, stdev=1904.96
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10814],
     | 30.00th=[11207], 40.00th=[11600], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12649], 80.00th=[13173], 90.00th=[13960], 95.00th=[15401],
     | 99.00th=[19792], 99.50th=[21365], 99.90th=[23200], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 2448, max= 2816, per=25.48%, avg=2664.00, stdev=192.17, samples=3
   iops        : min=  612, max=  704, avg=666.00, stdev=48.04, samples=3
  lat (msec)   : 2=0.04%, 4=0.04%, 10=9.88%, 20=88.87%, 50=1.17%
  cpu          : usr=1.15%, sys=8.88%, ctx=2380, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21693: Fri May 16 10:12:42 2025
   read: IOPS=627, BW=2510KiB/s (2570kB/s)(4984KiB/1986msec)
    slat (usec): min=12, max=1794, avg=725.22, stdev=172.65
    clat (usec): min=1211, max=20780, avg=11721.62, stdev=2076.16
     lat (usec): min=1225, max=21940, avg=12448.38, stdev=2137.10
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 8979], 10.00th=[ 9634], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11076], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12387], 80.00th=[13042], 90.00th=[13960], 95.00th=[15270],
     | 99.00th=[18744], 99.50th=[19530], 99.90th=[20841], 99.95th=[20841],
     | 99.99th=[20841]
   bw (  KiB/s): min= 2400, max= 2616, per=25.17%, avg=2528.00, stdev=113.42, samples=3
   iops        : min=  600, max=  654, avg=632.00, stdev=28.35, samples=3
  write: IOPS=661, BW=2647KiB/s (2710kB/s)(5256KiB/1986msec)
    slat (usec): min=11, max=226, avg=35.89, stdev=20.25
    clat (usec): min=6058, max=25744, avg=12269.87, stdev=2057.98
     lat (usec): min=6079, max=25762, avg=12306.85, stdev=2056.66
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10814],
     | 30.00th=[11207], 40.00th=[11600], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13304], 90.00th=[14222], 95.00th=[16188],
     | 99.00th=[20579], 99.50th=[21103], 99.90th=[25035], 99.95th=[25822],
     | 99.99th=[25822]
   bw (  KiB/s): min= 2416, max= 2768, per=25.07%, avg=2621.33, stdev=183.19, samples=3
   iops        : min=  604, max=  692, avg=655.33, stdev=45.80, samples=3
  lat (msec)   : 2=0.12%, 4=0.04%, 10=11.02%, 20=87.97%, 50=0.86%
  cpu          : usr=1.46%, sys=8.51%, ctx=2446, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21694: Fri May 16 10:12:42 2025
   read: IOPS=633, BW=2535KiB/s (2596kB/s)(5060KiB/1996msec)
    slat (usec): min=432, max=1854, avg=724.82, stdev=162.22
    clat (usec): min=7041, max=21877, avg=11820.94, stdev=2002.05
     lat (usec): min=7833, max=23077, avg=12547.49, stdev=2053.63
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10290],
     | 30.00th=[10683], 40.00th=[11207], 50.00th=[11600], 60.00th=[12125],
     | 70.00th=[12649], 80.00th=[13173], 90.00th=[14091], 95.00th=[15139],
     | 99.00th=[19792], 99.50th=[21103], 99.90th=[21890], 99.95th=[21890],
     | 99.99th=[21890]
   bw (  KiB/s): min= 2248, max= 2648, per=24.96%, avg=2506.67, stdev=224.33, samples=3
   iops        : min=  562, max=  662, avg=626.67, stdev=56.08, samples=3
  write: IOPS=648, BW=2595KiB/s (2657kB/s)(5180KiB/1996msec)
    slat (usec): min=11, max=272, avg=37.05, stdev=20.56
    clat (usec): min=8508, max=24533, avg=12206.57, stdev=1907.69
     lat (usec): min=8535, max=24547, avg=12244.59, stdev=1906.07
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9765], 10.00th=[10290], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11600], 50.00th=[11994], 60.00th=[12256],
     | 70.00th=[12780], 80.00th=[13304], 90.00th=[14091], 95.00th=[15664],
     | 99.00th=[19792], 99.50th=[20579], 99.90th=[22414], 99.95th=[24511],
     | 99.99th=[24511]
   bw (  KiB/s): min= 2440, max= 2680, per=24.79%, avg=2592.00, stdev=132.18, samples=3
   iops        : min=  610, max=  670, avg=648.00, stdev=33.05, samples=3
  lat (msec)   : 10=11.17%, 20=87.97%, 50=0.86%
  cpu          : usr=1.05%, sys=9.07%, ctx=2600, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9.81MiB/s (10.3MB/s), 2494KiB/s-2535KiB/s (2554kB/s-2596kB/s), io=19.6MiB (20.5MB), run=1986-1998msec
  WRITE: bw=10.2MiB/s (10.7MB/s), 2595KiB/s-2647KiB/s (2657kB/s-2710kB/s), io=20.4MiB (21.4MB), run=1986-1998msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=21699: Fri May 16 10:12:44 2025
   read: IOPS=5, BW=229MiB/s (240MB/s)(40.0MiB/175msec)
    slat (nsec): min=5981.0k, max=5981.0k, avg=5981012.00, stdev= 0.00
    clat (nsec): min=162960k, max=162960k, avg=162959900.00, stdev= 0.00
     lat (nsec): min=168950k, max=168950k, avg=168950289.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  163], 10.00th=[  163], 20.00th=[  163],
     | 30.00th=[  163], 40.00th=[  163], 50.00th=[  163], 60.00th=[  163],
     | 70.00th=[  163], 80.00th=[  163], 90.00th=[  163], 95.00th=[  163],
     | 99.00th=[  163], 99.50th=[  163], 99.90th=[  163], 99.95th=[  163],
     | 99.99th=[  163]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.45%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=40.0MiB (41.9MB), run=175-175msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck