== sanity test 398c: run fio to test AIO ================= 18:33:20 (1747262000)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.153312 s, 274 MB/s
osc.lustre-OST0000-osc-ffff88009c9b1000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18346: Wed May 14 18:33:28 2025
  write: IOPS=451, BW=1806KiB/s (1849kB/s)(10.0MiB/5671msec)
    slat (usec): min=12, max=402, avg=80.41, stdev=42.32
    clat (usec): min=8441, max=52729, avg=35283.07, stdev=3772.63
     lat (usec): min=8538, max=52798, avg=35365.80, stdev=3772.05
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[28705], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[37487], 80.00th=[38011], 90.00th=[38536], 95.00th=[39060],
     | 99.00th=[41681], 99.50th=[48497], 99.90th=[52691], 99.95th=[52691],
     | 99.99th=[52691]
   bw (  KiB/s): min= 1680, max= 1952, per=24.91%, avg=1798.55, stdev=76.97, samples=11
   iops        : min=  420, max=  488, avg=449.64, stdev=19.24, samples=11
  lat (msec)   : 10=0.04%, 20=0.31%, 50=99.41%, 100=0.23%
  cpu          : usr=0.90%, sys=4.20%, ctx=1236, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18347: Wed May 14 18:33:28 2025
  write: IOPS=451, BW=1807KiB/s (1850kB/s)(10.0MiB/5668msec)
    slat (usec): min=15, max=284, avg=81.81, stdev=40.12
    clat (usec): min=11460, max=65750, avg=35265.28, stdev=4013.07
     lat (usec): min=11590, max=65826, avg=35349.48, stdev=4013.19
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[28705], 10.00th=[30278], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[37487], 80.00th=[38011], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[42206], 99.50th=[48497], 99.90th=[62129], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min= 1680, max= 1984, per=24.98%, avg=1804.00, stdev=84.27, samples=11
   iops        : min=  420, max=  496, avg=451.00, stdev=21.07, samples=11
  lat (msec)   : 20=0.66%, 50=99.02%, 100=0.31%
  cpu          : usr=0.99%, sys=4.29%, ctx=1507, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18348: Wed May 14 18:33:28 2025
  write: IOPS=453, BW=1813KiB/s (1857kB/s)(10.0MiB/5647msec)
    slat (usec): min=14, max=387, avg=82.86, stdev=41.44
    clat (usec): min=3837, max=66212, avg=35159.49, stdev=4351.12
     lat (usec): min=3864, max=66270, avg=35244.73, stdev=4354.01
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[28443], 10.00th=[30278], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[37487], 80.00th=[38011], 90.00th=[38536], 95.00th=[39060],
     | 99.00th=[44827], 99.50th=[45876], 99.90th=[66323], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min= 1688, max= 2056, per=25.07%, avg=1810.55, stdev=94.46, samples=11
   iops        : min=  422, max=  514, avg=452.64, stdev=23.61, samples=11
  lat (msec)   : 4=0.04%, 10=0.04%, 20=1.29%, 50=98.20%, 100=0.43%
  cpu          : usr=0.83%, sys=4.66%, ctx=1548, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18349: Wed May 14 18:33:28 2025
  write: IOPS=451, BW=1805KiB/s (1849kB/s)(10.0MiB/5672msec)
    slat (usec): min=16, max=560, avg=81.86, stdev=42.72
    clat (usec): min=11434, max=58126, avg=35291.84, stdev=3853.90
     lat (usec): min=11549, max=58180, avg=35375.96, stdev=3853.44
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[28705], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[37487], 80.00th=[38011], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[42206], 99.50th=[47973], 99.90th=[57934], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1680, max= 1960, per=24.94%, avg=1801.09, stdev=80.36, samples=11
   iops        : min=  420, max=  490, avg=450.27, stdev=20.09, samples=11
  lat (msec)   : 20=0.70%, 50=99.10%, 100=0.20%
  cpu          : usr=0.76%, sys=4.50%, ctx=1488, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7221KiB/s (7395kB/s), 1805KiB/s-1813KiB/s (1849kB/s-1857kB/s), io=40.0MiB (41.9MB), run=5647-5672msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18364: Wed May 14 18:33:31 2025
   read: IOPS=628, BW=2514KiB/s (2574kB/s)(5048KiB/2008msec)
    slat (usec): min=449, max=2550, avg=759.67, stdev=183.30
    clat (usec): min=1435, max=25436, avg=11780.49, stdev=1942.20
     lat (usec): min=2290, max=26935, avg=12541.86, stdev=2011.23
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 8979], 10.00th=[ 9634], 20.00th=[10290],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11600], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[13173], 90.00th=[13960], 95.00th=[14746],
     | 99.00th=[17171], 99.50th=[20579], 99.90th=[24511], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 2272, max= 2720, per=25.08%, avg=2506.00, stdev=223.18, samples=4
   iops        : min=  568, max=  680, avg=626.50, stdev=55.79, samples=4
  write: IOPS=646, BW=2586KiB/s (2648kB/s)(5192KiB/2008msec)
    slat (usec): min=12, max=802, avg=37.41, stdev=28.74
    clat (usec): min=3782, max=26890, avg=12257.94, stdev=1814.43
     lat (usec): min=3814, max=26939, avg=12296.33, stdev=1816.02
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12125], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13304], 90.00th=[13960], 95.00th=[14615],
     | 99.00th=[21103], 99.50th=[22938], 99.90th=[25560], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 2456, max= 2728, per=24.84%, avg=2584.00, stdev=132.58, samples=4
   iops        : min=  614, max=  682, avg=646.00, stdev=33.15, samples=4
  lat (msec)   : 2=0.04%, 4=0.04%, 10=10.04%, 20=89.10%, 50=0.78%
  cpu          : usr=1.05%, sys=9.52%, ctx=2177, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18365: Wed May 14 18:33:31 2025
   read: IOPS=619, BW=2479KiB/s (2539kB/s)(4976KiB/2007msec)
    slat (usec): min=447, max=2625, avg=757.94, stdev=179.43
    clat (usec): min=3564, max=25370, avg=11754.43, stdev=2018.37
     lat (usec): min=5161, max=27030, avg=12514.33, stdev=2069.72
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11207], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[14091], 95.00th=[15008],
     | 99.00th=[18220], 99.50th=[21890], 99.90th=[25297], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 2280, max= 2664, per=24.69%, avg=2468.00, stdev=181.55, samples=4
   iops        : min=  570, max=  666, avg=617.00, stdev=45.39, samples=4
  write: IOPS=655, BW=2623KiB/s (2686kB/s)(5264KiB/2007msec)
    slat (usec): min=12, max=680, avg=37.62, stdev=26.97
    clat (usec): min=8109, max=24969, avg=12337.94, stdev=1821.58
     lat (usec): min=8181, max=25046, avg=12376.54, stdev=1822.53
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[10028], 10.00th=[10421], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11731], 50.00th=[12125], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13304], 90.00th=[14091], 95.00th=[14877],
     | 99.00th=[20579], 99.50th=[22676], 99.90th=[25035], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 2456, max= 2832, per=25.20%, avg=2622.00, stdev=168.74, samples=4
   iops        : min=  614, max=  708, avg=655.50, stdev=42.19, samples=4
  lat (msec)   : 4=0.04%, 10=10.55%, 20=88.55%, 50=0.86%
  cpu          : usr=1.55%, sys=8.87%, ctx=2191, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18366: Wed May 14 18:33:31 2025
   read: IOPS=620, BW=2483KiB/s (2543kB/s)(4984KiB/2007msec)
    slat (usec): min=435, max=2046, avg=765.05, stdev=166.86
    clat (usec): min=7106, max=24344, avg=11756.97, stdev=1944.70
     lat (usec): min=7938, max=26115, avg=12523.88, stdev=2004.69
    clat percentiles (usec):
     |  1.00th=[ 8225],  5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11076], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12518], 80.00th=[13173], 90.00th=[14091], 95.00th=[15008],
     | 99.00th=[18220], 99.50th=[19530], 99.90th=[23200], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 2376, max= 2576, per=24.83%, avg=2482.00, stdev=97.19, samples=4
   iops        : min=  594, max=  644, avg=620.50, stdev=24.30, samples=4
  write: IOPS=654, BW=2619KiB/s (2682kB/s)(5256KiB/2007msec)
    slat (usec): min=11, max=520, avg=37.85, stdev=22.18
    clat (usec): min=8059, max=24948, avg=12308.67, stdev=1870.64
     lat (usec): min=8095, max=25005, avg=12347.51, stdev=1872.79
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125], 60.00th=[12518],
     | 70.00th=[12911], 80.00th=[13435], 90.00th=[14222], 95.00th=[14877],
     | 99.00th=[21627], 99.50th=[22414], 99.90th=[24511], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 2344, max= 2904, per=25.07%, avg=2608.00, stdev=289.33, samples=4
   iops        : min=  586, max=  726, avg=652.00, stdev=72.33, samples=4
  lat (msec)   : 10=10.90%, 20=88.24%, 50=0.86%
  cpu          : usr=1.30%, sys=9.47%, ctx=2165, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18367: Wed May 14 18:33:31 2025
   read: IOPS=633, BW=2535KiB/s (2596kB/s)(5060KiB/1996msec)
    slat (usec): min=60, max=2502, avg=761.18, stdev=188.85
    clat (usec): min=6822, max=25377, avg=11757.99, stdev=1981.91
     lat (usec): min=7399, max=27766, avg=12520.99, stdev=2062.07
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 9241], 10.00th=[ 9634], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11076], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12518], 80.00th=[13042], 90.00th=[13960], 95.00th=[14877],
     | 99.00th=[18744], 99.50th=[22152], 99.90th=[24773], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 2320, max= 2696, per=24.84%, avg=2482.67, stdev=193.05, samples=3
   iops        : min=  580, max=  674, avg=620.67, stdev=48.26, samples=3
  write: IOPS=648, BW=2595KiB/s (2657kB/s)(5180KiB/1996msec)
    slat (usec): min=12, max=422, avg=37.85, stdev=23.06
    clat (usec): min=8008, max=25403, avg=12280.56, stdev=1745.72
     lat (usec): min=8051, max=25431, avg=12319.48, stdev=1747.08
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10028], 10.00th=[10552], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11731], 50.00th=[12125], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13304], 90.00th=[13960], 95.00th=[14615],
     | 99.00th=[20055], 99.50th=[21890], 99.90th=[25297], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 2464, max= 2680, per=24.58%, avg=2557.33, stdev=110.95, samples=3
   iops        : min=  616, max=  670, avg=639.33, stdev=27.74, samples=3
  lat (msec)   : 10=10.00%, 20=89.10%, 50=0.90%
  cpu          : usr=1.10%, sys=9.47%, ctx=2192, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9994KiB/s (10.2MB/s), 2479KiB/s-2535KiB/s (2539kB/s-2596kB/s), io=19.6MiB (20.5MB), run=1996-2008msec
  WRITE: bw=10.2MiB/s (10.7MB/s), 2586KiB/s-2623KiB/s (2648kB/s-2686kB/s), io=20.4MiB (21.4MB), run=1996-2008msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18372: Wed May 14 18:33:32 2025
   read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec)
    slat (nsec): min=7307.5k, max=7307.5k, avg=7307498.00, stdev= 0.00
    clat (nsec): min=91238k, max=91238k, avg=91238048.00, stdev= 0.00
     lat (nsec): min=98558k, max=98558k, avg=98558371.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[91751],  5.00th=[91751], 10.00th=[91751], 20.00th=[91751],
     | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751],
     | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751],
     | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.86%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck