== sanity test 398c: run fio to test AIO ================= 21:44:57 (1736995497)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0956068 s, 439 MB/s
osc.lustre-OST0000-osc-ffff88008e15a800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=21605: Wed Jan 15 21:45:01 2025
  write: IOPS=922, BW=3691KiB/s (3780kB/s)(10.0MiB/2774msec)
    slat (usec): min=12, max=300, avg=42.87, stdev=20.56
    clat (usec): min=3624, max=35456, avg=17245.97, stdev=2165.22
     lat (usec): min=3672, max=35474, avg=17290.10, stdev=2166.28
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13435], 10.00th=[14877], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[16909], 50.00th=[17433], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[24511], 99.50th=[26084], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[35390]
   bw (  KiB/s): min= 3488, max= 3960, per=24.98%, avg=3688.00, stdev=195.55, samples=5
   iops        : min=  872, max=  990, avg=922.00, stdev=48.89, samples=5
  lat (msec)   : 4=0.04%, 10=0.04%, 20=94.80%, 50=5.12%
  cpu          : usr=0.65%, sys=5.01%, ctx=1296, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21606: Wed Jan 15 21:45:01 2025
  write: IOPS=922, BW=3691KiB/s (3780kB/s)(10.0MiB/2774msec)
    slat (usec): min=12, max=394, avg=43.01, stdev=21.37
    clat (usec): min=6255, max=35193, avg=17246.22, stdev=2200.42
     lat (usec): min=6305, max=35250, avg=17290.48, stdev=2201.49
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13435], 10.00th=[14877], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[16909], 50.00th=[17433], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[24511], 99.50th=[25822], 99.90th=[33817], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 3488, max= 3992, per=24.99%, avg=3689.60, stdev=198.43, samples=5
   iops        : min=  872, max=  998, avg=922.40, stdev=49.61, samples=5
  lat (msec)   : 10=0.20%, 20=94.06%, 50=5.74%
  cpu          : usr=0.76%, sys=5.01%, ctx=1331, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21607: Wed Jan 15 21:45:01 2025
  write: IOPS=927, BW=3710KiB/s (3799kB/s)(10.0MiB/2760msec)
    slat (usec): min=12, max=248, avg=42.13, stdev=18.39
    clat (usec): min=3865, max=34112, avg=17178.89, stdev=2295.39
     lat (usec): min=3899, max=34133, avg=17222.26, stdev=2295.28
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[13435], 10.00th=[14746], 20.00th=[15795],
     | 30.00th=[16450], 40.00th=[16909], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18482], 90.00th=[19268], 95.00th=[19792],
     | 99.00th=[23987], 99.50th=[27395], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[34341]
   bw (  KiB/s): min= 3456, max= 4095, per=25.12%, avg=3708.60, stdev=254.72, samples=5
   iops        : min=  864, max= 1023, avg=927.00, stdev=63.40, samples=5
  lat (msec)   : 4=0.04%, 10=0.90%, 20=94.45%, 50=4.61%
  cpu          : usr=1.01%, sys=4.75%, ctx=1163, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21608: Wed Jan 15 21:45:01 2025
  write: IOPS=923, BW=3694KiB/s (3783kB/s)(10.0MiB/2772msec)
    slat (usec): min=12, max=274, avg=40.94, stdev=19.32
    clat (usec): min=6280, max=31858, avg=17237.44, stdev=2182.98
     lat (usec): min=6319, max=31926, avg=17279.54, stdev=2183.42
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13435], 10.00th=[14877], 20.00th=[15664],
     | 30.00th=[16581], 40.00th=[16909], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18482], 90.00th=[19268], 95.00th=[20055],
     | 99.00th=[24511], 99.50th=[25822], 99.90th=[31851], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 3488, max= 3992, per=24.97%, avg=3686.40, stdev=199.47, samples=5
   iops        : min=  872, max=  998, avg=921.60, stdev=49.87, samples=5
  lat (msec)   : 10=0.27%, 20=94.41%, 50=5.31%
  cpu          : usr=1.01%, sys=4.44%, ctx=1177, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=14.4MiB/s (15.1MB/s), 3691KiB/s-3710KiB/s (3780kB/s-3799kB/s), io=40.0MiB (41.9MB), run=2760-2774msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=21620: Wed Jan 15 21:45:04 2025
   read: IOPS=510, BW=2044KiB/s (2093kB/s)(5048KiB/2470msec)
    slat (usec): min=510, max=2248, avg=942.44, stdev=211.99
    clat (usec): min=2000, max=29826, avg=14601.92, stdev=2979.95
     lat (usec): min=3843, max=31658, avg=15546.30, stdev=3049.56
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10683], 10.00th=[11338], 20.00th=[12256],
     | 30.00th=[12911], 40.00th=[13698], 50.00th=[14222], 60.00th=[14746],
     | 70.00th=[15533], 80.00th=[16581], 90.00th=[18482], 95.00th=[20055],
     | 99.00th=[23987], 99.50th=[25035], 99.90th=[29492], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1792, max= 2307, per=25.42%, avg=2064.75, stdev=229.95, samples=4
   iops        : min=  448, max=  576, avg=516.00, stdev=57.22, samples=4
  write: IOPS=525, BW=2102KiB/s (2152kB/s)(5192KiB/2470msec)
    slat (usec): min=12, max=1091, avg=47.96, stdev=44.71
    clat (usec): min=1946, max=27761, avg=15118.52, stdev=2774.68
     lat (usec): min=1991, max=27813, avg=15167.70, stdev=2775.18
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[11731], 10.00th=[12256], 20.00th=[13042],
     | 30.00th=[13698], 40.00th=[14091], 50.00th=[14615], 60.00th=[15270],
     | 70.00th=[15926], 80.00th=[16909], 90.00th=[18482], 95.00th=[20055],
     | 99.00th=[25035], 99.50th=[25822], 99.90th=[27395], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1960, max= 2435, per=25.88%, avg=2188.75, stdev=194.87, samples=4
   iops        : min=  490, max=  608, avg=547.00, stdev=48.40, samples=4
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.98%, 20=93.63%, 50=5.23%
  cpu          : usr=1.13%, sys=8.87%, ctx=2338, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21621: Wed Jan 15 21:45:04 2025
   read: IOPS=504, BW=2016KiB/s (2065kB/s)(4976KiB/2468msec)
    slat (usec): min=481, max=2305, avg=933.28, stdev=209.02
    clat (usec): min=7218, max=29929, avg=14578.24, stdev=3011.86
     lat (usec): min=8256, max=31265, avg=15513.36, stdev=3086.37
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[10683], 10.00th=[11338], 20.00th=[12256],
     | 30.00th=[12911], 40.00th=[13566], 50.00th=[14091], 60.00th=[14746],
     | 70.00th=[15533], 80.00th=[16319], 90.00th=[18220], 95.00th=[20317],
     | 99.00th=[24773], 99.50th=[25822], 99.90th=[29492], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1808, max= 2259, per=25.27%, avg=2052.75, stdev=201.34, samples=4
   iops        : min=  452, max=  564, avg=513.00, stdev=50.08, samples=4
  write: IOPS=533, BW=2133KiB/s (2184kB/s)(5264KiB/2468msec)
    slat (usec): min=12, max=393, avg=47.07, stdev=26.21
    clat (usec): min=9072, max=27804, avg=15158.81, stdev=2782.12
     lat (usec): min=9126, max=27840, avg=15207.09, stdev=2785.21
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[11731], 10.00th=[12387], 20.00th=[13042],
     | 30.00th=[13566], 40.00th=[14091], 50.00th=[14615], 60.00th=[15270],
     | 70.00th=[15926], 80.00th=[16909], 90.00th=[18482], 95.00th=[20579],
     | 99.00th=[25035], 99.50th=[26346], 99.90th=[27395], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1904, max= 2491, per=26.16%, avg=2212.75, stdev=240.49, samples=4
   iops        : min=  476, max=  622, avg=553.00, stdev=59.83, samples=4
  lat (msec)   : 10=1.05%, 20=93.24%, 50=5.70%
  cpu          : usr=0.93%, sys=9.20%, ctx=2277, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21622: Wed Jan 15 21:45:04 2025
   read: IOPS=505, BW=2024KiB/s (2072kB/s)(4984KiB/2463msec)
    slat (usec): min=12, max=2250, avg=934.16, stdev=219.99
    clat (usec): min=1175, max=34271, avg=14459.44, stdev=2970.55
     lat (usec): min=1194, max=35360, avg=15395.48, stdev=3058.74
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[10814], 10.00th=[11338], 20.00th=[12256],
     | 30.00th=[12911], 40.00th=[13435], 50.00th=[14222], 60.00th=[14746],
     | 70.00th=[15270], 80.00th=[16057], 90.00th=[17695], 95.00th=[19792],
     | 99.00th=[24249], 99.50th=[27919], 99.90th=[33162], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1648, max= 2376, per=25.65%, avg=2084.00, stdev=308.77, samples=4
   iops        : min=  412, max=  594, avg=521.00, stdev=77.19, samples=4
  write: IOPS=533, BW=2134KiB/s (2185kB/s)(5256KiB/2463msec)
    slat (usec): min=12, max=216, avg=44.98, stdev=20.35
    clat (usec): min=3144, max=38826, avg=15260.21, stdev=3168.31
     lat (usec): min=3202, max=38859, avg=15306.41, stdev=3170.39
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[11863], 10.00th=[12256], 20.00th=[13042],
     | 30.00th=[13698], 40.00th=[14222], 50.00th=[14746], 60.00th=[15270],
     | 70.00th=[15926], 80.00th=[16909], 90.00th=[18482], 95.00th=[20841],
     | 99.00th=[26346], 99.50th=[28443], 99.90th=[38536], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 2080, max= 2368, per=25.99%, avg=2198.00, stdev=128.56, samples=4
   iops        : min=  520, max=  592, avg=549.50, stdev=32.14, samples=4
  lat (msec)   : 2=0.08%, 4=0.04%, 10=1.13%, 20=93.40%, 50=5.35%
  cpu          : usr=1.42%, sys=8.85%, ctx=2245, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21623: Wed Jan 15 21:45:04 2025
   read: IOPS=515, BW=2063KiB/s (2112kB/s)(5060KiB/2453msec)
    slat (usec): min=17, max=2146, avg=934.47, stdev=229.59
    clat (usec): min=1284, max=36121, avg=14484.99, stdev=3124.80
     lat (usec): min=1419, max=36992, avg=15421.59, stdev=3208.21
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10814], 10.00th=[11338], 20.00th=[12256],
     | 30.00th=[12780], 40.00th=[13435], 50.00th=[14091], 60.00th=[14746],
     | 70.00th=[15401], 80.00th=[16319], 90.00th=[18220], 95.00th=[19792],
     | 99.00th=[25560], 99.50th=[27657], 99.90th=[33162], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1816, max= 2363, per=26.01%, avg=2112.75, stdev=229.52, samples=4
   iops        : min=  454, max=  590, avg=528.00, stdev=57.11, samples=4
  write: IOPS=527, BW=2112KiB/s (2162kB/s)(5180KiB/2453msec)
    slat (usec): min=13, max=261, avg=47.48, stdev=26.70
    clat (usec): min=2260, max=37062, avg=15103.51, stdev=3001.61
     lat (usec): min=2296, max=37114, avg=15152.26, stdev=3003.92
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[11469], 10.00th=[12125], 20.00th=[13042],
     | 30.00th=[13698], 40.00th=[14091], 50.00th=[14615], 60.00th=[15139],
     | 70.00th=[15926], 80.00th=[16712], 90.00th=[18482], 95.00th=[20579],
     | 99.00th=[24511], 99.50th=[25822], 99.90th=[36963], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1904, max= 2419, per=25.76%, avg=2178.75, stdev=211.42, samples=4
   iops        : min=  476, max=  604, avg=544.50, stdev=52.57, samples=4
  lat (msec)   : 2=0.12%, 4=0.12%, 10=1.29%, 20=93.20%, 50=5.27%
  cpu          : usr=1.18%, sys=9.05%, ctx=2394, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=8125KiB/s (8320kB/s), 2016KiB/s-2063KiB/s (2065kB/s-2112kB/s), io=19.6MiB (20.5MB), run=2453-2470msec
  WRITE: bw=8458KiB/s (8661kB/s), 2102KiB/s-2134KiB/s (2152kB/s-2185kB/s), io=20.4MiB (21.4MB), run=2453-2470msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=21629: Wed Jan 15 21:45:06 2025
   read: IOPS=11, BW=465MiB/s (488MB/s)(40.0MiB/86msec)
    slat (nsec): min=7552.8k, max=7552.8k, avg=7552840.00, stdev= 0.00
    clat (nsec): min=74781k, max=74781k, avg=74781104.00, stdev= 0.00
     lat (nsec): min=82349k, max=82349k, avg=82349069.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[74974],  5.00th=[74974], 10.00th=[74974], 20.00th=[74974],
     | 30.00th=[74974], 40.00th=[74974], 50.00th=[74974], 60.00th=[74974],
     | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974],
     | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.41%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=465MiB/s (488MB/s), 465MiB/s-465MiB/s (488MB/s-488MB/s), io=40.0MiB (41.9MB), run=86-86msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck