== sanity test 398c: run fio to test AIO ================= 01:19:34 (1789449574) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.374643 s, 112 MB/s osc.lustre-OST0000-osc-ffff936eb119d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146471: Tue Sep 15 01:22:21 2026 write: IOPS=16, BW=64.7KiB/s (66.3kB/s)(10.0MiB/158203msec); 0 zone resets slat (usec): min=23, max=20319, avg=200.65, stdev=841.08 clat (msec): min=172, max=2048, avg=988.17, stdev=254.47 lat (msec): min=172, max=2048, avg=988.38, stdev=254.50 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 625], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1653], 99.50th=[ 1854], 99.90th=[ 2005], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 221, per=25.18%, avg=64.96, stdev=31.03, samples=310 iops : min= 1, max= 55, avg=15.81, stdev= 7.77, samples=310 lat (msec) : 250=0.62%, 500=2.73%, 750=9.06%, 1000=42.38%, 2000=45.08% lat (msec) : >=2000=0.12% cpu : usr=0.04%, sys=0.21%, ctx=2243, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146472: Tue Sep 15 01:22:21 2026 write: IOPS=16, BW=64.7KiB/s (66.3kB/s)(10.0MiB/158204msec); 0 zone resets slat (usec): min=29, max=13731, avg=186.94, stdev=678.46 clat (msec): min=308, max=2009, avg=987.65, stdev=254.09 lat (msec): min=308, max=2010, avg=987.84, stdev=254.08 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 625], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1687], 99.50th=[ 1854], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 158, per=24.88%, avg=64.18, stdev=28.89, samples=314 iops : min= 1, max= 39, avg=15.61, stdev= 7.24, samples=314 lat (msec) : 500=3.44%, 750=10.23%, 1000=41.76%, 2000=44.53%, >=2000=0.04% cpu : usr=0.04%, sys=0.20%, ctx=2056, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146473: Tue Sep 15 01:22:21 2026 write: IOPS=16, BW=64.8KiB/s (66.4kB/s)(10.0MiB/158009msec); 0 zone resets slat (usec): min=23, max=13911, avg=196.78, stdev=686.77 clat (msec): min=174, max=2005, avg=986.49, stdev=254.28 lat (msec): min=174, max=2006, avg=986.69, stdev=254.28 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1028], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 1921], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 222, per=25.12%, avg=64.81, stdev=26.02, samples=311 iops : min= 1, max= 55, avg=15.77, stdev= 6.49, samples=311 lat (msec) : 250=0.39%, 500=3.05%, 750=8.71%, 1000=43.32%, 2000=44.45% lat (msec) : >=2000=0.08% cpu : usr=0.04%, sys=0.21%, ctx=2043, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146474: Tue Sep 15 01:22:21 2026 write: IOPS=16, BW=64.8KiB/s (66.3kB/s)(10.0MiB/158107msec); 0 zone resets slat (usec): min=23, max=13272, avg=174.69, stdev=622.89 clat (msec): min=292, max=1930, avg=986.81, stdev=250.74 lat (msec): min=292, max=1930, avg=986.98, stdev=250.74 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 651], 10.00th=[ 735], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1401], | 99.00th=[ 1687], 99.50th=[ 1854], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 253, per=25.04%, avg=64.61, stdev=30.17, samples=312 iops : min= 1, max= 63, avg=15.72, stdev= 7.52, samples=312 lat (msec) : 500=3.20%, 750=7.85%, 1000=45.31%, 2000=43.63% cpu : usr=0.04%, sys=0.19%, ctx=2027, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=259KiB/s (265kB/s), 64.7KiB/s-64.8KiB/s (66.3kB/s-66.4kB/s), io=40.0MiB (41.9MB), run=158009-158204msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146507: Tue Sep 15 01:24:28 2026 read: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5048KiB/116374msec) slat (usec): min=2505, max=52930, avg=13764.37, stdev=8153.32 clat (msec): min=8, max=1252, avg=443.01, stdev=131.73 lat (msec): min=14, max=1260, avg=456.78, stdev=133.37 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 266], 10.00th=[ 300], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 535], 90.00th=[ 609], 95.00th=[ 667], | 99.00th=[ 860], 99.50th=[ 911], 99.90th=[ 1217], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 104, per=25.12%, avg=43.21, stdev=21.06, samples=229 iops : min= 1, max= 26, avg=10.33, stdev= 5.30, samples=229 write: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5192KiB/116374msec); 0 zone resets slat (usec): min=28, max=15336, avg=209.45, stdev=690.98 clat (msec): min=351, max=2198, avg=987.60, stdev=300.20 lat (msec): min=351, max=2199, avg=987.81, stdev=300.17 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1385], 95.00th=[ 1586], | 99.00th=[ 1871], 99.50th=[ 2056], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 120, per=24.76%, avg=44.33, stdev=20.73, samples=229 iops : min= 1, max= 30, avg=10.62, stdev= 5.23, samples=229 lat (msec) : 10=0.04%, 50=0.04%, 100=0.20%, 250=1.02%, 500=35.66% lat (msec) : 750=21.60%, 1000=18.52%, 2000=22.62%, >=2000=0.31% cpu : usr=0.07%, sys=0.52%, ctx=4384, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146508: Tue Sep 15 01:24:28 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(4976KiB/116601msec) slat (usec): min=2514, max=62468, avg=13876.25, stdev=8216.91 clat (msec): min=16, max=1134, avg=440.20, stdev=124.33 lat (msec): min=21, max=1147, avg=454.08, stdev=126.00 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 266], 10.00th=[ 300], 20.00th=[ 338], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 456], | 70.00th=[ 489], 80.00th=[ 535], 90.00th=[ 600], 95.00th=[ 659], | 99.00th=[ 785], 99.50th=[ 810], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 111, per=24.80%, avg=42.66, stdev=21.66, samples=226 iops : min= 1, max= 27, avg=10.20, stdev= 5.42, samples=226 write: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5264KiB/116601msec); 0 zone resets slat (usec): min=32, max=22973, avg=221.40, stdev=907.25 clat (msec): min=151, max=2212, avg=986.66, stdev=318.74 lat (msec): min=151, max=2212, avg=986.88, stdev=318.74 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 542], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 810], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1385], 95.00th=[ 1603], | 99.00th=[ 1871], 99.50th=[ 2106], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 128, per=24.79%, avg=44.37, stdev=20.22, samples=231 iops : min= 1, max= 32, avg=10.63, stdev= 5.10, samples=231 lat (msec) : 20=0.04%, 50=0.04%, 100=0.08%, 250=1.72%, 500=35.31% lat (msec) : 750=21.91%, 1000=18.09%, 2000=22.46%, >=2000=0.35% cpu : usr=0.08%, sys=0.49%, ctx=4332, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146509: Tue Sep 15 01:24:28 2026 read: IOPS=10, BW=42.9KiB/s (43.9kB/s)(4984KiB/116300msec) slat (usec): min=33, max=519165, avg=14494.81, stdev=16366.79 clat (msec): min=18, max=1118, avg=444.31, stdev=129.76 lat (msec): min=33, max=1129, avg=458.81, stdev=130.93 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 266], 10.00th=[ 309], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 464], | 70.00th=[ 502], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 802], 99.50th=[ 869], 99.90th=[ 995], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 110, per=25.11%, avg=43.19, stdev=20.41, samples=227 iops : min= 1, max= 27, avg=10.33, stdev= 5.16, samples=227 write: IOPS=11, BW=45.2KiB/s (46.3kB/s)(5256KiB/116300msec); 0 zone resets slat (usec): min=33, max=15780, avg=242.62, stdev=887.96 clat (msec): min=78, max=2223, avg=979.68, stdev=315.10 lat (msec): min=79, max=2223, avg=979.93, stdev=315.14 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 567], 10.00th=[ 651], 20.00th=[ 726], | 30.00th=[ 793], 40.00th=[ 860], 50.00th=[ 936], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1385], 95.00th=[ 1603], | 99.00th=[ 1854], 99.50th=[ 2089], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 120, per=25.21%, avg=45.12, stdev=21.86, samples=227 iops : min= 1, max= 30, avg=10.81, stdev= 5.50, samples=227 lat (msec) : 20=0.04%, 50=0.16%, 100=0.20%, 250=1.52%, 500=33.48% lat (msec) : 750=24.06%, 1000=18.59%, 2000=21.56%, >=2000=0.39% cpu : usr=0.08%, sys=0.54%, ctx=4661, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146510: Tue Sep 15 01:24:28 2026 read: IOPS=10, BW=43.5KiB/s (44.6kB/s)(5060KiB/116298msec) slat (usec): min=2597, max=60102, avg=13799.84, stdev=8121.67 clat (msec): min=27, max=1227, avg=450.88, stdev=131.57 lat (msec): min=41, max=1256, avg=464.69, stdev=133.47 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 464], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 869], 99.50th=[ 919], 99.90th=[ 1133], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 111, per=25.33%, avg=43.57, stdev=19.77, samples=228 iops : min= 1, max= 27, avg=10.42, stdev= 4.95, samples=228 write: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5180KiB/116298msec); 0 zone resets slat (usec): min=33, max=12258, avg=193.34, stdev=597.76 clat (msec): min=308, max=2161, avg=979.85, stdev=303.36 lat (msec): min=308, max=2162, avg=980.04, stdev=303.37 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 927], 60.00th=[ 1011], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1385], 95.00th=[ 1552], | 99.00th=[ 1838], 99.50th=[ 2106], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8, max= 103, per=24.78%, avg=44.36, stdev=19.27, samples=228 iops : min= 2, max= 25, avg=10.62, stdev= 4.87, samples=228 lat (msec) : 50=0.12%, 100=0.08%, 250=0.90%, 500=34.30%, 750=23.55% lat (msec) : 1000=20.16%, 2000=20.51%, >=2000=0.39% cpu : usr=0.07%, sys=0.46%, ctx=4234, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172KiB/s (176kB/s), 42.7KiB/s-43.5KiB/s (43.7kB/s-44.6kB/s), io=19.6MiB (20.5MB), run=116298-116601msec WRITE: bw=179KiB/s (183kB/s), 44.5KiB/s-45.2KiB/s (45.6kB/s-46.3kB/s), io=20.4MiB (21.4MB), run=116298-116601msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146531: Tue Sep 15 01:24:38 2026 read: IOPS=2, BW=87.7MiB/s (91.0MB/s)(40.0MiB/456msec) slat (nsec): min=36695k, max=36695k, avg=36694780.00, stdev= 0.00 clat (nsec): min=396212k, max=396212k, avg=396211925.00, stdev= 0.00 lat (nsec): min=432912k, max=432912k, avg=432911652.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.98%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=87.7MiB/s (91.0MB/s), 87.7MiB/s-87.7MiB/s (91.0MB/s-91.0MB/s), io=40.0MiB (41.9MB), run=456-456msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout