== sanity test 398c: run fio to test AIO ================= 06:41:50 (1789382510) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.490573 s, 85.5 MB/s osc.lustre-OST0000-osc-ffff8a36787b5800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145826: Mon Sep 14 06:44:36 2026 write: IOPS=16, BW=65.0KiB/s (66.6kB/s)(10.0MiB/157498msec); 0 zone resets slat (usec): min=26, max=11623, avg=169.53, stdev=539.13 clat (msec): min=217, max=2111, avg=983.24, stdev=264.71 lat (msec): min=220, max=2111, avg=983.41, stdev=264.71 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 584], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1653], 99.50th=[ 1770], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 246, per=24.93%, avg=64.57, stdev=29.44, samples=312 iops : min= 1, max= 61, avg=15.69, stdev= 7.41, samples=312 lat (msec) : 250=1.29%, 500=2.30%, 750=11.95%, 1000=39.14%, 2000=45.12% lat (msec) : >=2000=0.20% cpu : usr=0.04%, sys=0.19%, ctx=1874, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145827: Mon Sep 14 06:44:36 2026 write: IOPS=16, BW=65.0KiB/s (66.6kB/s)(10.0MiB/157496msec); 0 zone resets slat (usec): min=25, max=12573, avg=213.85, stdev=775.88 clat (msec): min=134, max=1904, avg=983.14, stdev=262.07 lat (msec): min=135, max=1904, avg=983.35, stdev=262.08 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1586], 99.50th=[ 1670], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 254, per=25.59%, avg=66.29, stdev=32.06, samples=304 iops : min= 1, max= 63, avg=16.12, stdev= 8.05, samples=304 lat (msec) : 250=1.41%, 500=2.07%, 750=12.77%, 1000=36.80%, 2000=46.95% cpu : usr=0.04%, sys=0.20%, ctx=1844, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145828: Mon Sep 14 06:44:36 2026 write: IOPS=16, BW=65.1KiB/s (66.7kB/s)(10.0MiB/157267msec); 0 zone resets slat (usec): min=25, max=17289, avg=181.19, stdev=708.42 clat (msec): min=153, max=2112, avg=982.40, stdev=271.68 lat (msec): min=153, max=2112, avg=982.58, stdev=271.69 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 567], 10.00th=[ 676], 20.00th=[ 776], | 30.00th=[ 852], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1720], 99.50th=[ 1905], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 198, per=25.00%, avg=64.76, stdev=29.15, samples=311 iops : min= 1, max= 49, avg=15.74, stdev= 7.35, samples=311 lat (msec) : 250=0.98%, 500=2.62%, 750=13.09%, 1000=37.89%, 2000=45.27% lat (msec) : >=2000=0.16% cpu : usr=0.03%, sys=0.21%, ctx=1775, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145829: Mon Sep 14 06:44:36 2026 write: IOPS=16, BW=64.0KiB/s (66.6kB/s)(10.0MiB/157556msec); 0 zone resets slat (usec): min=22, max=16394, avg=198.94, stdev=779.75 clat (msec): min=177, max=2031, avg=983.45, stdev=265.58 lat (msec): min=178, max=2031, avg=983.65, stdev=265.59 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 592], 10.00th=[ 684], 20.00th=[ 776], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1636], 99.50th=[ 1770], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 254, per=24.85%, avg=64.36, stdev=32.80, samples=313 iops : min= 1, max= 63, avg=15.64, stdev= 8.22, samples=313 lat (msec) : 250=0.90%, 500=2.62%, 750=13.59%, 1000=38.40%, 2000=44.45% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.22%, ctx=1872, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=260KiB/s (266kB/s), 64.0KiB/s-65.1KiB/s (66.6kB/s-66.7kB/s), io=40.0MiB (41.9MB), run=157267-157556msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145861: Mon Sep 14 06:46:25 2026 read: IOPS=12, BW=48.9KiB/s (50.1kB/s)(5048KiB/103196msec) slat (usec): min=2983, max=54124, avg=13459.88, stdev=7013.54 clat (msec): min=22, max=1028, avg=400.86, stdev=95.85 lat (msec): min=34, max=1060, avg=414.32, stdev=96.81 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 271], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 468], 90.00th=[ 514], 95.00th=[ 575], | 99.00th=[ 676], 99.50th=[ 726], 99.90th=[ 944], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 128, per=24.86%, avg=48.23, stdev=22.11, samples=205 iops : min= 1, max= 32, avg=11.59, stdev= 5.60, samples=205 write: IOPS=12, BW=50.3KiB/s (51.5kB/s)(5192KiB/103196msec); 0 zone resets slat (usec): min=29, max=13479, avg=213.01, stdev=760.85 clat (msec): min=188, max=1437, avg=865.69, stdev=207.84 lat (msec): min=188, max=1437, avg=865.90, stdev=207.84 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 514], 10.00th=[ 609], 20.00th=[ 693], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1045], 90.00th=[ 1150], 95.00th=[ 1217], | 99.00th=[ 1334], 99.50th=[ 1385], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 6, max= 119, per=24.72%, avg=49.93, stdev=20.50, samples=204 iops : min= 1, max= 29, avg=12.01, stdev= 5.11, samples=204 lat (msec) : 50=0.12%, 100=0.20%, 250=0.94%, 500=44.06%, 750=18.71% lat (msec) : 1000=22.89%, 2000=13.09% cpu : usr=0.09%, sys=0.58%, ctx=3940, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145862: Mon Sep 14 06:46:25 2026 read: IOPS=12, BW=48.2KiB/s (49.3kB/s)(4976KiB/103318msec) slat (usec): min=2588, max=59683, avg=13442.80, stdev=7070.74 clat (msec): min=28, max=739, avg=400.71, stdev=91.07 lat (msec): min=32, max=764, avg=414.16, stdev=92.14 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 355], 40.00th=[ 376], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 464], 90.00th=[ 514], 95.00th=[ 567], | 99.00th=[ 667], 99.50th=[ 718], 99.90th=[ 735], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 7, max= 128, per=24.77%, avg=48.05, stdev=23.03, samples=203 iops : min= 1, max= 32, avg=11.55, stdev= 5.79, samples=203 write: IOPS=12, BW=50.9KiB/s (52.2kB/s)(5264KiB/103318msec); 0 zone resets slat (usec): min=30, max=13463, avg=223.24, stdev=753.31 clat (msec): min=292, max=1406, avg=862.14, stdev=206.89 lat (msec): min=292, max=1406, avg=862.36, stdev=206.88 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 510], 10.00th=[ 600], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1217], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 112, per=24.90%, avg=50.29, stdev=20.47, samples=205 iops : min= 1, max= 28, avg=12.10, stdev= 5.12, samples=205 lat (msec) : 50=0.08%, 100=0.35%, 250=0.94%, 500=43.95%, 750=17.89% lat (msec) : 1000=23.91%, 2000=12.89% cpu : usr=0.09%, sys=0.59%, ctx=4408, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145863: Mon Sep 14 06:46:25 2026 read: IOPS=12, BW=48.3KiB/s (49.4kB/s)(4984KiB/103263msec) slat (usec): min=39, max=54416, avg=13447.33, stdev=7397.19 clat (msec): min=17, max=793, avg=400.90, stdev=96.71 lat (msec): min=21, max=809, avg=414.36, stdev=97.67 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 376], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 468], 90.00th=[ 514], 95.00th=[ 567], | 99.00th=[ 701], 99.50th=[ 718], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 7, max= 118, per=24.60%, avg=47.73, stdev=22.86, samples=205 iops : min= 1, max= 29, avg=11.46, stdev= 5.73, samples=205 write: IOPS=12, BW=50.9KiB/s (52.1kB/s)(5256KiB/103263msec); 0 zone resets slat (usec): min=28, max=24226, avg=232.50, stdev=1021.05 clat (msec): min=190, max=1407, avg=863.58, stdev=212.70 lat (msec): min=190, max=1407, avg=863.81, stdev=212.69 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 498], 10.00th=[ 600], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 986], 80.00th=[ 1045], 90.00th=[ 1133], 95.00th=[ 1217], | 99.00th=[ 1351], 99.50th=[ 1385], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 127, per=24.79%, avg=50.07, stdev=21.27, samples=205 iops : min= 1, max= 31, avg=12.05, stdev= 5.35, samples=205 lat (msec) : 20=0.04%, 50=0.23%, 100=0.31%, 250=1.05%, 500=43.55% lat (msec) : 750=18.20%, 1000=22.73%, 2000=13.87% cpu : usr=0.09%, sys=0.58%, ctx=3891, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145864: Mon Sep 14 06:46:25 2026 read: IOPS=12, BW=49.1KiB/s (50.3kB/s)(5060KiB/103082msec) slat (usec): min=2419, max=47810, avg=13465.95, stdev=7273.96 clat (msec): min=18, max=1240, avg=403.30, stdev=98.44 lat (msec): min=27, max=1258, avg=416.78, stdev=99.52 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 279], 10.00th=[ 300], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 472], 90.00th=[ 531], 95.00th=[ 575], | 99.00th=[ 667], 99.50th=[ 709], 99.90th=[ 1028], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 112, per=25.25%, avg=48.99, stdev=19.80, samples=203 iops : min= 1, max= 28, avg=11.78, stdev= 4.97, samples=203 write: IOPS=12, BW=50.3KiB/s (51.5kB/s)(5180KiB/103082msec); 0 zone resets slat (usec): min=31, max=15910, avg=222.03, stdev=756.65 clat (msec): min=190, max=1724, avg=863.44, stdev=213.73 lat (msec): min=190, max=1724, avg=863.67, stdev=213.73 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 514], 10.00th=[ 609], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 852], 60.00th=[ 894], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1133], 95.00th=[ 1217], | 99.00th=[ 1368], 99.50th=[ 1670], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 8, max= 103, per=24.69%, avg=49.88, stdev=19.45, samples=203 iops : min= 2, max= 25, avg=12.00, stdev= 4.91, samples=203 lat (msec) : 20=0.04%, 100=0.20%, 250=1.21%, 500=43.28%, 750=19.06% lat (msec) : 1000=23.44%, 2000=12.77% cpu : usr=0.08%, sys=0.57%, ctx=4179, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=194KiB/s (199kB/s), 48.2KiB/s-49.1KiB/s (49.3kB/s-50.3kB/s), io=19.6MiB (20.5MB), run=103082-103318msec WRITE: bw=202KiB/s (207kB/s), 50.3KiB/s-50.9KiB/s (51.5kB/s-52.2kB/s), io=20.4MiB (21.4MB), run=103082-103318msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145883: Mon Sep 14 06:46:33 2026 read: IOPS=2, BW=88.3MiB/s (92.6MB/s)(40.0MiB/453msec) slat (nsec): min=30474k, max=30474k, avg=30474262.00, stdev= 0.00 clat (nsec): min=406474k, max=406474k, avg=406474472.00, stdev= 0.00 lat (nsec): min=436953k, max=436953k, avg=436953124.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.88%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=88.3MiB/s (92.6MB/s), 88.3MiB/s-88.3MiB/s (92.6MB/s-92.6MB/s), io=40.0MiB (41.9MB), run=453-453msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout