== sanity test 398c: run fio to test AIO ================= 04:21:24 (1788510084) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.445228 s, 94.2 MB/s osc.lustre-OST0000-osc-ffff972788431000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146069: Fri Sep 4 04:24:04 2026 write: IOPS=16, BW=67.6KiB/s (69.2kB/s)(10.0MiB/151522msec); 0 zone resets slat (usec): min=28, max=13768, avg=229.36, stdev=685.26 clat (msec): min=243, max=1561, avg=945.92, stdev=206.47 lat (msec): min=243, max=1562, avg=946.15, stdev=206.49 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 617], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 852], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 995], | 70.00th=[ 1053], 80.00th=[ 1116], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1401], 99.50th=[ 1469], 99.90th=[ 1552], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 254, per=25.05%, avg=67.62, stdev=31.54, samples=298 iops : min= 1, max= 63, avg=16.47, stdev= 7.90, samples=298 lat (msec) : 250=0.55%, 500=2.15%, 750=11.68%, 1000=46.68%, 2000=38.95% cpu : usr=0.06%, sys=0.22%, ctx=1749, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146070: Fri Sep 4 04:24:04 2026 write: IOPS=16, BW=67.7KiB/s (69.3kB/s)(10.0MiB/151354msec); 0 zone resets slat (usec): min=28, max=22332, avg=235.70, stdev=851.16 clat (msec): min=127, max=1551, avg=944.92, stdev=211.36 lat (msec): min=127, max=1551, avg=945.16, stdev=211.34 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 617], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 953], 60.00th=[ 1003], | 70.00th=[ 1053], 80.00th=[ 1116], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1401], 99.50th=[ 1452], 99.90th=[ 1536], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 198, per=24.87%, avg=67.15, stdev=26.20, samples=300 iops : min= 1, max= 49, avg=16.35, stdev= 6.57, samples=300 lat (msec) : 250=0.70%, 500=2.54%, 750=11.56%, 1000=45.59%, 2000=39.61% cpu : usr=0.04%, sys=0.24%, ctx=1768, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146071: Fri Sep 4 04:24:04 2026 write: IOPS=16, BW=67.7KiB/s (69.3kB/s)(10.0MiB/151233msec); 0 zone resets slat (usec): min=28, max=19677, avg=233.66, stdev=808.08 clat (msec): min=77, max=1569, avg=944.54, stdev=211.79 lat (msec): min=77, max=1569, avg=944.78, stdev=211.74 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 617], 10.00th=[ 701], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1116], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1368], 99.50th=[ 1469], 99.90th=[ 1552], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 15, max= 200, per=24.87%, avg=67.16, stdev=27.45, samples=300 iops : min= 3, max= 50, avg=16.35, stdev= 6.89, samples=300 lat (msec) : 100=0.23%, 250=0.43%, 500=2.66%, 750=11.41%, 1000=45.39% lat (msec) : 2000=39.88% cpu : usr=0.05%, sys=0.23%, ctx=1684, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146072: Fri Sep 4 04:24:04 2026 write: IOPS=16, BW=67.6KiB/s (69.2kB/s)(10.0MiB/151446msec); 0 zone resets slat (usec): min=28, max=17927, avg=235.02, stdev=790.40 clat (msec): min=206, max=1520, avg=945.62, stdev=208.10 lat (msec): min=206, max=1520, avg=945.85, stdev=208.06 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 995], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1385], 99.50th=[ 1418], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1519] bw ( KiB/s): min= 8, max= 192, per=24.87%, avg=67.16, stdev=25.08, samples=300 iops : min= 2, max= 48, avg=16.35, stdev= 6.30, samples=300 lat (msec) : 250=0.62%, 500=2.58%, 750=10.90%, 1000=46.37%, 2000=39.53% cpu : usr=0.05%, sys=0.25%, ctx=1839, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=270KiB/s (277kB/s), 67.6KiB/s-67.7KiB/s (69.2kB/s-69.3kB/s), io=40.0MiB (41.9MB), run=151233-151522msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146103: Fri Sep 4 04:25:53 2026 read: IOPS=12, BW=50.4KiB/s (51.6kB/s)(5048KiB/100177msec) slat (usec): min=3126, max=51574, avg=13533.87, stdev=7375.74 clat (msec): min=12, max=1291, avg=397.64, stdev=112.13 lat (msec): min=16, max=1308, avg=411.18, stdev=113.60 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 253], 10.00th=[ 271], 20.00th=[ 309], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 414], | 70.00th=[ 443], 80.00th=[ 481], 90.00th=[ 531], 95.00th=[ 575], | 99.00th=[ 693], 99.50th=[ 860], 99.90th=[ 1083], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 15, max= 104, per=25.26%, avg=50.02, stdev=20.89, samples=198 iops : min= 3, max= 26, avg=12.04, stdev= 5.28, samples=198 write: IOPS=12, BW=51.8KiB/s (53.1kB/s)(5192KiB/100177msec); 0 zone resets slat (usec): min=36, max=12486, avg=291.25, stdev=735.43 clat (msec): min=240, max=1712, avg=832.36, stdev=245.96 lat (msec): min=240, max=1712, avg=832.65, stdev=245.95 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 456], 10.00th=[ 518], 20.00th=[ 625], | 30.00th=[ 684], 40.00th=[ 760], 50.00th=[ 818], 60.00th=[ 894], | 70.00th=[ 953], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1502], 99.50th=[ 1586], 99.90th=[ 1670], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 119, per=24.98%, avg=51.46, stdev=20.15, samples=198 iops : min= 1, max= 29, avg=12.40, stdev= 5.00, samples=198 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=2.03%, 500=43.67% lat (msec) : 750=22.70%, 1000=19.30%, 2000=12.03% cpu : usr=0.10%, sys=0.63%, ctx=4840, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146104: Fri Sep 4 04:25:53 2026 read: IOPS=12, BW=49.3KiB/s (50.5kB/s)(4976KiB/100996msec) slat (msec): min=2, max=373, avg=13.97, stdev=16.89 clat (msec): min=35, max=1254, avg=401.26, stdev=108.49 lat (msec): min=49, max=1261, avg=415.23, stdev=110.30 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 257], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 363], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 584], | 99.00th=[ 651], 99.50th=[ 676], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 120, per=24.50%, avg=48.50, stdev=22.26, samples=201 iops : min= 1, max= 30, avg=11.66, stdev= 5.56, samples=201 write: IOPS=13, BW=52.1KiB/s (53.4kB/s)(5264KiB/100996msec); 0 zone resets slat (usec): min=28, max=19131, avg=315.34, stdev=1031.47 clat (msec): min=248, max=1659, avg=833.47, stdev=249.95 lat (msec): min=248, max=1659, avg=833.79, stdev=249.94 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 443], 10.00th=[ 518], 20.00th=[ 625], | 30.00th=[ 684], 40.00th=[ 751], 50.00th=[ 827], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1452], 99.50th=[ 1586], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 15, max= 127, per=25.36%, avg=52.25, stdev=22.52, samples=198 iops : min= 3, max= 31, avg=12.60, stdev= 5.64, samples=198 lat (msec) : 50=0.23%, 100=0.16%, 250=1.64%, 500=42.85%, 750=23.91% lat (msec) : 1000=18.20%, 2000=13.01% cpu : usr=0.09%, sys=0.64%, ctx=4600, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146105: Fri Sep 4 04:25:53 2026 read: IOPS=12, BW=49.4KiB/s (50.6kB/s)(4984KiB/100902msec) slat (usec): min=2784, max=62754, avg=13366.30, stdev=7175.08 clat (msec): min=15, max=755, avg=392.42, stdev=103.93 lat (msec): min=24, max=792, avg=405.79, stdev=105.21 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 247], 10.00th=[ 271], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 439], 80.00th=[ 477], 90.00th=[ 531], 95.00th=[ 575], | 99.00th=[ 651], 99.50th=[ 667], 99.90th=[ 743], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 7, max= 183, per=25.01%, avg=49.52, stdev=25.14, samples=198 iops : min= 1, max= 45, avg=11.91, stdev= 6.28, samples=198 write: IOPS=13, BW=52.1KiB/s (53.3kB/s)(5256KiB/100902msec); 0 zone resets slat (usec): min=29, max=10767, avg=314.28, stdev=848.37 clat (msec): min=241, max=1671, avg=841.30, stdev=245.96 lat (msec): min=241, max=1671, avg=841.62, stdev=245.97 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 460], 10.00th=[ 527], 20.00th=[ 625], | 30.00th=[ 693], 40.00th=[ 760], 50.00th=[ 835], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1045], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1469], 99.50th=[ 1586], 99.90th=[ 1603], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 127, per=24.93%, avg=51.36, stdev=21.46, samples=200 iops : min= 1, max= 31, avg=12.37, stdev= 5.38, samples=200 lat (msec) : 20=0.04%, 50=0.16%, 100=0.35%, 250=2.38%, 500=42.30% lat (msec) : 750=23.32%, 1000=17.62%, 2000=13.83% cpu : usr=0.10%, sys=0.61%, ctx=4212, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146106: Fri Sep 4 04:25:53 2026 read: IOPS=12, BW=50.4KiB/s (51.7kB/s)(5060KiB/100316msec) slat (usec): min=52, max=390902, avg=13610.51, stdev=15405.59 clat (msec): min=11, max=1038, avg=395.55, stdev=107.67 lat (msec): min=11, max=1052, avg=409.17, stdev=108.68 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 251], 10.00th=[ 275], 20.00th=[ 309], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 414], | 70.00th=[ 443], 80.00th=[ 481], 90.00th=[ 535], 95.00th=[ 575], | 99.00th=[ 684], 99.50th=[ 726], 99.90th=[ 827], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 112, per=25.38%, avg=50.25, stdev=22.57, samples=198 iops : min= 1, max= 28, avg=12.10, stdev= 5.68, samples=198 write: IOPS=12, BW=51.6KiB/s (52.9kB/s)(5180KiB/100316msec); 0 zone resets slat (usec): min=34, max=13040, avg=290.05, stdev=749.05 clat (msec): min=120, max=1667, avg=838.59, stdev=249.63 lat (msec): min=120, max=1668, avg=838.89, stdev=249.62 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 472], 10.00th=[ 535], 20.00th=[ 617], | 30.00th=[ 693], 40.00th=[ 768], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1267], | 99.00th=[ 1519], 99.50th=[ 1569], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 127, per=24.97%, avg=51.44, stdev=22.24, samples=197 iops : min= 1, max= 31, avg=12.39, stdev= 5.54, samples=197 lat (msec) : 20=0.04%, 50=0.23%, 100=0.23%, 250=2.23%, 500=42.42% lat (msec) : 750=23.05%, 1000=19.10%, 2000=12.70% cpu : usr=0.10%, sys=0.63%, ctx=4394, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=199KiB/s (203kB/s), 49.3KiB/s-50.4KiB/s (50.5kB/s-51.7kB/s), io=19.6MiB (20.5MB), run=100177-100996msec WRITE: bw=207KiB/s (212kB/s), 51.6KiB/s-52.1KiB/s (52.9kB/s-53.4kB/s), io=20.4MiB (21.4MB), run=100177-100996msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146124: Fri Sep 4 04:26:01 2026 read: IOPS=1, BW=67.5MiB/s (70.7MB/s)(40.0MiB/593msec) slat (nsec): min=68995k, max=68995k, avg=68995086.00, stdev= 0.00 clat (nsec): min=510918k, max=510918k, avg=510918178.00, stdev= 0.00 lat (nsec): min=579918k, max=579918k, avg=579917688.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.03%, ctx=24, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.5MiB/s (70.7MB/s), 67.5MiB/s-67.5MiB/s (70.7MB/s-70.7MB/s), io=40.0MiB (41.9MB), run=593-593msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout