== sanity test 398c: run fio to test AIO ================= 03:33:21 (1786952001) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.719105 s, 58.3 MB/s osc.lustre-OST0000-osc-ffffa0b62ed55800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144727: Mon Aug 17 03:36:22 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168160msec); 0 zone resets slat (usec): min=40, max=25036, avg=262.15, stdev=944.68 clat (msec): min=387, max=2471, avg=1049.92, stdev=319.55 lat (msec): min=387, max=2471, avg=1050.19, stdev=319.60 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1150], 80.00th=[ 1301], 90.00th=[ 1485], 95.00th=[ 1653], | 99.00th=[ 1921], 99.50th=[ 1989], 99.90th=[ 2400], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 191, per=25.24%, avg=61.33, stdev=31.51, samples=328 iops : min= 1, max= 47, avg=14.88, stdev= 7.91, samples=328 lat (msec) : 500=3.52%, 750=9.92%, 1000=37.42%, 2000=48.67%, >=2000=0.47% cpu : usr=0.04%, sys=0.28%, ctx=2025, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144728: Mon Aug 17 03:36:22 2026 write: IOPS=15, BW=60.0KiB/s (62.5kB/s)(10.0MiB/167881msec); 0 zone resets slat (usec): min=38, max=16218, avg=225.56, stdev=728.15 clat (msec): min=122, max=3060, avg=1048.43, stdev=333.48 lat (msec): min=122, max=3061, avg=1048.65, stdev=333.52 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1062], | 70.00th=[ 1150], 80.00th=[ 1301], 90.00th=[ 1485], 95.00th=[ 1670], | 99.00th=[ 1989], 99.50th=[ 2123], 99.90th=[ 3004], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 176, per=26.07%, avg=63.34, stdev=38.74, samples=317 iops : min= 1, max= 44, avg=15.38, stdev= 9.72, samples=317 lat (msec) : 250=0.62%, 500=3.12%, 750=9.88%, 1000=38.24%, 2000=47.27% lat (msec) : >=2000=0.86% cpu : usr=0.04%, sys=0.24%, ctx=1796, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144729: Mon Aug 17 03:36:22 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168161msec); 0 zone resets slat (usec): min=37, max=16203, avg=245.17, stdev=724.51 clat (msec): min=155, max=2435, avg=1050.08, stdev=327.54 lat (msec): min=156, max=2435, avg=1050.33, stdev=327.56 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 617], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1485], 95.00th=[ 1687], | 99.00th=[ 1989], 99.50th=[ 2165], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 176, per=25.11%, avg=61.02, stdev=31.32, samples=329 iops : min= 1, max= 44, avg=14.81, stdev= 7.88, samples=329 lat (msec) : 250=0.62%, 500=2.97%, 750=10.00%, 1000=37.62%, 2000=47.85% lat (msec) : >=2000=0.94% cpu : usr=0.04%, sys=0.25%, ctx=1840, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144730: Mon Aug 17 03:36:22 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168067msec); 0 zone resets slat (usec): min=39, max=18068, avg=267.79, stdev=864.82 clat (msec): min=117, max=2327, avg=1049.37, stdev=322.58 lat (msec): min=117, max=2327, avg=1049.64, stdev=322.61 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 625], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1703], | 99.00th=[ 1989], 99.50th=[ 2056], 99.90th=[ 2232], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 199, per=25.09%, avg=60.96, stdev=31.51, samples=330 iops : min= 1, max= 49, avg=14.80, stdev= 7.93, samples=330 lat (msec) : 250=0.12%, 500=3.32%, 750=9.61%, 1000=38.01%, 2000=48.12% lat (msec) : >=2000=0.82% cpu : usr=0.04%, sys=0.26%, ctx=1959, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=244KiB/s (249kB/s), 60.9KiB/s-60.0KiB/s (62.4kB/s-62.5kB/s), io=40.0MiB (41.9MB), run=167881-168161msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144766: Mon Aug 17 03:38:22 2026 read: IOPS=11, BW=46.4KiB/s (47.5kB/s)(5048KiB/108785msec) slat (usec): min=2585, max=90009, avg=15025.88, stdev=8351.75 clat (msec): min=24, max=881, avg=427.92, stdev=126.47 lat (msec): min=41, max=915, avg=442.96, stdev=127.50 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 405], 60.00th=[ 439], | 70.00th=[ 468], 80.00th=[ 514], 90.00th=[ 600], 95.00th=[ 676], | 99.00th=[ 810], 99.50th=[ 844], 99.90th=[ 877], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 128, per=25.09%, avg=46.16, stdev=21.61, samples=214 iops : min= 1, max= 32, avg=11.05, stdev= 5.48, samples=214 write: IOPS=11, BW=47.7KiB/s (48.9kB/s)(5192KiB/108785msec); 0 zone resets slat (usec): min=41, max=36202, avg=413.58, stdev=1565.26 clat (msec): min=246, max=2015, avg=907.65, stdev=253.67 lat (msec): min=246, max=2016, avg=908.07, stdev=253.67 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 944], | 70.00th=[ 1011], 80.00th=[ 1116], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1636], 99.50th=[ 1871], 99.90th=[ 1972], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 112, per=24.45%, avg=46.94, stdev=20.60, samples=216 iops : min= 1, max= 28, avg=11.25, stdev= 5.18, samples=216 lat (msec) : 50=0.12%, 100=0.12%, 250=1.68%, 500=38.52%, 750=21.33% lat (msec) : 1000=22.11%, 2000=16.09%, >=2000=0.04% cpu : usr=0.09%, sys=0.69%, ctx=5018, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144767: Mon Aug 17 03:38:22 2026 read: IOPS=11, BW=45.8KiB/s (46.9kB/s)(4976KiB/108716msec) slat (usec): min=230, max=64872, avg=15274.59, stdev=8688.09 clat (msec): min=11, max=950, avg=432.84, stdev=134.66 lat (msec): min=24, max=979, avg=448.12, stdev=136.19 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 262], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 472], 80.00th=[ 518], 90.00th=[ 617], 95.00th=[ 709], | 99.00th=[ 844], 99.50th=[ 869], 99.90th=[ 944], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 104, per=24.65%, avg=45.36, stdev=20.91, samples=214 iops : min= 1, max= 26, avg=10.86, stdev= 5.29, samples=214 write: IOPS=12, BW=48.4KiB/s (49.6kB/s)(5264KiB/108716msec); 0 zone resets slat (usec): min=55, max=15482, avg=344.58, stdev=1014.90 clat (msec): min=96, max=1921, avg=897.00, stdev=259.39 lat (msec): min=96, max=1922, avg=897.34, stdev=259.38 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 498], 10.00th=[ 609], 20.00th=[ 693], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1586], 99.50th=[ 1770], 99.90th=[ 1905], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 121, per=25.19%, avg=48.36, stdev=20.64, samples=213 iops : min= 1, max= 30, avg=11.61, stdev= 5.19, samples=213 lat (msec) : 20=0.04%, 50=0.23%, 100=0.12%, 250=2.11%, 500=37.07% lat (msec) : 750=21.60%, 1000=22.62%, 2000=16.21% cpu : usr=0.08%, sys=0.67%, ctx=4586, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144768: Mon Aug 17 03:38:22 2026 read: IOPS=11, BW=45.8KiB/s (46.9kB/s)(4984KiB/108784msec) slat (usec): min=3163, max=90636, avg=15113.79, stdev=8769.34 clat (msec): min=16, max=973, avg=430.70, stdev=126.74 lat (msec): min=28, max=985, avg=445.82, stdev=127.81 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 271], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 468], 80.00th=[ 514], 90.00th=[ 592], 95.00th=[ 676], | 99.00th=[ 827], 99.50th=[ 860], 99.90th=[ 961], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 103, per=24.59%, avg=45.25, stdev=21.11, samples=216 iops : min= 1, max= 25, avg=10.83, stdev= 5.30, samples=216 write: IOPS=12, BW=48.3KiB/s (49.5kB/s)(5256KiB/108784msec); 0 zone resets slat (usec): min=54, max=23835, avg=312.66, stdev=966.16 clat (msec): min=195, max=1882, avg=899.52, stdev=250.37 lat (msec): min=195, max=1883, avg=899.84, stdev=250.40 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 518], 10.00th=[ 600], 20.00th=[ 693], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 1011], 80.00th=[ 1116], 90.00th=[ 1250], 95.00th=[ 1334], | 99.00th=[ 1485], 99.50th=[ 1552], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 104, per=24.82%, avg=47.66, stdev=19.69, samples=215 iops : min= 1, max= 26, avg=11.43, stdev= 4.95, samples=215 lat (msec) : 20=0.04%, 50=0.12%, 100=0.04%, 250=1.52%, 500=38.48% lat (msec) : 750=21.41%, 1000=22.19%, 2000=16.21% cpu : usr=0.09%, sys=0.62%, ctx=4530, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144769: Mon Aug 17 03:38:22 2026 read: IOPS=11, BW=46.8KiB/s (47.0kB/s)(5060KiB/108005msec) slat (usec): min=2803, max=58206, avg=15032.65, stdev=8356.92 clat (msec): min=15, max=954, avg=422.92, stdev=133.57 lat (msec): min=21, max=978, avg=437.97, stdev=134.81 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 249], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 351], 40.00th=[ 384], 50.00th=[ 405], 60.00th=[ 430], | 70.00th=[ 464], 80.00th=[ 506], 90.00th=[ 600], 95.00th=[ 684], | 99.00th=[ 844], 99.50th=[ 860], 99.90th=[ 902], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 127, per=25.44%, avg=46.80, stdev=21.80, samples=212 iops : min= 1, max= 31, avg=11.22, stdev= 5.44, samples=212 write: IOPS=11, BW=47.0KiB/s (49.1kB/s)(5180KiB/108005msec); 0 zone resets slat (usec): min=56, max=11714, avg=289.26, stdev=697.88 clat (msec): min=185, max=2019, avg=904.66, stdev=254.67 lat (msec): min=185, max=2019, avg=904.95, stdev=254.68 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 510], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 944], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1552], 99.50th=[ 1770], 99.90th=[ 1989], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 104, per=24.59%, avg=47.21, stdev=19.55, samples=214 iops : min= 1, max= 26, avg=11.32, stdev= 4.94, samples=214 lat (msec) : 20=0.08%, 50=0.04%, 100=0.27%, 250=2.50%, 500=38.44% lat (msec) : 750=19.77%, 1000=23.12%, 2000=15.74%, >=2000=0.04% cpu : usr=0.10%, sys=0.67%, ctx=4537, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=184KiB/s (189kB/s), 45.8KiB/s-46.8KiB/s (46.9kB/s-47.0kB/s), io=19.6MiB (20.5MB), run=108005-108785msec WRITE: bw=192KiB/s (197kB/s), 47.7KiB/s-48.4KiB/s (48.9kB/s-49.6kB/s), io=20.4MiB (21.4MB), run=108005-108785msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144788: Mon Aug 17 03:38:33 2026 read: IOPS=1, BW=52.7MiB/s (55.3MB/s)(40.0MiB/759msec) slat (nsec): min=52959k, max=52959k, avg=52958937.00, stdev= 0.00 clat (nsec): min=662683k, max=662683k, avg=662682835.00, stdev= 0.00 lat (nsec): min=715647k, max=715647k, avg=715647254.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 659], 5.00th=[ 659], 10.00th=[ 659], 20.00th=[ 659], | 30.00th=[ 659], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.19%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=52.7MiB/s (55.3MB/s), 52.7MiB/s-52.7MiB/s (55.3MB/s-55.3MB/s), io=40.0MiB (41.9MB), run=759-759msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout