== sanity test 398c: run fio to test AIO ================= 08:41:46 (1787834506) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.758632 s, 55.3 MB/s osc.lustre-OST0000-osc-ffff951cc4fca000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145488: Thu Aug 27 08:44:40 2026 write: IOPS=15, BW=62.5KiB/s (64.0kB/s)(10.0MiB/163810msec); 0 zone resets slat (usec): min=21, max=18469, avg=178.69, stdev=737.99 clat (msec): min=171, max=1984, avg=1023.01, stdev=260.00 lat (msec): min=171, max=1984, avg=1023.19, stdev=259.98 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 676], 10.00th=[ 735], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 919], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1502], | 99.00th=[ 1871], 99.50th=[ 1921], 99.90th=[ 1955], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 128, per=25.09%, avg=62.72, stdev=30.43, samples=321 iops : min= 1, max= 32, avg=15.23, stdev= 7.65, samples=321 lat (msec) : 250=0.12%, 500=0.55%, 750=11.68%, 1000=38.28%, 2000=49.38% cpu : usr=0.04%, sys=0.21%, ctx=1679, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145489: Thu Aug 27 08:44:40 2026 write: IOPS=15, BW=62.6KiB/s (64.1kB/s)(10.0MiB/163682msec); 0 zone resets slat (usec): min=22, max=14818, avg=173.99, stdev=664.62 clat (msec): min=276, max=2095, avg=1021.54, stdev=258.77 lat (msec): min=277, max=2095, avg=1021.72, stdev=258.77 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 667], 10.00th=[ 726], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1821], 99.50th=[ 1871], 99.90th=[ 2022], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=25.81%, avg=64.53, stdev=28.40, samples=312 iops : min= 1, max= 32, avg=15.69, stdev= 7.11, samples=312 lat (msec) : 500=0.62%, 750=12.54%, 1000=38.48%, 2000=48.20%, >=2000=0.16% cpu : usr=0.04%, sys=0.21%, ctx=1583, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145490: Thu Aug 27 08:44:40 2026 write: IOPS=15, BW=62.6KiB/s (64.1kB/s)(10.0MiB/163685msec); 0 zone resets slat (usec): min=22, max=13806, avg=161.39, stdev=564.58 clat (msec): min=226, max=1961, avg=1021.61, stdev=260.66 lat (msec): min=227, max=1962, avg=1021.77, stdev=260.68 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 684], 10.00th=[ 726], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1519], | 99.00th=[ 1821], 99.50th=[ 1905], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 128, per=25.42%, avg=63.54, stdev=30.19, samples=317 iops : min= 1, max= 32, avg=15.44, stdev= 7.61, samples=317 lat (msec) : 250=0.62%, 500=0.08%, 750=13.05%, 1000=37.66%, 2000=48.59% cpu : usr=0.04%, sys=0.19%, ctx=1359, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145491: Thu Aug 27 08:44:40 2026 write: IOPS=15, BW=62.5KiB/s (64.0kB/s)(10.0MiB/163803msec); 0 zone resets slat (usec): min=23, max=23489, avg=170.78, stdev=792.15 clat (msec): min=245, max=2017, avg=1022.24, stdev=262.46 lat (msec): min=245, max=2017, avg=1022.42, stdev=262.49 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 659], 10.00th=[ 726], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1519], | 99.00th=[ 1854], 99.50th=[ 1888], 99.90th=[ 1972], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 128, per=24.87%, avg=62.18, stdev=30.66, samples=324 iops : min= 1, max= 32, avg=15.09, stdev= 7.68, samples=324 lat (msec) : 250=0.04%, 500=0.31%, 750=13.59%, 1000=35.94%, 2000=50.04% lat (msec) : >=2000=0.08% cpu : usr=0.04%, sys=0.21%, ctx=1568, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=250KiB/s (256kB/s), 62.5KiB/s-62.6KiB/s (64.0kB/s-64.1kB/s), io=40.0MiB (41.9MB), run=163682-163810msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145525: Thu Aug 27 08:46:44 2026 read: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5048KiB/113060msec) slat (usec): min=2571, max=66691, avg=14515.68, stdev=8434.38 clat (msec): min=25, max=972, avg=427.21, stdev=123.78 lat (msec): min=36, max=983, avg=441.73, stdev=125.11 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 255], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 523], 90.00th=[ 584], 95.00th=[ 659], | 99.00th=[ 760], 99.50th=[ 776], 99.90th=[ 885], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 112, per=25.02%, avg=44.29, stdev=20.84, samples=223 iops : min= 1, max= 28, avg=10.58, stdev= 5.25, samples=223 write: IOPS=11, BW=45.9KiB/s (47.0kB/s)(5192KiB/113060msec); 0 zone resets slat (usec): min=23, max=30528, avg=287.94, stdev=1440.58 clat (msec): min=267, max=1779, avg=960.08, stdev=251.37 lat (msec): min=267, max=1779, avg=960.38, stdev=251.40 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 600], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1435], | 99.00th=[ 1620], 99.50th=[ 1687], 99.90th=[ 1737], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 119, per=24.61%, avg=45.28, stdev=18.64, samples=224 iops : min= 1, max= 29, avg=10.83, stdev= 4.64, samples=224 lat (msec) : 50=0.12%, 100=0.35%, 250=1.88%, 500=35.78%, 750=21.05% lat (msec) : 1000=19.96%, 2000=20.86% cpu : usr=0.08%, sys=0.61%, ctx=4252, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145526: Thu Aug 27 08:46:44 2026 read: IOPS=10, BW=43.9KiB/s (44.0kB/s)(4976KiB/113298msec) slat (usec): min=3016, max=78179, avg=15065.33, stdev=8637.20 clat (msec): min=17, max=944, avg=435.08, stdev=126.93 lat (msec): min=31, max=968, avg=450.15, stdev=128.01 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 262], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 422], 60.00th=[ 456], | 70.00th=[ 493], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 776], 99.50th=[ 810], 99.90th=[ 894], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 112, per=24.56%, avg=43.47, stdev=20.53, samples=223 iops : min= 1, max= 28, avg=10.38, stdev= 5.18, samples=223 write: IOPS=11, BW=46.5KiB/s (47.6kB/s)(5264KiB/113298msec); 0 zone resets slat (usec): min=27, max=15807, avg=270.05, stdev=914.82 clat (msec): min=98, max=1837, avg=948.45, stdev=260.16 lat (msec): min=99, max=1838, avg=948.72, stdev=260.16 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 558], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1603], 99.50th=[ 1703], 99.90th=[ 1804], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 111, per=24.89%, avg=45.80, stdev=20.12, samples=225 iops : min= 1, max= 27, avg=10.96, stdev= 5.06, samples=225 lat (msec) : 20=0.04%, 50=0.04%, 100=0.27%, 250=1.88%, 500=34.34% lat (msec) : 750=22.58%, 1000=20.55%, 2000=20.31% cpu : usr=0.10%, sys=0.57%, ctx=4125, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145527: Thu Aug 27 08:46:44 2026 read: IOPS=11, BW=44.1KiB/s (45.2kB/s)(4984KiB/113009msec) slat (usec): min=54, max=79582, avg=15038.32, stdev=8808.13 clat (msec): min=11, max=992, avg=439.47, stdev=132.86 lat (msec): min=23, max=1004, avg=454.51, stdev=133.60 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 251], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 464], | 70.00th=[ 498], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 667], | 99.00th=[ 776], 99.50th=[ 810], 99.90th=[ 961], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 135, per=24.79%, avg=43.88, stdev=20.43, samples=223 iops : min= 1, max= 33, avg=10.48, stdev= 5.11, samples=223 write: IOPS=11, BW=46.5KiB/s (47.6kB/s)(5256KiB/113009msec); 0 zone resets slat (usec): min=26, max=12393, avg=246.36, stdev=711.64 clat (msec): min=141, max=1726, avg=943.77, stdev=259.53 lat (msec): min=141, max=1726, avg=944.02, stdev=259.57 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 726], | 30.00th=[ 793], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1284], 95.00th=[ 1401], | 99.00th=[ 1620], 99.50th=[ 1653], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 111, per=24.97%, avg=45.95, stdev=20.31, samples=223 iops : min= 1, max= 27, avg=11.00, stdev= 5.10, samples=223 lat (msec) : 20=0.04%, 50=0.31%, 100=0.27%, 250=2.03%, 500=33.24% lat (msec) : 750=23.67%, 1000=20.16%, 2000=20.27% cpu : usr=0.09%, sys=0.61%, ctx=4363, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145528: Thu Aug 27 08:46:44 2026 read: IOPS=11, BW=44.7KiB/s (45.8kB/s)(5060KiB/113169msec) slat (usec): min=2435, max=62242, avg=14639.33, stdev=8514.88 clat (msec): min=13, max=944, avg=439.73, stdev=129.35 lat (msec): min=20, max=968, avg=454.37, stdev=130.77 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 253], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 456], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 776], 99.50th=[ 802], 99.90th=[ 902], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 119, per=25.36%, avg=44.90, stdev=20.02, samples=221 iops : min= 1, max= 29, avg=10.74, stdev= 5.04, samples=221 write: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5180KiB/113169msec); 0 zone resets slat (usec): min=30, max=22449, avg=252.27, stdev=1036.87 clat (msec): min=208, max=1723, avg=951.24, stdev=256.01 lat (msec): min=208, max=1723, avg=951.50, stdev=256.00 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1418], | 99.00th=[ 1569], 99.50th=[ 1620], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 103, per=24.68%, avg=45.41, stdev=20.40, samples=223 iops : min= 1, max= 25, avg=10.86, stdev= 5.11, samples=223 lat (msec) : 20=0.08%, 50=0.04%, 100=0.27%, 250=2.07%, 500=33.71% lat (msec) : 750=23.05%, 1000=20.90%, 2000=19.88% cpu : usr=0.07%, sys=0.56%, ctx=4454, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177KiB/s (181kB/s), 43.9KiB/s-44.7KiB/s (44.0kB/s-45.8kB/s), io=19.6MiB (20.5MB), run=113009-113298msec WRITE: bw=184KiB/s (189kB/s), 45.8KiB/s-46.5KiB/s (46.9kB/s-47.6kB/s), io=20.4MiB (21.4MB), run=113009-113298msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145549: Thu Aug 27 08:46:54 2026 read: IOPS=1, BW=73.1MiB/s (76.7MB/s)(40.0MiB/547msec) slat (nsec): min=73218k, max=73218k, avg=73217896.00, stdev= 0.00 clat (nsec): min=449580k, max=449580k, avg=449579977.00, stdev= 0.00 lat (nsec): min=522803k, max=522803k, avg=522803060.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] lat (msec) : 500=100.00% cpu : usr=0.18%, sys=3.30%, ctx=44, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=73.1MiB/s (76.7MB/s), 73.1MiB/s-73.1MiB/s (76.7MB/s-76.7MB/s), io=40.0MiB (41.9MB), run=547-547msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout