== sanity test 398c: run fio to test AIO ================= 13:18:19 (1788801499) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.481345 s, 87.1 MB/s osc.lustre-OST0000-osc-ffff90eb88776800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145486: Mon Sep 7 13:21:05 2026 write: IOPS=16, BW=65.9KiB/s (67.5kB/s)(10.0MiB/155351msec); 0 zone resets slat (usec): min=28, max=12276, avg=174.42, stdev=604.23 clat (msec): min=185, max=1987, avg=970.33, stdev=262.01 lat (msec): min=185, max=1987, avg=970.50, stdev=262.03 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 609], 10.00th=[ 684], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1787], 99.50th=[ 1888], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 256, per=25.38%, avg=66.76, stdev=29.01, samples=302 iops : min= 1, max= 64, avg=16.26, stdev= 7.27, samples=302 lat (msec) : 250=0.62%, 500=1.88%, 750=14.96%, 1000=43.12%, 2000=39.41% cpu : usr=0.03%, sys=0.20%, ctx=1493, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145487: Mon Sep 7 13:21:05 2026 write: IOPS=16, BW=65.0KiB/s (67.6kB/s)(10.0MiB/155207msec); 0 zone resets slat (usec): min=29, max=17128, avg=192.47, stdev=800.40 clat (msec): min=172, max=1988, avg=968.94, stdev=262.39 lat (msec): min=173, max=1988, avg=969.14, stdev=262.41 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1687], 99.50th=[ 1854], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 256, per=25.05%, avg=65.88, stdev=28.79, samples=306 iops : min= 1, max= 64, avg=16.05, stdev= 7.25, samples=306 lat (msec) : 250=0.62%, 500=1.48%, 750=16.25%, 1000=41.02%, 2000=40.62% cpu : usr=0.06%, sys=0.19%, ctx=1603, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145488: Mon Sep 7 13:21:05 2026 write: IOPS=16, BW=65.8KiB/s (67.4kB/s)(10.0MiB/155544msec); 0 zone resets slat (usec): min=29, max=20839, avg=208.30, stdev=816.43 clat (msec): min=203, max=1897, avg=970.87, stdev=254.02 lat (msec): min=203, max=1898, avg=971.08, stdev=254.06 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 651], 10.00th=[ 693], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1284], 95.00th=[ 1418], | 99.00th=[ 1687], 99.50th=[ 1737], 99.90th=[ 1888], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 200, per=25.82%, avg=67.90, stdev=31.67, samples=297 iops : min= 1, max= 50, avg=16.56, stdev= 7.92, samples=297 lat (msec) : 250=0.62%, 500=1.88%, 750=14.92%, 1000=41.68%, 2000=40.90% cpu : usr=0.05%, sys=0.22%, ctx=1784, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145489: Mon Sep 7 13:21:05 2026 write: IOPS=16, BW=65.8KiB/s (67.4kB/s)(10.0MiB/155615msec); 0 zone resets slat (usec): min=28, max=16545, avg=216.32, stdev=838.97 clat (msec): min=66, max=1894, avg=971.28, stdev=257.20 lat (msec): min=66, max=1894, avg=971.50, stdev=257.19 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 625], 10.00th=[ 693], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1737], 99.50th=[ 1804], 99.90th=[ 1854], 99.95th=[ 1871], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 144, per=24.93%, avg=65.56, stdev=30.75, samples=307 iops : min= 1, max= 36, avg=15.97, stdev= 7.69, samples=307 lat (msec) : 100=0.08%, 250=0.12%, 500=2.46%, 750=13.95%, 1000=43.87% lat (msec) : 2000=39.53% cpu : usr=0.04%, sys=0.23%, ctx=2046, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=263KiB/s (270kB/s), 65.8KiB/s-65.0KiB/s (67.4kB/s-67.6kB/s), io=40.0MiB (41.9MB), run=155207-155615msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145522: Mon Sep 7 13:23:08 2026 read: IOPS=11, BW=45.2KiB/s (46.3kB/s)(5048KiB/111573msec) slat (usec): min=3183, max=73859, avg=14539.85, stdev=8037.70 clat (msec): min=19, max=1060, avg=437.06, stdev=115.96 lat (msec): min=29, max=1085, avg=451.60, stdev=117.32 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 514], 90.00th=[ 592], 95.00th=[ 651], | 99.00th=[ 776], 99.50th=[ 827], 99.90th=[ 919], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 112, per=25.10%, avg=44.93, stdev=21.80, samples=220 iops : min= 1, max= 28, avg=10.74, stdev= 5.50, samples=220 write: IOPS=11, BW=46.5KiB/s (47.7kB/s)(5192KiB/111573msec); 0 zone resets slat (usec): min=37, max=18961, avg=260.41, stdev=1054.66 clat (msec): min=339, max=1784, avg=932.65, stdev=241.51 lat (msec): min=340, max=1784, avg=932.91, stdev=241.55 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 961], | 70.00th=[ 1045], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1603], 99.50th=[ 1670], 99.90th=[ 1770], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 95, per=24.60%, avg=46.01, stdev=18.27, samples=221 iops : min= 1, max= 23, avg=11.01, stdev= 4.57, samples=221 lat (msec) : 20=0.04%, 100=0.27%, 250=0.86%, 500=37.03%, 750=23.12% lat (msec) : 1000=21.02%, 2000=17.66% cpu : usr=0.10%, sys=0.55%, ctx=4011, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145523: Mon Sep 7 13:23:08 2026 read: IOPS=11, BW=44.7KiB/s (45.7kB/s)(4976KiB/111379msec) slat (usec): min=3110, max=58781, avg=14780.71, stdev=8153.98 clat (msec): min=27, max=1008, avg=431.46, stdev=121.36 lat (msec): min=37, max=1025, avg=446.25, stdev=122.94 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 472], 80.00th=[ 510], 90.00th=[ 575], 95.00th=[ 667], | 99.00th=[ 844], 99.50th=[ 860], 99.90th=[ 995], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 7, max= 104, per=24.58%, avg=44.00, stdev=20.45, samples=221 iops : min= 1, max= 26, avg=10.51, stdev= 5.18, samples=221 write: IOPS=11, BW=47.3KiB/s (48.4kB/s)(5264KiB/111379msec); 0 zone resets slat (usec): min=32, max=16485, avg=252.37, stdev=804.71 clat (msec): min=169, max=1733, avg=929.35, stdev=252.10 lat (msec): min=170, max=1733, avg=929.61, stdev=252.17 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 709], | 30.00th=[ 776], 40.00th=[ 844], 50.00th=[ 902], 60.00th=[ 969], | 70.00th=[ 1045], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1636], 99.50th=[ 1653], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 103, per=24.97%, avg=46.69, stdev=19.63, samples=221 iops : min= 1, max= 25, avg=11.18, stdev= 4.89, samples=221 lat (msec) : 50=0.04%, 100=0.20%, 250=0.94%, 500=37.70%, 750=22.42% lat (msec) : 1000=20.31%, 2000=18.40% cpu : usr=0.10%, sys=0.53%, ctx=4518, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145524: Mon Sep 7 13:23:08 2026 read: IOPS=11, BW=44.7KiB/s (45.8kB/s)(4984KiB/111437msec) slat (usec): min=36, max=389924, avg=14704.45, stdev=13421.06 clat (msec): min=13, max=1048, avg=430.19, stdev=113.82 lat (msec): min=22, max=1061, avg=444.90, stdev=115.01 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 439], | 70.00th=[ 460], 80.00th=[ 506], 90.00th=[ 575], 95.00th=[ 634], | 99.00th=[ 793], 99.50th=[ 852], 99.90th=[ 1011], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 7, max= 119, per=24.97%, avg=44.69, stdev=20.64, samples=219 iops : min= 1, max= 29, avg=10.68, stdev= 5.17, samples=219 write: IOPS=11, BW=47.2KiB/s (48.3kB/s)(5256KiB/111437msec); 0 zone resets slat (usec): min=30, max=19230, avg=235.81, stdev=770.30 clat (msec): min=195, max=1821, avg=934.00, stdev=253.95 lat (msec): min=195, max=1822, avg=934.24, stdev=254.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 969], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1636], 99.50th=[ 1687], 99.90th=[ 1804], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 104, per=24.93%, avg=46.62, stdev=19.92, samples=220 iops : min= 1, max= 26, avg=11.16, stdev= 5.03, samples=220 lat (msec) : 20=0.04%, 50=0.08%, 100=0.31%, 250=0.90%, 500=38.40% lat (msec) : 750=21.29%, 1000=20.27%, 2000=18.71% cpu : usr=0.08%, sys=0.58%, ctx=4370, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145525: Mon Sep 7 13:23:08 2026 read: IOPS=11, BW=45.9KiB/s (46.0kB/s)(5060KiB/110356msec) slat (usec): min=2895, max=76810, avg=14335.05, stdev=8005.04 clat (msec): min=35, max=873, avg=429.63, stdev=106.36 lat (msec): min=43, max=885, avg=443.97, stdev=107.79 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 472], 80.00th=[ 510], 90.00th=[ 567], 95.00th=[ 617], | 99.00th=[ 726], 99.50th=[ 818], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 119, per=25.48%, avg=45.61, stdev=20.43, samples=218 iops : min= 1, max= 29, avg=10.91, stdev= 5.13, samples=218 write: IOPS=11, BW=46.9KiB/s (48.1kB/s)(5180KiB/110356msec); 0 zone resets slat (usec): min=32, max=13238, avg=263.03, stdev=845.42 clat (msec): min=221, max=1925, avg=927.02, stdev=251.71 lat (msec): min=222, max=1925, avg=927.28, stdev=251.70 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 584], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 894], 60.00th=[ 953], | 70.00th=[ 1045], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1687], 99.50th=[ 1687], 99.90th=[ 1770], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 95, per=24.80%, avg=46.37, stdev=19.08, samples=218 iops : min= 1, max= 23, avg=11.10, stdev= 4.77, samples=218 lat (msec) : 50=0.04%, 100=0.08%, 250=1.29%, 500=37.81%, 750=22.93% lat (msec) : 1000=20.55%, 2000=17.30% cpu : usr=0.11%, sys=0.56%, ctx=4361, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=180KiB/s (184kB/s), 44.7KiB/s-45.9KiB/s (45.7kB/s-46.0kB/s), io=19.6MiB (20.5MB), run=110356-111573msec WRITE: bw=187KiB/s (192kB/s), 46.5KiB/s-47.3KiB/s (47.7kB/s-48.4kB/s), io=20.4MiB (21.4MB), run=110356-111573msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145546: Mon Sep 7 13:23:17 2026 read: IOPS=1, BW=63.5MiB/s (66.6MB/s)(40.0MiB/630msec) slat (nsec): min=32955k, max=32955k, avg=32955064.00, stdev= 0.00 clat (nsec): min=583001k, max=583001k, avg=583000500.00, stdev= 0.00 lat (nsec): min=615961k, max=615961k, avg=615960936.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.38%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.5MiB/s (66.6MB/s), 63.5MiB/s-63.5MiB/s (66.6MB/s-66.6MB/s), io=40.0MiB (41.9MB), run=630-630msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout