== sanity test 398c: run fio to test AIO ================= 22:34:30 (1788834870) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.502869 s, 83.4 MB/s osc.lustre-OST0000-osc-ffff8b5c84c2f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146316: Mon Sep 7 22:37:24 2026 write: IOPS=15, BW=62.3KiB/s (63.8kB/s)(10.0MiB/164389msec); 0 zone resets slat (usec): min=24, max=15878, avg=162.68, stdev=671.88 clat (msec): min=361, max=2082, avg=1026.20, stdev=258.27 lat (msec): min=361, max=2082, avg=1026.37, stdev=258.29 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 634], 10.00th=[ 743], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1754], 99.50th=[ 1838], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 255, per=25.29%, avg=62.97, stdev=30.62, samples=320 iops : min= 1, max= 63, avg=15.29, stdev= 7.69, samples=320 lat (msec) : 500=3.12%, 750=7.19%, 1000=38.55%, 2000=50.98%, >=2000=0.16% cpu : usr=0.04%, sys=0.17%, ctx=1281, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146317: Mon Sep 7 22:37:24 2026 write: IOPS=15, BW=62.4KiB/s (63.9kB/s)(10.0MiB/164114msec); 0 zone resets slat (usec): min=21, max=12833, avg=202.61, stdev=803.19 clat (msec): min=200, max=2145, avg=1025.14, stdev=271.16 lat (msec): min=200, max=2145, avg=1025.34, stdev=271.20 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 617], 10.00th=[ 743], 20.00th=[ 835], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1502], | 99.00th=[ 1854], 99.50th=[ 1972], 99.90th=[ 2140], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 255, per=25.45%, avg=63.37, stdev=38.99, samples=318 iops : min= 1, max= 63, avg=15.40, stdev= 9.80, samples=318 lat (msec) : 250=0.62%, 500=2.07%, 750=8.05%, 1000=38.09%, 2000=50.94% lat (msec) : >=2000=0.23% cpu : usr=0.04%, sys=0.19%, ctx=1600, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146318: Mon Sep 7 22:37:24 2026 write: IOPS=15, BW=62.4KiB/s (63.9kB/s)(10.0MiB/163998msec); 0 zone resets slat (usec): min=22, max=15791, avg=194.71, stdev=782.35 clat (msec): min=188, max=2102, avg=1024.04, stdev=279.19 lat (msec): min=188, max=2102, avg=1024.24, stdev=279.25 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 634], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1519], | 99.00th=[ 1921], 99.50th=[ 1989], 99.90th=[ 2089], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 255, per=26.64%, avg=66.32, stdev=36.08, samples=304 iops : min= 1, max= 63, avg=16.15, stdev= 9.05, samples=304 lat (msec) : 250=0.70%, 500=2.73%, 750=8.01%, 1000=38.16%, 2000=49.92% lat (msec) : >=2000=0.47% cpu : usr=0.04%, sys=0.19%, ctx=1512, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146319: Mon Sep 7 22:37:24 2026 write: IOPS=15, BW=62.3KiB/s (63.8kB/s)(10.0MiB/164345msec); 0 zone resets slat (usec): min=22, max=22762, avg=182.33, stdev=716.13 clat (msec): min=184, max=2084, avg=1026.05, stdev=268.19 lat (msec): min=185, max=2084, avg=1026.23, stdev=268.25 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 642], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1804], 99.50th=[ 1921], 99.90th=[ 1955], 99.95th=[ 2005], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 183, per=24.89%, avg=61.98, stdev=32.25, samples=325 iops : min= 1, max= 45, avg=15.06, stdev= 8.07, samples=325 lat (msec) : 250=0.04%, 500=3.16%, 750=8.52%, 1000=36.88%, 2000=51.33% lat (msec) : >=2000=0.08% cpu : usr=0.05%, sys=0.18%, ctx=1685, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=249KiB/s (255kB/s), 62.3KiB/s-62.4KiB/s (63.8kB/s-63.9kB/s), io=40.0MiB (41.9MB), run=163998-164389msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146352: Mon Sep 7 22:39:26 2026 read: IOPS=11, BW=44.6KiB/s (45.6kB/s)(5048KiB/113257msec) slat (usec): min=2817, max=63192, avg=14667.62, stdev=8288.45 clat (msec): min=11, max=916, avg=439.05, stdev=125.01 lat (msec): min=17, max=934, avg=453.72, stdev=126.72 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 451], | 70.00th=[ 485], 80.00th=[ 527], 90.00th=[ 609], 95.00th=[ 684], | 99.00th=[ 776], 99.50th=[ 827], 99.90th=[ 885], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 7, max= 119, per=24.97%, avg=43.96, stdev=20.65, samples=225 iops : min= 1, max= 29, avg=10.48, stdev= 5.20, samples=225 write: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5192KiB/113257msec); 0 zone resets slat (usec): min=32, max=27602, avg=223.38, stdev=958.28 clat (msec): min=311, max=2126, avg=952.76, stdev=277.27 lat (msec): min=311, max=2126, avg=952.99, stdev=277.25 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 969], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1418], | 99.00th=[ 1905], 99.50th=[ 2056], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 95, per=24.73%, avg=45.26, stdev=19.22, samples=224 iops : min= 1, max= 23, avg=10.81, stdev= 4.83, samples=224 lat (msec) : 20=0.12%, 50=0.12%, 100=0.27%, 250=0.98%, 500=36.52% lat (msec) : 750=21.37%, 1000=21.76%, 2000=18.48%, >=2000=0.39% cpu : usr=0.08%, sys=0.50%, ctx=4313, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146353: Mon Sep 7 22:39:26 2026 read: IOPS=10, BW=43.8KiB/s (44.9kB/s)(4976KiB/113581msec) slat (usec): min=2775, max=56388, avg=14524.16, stdev=7753.85 clat (msec): min=39, max=875, avg=439.15, stdev=121.76 lat (msec): min=51, max=900, avg=453.68, stdev=123.66 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 485], 80.00th=[ 523], 90.00th=[ 600], 95.00th=[ 676], | 99.00th=[ 785], 99.50th=[ 827], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 119, per=24.73%, avg=43.52, stdev=20.96, samples=221 iops : min= 1, max= 29, avg=10.37, stdev= 5.28, samples=221 write: IOPS=11, BW=46.3KiB/s (47.5kB/s)(5264KiB/113581msec); 0 zone resets slat (usec): min=28, max=18527, avg=241.45, stdev=853.61 clat (msec): min=46, max=2087, avg=950.23, stdev=287.36 lat (msec): min=47, max=2087, avg=950.47, stdev=287.36 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 793], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 978], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1435], | 99.00th=[ 1955], 99.50th=[ 2039], 99.90th=[ 2072], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 103, per=25.00%, avg=45.75, stdev=20.12, samples=224 iops : min= 1, max= 25, avg=10.93, stdev= 5.03, samples=224 lat (msec) : 50=0.16%, 100=0.35%, 250=0.78%, 500=36.52%, 750=21.13% lat (msec) : 1000=21.76%, 2000=18.98%, >=2000=0.31% cpu : usr=0.10%, sys=0.55%, ctx=4307, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146354: Mon Sep 7 22:39:26 2026 read: IOPS=10, BW=43.9KiB/s (44.0kB/s)(4984KiB/113462msec) slat (usec): min=2653, max=52823, avg=14511.03, stdev=8186.11 clat (msec): min=20, max=894, avg=439.46, stdev=123.33 lat (msec): min=36, max=909, avg=453.98, stdev=125.40 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 485], 80.00th=[ 527], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 802], 99.50th=[ 844], 99.90th=[ 869], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 104, per=24.79%, avg=43.63, stdev=20.29, samples=224 iops : min= 1, max= 26, avg=10.40, stdev= 5.12, samples=224 write: IOPS=11, BW=46.3KiB/s (47.4kB/s)(5256KiB/113462msec); 0 zone resets slat (usec): min=33, max=16068, avg=224.74, stdev=747.86 clat (msec): min=286, max=2138, avg=948.63, stdev=276.34 lat (msec): min=286, max=2138, avg=948.86, stdev=276.35 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1284], 95.00th=[ 1435], | 99.00th=[ 1888], 99.50th=[ 2022], 99.90th=[ 2089], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 119, per=24.93%, avg=45.61, stdev=20.63, samples=225 iops : min= 1, max= 29, avg=10.89, stdev= 5.17, samples=225 lat (msec) : 50=0.20%, 100=0.35%, 250=0.78%, 500=36.09%, 750=21.37% lat (msec) : 1000=22.42%, 2000=18.48%, >=2000=0.31% cpu : usr=0.08%, sys=0.49%, ctx=4185, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146355: Mon Sep 7 22:39:26 2026 read: IOPS=11, BW=44.7KiB/s (45.7kB/s)(5060KiB/113257msec) slat (usec): min=44, max=58806, avg=14029.70, stdev=7718.62 clat (msec): min=26, max=943, avg=442.93, stdev=125.50 lat (msec): min=28, max=949, avg=456.96, stdev=127.60 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 355], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 489], 80.00th=[ 523], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 818], 99.50th=[ 860], 99.90th=[ 927], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 104, per=25.25%, avg=44.44, stdev=20.26, samples=223 iops : min= 1, max= 26, avg=10.60, stdev= 5.11, samples=223 write: IOPS=11, BW=45.7KiB/s (46.8kB/s)(5180KiB/113257msec); 0 zone resets slat (usec): min=33, max=14097, avg=243.54, stdev=823.10 clat (msec): min=137, max=2104, avg=952.09, stdev=283.53 lat (msec): min=137, max=2104, avg=952.33, stdev=283.55 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 550], 10.00th=[ 634], 20.00th=[ 735], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1452], | 99.00th=[ 1955], 99.50th=[ 1989], 99.90th=[ 2039], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 95, per=24.55%, avg=44.93, stdev=18.47, samples=225 iops : min= 1, max= 23, avg=10.72, stdev= 4.65, samples=225 lat (msec) : 50=0.27%, 100=0.23%, 250=1.21%, 500=36.91%, 750=20.59% lat (msec) : 1000=21.25%, 2000=19.38%, >=2000=0.16% cpu : usr=0.08%, sys=0.52%, ctx=4503, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177KiB/s (181kB/s), 43.8KiB/s-44.7KiB/s (44.9kB/s-45.7kB/s), io=19.6MiB (20.5MB), run=113257-113581msec WRITE: bw=184KiB/s (188kB/s), 45.7KiB/s-46.3KiB/s (46.8kB/s-47.5kB/s), io=20.4MiB (21.4MB), run=113257-113581msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146376: Mon Sep 7 22:39:36 2026 read: IOPS=1, BW=68.5MiB/s (71.8MB/s)(40.0MiB/584msec) slat (nsec): min=40690k, max=40690k, avg=40689575.00, stdev= 0.00 clat (nsec): min=500502k, max=500502k, avg=500501539.00, stdev= 0.00 lat (nsec): min=541218k, max=541218k, avg=541217764.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.57%, ctx=22, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.5MiB/s (71.8MB/s), 68.5MiB/s-68.5MiB/s (71.8MB/s-71.8MB/s), io=40.0MiB (41.9MB), run=584-584msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout