== sanity test 398c: run fio to test AIO ================= 05:18:35 (1788513515) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.579925 s, 72.3 MB/s osc.lustre-OST0000-osc-ffff8a9b02dae000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146028: Fri Sep 4 05:21:48 2026 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(10.0MiB/179227msec); 0 zone resets slat (usec): min=53, max=19064, avg=359.12, stdev=1010.15 clat (msec): min=222, max=2217, avg=1118.10, stdev=261.72 lat (msec): min=222, max=2218, avg=1118.46, stdev=261.72 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 709], 10.00th=[ 810], 20.00th=[ 911], | 30.00th=[ 986], 40.00th=[ 1053], 50.00th=[ 1116], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1435], 95.00th=[ 1569], | 99.00th=[ 1854], 99.50th=[ 1955], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 127, per=25.32%, avg=57.72, stdev=27.92, samples=348 iops : min= 1, max= 31, avg=13.92, stdev= 6.98, samples=348 lat (msec) : 250=0.62%, 750=6.17%, 1000=25.74%, 2000=67.07%, >=2000=0.39% cpu : usr=0.04%, sys=0.30%, ctx=1911, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146029: Fri Sep 4 05:21:48 2026 write: IOPS=14, BW=57.0KiB/s (58.4kB/s)(10.0MiB/179559msec); 0 zone resets slat (usec): min=52, max=30903, avg=391.20, stdev=1349.11 clat (msec): min=231, max=2137, avg=1120.37, stdev=258.21 lat (msec): min=231, max=2138, avg=1120.76, stdev=258.16 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 718], 10.00th=[ 818], 20.00th=[ 911], | 30.00th=[ 986], 40.00th=[ 1053], 50.00th=[ 1116], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1435], 95.00th=[ 1552], | 99.00th=[ 1888], 99.50th=[ 1921], 99.90th=[ 2072], 99.95th=[ 2106], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 119, per=24.74%, avg=56.42, stdev=22.47, samples=356 iops : min= 1, max= 29, avg=13.59, stdev= 5.64, samples=356 lat (msec) : 250=0.23%, 500=0.43%, 750=5.70%, 1000=26.48%, 2000=66.95% lat (msec) : >=2000=0.20% cpu : usr=0.05%, sys=0.29%, ctx=1854, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146030: Fri Sep 4 05:21:48 2026 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(10.0MiB/179323msec); 0 zone resets slat (usec): min=52, max=23287, avg=423.88, stdev=1331.48 clat (msec): min=207, max=2217, avg=1119.67, stdev=254.32 lat (msec): min=209, max=2219, avg=1120.10, stdev=254.26 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 701], 10.00th=[ 818], 20.00th=[ 919], | 30.00th=[ 986], 40.00th=[ 1053], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1418], 95.00th=[ 1552], | 99.00th=[ 1804], 99.50th=[ 1938], 99.90th=[ 2198], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 127, per=25.39%, avg=57.89, stdev=26.70, samples=347 iops : min= 1, max= 31, avg=13.96, stdev= 6.70, samples=347 lat (msec) : 250=0.20%, 500=0.66%, 750=5.47%, 1000=26.37%, 2000=66.99% lat (msec) : >=2000=0.31% cpu : usr=0.06%, sys=0.31%, ctx=2151, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146031: Fri Sep 4 05:21:48 2026 write: IOPS=14, BW=57.0KiB/s (58.4kB/s)(10.0MiB/179561msec); 0 zone resets slat (usec): min=52, max=23310, avg=394.89, stdev=1264.99 clat (msec): min=356, max=2199, avg=1120.56, stdev=253.07 lat (msec): min=356, max=2200, avg=1120.96, stdev=253.04 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 726], 10.00th=[ 818], 20.00th=[ 911], | 30.00th=[ 986], 40.00th=[ 1045], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1435], 95.00th=[ 1552], | 99.00th=[ 1804], 99.50th=[ 1921], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 128, per=25.38%, avg=57.88, stdev=26.42, samples=347 iops : min= 1, max= 32, avg=13.96, stdev= 6.63, samples=347 lat (msec) : 500=0.66%, 750=5.23%, 1000=26.48%, 2000=67.38%, >=2000=0.23% cpu : usr=0.05%, sys=0.31%, ctx=2234, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=228KiB/s (234kB/s), 57.0KiB/s-57.1KiB/s (58.4kB/s-58.5kB/s), io=40.0MiB (41.9MB), run=179227-179561msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146067: Fri Sep 4 05:24:03 2026 read: IOPS=10, BW=40.4KiB/s (41.3kB/s)(5048KiB/125026msec) slat (usec): min=3406, max=62194, avg=16197.74, stdev=8826.45 clat (msec): min=16, max=1000, avg=477.75, stdev=139.02 lat (msec): min=25, max=1039, avg=493.96, stdev=140.43 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 430], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 535], 80.00th=[ 584], 90.00th=[ 659], 95.00th=[ 718], | 99.00th=[ 869], 99.50th=[ 911], 99.90th=[ 986], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 126, per=25.03%, avg=40.04, stdev=20.80, samples=246 iops : min= 1, max= 31, avg= 9.48, stdev= 5.21, samples=246 write: IOPS=10, BW=41.5KiB/s (42.5kB/s)(5192KiB/125026msec); 0 zone resets slat (usec): min=57, max=15255, avg=519.49, stdev=1344.71 clat (msec): min=336, max=1768, avg=1057.01, stdev=300.32 lat (msec): min=336, max=1769, avg=1057.54, stdev=300.26 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 776], | 30.00th=[ 869], 40.00th=[ 953], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1720], 99.50th=[ 1737], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 96, per=25.12%, avg=41.70, stdev=18.67, samples=243 iops : min= 1, max= 24, avg= 9.88, stdev= 4.70, samples=243 lat (msec) : 20=0.04%, 50=0.08%, 100=0.20%, 250=0.82%, 500=29.80% lat (msec) : 750=24.84%, 1000=16.64%, 2000=27.58% cpu : usr=0.10%, sys=0.69%, ctx=4656, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146068: Fri Sep 4 05:24:03 2026 read: IOPS=9, BW=39.7KiB/s (40.7kB/s)(4976KiB/125249msec) slat (usec): min=3395, max=80716, avg=16420.21, stdev=9048.55 clat (msec): min=21, max=1006, avg=483.52, stdev=142.52 lat (msec): min=32, max=1023, avg=499.96, stdev=144.09 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 284], 10.00th=[ 326], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 439], 50.00th=[ 472], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 877], 99.50th=[ 911], 99.90th=[ 927], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 110, per=24.64%, avg=39.43, stdev=19.60, samples=245 iops : min= 1, max= 27, avg= 9.32, stdev= 4.91, samples=245 write: IOPS=10, BW=42.0KiB/s (43.0kB/s)(5264KiB/125249msec); 0 zone resets slat (usec): min=55, max=21741, avg=482.80, stdev=1235.50 clat (msec): min=298, max=1847, avg=1047.75, stdev=306.91 lat (msec): min=298, max=1847, avg=1048.23, stdev=306.89 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 776], | 30.00th=[ 860], 40.00th=[ 936], 50.00th=[ 1011], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1603], | 99.00th=[ 1754], 99.50th=[ 1770], 99.90th=[ 1821], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 103, per=25.04%, avg=41.57, stdev=19.78, samples=247 iops : min= 1, max= 25, avg= 9.86, stdev= 4.98, samples=247 lat (msec) : 50=0.20%, 100=0.31%, 250=0.66%, 500=28.16%, 750=26.29% lat (msec) : 1000=17.81%, 2000=26.56% cpu : usr=0.10%, sys=0.71%, ctx=4641, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146069: Fri Sep 4 05:24:03 2026 read: IOPS=9, BW=39.8KiB/s (40.8kB/s)(4984KiB/125198msec) slat (usec): min=80, max=60418, avg=16057.86, stdev=8928.84 clat (msec): min=12, max=940, avg=477.95, stdev=141.37 lat (msec): min=12, max=948, avg=494.02, stdev=143.02 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 368], | 30.00th=[ 393], 40.00th=[ 426], 50.00th=[ 464], 60.00th=[ 498], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 667], 95.00th=[ 743], | 99.00th=[ 852], 99.50th=[ 902], 99.90th=[ 927], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 95, per=24.97%, avg=39.95, stdev=20.00, samples=244 iops : min= 1, max= 23, avg= 9.45, stdev= 5.02, samples=244 write: IOPS=10, BW=41.0KiB/s (42.0kB/s)(5256KiB/125198msec); 0 zone resets slat (usec): min=56, max=24916, avg=488.07, stdev=1251.88 clat (msec): min=173, max=1826, avg=1054.70, stdev=308.43 lat (msec): min=173, max=1830, avg=1055.19, stdev=308.45 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 584], 10.00th=[ 659], 20.00th=[ 785], | 30.00th=[ 877], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1586], | 99.00th=[ 1754], 99.50th=[ 1787], 99.90th=[ 1804], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 127, per=25.15%, avg=41.75, stdev=19.06, samples=245 iops : min= 1, max= 31, avg= 9.90, stdev= 4.78, samples=245 lat (msec) : 20=0.12%, 50=0.08%, 100=0.20%, 250=0.86%, 500=29.80% lat (msec) : 750=24.26%, 1000=16.48%, 2000=28.20% cpu : usr=0.10%, sys=0.71%, ctx=4738, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146070: Fri Sep 4 05:24:03 2026 read: IOPS=10, BW=40.4KiB/s (41.4kB/s)(5060KiB/125242msec) slat (usec): min=3008, max=65521, avg=16162.61, stdev=9250.65 clat (msec): min=24, max=1006, avg=480.83, stdev=138.66 lat (msec): min=34, max=1024, avg=497.01, stdev=139.95 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 292], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 464], 60.00th=[ 498], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 659], 95.00th=[ 735], | 99.00th=[ 902], 99.50th=[ 919], 99.90th=[ 995], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 112, per=25.18%, avg=40.29, stdev=20.31, samples=245 iops : min= 1, max= 28, avg= 9.53, stdev= 5.11, samples=245 write: IOPS=10, BW=41.4KiB/s (42.4kB/s)(5180KiB/125242msec); 0 zone resets slat (usec): min=54, max=13044, avg=454.87, stdev=1121.40 clat (msec): min=279, max=1827, avg=1058.21, stdev=308.81 lat (msec): min=279, max=1832, avg=1058.68, stdev=308.86 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 592], 10.00th=[ 667], 20.00th=[ 776], | 30.00th=[ 869], 40.00th=[ 953], 50.00th=[ 1036], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1603], | 99.00th=[ 1754], 99.50th=[ 1770], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 111, per=24.63%, avg=40.89, stdev=18.87, samples=246 iops : min= 1, max= 27, avg= 9.68, stdev= 4.73, samples=246 lat (msec) : 50=0.08%, 100=0.23%, 250=0.47%, 500=30.31%, 750=24.53% lat (msec) : 1000=16.84%, 2000=27.54% cpu : usr=0.10%, sys=0.60%, ctx=4477, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=160KiB/s (164kB/s), 39.7KiB/s-40.4KiB/s (40.7kB/s-41.4kB/s), io=19.6MiB (20.5MB), run=125026-125249msec WRITE: bw=167KiB/s (171kB/s), 41.4KiB/s-42.0KiB/s (42.4kB/s-43.0kB/s), io=20.4MiB (21.4MB), run=125026-125249msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146093: Fri Sep 4 05:24:15 2026 read: IOPS=1, BW=55.9MiB/s (58.6MB/s)(40.0MiB/716msec) slat (nsec): min=48859k, max=48859k, avg=48859226.00, stdev= 0.00 clat (nsec): min=641930k, max=641930k, avg=641930309.00, stdev= 0.00 lat (nsec): min=690794k, max=690794k, avg=690794008.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.40%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=55.9MiB/s (58.6MB/s), 55.9MiB/s-55.9MiB/s (58.6MB/s-58.6MB/s), io=40.0MiB (41.9MB), run=716-716msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout