== sanity test 398c: run fio to test AIO ================= 15:22:10 (1787858530) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.439652 s, 95.4 MB/s osc.lustre-OST0000-osc-ffff9038784fb000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146603: Thu Aug 27 15:24:50 2026 write: IOPS=17, BW=68.3KiB/s (69.0kB/s)(10.0MiB/149819msec); 0 zone resets slat (usec): min=22, max=30316, avg=220.82, stdev=1170.98 clat (msec): min=87, max=1831, avg=935.77, stdev=256.22 lat (msec): min=87, max=1831, avg=935.99, stdev=256.25 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 493], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 919], 60.00th=[ 978], | 70.00th=[ 1045], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1636], 99.50th=[ 1670], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 207, per=25.02%, avg=68.06, stdev=29.25, samples=296 iops : min= 1, max= 51, avg=16.57, stdev= 7.36, samples=296 lat (msec) : 100=0.04%, 250=0.62%, 500=4.38%, 750=15.47%, 1000=43.48% lat (msec) : 2000=36.02% cpu : usr=0.05%, sys=0.20%, ctx=1721, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146604: Thu Aug 27 15:24:50 2026 write: IOPS=17, BW=68.2KiB/s (69.9kB/s)(10.0MiB/150065msec); 0 zone resets slat (usec): min=22, max=18105, avg=201.28, stdev=903.27 clat (msec): min=218, max=1830, avg=936.71, stdev=250.98 lat (msec): min=219, max=1830, avg=936.92, stdev=251.00 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 502], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 969], | 70.00th=[ 1045], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1620], 99.50th=[ 1754], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 168, per=25.33%, avg=68.90, stdev=29.54, samples=292 iops : min= 1, max= 42, avg=16.78, stdev= 7.45, samples=292 lat (msec) : 250=0.66%, 500=4.30%, 750=14.80%, 1000=44.57%, 2000=35.66% cpu : usr=0.04%, sys=0.25%, ctx=1843, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146605: Thu Aug 27 15:24:50 2026 write: IOPS=17, BW=68.3KiB/s (69.0kB/s)(10.0MiB/149817msec); 0 zone resets slat (usec): min=22, max=30956, avg=230.36, stdev=1205.89 clat (msec): min=244, max=1832, avg=935.27, stdev=252.72 lat (msec): min=244, max=1832, avg=935.50, stdev=252.78 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 472], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1045], 80.00th=[ 1116], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1620], 99.50th=[ 1720], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 215, per=25.12%, avg=68.33, stdev=29.82, samples=295 iops : min= 1, max= 53, avg=16.63, stdev= 7.52, samples=295 lat (msec) : 250=0.62%, 500=5.00%, 750=13.40%, 1000=44.26%, 2000=36.72% cpu : usr=0.04%, sys=0.26%, ctx=1701, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146606: Thu Aug 27 15:24:50 2026 write: IOPS=17, BW=68.3KiB/s (69.9kB/s)(10.0MiB/150011msec); 0 zone resets slat (usec): min=23, max=24168, avg=226.18, stdev=1033.87 clat (msec): min=231, max=1830, avg=936.43, stdev=249.60 lat (msec): min=231, max=1830, avg=936.65, stdev=249.65 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 506], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 919], 60.00th=[ 978], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1603], 99.50th=[ 1670], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 167, per=25.11%, avg=68.30, stdev=29.70, samples=295 iops : min= 1, max= 41, avg=16.63, stdev= 7.47, samples=295 lat (msec) : 250=0.62%, 500=3.91%, 750=15.16%, 1000=44.61%, 2000=35.70% cpu : usr=0.05%, sys=0.24%, ctx=1781, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=273KiB/s (279kB/s), 68.2KiB/s-68.3KiB/s (69.9kB/s-69.0kB/s), io=40.0MiB (41.9MB), run=149817-150065msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146637: Thu Aug 27 15:26:35 2026 read: IOPS=12, BW=51.4KiB/s (52.6kB/s)(5048KiB/98203msec) slat (usec): min=2522, max=59688, avg=13792.86, stdev=7945.46 clat (msec): min=38, max=940, avg=386.33, stdev=115.98 lat (msec): min=50, max=984, avg=400.13, stdev=117.87 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 247], 10.00th=[ 271], 20.00th=[ 300], | 30.00th=[ 326], 40.00th=[ 347], 50.00th=[ 368], 60.00th=[ 393], | 70.00th=[ 422], 80.00th=[ 460], 90.00th=[ 518], 95.00th=[ 609], | 99.00th=[ 818], 99.50th=[ 852], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 136, per=24.85%, avg=50.70, stdev=23.54, samples=195 iops : min= 1, max= 34, avg=12.15, stdev= 5.89, samples=195 write: IOPS=13, BW=52.9KiB/s (54.1kB/s)(5192KiB/98203msec); 0 zone resets slat (usec): min=23, max=22154, avg=250.15, stdev=1059.90 clat (msec): min=245, max=1767, avg=818.33, stdev=222.88 lat (msec): min=245, max=1767, avg=818.58, stdev=222.85 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 502], 10.00th=[ 567], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 844], | 70.00th=[ 919], 80.00th=[ 995], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1485], 99.50th=[ 1670], 99.90th=[ 1754], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 120, per=24.63%, avg=52.21, stdev=21.64, samples=195 iops : min= 1, max= 30, avg=12.53, stdev= 5.37, samples=195 lat (msec) : 50=0.16%, 100=0.20%, 250=2.46%, 500=42.93%, 750=25.31% lat (msec) : 1000=18.83%, 2000=10.12% cpu : usr=0.09%, sys=0.71%, ctx=4358, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146638: Thu Aug 27 15:26:35 2026 read: IOPS=12, BW=50.7KiB/s (51.9kB/s)(4976KiB/98208msec) slat (usec): min=2357, max=81735, avg=13712.97, stdev=8030.91 clat (msec): min=44, max=940, avg=390.93, stdev=111.35 lat (msec): min=53, max=958, avg=404.65, stdev=112.64 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 249], 10.00th=[ 271], 20.00th=[ 300], | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 401], | 70.00th=[ 435], 80.00th=[ 468], 90.00th=[ 523], 95.00th=[ 600], | 99.00th=[ 760], 99.50th=[ 835], 99.90th=[ 911], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 144, per=24.62%, avg=50.22, stdev=22.25, samples=194 iops : min= 1, max= 36, avg=12.03, stdev= 5.59, samples=194 write: IOPS=13, BW=53.6KiB/s (54.9kB/s)(5264KiB/98208msec); 0 zone resets slat (usec): min=25, max=10234, avg=234.53, stdev=611.62 clat (msec): min=161, max=1760, avg=809.32, stdev=222.73 lat (msec): min=161, max=1761, avg=809.55, stdev=222.75 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 477], 10.00th=[ 558], 20.00th=[ 634], | 30.00th=[ 684], 40.00th=[ 726], 50.00th=[ 785], 60.00th=[ 844], | 70.00th=[ 911], 80.00th=[ 995], 90.00th=[ 1099], 95.00th=[ 1183], | 99.00th=[ 1418], 99.50th=[ 1670], 99.90th=[ 1687], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 15, max= 126, per=25.09%, avg=53.18, stdev=20.82, samples=194 iops : min= 3, max= 31, avg=12.78, stdev= 5.19, samples=194 lat (msec) : 50=0.08%, 100=0.08%, 250=2.54%, 500=42.15%, 750=25.47% lat (msec) : 1000=19.65%, 2000=10.04% cpu : usr=0.12%, sys=0.62%, ctx=4349, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146639: Thu Aug 27 15:26:35 2026 read: IOPS=12, BW=50.9KiB/s (52.1kB/s)(4984KiB/97988msec) slat (usec): min=47, max=53088, avg=13328.53, stdev=7670.20 clat (msec): min=23, max=1323, avg=376.08, stdev=112.35 lat (msec): min=28, max=1331, avg=389.42, stdev=114.01 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 230], 10.00th=[ 255], 20.00th=[ 296], | 30.00th=[ 321], 40.00th=[ 342], 50.00th=[ 363], 60.00th=[ 388], | 70.00th=[ 418], 80.00th=[ 451], 90.00th=[ 506], 95.00th=[ 558], | 99.00th=[ 776], 99.50th=[ 818], 99.90th=[ 877], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 119, per=25.02%, avg=51.04, stdev=24.94, samples=192 iops : min= 1, max= 29, avg=12.24, stdev= 6.23, samples=192 write: IOPS=13, BW=53.6KiB/s (54.9kB/s)(5256KiB/97988msec); 0 zone resets slat (usec): min=29, max=23231, avg=233.59, stdev=915.54 clat (msec): min=148, max=1826, avg=822.91, stdev=234.50 lat (msec): min=149, max=1826, avg=823.15, stdev=234.48 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 502], 10.00th=[ 575], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 852], | 70.00th=[ 919], 80.00th=[ 1003], 90.00th=[ 1116], 95.00th=[ 1234], | 99.00th=[ 1569], 99.50th=[ 1737], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 111, per=24.94%, avg=52.87, stdev=19.80, samples=194 iops : min= 1, max= 27, avg=12.70, stdev= 4.94, samples=194 lat (msec) : 50=0.23%, 100=0.27%, 250=4.06%, 500=41.52%, 750=24.18% lat (msec) : 1000=19.14%, 2000=10.59% cpu : usr=0.12%, sys=0.64%, ctx=4372, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146640: Thu Aug 27 15:26:35 2026 read: IOPS=12, BW=51.6KiB/s (52.8kB/s)(5060KiB/98100msec) slat (usec): min=2343, max=52436, avg=13333.10, stdev=7492.45 clat (msec): min=16, max=982, avg=381.13, stdev=114.05 lat (msec): min=20, max=1003, avg=394.47, stdev=115.24 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 247], 10.00th=[ 264], 20.00th=[ 296], | 30.00th=[ 326], 40.00th=[ 347], 50.00th=[ 368], 60.00th=[ 393], | 70.00th=[ 422], 80.00th=[ 456], 90.00th=[ 506], 95.00th=[ 567], | 99.00th=[ 818], 99.50th=[ 852], 99.90th=[ 978], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 135, per=25.45%, avg=51.92, stdev=25.54, samples=191 iops : min= 1, max= 33, avg=12.45, stdev= 6.41, samples=191 write: IOPS=13, BW=52.8KiB/s (54.1kB/s)(5180KiB/98100msec); 0 zone resets slat (usec): min=26, max=22740, avg=282.57, stdev=975.69 clat (msec): min=253, max=1797, avg=823.55, stdev=227.75 lat (msec): min=255, max=1797, avg=823.84, stdev=227.73 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 498], 10.00th=[ 558], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 852], | 70.00th=[ 927], 80.00th=[ 1020], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1552], 99.50th=[ 1653], 99.90th=[ 1787], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 111, per=24.54%, avg=52.03, stdev=21.70, samples=195 iops : min= 1, max= 27, avg=12.48, stdev= 5.39, samples=195 lat (msec) : 20=0.04%, 50=0.16%, 100=0.27%, 250=2.70%, 500=43.24% lat (msec) : 750=23.59%, 1000=18.91%, 2000=11.09% cpu : usr=0.10%, sys=0.64%, ctx=4247, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=204KiB/s (209kB/s), 50.7KiB/s-51.6KiB/s (51.9kB/s-52.8kB/s), io=19.6MiB (20.5MB), run=97988-98208msec WRITE: bw=213KiB/s (218kB/s), 52.8KiB/s-53.6KiB/s (54.1kB/s-54.9kB/s), io=20.4MiB (21.4MB), run=97988-98208msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146658: Thu Aug 27 15:26:46 2026 read: IOPS=1, BW=63.4MiB/s (66.5MB/s)(40.0MiB/631msec) slat (nsec): min=24179k, max=24179k, avg=24179388.00, stdev= 0.00 clat (nsec): min=551320k, max=551320k, avg=551320229.00, stdev= 0.00 lat (nsec): min=575504k, max=575504k, avg=575504307.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.06%, ctx=6, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.4MiB/s (66.5MB/s), 63.4MiB/s-63.4MiB/s (66.5MB/s-66.5MB/s), io=40.0MiB (41.9MB), run=631-631msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout