== sanity test 398c: run fio to test AIO ================= 08:35:42 (1787747742) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.270034 s, 155 MB/s osc.lustre-OST0000-osc-ffffa0575f479800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146506: Wed Aug 26 08:38:43 2026 write: IOPS=14, BW=59.2KiB/s (60.6kB/s)(10.0MiB/172958msec); 0 zone resets slat (usec): min=23, max=14135, avg=216.34, stdev=813.22 clat (msec): min=108, max=2127, avg=1079.67, stdev=251.55 lat (msec): min=108, max=2128, avg=1079.89, stdev=251.53 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 667], 10.00th=[ 802], 20.00th=[ 902], | 30.00th=[ 978], 40.00th=[ 1028], 50.00th=[ 1083], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1267], 90.00th=[ 1385], 95.00th=[ 1469], | 99.00th=[ 1653], 99.50th=[ 1754], 99.90th=[ 2106], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 8, max= 255, per=25.24%, avg=59.57, stdev=28.89, samples=338 iops : min= 2, max= 63, avg=14.44, stdev= 7.25, samples=338 lat (msec) : 250=0.08%, 500=2.46%, 750=5.35%, 1000=25.98%, 2000=65.94% lat (msec) : >=2000=0.20% cpu : usr=0.05%, sys=0.20%, ctx=1818, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146507: Wed Aug 26 08:38:43 2026 write: IOPS=14, BW=59.1KiB/s (60.5kB/s)(10.0MiB/173176msec); 0 zone resets slat (usec): min=23, max=21597, avg=248.59, stdev=1112.94 clat (msec): min=290, max=2213, avg=1081.21, stdev=250.68 lat (msec): min=290, max=2213, avg=1081.46, stdev=250.69 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 676], 10.00th=[ 802], 20.00th=[ 902], | 30.00th=[ 978], 40.00th=[ 1028], 50.00th=[ 1083], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1267], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1720], 99.50th=[ 1804], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 255, per=25.86%, avg=61.03, stdev=29.63, samples=330 iops : min= 1, max= 63, avg=14.81, stdev= 7.41, samples=330 lat (msec) : 500=2.58%, 750=4.88%, 1000=26.80%, 2000=65.39%, >=2000=0.35% cpu : usr=0.04%, sys=0.18%, ctx=1632, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146508: Wed Aug 26 08:38:43 2026 write: IOPS=14, BW=59.3KiB/s (60.8kB/s)(10.0MiB/172543msec); 0 zone resets slat (usec): min=22, max=15044, avg=193.37, stdev=742.39 clat (msec): min=142, max=2209, avg=1077.81, stdev=256.55 lat (msec): min=144, max=2209, avg=1078.00, stdev=256.55 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 667], 10.00th=[ 793], 20.00th=[ 894], | 30.00th=[ 961], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1150], | 70.00th=[ 1200], 80.00th=[ 1267], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 1670], 99.50th=[ 1754], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 191, per=25.38%, avg=59.89, stdev=26.65, samples=336 iops : min= 1, max= 47, avg=14.52, stdev= 6.66, samples=336 lat (msec) : 250=0.74%, 500=1.99%, 750=5.08%, 1000=27.66%, 2000=64.41% lat (msec) : >=2000=0.12% cpu : usr=0.04%, sys=0.18%, ctx=1695, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146509: Wed Aug 26 08:38:43 2026 write: IOPS=14, BW=59.1KiB/s (60.6kB/s)(10.0MiB/173173msec); 0 zone resets slat (usec): min=25, max=13823, avg=186.51, stdev=678.54 clat (msec): min=230, max=2046, avg=1080.89, stdev=251.76 lat (msec): min=231, max=2046, avg=1081.08, stdev=251.78 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 667], 10.00th=[ 802], 20.00th=[ 902], | 30.00th=[ 969], 40.00th=[ 1028], 50.00th=[ 1083], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 1653], 99.50th=[ 1737], 99.90th=[ 2022], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 191, per=25.09%, avg=59.21, stdev=27.45, samples=340 iops : min= 1, max= 47, avg=14.36, stdev= 6.86, samples=340 lat (msec) : 250=0.94%, 500=1.80%, 750=4.69%, 1000=26.41%, 2000=66.05% lat (msec) : >=2000=0.12% cpu : usr=0.04%, sys=0.18%, ctx=1836, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=237KiB/s (242kB/s), 59.1KiB/s-59.3KiB/s (60.5kB/s-60.8kB/s), io=40.0MiB (41.9MB), run=172543-173176msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146545: Wed Aug 26 08:40:54 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(5048KiB/120671msec) slat (usec): min=36, max=76589, avg=14671.65, stdev=9040.78 clat (msec): min=10, max=1112, avg=472.17, stdev=153.37 lat (msec): min=28, max=1169, avg=486.85, stdev=155.72 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 257], 10.00th=[ 300], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 535], 80.00th=[ 592], 90.00th=[ 676], 95.00th=[ 743], | 99.00th=[ 877], 99.50th=[ 944], 99.90th=[ 1070], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 144, per=25.60%, avg=42.24, stdev=21.92, samples=234 iops : min= 1, max= 36, avg=10.11, stdev= 5.53, samples=234 write: IOPS=10, BW=43.0KiB/s (44.1kB/s)(5192KiB/120671msec); 0 zone resets slat (usec): min=32, max=14973, avg=247.59, stdev=847.17 clat (msec): min=162, max=2151, avg=1012.92, stdev=325.73 lat (msec): min=163, max=2151, avg=1013.17, stdev=325.73 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 743], | 30.00th=[ 835], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1435], 95.00th=[ 1636], | 99.00th=[ 1972], 99.50th=[ 2022], 99.90th=[ 2123], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 104, per=25.40%, avg=43.44, stdev=19.26, samples=234 iops : min= 1, max= 26, avg=10.42, stdev= 4.86, samples=234 lat (msec) : 20=0.04%, 50=0.35%, 100=0.39%, 250=1.72%, 500=30.00% lat (msec) : 750=24.80%, 1000=18.44%, 2000=23.87%, >=2000=0.39% cpu : usr=0.09%, sys=0.51%, ctx=4394, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146546: Wed Aug 26 08:40:54 2026 read: IOPS=10, BW=40.0KiB/s (41.9kB/s)(4976KiB/121482msec) slat (usec): min=2519, max=83517, avg=14555.34, stdev=9136.87 clat (msec): min=21, max=1152, avg=479.26, stdev=153.87 lat (msec): min=28, max=1162, avg=493.82, stdev=155.75 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 355], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 493], | 70.00th=[ 542], 80.00th=[ 600], 90.00th=[ 693], 95.00th=[ 760], | 99.00th=[ 978], 99.50th=[ 1011], 99.90th=[ 1133], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 104, per=24.58%, avg=40.56, stdev=21.51, samples=240 iops : min= 1, max= 26, avg= 9.70, stdev= 5.42, samples=240 write: IOPS=10, BW=43.3KiB/s (44.4kB/s)(5264KiB/121482msec); 0 zone resets slat (usec): min=24, max=23028, avg=306.37, stdev=1190.06 clat (msec): min=103, max=2123, avg=1007.34, stdev=323.49 lat (msec): min=103, max=2123, avg=1007.65, stdev=323.49 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 535], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 978], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1636], | 99.00th=[ 1905], 99.50th=[ 2005], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 120, per=25.17%, avg=43.05, stdev=19.94, samples=240 iops : min= 1, max= 30, avg=10.33, stdev= 5.04, samples=240 lat (msec) : 50=0.04%, 100=0.23%, 250=0.90%, 500=31.05%, 750=24.26% lat (msec) : 1000=19.06%, 2000=24.18%, >=2000=0.27% cpu : usr=0.08%, sys=0.52%, ctx=4383, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146547: Wed Aug 26 08:40:54 2026 read: IOPS=10, BW=41.0KiB/s (42.0kB/s)(4984KiB/121480msec) slat (usec): min=2846, max=74954, avg=14806.71, stdev=9244.82 clat (msec): min=27, max=1171, avg=472.82, stdev=146.37 lat (msec): min=36, max=1219, avg=487.63, stdev=148.68 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 493], | 70.00th=[ 527], 80.00th=[ 584], 90.00th=[ 659], 95.00th=[ 743], | 99.00th=[ 894], 99.50th=[ 969], 99.90th=[ 1150], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 112, per=25.23%, avg=41.63, stdev=22.48, samples=235 iops : min= 1, max= 28, avg= 9.97, stdev= 5.66, samples=235 write: IOPS=10, BW=43.3KiB/s (44.3kB/s)(5256KiB/121480msec); 0 zone resets slat (usec): min=24, max=14736, avg=295.77, stdev=1073.25 clat (msec): min=245, max=2151, avg=1014.44, stdev=330.18 lat (msec): min=246, max=2151, avg=1014.74, stdev=330.20 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 518], 10.00th=[ 617], 20.00th=[ 735], | 30.00th=[ 835], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1670], | 99.00th=[ 1921], 99.50th=[ 1989], 99.90th=[ 2072], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 120, per=24.96%, avg=42.67, stdev=19.20, samples=240 iops : min= 1, max= 30, avg=10.23, stdev= 4.84, samples=240 lat (msec) : 50=0.20%, 100=0.08%, 250=1.33%, 500=31.09%, 750=24.69% lat (msec) : 1000=17.73%, 2000=24.73%, >=2000=0.16% cpu : usr=0.08%, sys=0.53%, ctx=4647, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146548: Wed Aug 26 08:40:54 2026 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5060KiB/120722msec) slat (usec): min=2304, max=70896, avg=14529.40, stdev=8543.93 clat (msec): min=14, max=1163, avg=478.04, stdev=144.22 lat (msec): min=19, max=1189, avg=492.58, stdev=146.05 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 498], | 70.00th=[ 542], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 894], 99.50th=[ 969], 99.90th=[ 1133], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 135, per=25.44%, avg=41.98, stdev=20.78, samples=236 iops : min= 1, max= 33, avg=10.05, stdev= 5.21, samples=236 write: IOPS=10, BW=42.9KiB/s (43.9kB/s)(5180KiB/120722msec); 0 zone resets slat (usec): min=36, max=17981, avg=289.10, stdev=1051.52 clat (msec): min=229, max=2053, avg=1007.85, stdev=311.18 lat (msec): min=229, max=2053, avg=1008.14, stdev=311.16 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 735], | 30.00th=[ 844], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1250], 90.00th=[ 1418], 95.00th=[ 1603], | 99.00th=[ 1888], 99.50th=[ 1955], 99.90th=[ 2022], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 112, per=24.95%, avg=42.67, stdev=20.70, samples=238 iops : min= 1, max= 28, avg=10.23, stdev= 5.20, samples=238 lat (msec) : 20=0.04%, 100=0.08%, 250=0.98%, 500=30.90%, 750=26.02% lat (msec) : 1000=18.05%, 2000=23.83%, >=2000=0.12% cpu : usr=0.07%, sys=0.48%, ctx=4247, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=165KiB/s (169kB/s), 40.0KiB/s-41.9KiB/s (41.9kB/s-42.9kB/s), io=19.6MiB (20.5MB), run=120671-121482msec WRITE: bw=172KiB/s (176kB/s), 42.9KiB/s-43.3KiB/s (43.9kB/s-44.4kB/s), io=20.4MiB (21.4MB), run=120671-121482msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146569: Wed Aug 26 08:41:03 2026 read: IOPS=1, BW=68.8MiB/s (72.2MB/s)(40.0MiB/581msec) slat (nsec): min=49009k, max=49009k, avg=49008631.00, stdev= 0.00 clat (nsec): min=493170k, max=493170k, avg=493169978.00, stdev= 0.00 lat (nsec): min=542183k, max=542183k, avg=542183398.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 493], 10.00th=[ 493], 20.00th=[ 493], | 30.00th=[ 493], 40.00th=[ 493], 50.00th=[ 493], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] lat (msec) : 500=100.00% cpu : usr=0.17%, sys=2.24%, ctx=44, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.8MiB/s (72.2MB/s), 68.8MiB/s-68.8MiB/s (72.2MB/s-72.2MB/s), io=40.0MiB (41.9MB), run=581-581msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout