== sanity test 398c: run fio to test AIO ================= 02:21:45 (1790662905) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.400647 s, 105 MB/s osc.lustre-OST0000-osc-ffff94901029f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=176131: Tue Sep 29 02:24:36 2026 write: IOPS=15, BW=63.3KiB/s (64.8kB/s)(10.0MiB/161852msec); 0 zone resets slat (usec): min=30, max=22467, avg=326.96, stdev=1092.43 clat (msec): min=159, max=2089, avg=1010.66, stdev=252.20 lat (msec): min=159, max=2090, avg=1010.99, stdev=252.21 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 600], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1586], 99.50th=[ 1653], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=25.27%, avg=63.67, stdev=27.78, samples=316 iops : min= 1, max= 32, avg=15.38, stdev= 6.95, samples=316 lat (msec) : 250=0.62%, 500=1.99%, 750=10.51%, 1000=36.91%, 2000=49.80% lat (msec) : >=2000=0.16% cpu : usr=0.05%, sys=0.28%, ctx=2011, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=176132: Tue Sep 29 02:24:36 2026 write: IOPS=15, BW=63.1KiB/s (64.7kB/s)(10.0MiB/162184msec); 0 zone resets slat (usec): min=30, max=16297, avg=347.33, stdev=1081.89 clat (msec): min=159, max=1796, avg=1012.39, stdev=253.20 lat (msec): min=159, max=1796, avg=1012.74, stdev=253.23 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1653], 99.50th=[ 1670], 99.90th=[ 1754], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 128, per=25.59%, avg=64.48, stdev=30.61, samples=312 iops : min= 1, max= 32, avg=15.59, stdev= 7.69, samples=312 lat (msec) : 250=0.62%, 500=1.91%, 750=10.90%, 1000=36.68%, 2000=49.88% cpu : usr=0.05%, sys=0.27%, ctx=1882, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=176133: Tue Sep 29 02:24:36 2026 write: IOPS=15, BW=63.0KiB/s (64.5kB/s)(10.0MiB/162451msec); 0 zone resets slat (usec): min=28, max=16641, avg=303.14, stdev=895.24 clat (msec): min=216, max=1835, avg=1013.94, stdev=245.57 lat (msec): min=216, max=1835, avg=1014.24, stdev=245.58 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 617], 10.00th=[ 735], 20.00th=[ 818], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1217], 90.00th=[ 1318], 95.00th=[ 1401], | 99.00th=[ 1636], 99.50th=[ 1687], 99.90th=[ 1703], 99.95th=[ 1770], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 128, per=25.10%, avg=63.26, stdev=30.78, samples=318 iops : min= 1, max= 32, avg=15.29, stdev= 7.73, samples=318 lat (msec) : 250=0.62%, 500=1.95%, 750=9.30%, 1000=37.23%, 2000=50.90% cpu : usr=0.05%, sys=0.28%, ctx=1989, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=176134: Tue Sep 29 02:24:36 2026 write: IOPS=15, BW=63.0KiB/s (64.5kB/s)(10.0MiB/162450msec); 0 zone resets slat (usec): min=24, max=36880, avg=315.97, stdev=1169.26 clat (msec): min=124, max=1763, avg=1013.80, stdev=237.11 lat (msec): min=125, max=1763, avg=1014.12, stdev=237.10 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 617], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1217], 90.00th=[ 1318], 95.00th=[ 1385], | 99.00th=[ 1552], 99.50th=[ 1636], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 223, per=25.36%, avg=63.90, stdev=28.18, samples=315 iops : min= 1, max= 55, avg=15.44, stdev= 7.05, samples=315 lat (msec) : 250=0.04%, 500=2.11%, 750=9.84%, 1000=36.29%, 2000=51.72% cpu : usr=0.07%, sys=0.27%, ctx=2027, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=252KiB/s (258kB/s), 63.0KiB/s-63.3KiB/s (64.5kB/s-64.8kB/s), io=40.0MiB (41.9MB), run=161852-162451msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=176167: Tue Sep 29 02:26:45 2026 read: IOPS=10, BW=41.0KiB/s (43.0kB/s)(5048KiB/120204msec) slat (usec): min=3246, max=74548, avg=15736.82, stdev=8422.36 clat (msec): min=10, max=1773, avg=468.43, stdev=141.63 lat (msec): min=23, max=1782, avg=484.17, stdev=142.68 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 296], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 550], 90.00th=[ 625], 95.00th=[ 684], | 99.00th=[ 869], 99.50th=[ 1045], 99.90th=[ 1653], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 104, per=25.23%, avg=41.89, stdev=20.04, samples=236 iops : min= 1, max= 26, avg= 9.95, stdev= 5.02, samples=236 write: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5192KiB/120204msec); 0 zone resets slat (usec): min=32, max=21483, avg=446.34, stdev=1356.69 clat (msec): min=359, max=2249, avg=1006.67, stdev=269.50 lat (msec): min=359, max=2250, avg=1007.12, stdev=269.51 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 575], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1485], | 99.00th=[ 1737], 99.50th=[ 1787], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 104, per=24.92%, avg=43.11, stdev=19.88, samples=235 iops : min= 1, max= 26, avg=10.26, stdev= 4.98, samples=235 lat (msec) : 20=0.04%, 100=0.16%, 250=0.66%, 500=32.11%, 750=23.59% lat (msec) : 1000=18.09%, 2000=25.20%, >=2000=0.16% cpu : usr=0.12%, sys=0.80%, ctx=4868, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=176168: Tue Sep 29 02:26:45 2026 read: IOPS=10, BW=41.5KiB/s (42.5kB/s)(4976KiB/119933msec) slat (usec): min=53, max=52618, avg=15691.24, stdev=7972.37 clat (msec): min=12, max=1825, avg=465.75, stdev=149.63 lat (msec): min=24, max=1831, avg=481.45, stdev=150.25 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 292], 10.00th=[ 330], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 422], 50.00th=[ 447], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 919], 99.50th=[ 1519], 99.90th=[ 1603], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 104, per=25.57%, avg=42.44, stdev=19.61, samples=229 iops : min= 1, max= 26, avg=10.09, stdev= 4.90, samples=229 write: IOPS=10, BW=43.9KiB/s (44.9kB/s)(5264KiB/119933msec); 0 zone resets slat (usec): min=32, max=15081, avg=470.53, stdev=1237.49 clat (msec): min=122, max=2189, avg=1000.65, stdev=270.32 lat (msec): min=122, max=2190, avg=1001.13, stdev=270.30 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 919], 50.00th=[ 995], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 2056], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 107, per=25.44%, avg=44.01, stdev=19.25, samples=234 iops : min= 1, max= 26, avg=10.48, stdev= 4.84, samples=234 lat (msec) : 20=0.12%, 50=0.08%, 100=0.27%, 250=0.59%, 500=33.44% lat (msec) : 750=21.56%, 1000=18.12%, 2000=25.74%, >=2000=0.08% cpu : usr=0.12%, sys=0.79%, ctx=4931, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=176169: Tue Sep 29 02:26:45 2026 read: IOPS=10, BW=41.5KiB/s (42.5kB/s)(4984KiB/120204msec) slat (usec): min=3109, max=54377, avg=15489.68, stdev=8061.47 clat (msec): min=49, max=1048, avg=463.80, stdev=122.58 lat (msec): min=55, max=1076, avg=479.30, stdev=123.56 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 510], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 860], 99.50th=[ 1003], 99.90th=[ 1045], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 7, max= 127, per=24.98%, avg=41.47, stdev=19.81, samples=236 iops : min= 1, max= 31, avg= 9.85, stdev= 4.98, samples=236 write: IOPS=10, BW=43.7KiB/s (44.8kB/s)(5256KiB/120204msec); 0 zone resets slat (usec): min=37, max=23238, avg=400.54, stdev=1119.12 clat (msec): min=327, max=2080, avg=1005.27, stdev=263.54 lat (msec): min=327, max=2080, avg=1005.67, stdev=263.55 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 776], | 30.00th=[ 869], 40.00th=[ 919], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1703], 99.50th=[ 1787], 99.90th=[ 1989], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 119, per=24.94%, avg=43.15, stdev=19.04, samples=237 iops : min= 1, max= 29, avg=10.26, stdev= 4.79, samples=237 lat (msec) : 50=0.04%, 100=0.08%, 250=0.47%, 500=32.50%, 750=23.12% lat (msec) : 1000=17.85%, 2000=25.90%, >=2000=0.04% cpu : usr=0.12%, sys=0.73%, ctx=4845, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=176170: Tue Sep 29 02:26:45 2026 read: IOPS=10, BW=42.1KiB/s (43.1kB/s)(5060KiB/120196msec) slat (usec): min=3075, max=59968, avg=16136.24, stdev=8737.68 clat (msec): min=20, max=1494, avg=465.31, stdev=128.29 lat (msec): min=35, max=1513, avg=481.46, stdev=129.40 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 368], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 447], 60.00th=[ 481], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 667], | 99.00th=[ 835], 99.50th=[ 1045], 99.90th=[ 1368], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 111, per=25.53%, avg=42.38, stdev=20.69, samples=234 iops : min= 1, max= 27, avg=10.07, stdev= 5.19, samples=234 write: IOPS=10, BW=43.1KiB/s (44.1kB/s)(5180KiB/120196msec); 0 zone resets slat (usec): min=31, max=12676, avg=410.56, stdev=1089.43 clat (msec): min=306, max=2227, avg=1010.39, stdev=271.71 lat (msec): min=306, max=2227, avg=1010.80, stdev=271.69 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1720], 99.50th=[ 1989], 99.90th=[ 2106], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 104, per=24.84%, avg=42.98, stdev=17.89, samples=235 iops : min= 1, max= 26, avg=10.23, stdev= 4.49, samples=235 lat (msec) : 50=0.08%, 100=0.20%, 250=0.39%, 500=32.97%, 750=22.66% lat (msec) : 1000=17.93%, 2000=25.55%, >=2000=0.23% cpu : usr=0.11%, sys=0.73%, ctx=4322, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=167KiB/s (171kB/s), 41.5KiB/s-42.1KiB/s (42.5kB/s-43.1kB/s), io=19.6MiB (20.5MB), run=119933-120204msec WRITE: bw=174KiB/s (178kB/s), 43.1KiB/s-43.9KiB/s (44.1kB/s-44.9kB/s), io=20.4MiB (21.4MB), run=119933-120204msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=176193: Tue Sep 29 02:26:55 2026 read: IOPS=2, BW=88.5MiB/s (92.8MB/s)(40.0MiB/452msec) slat (nsec): min=44699k, max=44699k, avg=44698591.00, stdev= 0.00 clat (nsec): min=384785k, max=384785k, avg=384784507.00, stdev= 0.00 lat (nsec): min=429489k, max=429489k, avg=429488918.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.44%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=88.5MiB/s (92.8MB/s), 88.5MiB/s-88.5MiB/s (92.8MB/s-92.8MB/s), io=40.0MiB (41.9MB), run=452-452msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout