== sanity test 398c: run fio to test AIO ================= 04:27:25 (1787387245) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.796287 s, 52.7 MB/s osc.lustre-OST0000-osc-ffff8b4ec7201800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146002: Sat Aug 22 04:31:11 2026 write: IOPS=12, BW=48.3KiB/s (49.4kB/s)(10.0MiB/212153msec); 0 zone resets slat (usec): min=50, max=25955, avg=396.95, stdev=1446.16 clat (msec): min=257, max=3655, avg=1324.40, stdev=452.90 lat (msec): min=257, max=3655, avg=1324.80, stdev=452.90 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 693], 10.00th=[ 827], 20.00th=[ 986], | 30.00th=[ 1083], 40.00th=[ 1183], 50.00th=[ 1267], 60.00th=[ 1385], | 70.00th=[ 1519], 80.00th=[ 1636], 90.00th=[ 1871], 95.00th=[ 2123], | 99.00th=[ 2769], 99.50th=[ 3004], 99.90th=[ 3608], 99.95th=[ 3608], | 99.99th=[ 3641] bw ( KiB/s): min= 7, max= 254, per=25.41%, avg=49.03, stdev=29.41, samples=409 iops : min= 1, max= 63, avg=11.75, stdev= 7.37, samples=409 lat (msec) : 500=2.66%, 750=4.14%, 1000=15.62%, 2000=70.35%, >=2000=7.23% cpu : usr=0.04%, sys=0.25%, ctx=1842, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146003: Sat Aug 22 04:31:11 2026 write: IOPS=12, BW=48.3KiB/s (49.5kB/s)(10.0MiB/211866msec); 0 zone resets slat (usec): min=50, max=21976, avg=344.31, stdev=1060.28 clat (msec): min=86, max=3716, avg=1322.70, stdev=451.57 lat (msec): min=87, max=3716, avg=1323.04, stdev=451.53 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 676], 10.00th=[ 818], 20.00th=[ 978], | 30.00th=[ 1083], 40.00th=[ 1183], 50.00th=[ 1284], 60.00th=[ 1385], | 70.00th=[ 1502], 80.00th=[ 1636], 90.00th=[ 1871], 95.00th=[ 2106], | 99.00th=[ 2735], 99.50th=[ 2869], 99.90th=[ 3574], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 7, max= 254, per=25.10%, avg=48.43, stdev=26.65, samples=414 iops : min= 1, max= 63, avg=11.59, stdev= 6.67, samples=414 lat (msec) : 100=0.04%, 250=0.04%, 500=2.70%, 750=4.45%, 1000=15.04% lat (msec) : 2000=70.74%, >=2000=6.99% cpu : usr=0.04%, sys=0.24%, ctx=1859, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146004: Sat Aug 22 04:31:11 2026 write: IOPS=12, BW=48.3KiB/s (49.4kB/s)(10.0MiB/212155msec); 0 zone resets slat (usec): min=37, max=15482, avg=379.85, stdev=1075.93 clat (msec): min=277, max=3656, avg=1324.77, stdev=455.83 lat (msec): min=277, max=3656, avg=1325.15, stdev=455.77 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 693], 10.00th=[ 827], 20.00th=[ 978], | 30.00th=[ 1083], 40.00th=[ 1183], 50.00th=[ 1284], 60.00th=[ 1385], | 70.00th=[ 1502], 80.00th=[ 1636], 90.00th=[ 1888], 95.00th=[ 2123], | 99.00th=[ 2702], 99.50th=[ 2970], 99.90th=[ 3641], 99.95th=[ 3641], | 99.99th=[ 3641] bw ( KiB/s): min= 7, max= 254, per=25.41%, avg=49.04, stdev=28.76, samples=409 iops : min= 1, max= 63, avg=11.75, stdev= 7.19, samples=409 lat (msec) : 500=2.54%, 750=5.43%, 1000=14.14%, 2000=70.62%, >=2000=7.27% cpu : usr=0.03%, sys=0.25%, ctx=2012, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146005: Sat Aug 22 04:31:11 2026 write: IOPS=12, BW=48.4KiB/s (49.5kB/s)(10.0MiB/211712msec); 0 zone resets slat (usec): min=47, max=23164, avg=354.41, stdev=1100.30 clat (msec): min=100, max=3463, avg=1322.20, stdev=451.57 lat (msec): min=100, max=3463, avg=1322.57, stdev=451.55 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 625], 10.00th=[ 818], 20.00th=[ 978], | 30.00th=[ 1083], 40.00th=[ 1183], 50.00th=[ 1267], 60.00th=[ 1401], | 70.00th=[ 1519], 80.00th=[ 1653], 90.00th=[ 1871], 95.00th=[ 2106], | 99.00th=[ 2702], 99.50th=[ 2970], 99.90th=[ 3440], 99.95th=[ 3440], | 99.99th=[ 3473] bw ( KiB/s): min= 7, max= 191, per=25.34%, avg=48.90, stdev=28.27, samples=410 iops : min= 1, max= 47, avg=11.71, stdev= 7.10, samples=410 lat (msec) : 250=0.55%, 500=2.27%, 750=4.96%, 1000=14.92%, 2000=70.43% lat (msec) : >=2000=6.88% cpu : usr=0.04%, sys=0.24%, ctx=1940, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=193KiB/s (198kB/s), 48.3KiB/s-48.4KiB/s (49.4kB/s-49.5kB/s), io=40.0MiB (41.9MB), run=211712-212155msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146045: Sat Aug 22 04:33:27 2026 read: IOPS=10, BW=40.1KiB/s (41.1kB/s)(5048KiB/125799msec) slat (usec): min=3532, max=73318, avg=16131.59, stdev=8998.30 clat (msec): min=13, max=943, avg=494.95, stdev=128.07 lat (msec): min=32, max=982, avg=511.09, stdev=130.08 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 317], 10.00th=[ 351], 20.00th=[ 397], | 30.00th=[ 426], 40.00th=[ 451], 50.00th=[ 481], 60.00th=[ 510], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 860], 99.50th=[ 919], 99.90th=[ 936], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 103, per=25.17%, avg=40.02, stdev=17.70, samples=246 iops : min= 1, max= 25, avg= 9.47, stdev= 4.48, samples=246 write: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5192KiB/125799msec); 0 zone resets slat (usec): min=55, max=29011, avg=437.15, stdev=1336.47 clat (msec): min=215, max=1997, avg=1050.62, stdev=270.30 lat (msec): min=215, max=1997, avg=1051.06, stdev=270.30 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 617], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1536], | 99.00th=[ 1770], 99.50th=[ 1838], 99.90th=[ 1938], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 96, per=24.59%, avg=40.57, stdev=17.38, samples=250 iops : min= 1, max= 24, avg= 9.60, stdev= 4.39, samples=250 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=0.39%, 500=27.85% lat (msec) : 750=25.51%, 1000=17.50%, 2000=28.48% cpu : usr=0.10%, sys=0.73%, ctx=4790, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146046: Sat Aug 22 04:33:27 2026 read: IOPS=9, BW=39.6KiB/s (40.6kB/s)(4976KiB/125584msec) slat (usec): min=3199, max=58981, avg=16401.49, stdev=8809.61 clat (msec): min=14, max=952, avg=493.88, stdev=128.03 lat (msec): min=32, max=971, avg=510.29, stdev=129.25 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 317], 10.00th=[ 351], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 451], 50.00th=[ 485], 60.00th=[ 518], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 835], 99.50th=[ 852], 99.90th=[ 927], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 111, per=24.68%, avg=39.23, stdev=19.88, samples=247 iops : min= 1, max= 27, avg= 9.27, stdev= 5.01, samples=247 write: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5264KiB/125584msec); 0 zone resets slat (usec): min=43, max=15485, avg=435.45, stdev=1178.07 clat (msec): min=232, max=2015, avg=1041.81, stdev=277.70 lat (msec): min=232, max=2016, avg=1042.25, stdev=277.71 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 617], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1401], 95.00th=[ 1536], | 99.00th=[ 1770], 99.50th=[ 1854], 99.90th=[ 2005], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 119, per=25.28%, avg=41.70, stdev=19.21, samples=247 iops : min= 1, max= 29, avg= 9.89, stdev= 4.85, samples=247 lat (msec) : 20=0.04%, 50=0.12%, 100=0.20%, 250=0.51%, 500=26.17% lat (msec) : 750=26.60%, 1000=18.12%, 2000=28.16%, >=2000=0.08% cpu : usr=0.09%, sys=0.67%, ctx=4764, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146047: Sat Aug 22 04:33:27 2026 read: IOPS=9, BW=39.6KiB/s (40.5kB/s)(4984KiB/125867msec) slat (usec): min=3175, max=66321, avg=16214.45, stdev=8716.66 clat (msec): min=29, max=938, avg=493.98, stdev=123.98 lat (msec): min=45, max=951, avg=510.20, stdev=125.62 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 321], 10.00th=[ 355], 20.00th=[ 393], | 30.00th=[ 422], 40.00th=[ 451], 50.00th=[ 481], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 844], 99.50th=[ 869], 99.90th=[ 927], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 103, per=24.86%, avg=39.52, stdev=19.21, samples=247 iops : min= 1, max= 25, avg= 9.34, stdev= 4.85, samples=247 write: IOPS=10, BW=41.8KiB/s (42.8kB/s)(5256KiB/125867msec); 0 zone resets slat (usec): min=57, max=27887, avg=432.08, stdev=1341.07 clat (msec): min=223, max=1936, avg=1045.74, stdev=276.10 lat (msec): min=224, max=1936, avg=1046.17, stdev=276.12 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 600], 10.00th=[ 693], 20.00th=[ 818], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1267], 90.00th=[ 1401], 95.00th=[ 1536], | 99.00th=[ 1754], 99.50th=[ 1838], 99.90th=[ 1921], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 112, per=24.90%, avg=41.09, stdev=19.73, samples=249 iops : min= 1, max= 28, avg= 9.73, stdev= 4.95, samples=249 lat (msec) : 50=0.08%, 100=0.12%, 250=0.35%, 500=27.73%, 750=25.23% lat (msec) : 1000=17.81%, 2000=28.67% cpu : usr=0.09%, sys=0.65%, ctx=4737, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146048: Sat Aug 22 04:33:27 2026 read: IOPS=10, BW=40.5KiB/s (41.5kB/s)(5060KiB/124957msec) slat (usec): min=89, max=61860, avg=16234.37, stdev=8943.49 clat (msec): min=16, max=1176, avg=491.77, stdev=129.71 lat (msec): min=17, max=1181, avg=508.02, stdev=131.49 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 313], 10.00th=[ 351], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 451], 50.00th=[ 481], 60.00th=[ 510], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 659], 95.00th=[ 718], | 99.00th=[ 860], 99.50th=[ 894], 99.90th=[ 1150], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 112, per=25.32%, avg=40.26, stdev=19.04, samples=246 iops : min= 1, max= 28, avg= 9.52, stdev= 4.80, samples=246 write: IOPS=10, BW=41.5KiB/s (42.4kB/s)(5180KiB/124957msec); 0 zone resets slat (usec): min=47, max=14962, avg=402.90, stdev=1014.75 clat (msec): min=146, max=1998, avg=1046.38, stdev=278.64 lat (msec): min=146, max=2000, avg=1046.79, stdev=278.64 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 634], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1552], | 99.00th=[ 1804], 99.50th=[ 1821], 99.90th=[ 1989], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 103, per=24.82%, avg=40.96, stdev=19.06, samples=246 iops : min= 1, max= 25, avg= 9.70, stdev= 4.76, samples=246 lat (msec) : 20=0.08%, 50=0.20%, 100=0.12%, 250=0.82%, 500=27.34% lat (msec) : 750=25.43%, 1000=18.55%, 2000=27.46% cpu : usr=0.12%, sys=0.62%, ctx=4951, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=159KiB/s (163kB/s), 39.6KiB/s-40.5KiB/s (40.5kB/s-41.5kB/s), io=19.6MiB (20.5MB), run=124957-125867msec WRITE: bw=166KiB/s (170kB/s), 41.3KiB/s-41.9KiB/s (42.3kB/s-42.9kB/s), io=20.4MiB (21.4MB), run=124957-125867msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146071: Sat Aug 22 04:33:38 2026 read: IOPS=1, BW=67.8MiB/s (71.1MB/s)(40.0MiB/590msec) slat (nsec): min=38128k, max=38128k, avg=38128412.00, stdev= 0.00 clat (nsec): min=510858k, max=510858k, avg=510857572.00, stdev= 0.00 lat (nsec): min=548991k, max=548991k, avg=548990939.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] lat (msec) : 750=100.00% cpu : usr=0.17%, sys=2.21%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.8MiB/s (71.1MB/s), 67.8MiB/s-67.8MiB/s (71.1MB/s-71.1MB/s), io=40.0MiB (41.9MB), run=590-590msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout