== sanity test 398c: run fio to test AIO ================= 02:12:58 (1789452778) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.44194 s, 94.9 MB/s osc.lustre-OST0000-osc-ffffa018d3cb9800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146982: Tue Sep 15 02:15:42 2026 write: IOPS=16, BW=66.3KiB/s (67.9kB/s)(10.0MiB/154427msec); 0 zone resets slat (usec): min=27, max=16244, avg=233.15, stdev=776.11 clat (msec): min=333, max=2004, avg=963.93, stdev=239.36 lat (msec): min=333, max=2004, avg=964.17, stdev=239.37 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 642], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1687], 99.50th=[ 1905], 99.90th=[ 1938], 99.95th=[ 1955], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 128, per=25.08%, avg=66.45, stdev=26.62, samples=303 iops : min= 1, max= 32, avg=16.11, stdev= 6.68, samples=303 lat (msec) : 500=1.99%, 750=14.73%, 1000=43.55%, 2000=39.69%, >=2000=0.04% cpu : usr=0.04%, sys=0.22%, ctx=1571, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146983: Tue Sep 15 02:15:42 2026 write: IOPS=16, BW=66.4KiB/s (67.0kB/s)(10.0MiB/154265msec); 0 zone resets slat (usec): min=28, max=15225, avg=220.75, stdev=772.67 clat (msec): min=192, max=2029, avg=963.03, stdev=243.55 lat (msec): min=192, max=2029, avg=963.25, stdev=243.56 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 625], 10.00th=[ 701], 20.00th=[ 776], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1737], 99.50th=[ 1921], 99.90th=[ 1955], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 128, per=25.16%, avg=66.67, stdev=28.72, samples=302 iops : min= 1, max= 32, avg=16.17, stdev= 7.22, samples=302 lat (msec) : 250=0.62%, 500=1.17%, 750=14.45%, 1000=43.52%, 2000=40.16% lat (msec) : >=2000=0.08% cpu : usr=0.05%, sys=0.22%, ctx=1720, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146984: Tue Sep 15 02:15:42 2026 write: IOPS=16, BW=66.5KiB/s (68.1kB/s)(10.0MiB/153999msec); 0 zone resets slat (usec): min=24, max=12078, avg=252.14, stdev=912.23 clat (msec): min=220, max=2090, avg=961.59, stdev=242.39 lat (msec): min=221, max=2090, avg=961.84, stdev=242.40 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 776], | 30.00th=[ 827], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1670], 99.50th=[ 1871], 99.90th=[ 2022], 99.95th=[ 2039], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=25.24%, avg=66.89, stdev=31.91, samples=301 iops : min= 1, max= 32, avg=16.22, stdev= 7.97, samples=301 lat (msec) : 250=0.62%, 500=1.60%, 750=13.98%, 1000=43.83%, 2000=39.84% lat (msec) : >=2000=0.12% cpu : usr=0.05%, sys=0.23%, ctx=1743, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146985: Tue Sep 15 02:15:42 2026 write: IOPS=16, BW=66.5KiB/s (68.0kB/s)(10.0MiB/154093msec); 0 zone resets slat (usec): min=29, max=9616, avg=211.32, stdev=646.78 clat (msec): min=71, max=2042, avg=962.42, stdev=244.03 lat (msec): min=71, max=2043, avg=962.63, stdev=243.99 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 625], 10.00th=[ 701], 20.00th=[ 776], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1653], 99.50th=[ 1838], 99.90th=[ 1972], 99.95th=[ 2005], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 192, per=24.82%, avg=65.77, stdev=31.44, samples=306 iops : min= 1, max= 48, avg=15.94, stdev= 7.90, samples=306 lat (msec) : 100=0.20%, 250=0.27%, 500=1.68%, 750=14.10%, 1000=42.54% lat (msec) : 2000=41.13%, >=2000=0.08% cpu : usr=0.05%, sys=0.23%, ctx=1784, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=265KiB/s (272kB/s), 66.3KiB/s-66.5KiB/s (67.9kB/s-68.1kB/s), io=40.0MiB (41.9MB), run=153999-154427msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147019: Tue Sep 15 02:17:32 2026 read: IOPS=12, BW=49.6KiB/s (50.8kB/s)(5048KiB/101839msec) slat (usec): min=34, max=60780, avg=13234.89, stdev=6822.48 clat (msec): min=29, max=1612, avg=395.21, stdev=113.45 lat (msec): min=38, max=1620, avg=408.45, stdev=114.04 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 266], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 388], 60.00th=[ 405], | 70.00th=[ 426], 80.00th=[ 456], 90.00th=[ 502], 95.00th=[ 542], | 99.00th=[ 659], 99.50th=[ 693], 99.90th=[ 1469], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 8, max= 104, per=25.20%, avg=49.38, stdev=21.27, samples=201 iops : min= 2, max= 26, avg=11.90, stdev= 5.38, samples=201 write: IOPS=12, BW=50.0KiB/s (52.2kB/s)(5192KiB/101839msec); 0 zone resets slat (usec): min=34, max=8588, avg=245.54, stdev=620.15 clat (msec): min=212, max=2069, avg=856.62, stdev=206.47 lat (msec): min=212, max=2069, avg=856.86, stdev=206.49 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 558], 10.00th=[ 617], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1133], 95.00th=[ 1183], | 99.00th=[ 1368], 99.50th=[ 1469], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 96, per=24.96%, avg=50.93, stdev=18.79, samples=200 iops : min= 1, max= 24, avg=12.29, stdev= 4.67, samples=200 lat (msec) : 50=0.08%, 100=0.27%, 250=1.33%, 500=43.83%, 750=19.45% lat (msec) : 1000=23.24%, 2000=11.72%, >=2000=0.08% cpu : usr=0.09%, sys=0.64%, ctx=4289, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147020: Tue Sep 15 02:17:32 2026 read: IOPS=12, BW=48.6KiB/s (49.8kB/s)(4976KiB/102384msec) slat (usec): min=3012, max=56939, avg=13530.56, stdev=6823.01 clat (msec): min=22, max=963, avg=397.58, stdev=89.16 lat (msec): min=34, max=976, avg=411.12, stdev=89.57 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 330], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 409], | 70.00th=[ 430], 80.00th=[ 464], 90.00th=[ 514], 95.00th=[ 550], | 99.00th=[ 617], 99.50th=[ 667], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 111, per=24.64%, avg=48.30, stdev=21.60, samples=202 iops : min= 1, max= 27, avg=11.63, stdev= 5.41, samples=202 write: IOPS=12, BW=51.4KiB/s (52.6kB/s)(5264KiB/102384msec); 0 zone resets slat (usec): min=29, max=10035, avg=263.02, stdev=687.46 clat (msec): min=355, max=1416, avg=853.34, stdev=188.13 lat (msec): min=356, max=1417, avg=853.60, stdev=188.14 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 693], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1116], 95.00th=[ 1183], | 99.00th=[ 1250], 99.50th=[ 1301], 99.90th=[ 1401], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 111, per=25.08%, avg=51.17, stdev=20.43, samples=202 iops : min= 1, max= 27, avg=12.35, stdev= 5.12, samples=202 lat (msec) : 50=0.04%, 100=0.23%, 250=0.82%, 500=42.77%, 750=20.90% lat (msec) : 1000=23.24%, 2000=11.99% cpu : usr=0.08%, sys=0.64%, ctx=3973, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147021: Tue Sep 15 02:17:32 2026 read: IOPS=12, BW=48.7KiB/s (49.8kB/s)(4984KiB/102383msec) slat (msec): min=2, max=424, avg=13.88, stdev=16.93 clat (msec): min=25, max=1325, avg=398.39, stdev=101.23 lat (msec): min=32, max=1331, avg=412.28, stdev=102.15 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 330], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 409], | 70.00th=[ 426], 80.00th=[ 460], 90.00th=[ 502], 95.00th=[ 542], | 99.00th=[ 667], 99.50th=[ 961], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 118, per=24.75%, avg=48.51, stdev=21.96, samples=202 iops : min= 1, max= 29, avg=11.69, stdev= 5.49, samples=202 write: IOPS=12, BW=51.3KiB/s (52.6kB/s)(5256KiB/102383msec); 0 zone resets slat (usec): min=31, max=25910, avg=293.93, stdev=1229.72 clat (msec): min=121, max=1470, avg=853.06, stdev=201.15 lat (msec): min=122, max=1471, avg=853.36, stdev=201.11 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 802], 50.00th=[ 852], 60.00th=[ 894], | 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1133], 95.00th=[ 1200], | 99.00th=[ 1284], 99.50th=[ 1334], 99.90th=[ 1368], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 112, per=24.90%, avg=50.79, stdev=20.24, samples=202 iops : min= 1, max= 28, avg=12.25, stdev= 5.10, samples=202 lat (msec) : 50=0.12%, 100=0.12%, 250=1.17%, 500=44.14%, 750=18.87% lat (msec) : 1000=23.24%, 2000=12.34% cpu : usr=0.10%, sys=0.63%, ctx=4348, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147022: Tue Sep 15 02:17:32 2026 read: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5060KiB/101884msec) slat (usec): min=2859, max=46141, avg=13360.03, stdev=6924.82 clat (msec): min=23, max=708, avg=398.00, stdev=83.95 lat (msec): min=37, max=715, avg=411.37, stdev=84.25 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 359], 40.00th=[ 376], 50.00th=[ 393], 60.00th=[ 409], | 70.00th=[ 430], 80.00th=[ 464], 90.00th=[ 510], 95.00th=[ 550], | 99.00th=[ 625], 99.50th=[ 642], 99.90th=[ 701], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7, max= 118, per=25.28%, avg=49.56, stdev=20.85, samples=201 iops : min= 1, max= 29, avg=11.95, stdev= 5.28, samples=201 write: IOPS=12, BW=50.8KiB/s (52.1kB/s)(5180KiB/101884msec); 0 zone resets slat (usec): min=33, max=23275, avg=301.95, stdev=1128.03 clat (msec): min=354, max=1542, avg=854.07, stdev=196.19 lat (msec): min=354, max=1542, avg=854.37, stdev=196.16 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 550], 10.00th=[ 617], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 953], 80.00th=[ 1020], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1334], 99.50th=[ 1452], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 8, max= 119, per=24.70%, avg=50.39, stdev=21.08, samples=201 iops : min= 2, max= 29, avg=12.16, stdev= 5.26, samples=201 lat (msec) : 50=0.08%, 100=0.16%, 250=1.02%, 500=43.36%, 750=21.29% lat (msec) : 1000=22.50%, 2000=11.60% cpu : usr=0.11%, sys=0.58%, ctx=3676, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=196KiB/s (201kB/s), 48.6KiB/s-49.7KiB/s (49.8kB/s-50.9kB/s), io=19.6MiB (20.5MB), run=101839-102384msec WRITE: bw=204KiB/s (209kB/s), 50.8KiB/s-51.4KiB/s (52.1kB/s-52.6kB/s), io=20.4MiB (21.4MB), run=101839-102384msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147041: Tue Sep 15 02:17:43 2026 read: IOPS=1, BW=79.7MiB/s (83.6MB/s)(40.0MiB/502msec) slat (nsec): min=35264k, max=35264k, avg=35264251.00, stdev= 0.00 clat (nsec): min=445095k, max=445095k, avg=445094747.00, stdev= 0.00 lat (nsec): min=480364k, max=480364k, avg=480364498.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=100.00% cpu : usr=0.20%, sys=2.59%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=79.7MiB/s (83.6MB/s), 79.7MiB/s-79.7MiB/s (83.6MB/s-83.6MB/s), io=40.0MiB (41.9MB), run=502-502msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout