== sanity test 398c: run fio to test AIO ================= 10:38:47 (1789137527) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.528311 s, 79.4 MB/s osc.lustre-OST0000-osc-ffff8d4931f8d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144666: Fri Sep 11 10:41:54 2026 write: IOPS=14, BW=59.0KiB/s (60.4kB/s)(10.0MiB/173478msec); 0 zone resets slat (usec): min=23, max=20625, avg=197.27, stdev=814.63 clat (msec): min=177, max=2704, avg=1083.45, stdev=297.41 lat (msec): min=177, max=2704, avg=1083.65, stdev=297.41 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 844], | 30.00th=[ 936], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1452], 95.00th=[ 1569], | 99.00th=[ 1737], 99.50th=[ 1804], 99.90th=[ 2567], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 144, per=25.11%, avg=58.50, stdev=26.49, samples=344 iops : min= 1, max= 36, avg=14.19, stdev= 6.68, samples=344 lat (msec) : 250=0.08%, 500=2.34%, 750=10.12%, 1000=26.09%, 2000=60.94% lat (msec) : >=2000=0.43% cpu : usr=0.05%, sys=0.19%, ctx=1912, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144667: Fri Sep 11 10:41:54 2026 write: IOPS=14, BW=58.4KiB/s (59.8kB/s)(10.0MiB/175266msec); 0 zone resets slat (usec): min=23, max=21120, avg=209.90, stdev=1004.78 clat (msec): min=316, max=2167, avg=1093.56, stdev=290.54 lat (msec): min=316, max=2167, avg=1093.78, stdev=290.55 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 634], 10.00th=[ 726], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 1770], 99.50th=[ 1821], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 128, per=25.49%, avg=59.40, stdev=28.45, samples=339 iops : min= 1, max= 32, avg=14.41, stdev= 7.16, samples=339 lat (msec) : 500=1.37%, 750=10.55%, 1000=26.99%, 2000=60.90%, >=2000=0.20% cpu : usr=0.04%, sys=0.19%, ctx=1745, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144668: Fri Sep 11 10:41:54 2026 write: IOPS=14, BW=58.5KiB/s (59.9kB/s)(10.0MiB/175125msec); 0 zone resets slat (usec): min=21, max=23891, avg=182.59, stdev=783.55 clat (msec): min=297, max=2625, avg=1092.60, stdev=289.97 lat (msec): min=297, max=2625, avg=1092.78, stdev=289.96 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 625], 10.00th=[ 743], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1569], | 99.00th=[ 1754], 99.50th=[ 1838], 99.90th=[ 2265], 99.95th=[ 2534], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 128, per=25.42%, avg=59.23, stdev=27.79, samples=340 iops : min= 1, max= 32, avg=14.36, stdev= 6.99, samples=340 lat (msec) : 500=1.25%, 750=9.49%, 1000=27.89%, 2000=61.09%, >=2000=0.27% cpu : usr=0.04%, sys=0.18%, ctx=1734, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144669: Fri Sep 11 10:41:54 2026 write: IOPS=14, BW=58.7KiB/s (60.1kB/s)(10.0MiB/174533msec); 0 zone resets slat (usec): min=24, max=27004, avg=200.37, stdev=1017.22 clat (msec): min=283, max=2211, avg=1089.17, stdev=295.72 lat (msec): min=283, max=2211, avg=1089.37, stdev=295.73 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 609], 10.00th=[ 718], 20.00th=[ 844], | 30.00th=[ 927], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1603], | 99.00th=[ 1770], 99.50th=[ 1821], 99.90th=[ 2022], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 152, per=25.18%, avg=58.67, stdev=28.64, samples=343 iops : min= 1, max= 38, avg=14.23, stdev= 7.19, samples=343 lat (msec) : 500=2.46%, 750=9.26%, 1000=27.38%, 2000=60.66%, >=2000=0.23% cpu : usr=0.04%, sys=0.21%, ctx=1701, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=234KiB/s (239kB/s), 58.4KiB/s-59.0KiB/s (59.8kB/s-60.4kB/s), io=40.0MiB (41.9MB), run=173478-175266msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144705: Fri Sep 11 10:43:48 2026 read: IOPS=12, BW=48.7KiB/s (49.9kB/s)(5048KiB/103659msec) slat (msec): min=2, max=136, avg=14.40, stdev= 9.40 clat (msec): min=16, max=870, avg=400.45, stdev=120.24 lat (msec): min=27, max=898, avg=414.86, stdev=122.50 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 205], 10.00th=[ 249], 20.00th=[ 309], | 30.00th=[ 342], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 456], 80.00th=[ 489], 90.00th=[ 542], 95.00th=[ 609], | 99.00th=[ 751], 99.50th=[ 793], 99.90th=[ 852], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 7, max= 184, per=24.98%, avg=48.20, stdev=24.29, samples=205 iops : min= 1, max= 46, avg=11.60, stdev= 6.12, samples=205 write: IOPS=12, BW=50.1KiB/s (51.3kB/s)(5192KiB/103659msec); 0 zone resets slat (usec): min=26, max=26056, avg=244.61, stdev=1047.12 clat (msec): min=215, max=1682, avg=871.50, stdev=261.49 lat (msec): min=216, max=1683, avg=871.75, stdev=261.50 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 426], 10.00th=[ 535], 20.00th=[ 642], | 30.00th=[ 726], 40.00th=[ 810], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1318], | 99.00th=[ 1536], 99.50th=[ 1569], 99.90th=[ 1653], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 120, per=24.75%, avg=49.76, stdev=21.00, samples=205 iops : min= 1, max= 30, avg=11.99, stdev= 5.26, samples=205 lat (msec) : 20=0.08%, 50=0.16%, 100=0.12%, 250=4.65%, 500=39.49% lat (msec) : 750=20.27%, 1000=19.45%, 2000=15.78% cpu : usr=0.09%, sys=0.64%, ctx=5202, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144706: Fri Sep 11 10:43:48 2026 read: IOPS=12, BW=48.0KiB/s (49.2kB/s)(4976KiB/103616msec) slat (msec): min=2, max=133, avg=14.15, stdev= 8.96 clat (msec): min=24, max=892, avg=399.50, stdev=117.40 lat (msec): min=36, max=908, avg=413.65, stdev=119.43 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 209], 10.00th=[ 247], 20.00th=[ 309], | 30.00th=[ 342], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 426], | 70.00th=[ 451], 80.00th=[ 493], 90.00th=[ 550], 95.00th=[ 600], | 99.00th=[ 718], 99.50th=[ 751], 99.90th=[ 869], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 120, per=24.87%, avg=48.00, stdev=22.67, samples=203 iops : min= 1, max= 30, avg=11.55, stdev= 5.68, samples=203 write: IOPS=12, BW=50.8KiB/s (52.0kB/s)(5264KiB/103616msec); 0 zone resets slat (usec): min=22, max=13841, avg=253.21, stdev=876.71 clat (msec): min=200, max=1702, avg=866.44, stdev=261.16 lat (msec): min=200, max=1702, avg=866.69, stdev=261.19 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 422], 10.00th=[ 523], 20.00th=[ 642], | 30.00th=[ 735], 40.00th=[ 810], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1502], 99.50th=[ 1569], 99.90th=[ 1687], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 120, per=25.07%, avg=50.39, stdev=21.40, samples=205 iops : min= 1, max= 30, avg=12.15, stdev= 5.39, samples=205 lat (msec) : 50=0.23%, 100=0.12%, 250=4.88%, 500=38.75%, 750=20.78% lat (msec) : 1000=19.65%, 2000=15.59% cpu : usr=0.11%, sys=0.62%, ctx=4752, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144707: Fri Sep 11 10:43:48 2026 read: IOPS=12, BW=48.1KiB/s (49.2kB/s)(4984KiB/103687msec) slat (usec): min=2191, max=58056, avg=13955.56, stdev=8295.03 clat (msec): min=12, max=853, avg=395.80, stdev=120.80 lat (msec): min=19, max=899, avg=409.76, stdev=122.95 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 201], 10.00th=[ 241], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 363], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 489], 90.00th=[ 550], 95.00th=[ 600], | 99.00th=[ 709], 99.50th=[ 760], 99.90th=[ 818], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 7, max= 144, per=24.78%, avg=47.82, stdev=24.49, samples=205 iops : min= 1, max= 36, avg=11.51, stdev= 6.14, samples=205 write: IOPS=12, BW=50.7KiB/s (51.9kB/s)(5256KiB/103687msec); 0 zone resets slat (usec): min=25, max=27109, avg=271.65, stdev=1224.73 clat (msec): min=152, max=1683, avg=871.87, stdev=261.76 lat (msec): min=152, max=1683, avg=872.15, stdev=261.77 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 422], 10.00th=[ 527], 20.00th=[ 659], | 30.00th=[ 735], 40.00th=[ 810], 50.00th=[ 877], 60.00th=[ 944], | 70.00th=[ 1011], 80.00th=[ 1083], 90.00th=[ 1183], 95.00th=[ 1318], | 99.00th=[ 1502], 99.50th=[ 1552], 99.90th=[ 1670], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 112, per=24.93%, avg=50.10, stdev=21.12, samples=205 iops : min= 1, max= 28, avg=12.08, stdev= 5.33, samples=205 lat (msec) : 20=0.08%, 50=0.16%, 100=0.20%, 250=5.31%, 500=38.44% lat (msec) : 750=20.51%, 1000=18.75%, 2000=16.56% cpu : usr=0.10%, sys=0.63%, ctx=4648, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144708: Fri Sep 11 10:43:48 2026 read: IOPS=12, BW=49.0KiB/s (50.2kB/s)(5060KiB/103258msec) slat (usec): min=48, max=64529, avg=14008.40, stdev=8564.28 clat (msec): min=20, max=877, avg=401.69, stdev=126.89 lat (msec): min=20, max=898, avg=415.71, stdev=129.14 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 201], 10.00th=[ 241], 20.00th=[ 300], | 30.00th=[ 338], 40.00th=[ 372], 50.00th=[ 401], 60.00th=[ 430], | 70.00th=[ 460], 80.00th=[ 498], 90.00th=[ 550], 95.00th=[ 609], | 99.00th=[ 760], 99.50th=[ 785], 99.90th=[ 852], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 136, per=25.23%, avg=48.69, stdev=24.43, samples=204 iops : min= 1, max= 34, avg=11.72, stdev= 6.17, samples=204 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(5180KiB/103258msec); 0 zone resets slat (usec): min=24, max=29412, avg=269.78, stdev=1197.33 clat (msec): min=88, max=1785, avg=868.83, stdev=269.88 lat (msec): min=88, max=1785, avg=869.10, stdev=269.93 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 405], 10.00th=[ 535], 20.00th=[ 651], | 30.00th=[ 726], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1334], | 99.00th=[ 1552], 99.50th=[ 1703], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 131, per=24.79%, avg=49.84, stdev=22.04, samples=203 iops : min= 1, max= 32, avg=12.01, stdev= 5.53, samples=203 lat (msec) : 50=0.35%, 100=0.27%, 250=5.08%, 500=38.16%, 750=21.25% lat (msec) : 1000=19.41%, 2000=15.47% cpu : usr=0.10%, sys=0.62%, ctx=4816, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=194KiB/s (198kB/s), 48.0KiB/s-49.0KiB/s (49.2kB/s-50.2kB/s), io=19.6MiB (20.5MB), run=103258-103687msec WRITE: bw=201KiB/s (206kB/s), 50.1KiB/s-50.8KiB/s (51.3kB/s-52.0kB/s), io=20.4MiB (21.4MB), run=103258-103687msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144727: Fri Sep 11 10:43:59 2026 read: IOPS=2, BW=84.4MiB/s (88.5MB/s)(40.0MiB/474msec) slat (nsec): min=34007k, max=34007k, avg=34006729.00, stdev= 0.00 clat (nsec): min=412805k, max=412805k, avg=412805475.00, stdev= 0.00 lat (nsec): min=446841k, max=446841k, avg=446841464.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.53%, ctx=9, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=84.4MiB/s (88.5MB/s), 84.4MiB/s-84.4MiB/s (88.5MB/s-88.5MB/s), io=40.0MiB (41.9MB), run=474-474msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout