== sanity test 398c: run fio to test AIO ================= 11:20:37 (1788276037) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.664344 s, 63.1 MB/s osc.lustre-OST0000-osc-ffff96664c00f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146481: Tue Sep 1 11:23:52 2026 write: IOPS=13, BW=55.9KiB/s (57.3kB/s)(10.0MiB/183148msec); 0 zone resets slat (usec): min=23, max=36334, avg=191.29, stdev=1158.43 clat (msec): min=198, max=2812, avg=1143.96, stdev=429.77 lat (msec): min=198, max=2812, avg=1144.16, stdev=429.77 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 558], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1053], 60.00th=[ 1150], | 70.00th=[ 1284], 80.00th=[ 1469], 90.00th=[ 1737], 95.00th=[ 1989], | 99.00th=[ 2400], 99.50th=[ 2534], 99.90th=[ 2769], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 255, per=25.60%, avg=57.09, stdev=30.79, samples=352 iops : min= 1, max= 63, avg=13.76, stdev= 7.73, samples=352 lat (msec) : 250=0.62%, 500=3.24%, 750=8.24%, 1000=31.64%, 2000=51.37% lat (msec) : >=2000=4.88% cpu : usr=0.04%, sys=0.19%, ctx=1663, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146482: Tue Sep 1 11:23:52 2026 write: IOPS=13, BW=55.8KiB/s (57.1kB/s)(10.0MiB/183513msec); 0 zone resets slat (usec): min=23, max=15466, avg=182.32, stdev=835.81 clat (msec): min=298, max=2912, avg=1145.65, stdev=432.03 lat (msec): min=298, max=2915, avg=1145.84, stdev=432.03 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 550], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1150], | 70.00th=[ 1284], 80.00th=[ 1469], 90.00th=[ 1720], 95.00th=[ 1955], | 99.00th=[ 2567], 99.50th=[ 2735], 99.90th=[ 2869], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 128, per=26.21%, avg=58.45, stdev=32.20, samples=344 iops : min= 1, max= 32, avg=14.09, stdev= 8.06, samples=344 lat (msec) : 500=4.26%, 750=6.99%, 1000=34.06%, 2000=50.31%, >=2000=4.38% cpu : usr=0.05%, sys=0.21%, ctx=1927, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146483: Tue Sep 1 11:23:52 2026 write: IOPS=13, BW=55.8KiB/s (57.2kB/s)(10.0MiB/183437msec); 0 zone resets slat (usec): min=24, max=26928, avg=169.54, stdev=751.30 clat (msec): min=107, max=2816, avg=1145.05, stdev=435.05 lat (msec): min=107, max=2816, avg=1145.22, stdev=435.04 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 558], 10.00th=[ 709], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1045], 60.00th=[ 1167], | 70.00th=[ 1301], 80.00th=[ 1485], 90.00th=[ 1703], 95.00th=[ 1955], | 99.00th=[ 2567], 99.50th=[ 2702], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 174, per=26.13%, avg=58.28, stdev=30.48, samples=345 iops : min= 1, max= 43, avg=14.05, stdev= 7.62, samples=345 lat (msec) : 250=0.23%, 500=3.98%, 750=7.70%, 1000=33.20%, 2000=50.74% lat (msec) : >=2000=4.14% cpu : usr=0.04%, sys=0.18%, ctx=1872, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146484: Tue Sep 1 11:23:52 2026 write: IOPS=13, BW=55.9KiB/s (57.2kB/s)(10.0MiB/183302msec); 0 zone resets slat (usec): min=22, max=12669, avg=157.59, stdev=598.56 clat (msec): min=116, max=2770, avg=1144.55, stdev=431.24 lat (msec): min=116, max=2770, avg=1144.71, stdev=431.25 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 535], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 969], 50.00th=[ 1053], 60.00th=[ 1167], | 70.00th=[ 1284], 80.00th=[ 1469], 90.00th=[ 1703], 95.00th=[ 1989], | 99.00th=[ 2433], 99.50th=[ 2635], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 254, per=26.60%, avg=59.32, stdev=30.41, samples=339 iops : min= 1, max= 63, avg=14.31, stdev= 7.63, samples=339 lat (msec) : 250=0.66%, 500=3.36%, 750=7.89%, 1000=32.19%, 2000=51.05% lat (msec) : >=2000=4.84% cpu : usr=0.03%, sys=0.18%, ctx=1565, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=223KiB/s (229kB/s), 55.8KiB/s-55.9KiB/s (57.1kB/s-57.3kB/s), io=40.0MiB (41.9MB), run=183148-183513msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146520: Tue Sep 1 11:26:00 2026 read: IOPS=10, BW=43.3KiB/s (44.3kB/s)(5048KiB/116606msec) slat (usec): min=3031, max=64814, avg=15125.58, stdev=9214.37 clat (msec): min=13, max=1110, avg=448.09, stdev=149.36 lat (msec): min=21, max=1138, avg=463.22, stdev=151.53 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 502], 80.00th=[ 567], 90.00th=[ 659], 95.00th=[ 735], | 99.00th=[ 877], 99.50th=[ 936], 99.90th=[ 961], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 143, per=25.24%, avg=43.42, stdev=23.41, samples=228 iops : min= 1, max= 35, avg=10.37, stdev= 5.90, samples=228 write: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5192KiB/116606msec); 0 zone resets slat (usec): min=24, max=41469, avg=290.29, stdev=1662.83 clat (msec): min=265, max=2352, avg=983.58, stdev=361.06 lat (msec): min=265, max=2352, avg=983.87, stdev=361.06 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 502], 10.00th=[ 575], 20.00th=[ 667], | 30.00th=[ 760], 40.00th=[ 835], 50.00th=[ 936], 60.00th=[ 1028], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 2089], 99.50th=[ 2165], 99.90th=[ 2265], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 104, per=24.63%, avg=44.09, stdev=21.54, samples=230 iops : min= 1, max= 26, avg=10.54, stdev= 5.44, samples=230 lat (msec) : 20=0.04%, 100=0.12%, 250=1.91%, 500=35.04%, 750=24.77% lat (msec) : 1000=16.09%, 2000=21.13%, >=2000=0.90% cpu : usr=0.12%, sys=0.60%, ctx=4549, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146521: Tue Sep 1 11:26:00 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(4976KiB/116480msec) slat (usec): min=60, max=65148, avg=14953.88, stdev=9399.98 clat (msec): min=14, max=1272, avg=448.19, stdev=153.65 lat (msec): min=14, max=1291, avg=463.15, stdev=156.01 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 257], 10.00th=[ 292], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 388], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 489], 80.00th=[ 567], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 902], 99.50th=[ 969], 99.90th=[ 1083], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 7, max= 125, per=25.15%, avg=43.26, stdev=23.00, samples=225 iops : min= 1, max= 31, avg=10.33, stdev= 5.78, samples=225 write: IOPS=11, BW=45.2KiB/s (46.3kB/s)(5264KiB/116480msec); 0 zone resets slat (usec): min=27, max=34295, avg=271.24, stdev=1341.77 clat (msec): min=139, max=2359, avg=976.93, stdev=369.39 lat (msec): min=140, max=2359, avg=977.21, stdev=369.42 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 477], 10.00th=[ 558], 20.00th=[ 659], | 30.00th=[ 760], 40.00th=[ 844], 50.00th=[ 936], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1250], 90.00th=[ 1469], 95.00th=[ 1653], | 99.00th=[ 2089], 99.50th=[ 2165], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 112, per=25.26%, avg=45.21, stdev=22.90, samples=228 iops : min= 1, max= 28, avg=10.82, stdev= 5.78, samples=228 lat (msec) : 20=0.08%, 50=0.16%, 100=0.27%, 250=1.64%, 500=35.74% lat (msec) : 750=23.48%, 1000=16.48%, 2000=21.17%, >=2000=0.98% cpu : usr=0.09%, sys=0.58%, ctx=4795, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146522: Tue Sep 1 11:26:00 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(4984KiB/116655msec) slat (msec): min=2, max=103, avg=15.56, stdev= 9.93 clat (msec): min=23, max=1110, avg=448.62, stdev=155.80 lat (msec): min=35, max=1129, avg=464.19, stdev=158.15 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 255], 10.00th=[ 284], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 489], 80.00th=[ 567], 90.00th=[ 676], 95.00th=[ 743], | 99.00th=[ 936], 99.50th=[ 1020], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 143, per=25.45%, avg=43.77, stdev=23.53, samples=224 iops : min= 1, max= 35, avg=10.46, stdev= 5.93, samples=224 write: IOPS=11, BW=45.1KiB/s (46.1kB/s)(5256KiB/116655msec); 0 zone resets slat (usec): min=29, max=16587, avg=228.40, stdev=831.10 clat (msec): min=263, max=2333, avg=977.52, stdev=359.09 lat (msec): min=264, max=2333, avg=977.76, stdev=359.12 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 485], 10.00th=[ 575], 20.00th=[ 676], | 30.00th=[ 760], 40.00th=[ 844], 50.00th=[ 927], 60.00th=[ 1020], | 70.00th=[ 1116], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 2039], 99.50th=[ 2232], 99.90th=[ 2299], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 112, per=24.87%, avg=44.51, stdev=22.99, samples=230 iops : min= 1, max= 28, avg=10.64, stdev= 5.79, samples=230 lat (msec) : 50=0.08%, 100=0.16%, 250=1.91%, 500=35.74%, 750=23.32% lat (msec) : 1000=17.19%, 2000=20.90%, >=2000=0.70% cpu : usr=0.09%, sys=0.57%, ctx=4785, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146523: Tue Sep 1 11:26:00 2026 read: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5060KiB/116658msec) slat (usec): min=2456, max=76113, avg=14971.89, stdev=8838.49 clat (msec): min=33, max=1131, avg=445.43, stdev=148.35 lat (msec): min=38, max=1149, avg=460.41, stdev=150.13 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 262], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 443], | 70.00th=[ 485], 80.00th=[ 550], 90.00th=[ 659], 95.00th=[ 735], | 99.00th=[ 927], 99.50th=[ 969], 99.90th=[ 1070], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 112, per=25.66%, avg=44.13, stdev=22.33, samples=225 iops : min= 1, max= 28, avg=10.56, stdev= 5.66, samples=225 write: IOPS=11, BW=44.4KiB/s (45.5kB/s)(5180KiB/116658msec); 0 zone resets slat (usec): min=34, max=30550, avg=290.20, stdev=1552.90 clat (msec): min=278, max=2334, avg=987.89, stdev=367.55 lat (msec): min=278, max=2334, avg=988.18, stdev=367.53 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 485], 10.00th=[ 575], 20.00th=[ 667], | 30.00th=[ 768], 40.00th=[ 835], 50.00th=[ 927], 60.00th=[ 1028], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1485], 95.00th=[ 1670], | 99.00th=[ 2089], 99.50th=[ 2123], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 120, per=24.47%, avg=43.80, stdev=22.30, samples=231 iops : min= 1, max= 30, avg=10.47, stdev= 5.62, samples=231 lat (msec) : 50=0.08%, 100=0.20%, 250=1.72%, 500=36.37%, 750=23.59% lat (msec) : 1000=16.02%, 2000=21.09%, >=2000=0.94% cpu : usr=0.12%, sys=0.59%, ctx=4841, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172KiB/s (176kB/s), 42.7KiB/s-43.4KiB/s (43.7kB/s-44.4kB/s), io=19.6MiB (20.5MB), run=116480-116658msec WRITE: bw=179KiB/s (183kB/s), 44.4KiB/s-45.2KiB/s (45.5kB/s-46.3kB/s), io=20.4MiB (21.4MB), run=116480-116658msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146545: Tue Sep 1 11:26:13 2026 read: IOPS=2, BW=82.0MiB/s (87.0MB/s)(40.0MiB/482msec) slat (nsec): min=12570k, max=12570k, avg=12570258.00, stdev= 0.00 clat (nsec): min=452744k, max=452744k, avg=452744254.00, stdev= 0.00 lat (nsec): min=465319k, max=465319k, avg=465318536.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.08%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=82.0MiB/s (87.0MB/s), 82.0MiB/s-82.0MiB/s (87.0MB/s-87.0MB/s), io=40.0MiB (41.9MB), run=482-482msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout