== sanity test 398c: run fio to test AIO ================= 05:12:32 (1787735552) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.364853 s, 115 MB/s osc.lustre-OST0000-osc-ffff98c642fda000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=175220: Wed Aug 26 05:15:20 2026 write: IOPS=16, BW=64.7KiB/s (66.3kB/s)(10.0MiB/158223msec); 0 zone resets slat (usec): min=22, max=16426, avg=203.34, stdev=758.70 clat (msec): min=122, max=2355, avg=987.73, stdev=308.86 lat (msec): min=122, max=2355, avg=987.94, stdev=308.84 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 592], 10.00th=[ 684], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1368], 95.00th=[ 1536], | 99.00th=[ 2089], 99.50th=[ 2232], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 256, per=25.58%, avg=66.00, stdev=33.50, samples=305 iops : min= 1, max= 64, avg=16.05, stdev= 8.40, samples=305 lat (msec) : 250=0.62%, 500=2.50%, 750=14.61%, 1000=40.90%, 2000=40.12% lat (msec) : >=2000=1.25% cpu : usr=0.05%, sys=0.20%, ctx=1662, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=175221: Wed Aug 26 05:15:20 2026 write: IOPS=16, BW=64.7KiB/s (66.2kB/s)(10.0MiB/158347msec); 0 zone resets slat (usec): min=29, max=16107, avg=205.75, stdev=653.31 clat (msec): min=252, max=2411, avg=988.31, stdev=305.31 lat (msec): min=252, max=2411, avg=988.52, stdev=305.31 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 2005], 99.50th=[ 2265], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 159, per=25.19%, avg=64.99, stdev=28.23, samples=310 iops : min= 1, max= 39, avg=15.80, stdev= 7.08, samples=310 lat (msec) : 500=3.05%, 750=14.49%, 1000=40.94%, 2000=40.47%, >=2000=1.05% cpu : usr=0.05%, sys=0.22%, ctx=1690, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=175222: Wed Aug 26 05:15:20 2026 write: IOPS=16, BW=64.8KiB/s (66.4kB/s)(10.0MiB/158008msec); 0 zone resets slat (usec): min=27, max=15270, avg=225.23, stdev=876.56 clat (msec): min=174, max=2410, avg=986.87, stdev=312.33 lat (msec): min=174, max=2410, avg=987.09, stdev=312.31 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1368], 95.00th=[ 1536], | 99.00th=[ 2123], 99.50th=[ 2299], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 223, per=25.09%, avg=64.73, stdev=28.69, samples=311 iops : min= 1, max= 55, avg=15.72, stdev= 7.19, samples=311 lat (msec) : 250=0.62%, 500=2.73%, 750=15.35%, 1000=38.75%, 2000=41.17% lat (msec) : >=2000=1.37% cpu : usr=0.04%, sys=0.21%, ctx=1701, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=175223: Wed Aug 26 05:15:20 2026 write: IOPS=16, BW=64.8KiB/s (66.4kB/s)(10.0MiB/158004msec); 0 zone resets slat (usec): min=26, max=19444, avg=254.85, stdev=903.98 clat (msec): min=82, max=2402, avg=986.06, stdev=307.88 lat (msec): min=83, max=2402, avg=986.32, stdev=307.86 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 2056], 99.50th=[ 2265], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 192, per=25.02%, avg=64.56, stdev=28.08, samples=312 iops : min= 1, max= 48, avg=15.69, stdev= 7.07, samples=312 lat (msec) : 100=0.08%, 250=0.47%, 500=3.40%, 750=13.59%, 1000=40.04% lat (msec) : 2000=41.29%, >=2000=1.13% cpu : usr=0.04%, sys=0.22%, ctx=1784, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=259KiB/s (265kB/s), 64.7KiB/s-64.8KiB/s (66.2kB/s-66.4kB/s), io=40.0MiB (41.9MB), run=158004-158347msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=175257: Wed Aug 26 05:17:15 2026 read: IOPS=11, BW=47.2KiB/s (48.3kB/s)(5048KiB/106969msec) slat (usec): min=3206, max=53646, avg=13945.75, stdev=7644.63 clat (msec): min=29, max=929, avg=417.40, stdev=111.18 lat (msec): min=44, max=963, avg=431.36, stdev=112.73 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 363], 40.00th=[ 384], 50.00th=[ 405], 60.00th=[ 426], | 70.00th=[ 451], 80.00th=[ 489], 90.00th=[ 550], 95.00th=[ 609], | 99.00th=[ 810], 99.50th=[ 860], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 7, max= 136, per=24.97%, avg=46.70, stdev=21.98, samples=212 iops : min= 1, max= 34, avg=11.25, stdev= 5.52, samples=212 write: IOPS=12, BW=48.5KiB/s (49.7kB/s)(5192KiB/106969msec); 0 zone resets slat (usec): min=31, max=21322, avg=284.77, stdev=1039.81 clat (msec): min=279, max=1905, avg=896.47, stdev=255.84 lat (msec): min=279, max=1906, avg=896.76, stdev=255.85 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 693], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1368], | 99.00th=[ 1603], 99.50th=[ 1754], 99.90th=[ 1838], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 103, per=24.67%, avg=48.10, stdev=21.63, samples=212 iops : min= 1, max= 25, avg=11.60, stdev= 5.49, samples=212 lat (msec) : 50=0.12%, 100=0.16%, 250=1.25%, 500=41.21%, 750=20.35% lat (msec) : 1000=21.99%, 2000=14.92% cpu : usr=0.10%, sys=0.60%, ctx=4108, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175258: Wed Aug 26 05:17:15 2026 read: IOPS=11, BW=46.6KiB/s (47.7kB/s)(4976KiB/106874msec) slat (usec): min=36, max=63645, avg=13291.32, stdev=7052.14 clat (msec): min=16, max=950, avg=416.26, stdev=111.72 lat (msec): min=24, max=964, avg=429.56, stdev=113.11 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 338], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 401], 60.00th=[ 426], | 70.00th=[ 451], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 617], | 99.00th=[ 810], 99.50th=[ 835], 99.90th=[ 902], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 119, per=24.96%, avg=46.67, stdev=21.33, samples=209 iops : min= 1, max= 29, avg=11.24, stdev= 5.38, samples=209 write: IOPS=12, BW=49.3KiB/s (50.4kB/s)(5264KiB/106874msec); 0 zone resets slat (usec): min=27, max=11106, avg=228.75, stdev=616.68 clat (msec): min=144, max=1847, avg=892.27, stdev=264.07 lat (msec): min=144, max=1847, avg=892.51, stdev=264.07 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 489], 10.00th=[ 575], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1368], | 99.00th=[ 1687], 99.50th=[ 1720], 99.90th=[ 1838], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 136, per=25.01%, avg=48.77, stdev=21.44, samples=212 iops : min= 1, max= 34, avg=11.77, stdev= 5.43, samples=212 lat (msec) : 20=0.04%, 50=0.04%, 100=0.23%, 250=1.72%, 500=40.90% lat (msec) : 750=19.26%, 1000=23.44%, 2000=14.38% cpu : usr=0.10%, sys=0.58%, ctx=3991, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175259: Wed Aug 26 05:17:15 2026 read: IOPS=11, BW=46.5KiB/s (47.7kB/s)(4984KiB/107081msec) slat (msec): min=2, max=402, avg=14.31, stdev=15.89 clat (msec): min=13, max=1155, avg=414.83, stdev=115.26 lat (msec): min=27, max=1179, avg=429.15, stdev=117.11 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 275], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 405], 60.00th=[ 426], | 70.00th=[ 447], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 617], | 99.00th=[ 802], 99.50th=[ 894], 99.90th=[ 1062], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 128, per=24.74%, avg=46.25, stdev=22.86, samples=212 iops : min= 1, max= 32, avg=11.14, stdev= 5.75, samples=212 write: IOPS=12, BW=49.1KiB/s (50.3kB/s)(5256KiB/107081msec); 0 zone resets slat (usec): min=30, max=9778, avg=222.91, stdev=581.54 clat (msec): min=144, max=2393, avg=894.43, stdev=273.69 lat (msec): min=145, max=2393, avg=894.65, stdev=273.70 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 506], 10.00th=[ 575], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1703], 99.50th=[ 1838], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 120, per=24.90%, avg=48.55, stdev=21.82, samples=212 iops : min= 1, max= 30, avg=11.71, stdev= 5.52, samples=212 lat (msec) : 20=0.04%, 50=0.08%, 100=0.31%, 250=1.02%, 500=41.37% lat (msec) : 750=20.62%, 1000=21.60%, 2000=14.77%, >=2000=0.20% cpu : usr=0.09%, sys=0.56%, ctx=3987, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175260: Wed Aug 26 05:17:15 2026 read: IOPS=11, BW=47.5KiB/s (48.7kB/s)(5060KiB/106413msec) slat (usec): min=3178, max=62350, avg=13866.30, stdev=8023.69 clat (msec): min=21, max=950, avg=412.36, stdev=111.69 lat (msec): min=48, max=973, avg=426.23, stdev=113.41 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 271], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 401], 60.00th=[ 422], | 70.00th=[ 447], 80.00th=[ 485], 90.00th=[ 550], 95.00th=[ 617], | 99.00th=[ 776], 99.50th=[ 835], 99.90th=[ 927], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 112, per=25.24%, avg=47.19, stdev=22.06, samples=211 iops : min= 1, max= 28, avg=11.37, stdev= 5.53, samples=211 write: IOPS=12, BW=48.7KiB/s (49.8kB/s)(5180KiB/106413msec); 0 zone resets slat (usec): min=32, max=16267, avg=217.71, stdev=705.50 clat (msec): min=140, max=1858, avg=895.88, stdev=256.30 lat (msec): min=141, max=1858, avg=896.10, stdev=256.33 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 510], 10.00th=[ 584], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1653], 99.50th=[ 1720], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 111, per=24.64%, avg=48.05, stdev=21.24, samples=211 iops : min= 1, max= 27, avg=11.58, stdev= 5.38, samples=211 lat (msec) : 50=0.04%, 100=0.27%, 250=1.21%, 500=42.34%, 750=19.45% lat (msec) : 1000=21.84%, 2000=14.84% cpu : usr=0.09%, sys=0.55%, ctx=3705, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=187KiB/s (192kB/s), 46.5KiB/s-47.5KiB/s (47.7kB/s-48.7kB/s), io=19.6MiB (20.5MB), run=106413-107081msec WRITE: bw=195KiB/s (200kB/s), 48.5KiB/s-49.3KiB/s (49.7kB/s-50.4kB/s), io=20.4MiB (21.4MB), run=106413-107081msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=175279: Wed Aug 26 05:17:24 2026 read: IOPS=3, BW=127MiB/s (133MB/s)(40.0MiB/315msec) slat (nsec): min=19518k, max=19518k, avg=19518108.00, stdev= 0.00 clat (nsec): min=275108k, max=275108k, avg=275108377.00, stdev= 0.00 lat (nsec): min=294632k, max=294632k, avg=294632011.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.50%, ctx=28, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=40.0MiB (41.9MB), run=315-315msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout