== sanity test 398c: run fio to test AIO ================= 09:14:09 (1788614049) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.508783 s, 82.4 MB/s osc.lustre-OST0000-osc-ffff9e524ac0b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145543: Sat Sep 5 09:16:43 2026 write: IOPS=17, BW=71.0KiB/s (72.7kB/s)(10.0MiB/144208msec); 0 zone resets slat (usec): min=30, max=35253, avg=310.72, stdev=1108.91 clat (msec): min=188, max=1989, avg=899.74, stdev=215.24 lat (msec): min=188, max=1989, avg=900.05, stdev=215.28 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 600], 10.00th=[ 659], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 844], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 986], 80.00th=[ 1053], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1469], 99.50th=[ 1536], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 234, per=25.59%, avg=72.42, stdev=33.12, samples=278 iops : min= 1, max= 58, avg=17.57, stdev= 8.29, samples=278 lat (msec) : 250=1.25%, 500=1.95%, 750=18.20%, 1000=50.82%, 2000=27.77% cpu : usr=0.06%, sys=0.29%, ctx=1873, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145544: Sat Sep 5 09:16:43 2026 write: IOPS=17, BW=71.0KiB/s (72.8kB/s)(10.0MiB/144123msec); 0 zone resets slat (usec): min=30, max=22045, avg=328.79, stdev=1010.27 clat (msec): min=135, max=2098, avg=899.85, stdev=218.12 lat (msec): min=135, max=2107, avg=900.19, stdev=218.12 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 844], 50.00th=[ 894], 60.00th=[ 936], | 70.00th=[ 978], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1284], | 99.00th=[ 1469], 99.50th=[ 1569], 99.90th=[ 1989], 99.95th=[ 2056], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 234, per=24.86%, avg=70.36, stdev=28.05, samples=286 iops : min= 1, max= 58, avg=17.05, stdev= 7.04, samples=286 lat (msec) : 250=0.98%, 500=2.19%, 750=17.42%, 1000=51.76%, 2000=27.58% lat (msec) : >=2000=0.08% cpu : usr=0.06%, sys=0.32%, ctx=2123, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145545: Sat Sep 5 09:16:43 2026 write: IOPS=17, BW=70.0KiB/s (72.7kB/s)(10.0MiB/144326msec); 0 zone resets slat (usec): min=30, max=15081, avg=310.93, stdev=852.17 clat (msec): min=200, max=2057, avg=900.55, stdev=211.03 lat (msec): min=200, max=2058, avg=900.87, stdev=211.03 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 609], 10.00th=[ 667], 20.00th=[ 751], | 30.00th=[ 802], 40.00th=[ 844], 50.00th=[ 885], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1485], 99.50th=[ 1569], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 224, per=24.87%, avg=70.38, stdev=30.44, samples=286 iops : min= 1, max= 56, avg=17.06, stdev= 7.64, samples=286 lat (msec) : 250=0.78%, 500=1.95%, 750=16.56%, 1000=52.66%, 2000=28.01% lat (msec) : >=2000=0.04% cpu : usr=0.06%, sys=0.32%, ctx=1980, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145546: Sat Sep 5 09:16:43 2026 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144355msec); 0 zone resets slat (usec): min=28, max=16342, avg=286.75, stdev=847.50 clat (msec): min=183, max=2105, avg=900.68, stdev=212.33 lat (msec): min=183, max=2105, avg=900.97, stdev=212.36 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 609], 10.00th=[ 667], 20.00th=[ 751], | 30.00th=[ 793], 40.00th=[ 844], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1284], | 99.00th=[ 1452], 99.50th=[ 1552], 99.90th=[ 1653], 99.95th=[ 1787], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 207, per=24.87%, avg=70.39, stdev=27.78, samples=286 iops : min= 1, max= 51, avg=17.06, stdev= 7.00, samples=286 lat (msec) : 250=0.86%, 500=2.38%, 750=16.76%, 1000=51.29%, 2000=28.67% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.31%, ctx=2250, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=284KiB/s (291kB/s), 70.9KiB/s-71.0KiB/s (72.6kB/s-72.8kB/s), io=40.0MiB (41.9MB), run=144123-144355msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145578: Sat Sep 5 09:18:44 2026 read: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5048KiB/111488msec) slat (usec): min=36, max=63326, avg=15747.82, stdev=8839.57 clat (msec): min=31, max=897, avg=431.52, stdev=110.76 lat (msec): min=36, max=913, avg=447.27, stdev=112.81 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 279], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 443], | 70.00th=[ 468], 80.00th=[ 506], 90.00th=[ 584], 95.00th=[ 634], | 99.00th=[ 760], 99.50th=[ 768], 99.90th=[ 877], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 144, per=24.98%, avg=44.71, stdev=22.79, samples=221 iops : min= 1, max= 36, avg=10.65, stdev= 5.72, samples=221 write: IOPS=11, BW=46.6KiB/s (47.7kB/s)(5192KiB/111488msec); 0 zone resets slat (usec): min=36, max=22661, avg=387.50, stdev=1123.92 clat (msec): min=133, max=2022, avg=937.93, stdev=261.29 lat (msec): min=133, max=2022, avg=938.32, stdev=261.26 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1284], 95.00th=[ 1368], | 99.00th=[ 1620], 99.50th=[ 1804], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 96, per=24.74%, avg=46.27, stdev=19.80, samples=219 iops : min= 1, max= 24, avg=11.03, stdev= 5.00, samples=219 lat (msec) : 50=0.04%, 100=0.31%, 250=1.17%, 500=38.44%, 750=21.48% lat (msec) : 1000=18.79%, 2000=19.69%, >=2000=0.08% cpu : usr=0.12%, sys=0.84%, ctx=4804, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145579: Sat Sep 5 09:18:44 2026 read: IOPS=11, BW=44.6KiB/s (45.7kB/s)(4976KiB/111592msec) slat (usec): min=3033, max=62521, avg=15364.10, stdev=8119.12 clat (msec): min=24, max=785, avg=424.82, stdev=105.88 lat (msec): min=38, max=816, avg=440.20, stdev=107.49 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 342], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 443], | 70.00th=[ 464], 80.00th=[ 502], 90.00th=[ 558], 95.00th=[ 625], | 99.00th=[ 735], 99.50th=[ 751], 99.90th=[ 776], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 7, max= 128, per=24.66%, avg=44.15, stdev=22.33, samples=220 iops : min= 1, max= 32, avg=10.50, stdev= 5.61, samples=220 write: IOPS=11, BW=47.2KiB/s (48.3kB/s)(5264KiB/111592msec); 0 zone resets slat (usec): min=35, max=13596, avg=376.80, stdev=938.65 clat (msec): min=199, max=2038, avg=938.05, stdev=256.43 lat (msec): min=200, max=2038, avg=938.43, stdev=256.48 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 927], 60.00th=[ 1003], | 70.00th=[ 1070], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1670], 99.50th=[ 1787], 99.90th=[ 1921], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 112, per=24.79%, avg=46.35, stdev=20.15, samples=221 iops : min= 1, max= 28, avg=11.05, stdev= 5.04, samples=221 lat (msec) : 50=0.23%, 100=0.20%, 250=0.70%, 500=38.67%, 750=21.72% lat (msec) : 1000=17.93%, 2000=20.51%, >=2000=0.04% cpu : usr=0.14%, sys=0.78%, ctx=4969, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145580: Sat Sep 5 09:18:44 2026 read: IOPS=11, BW=44.7KiB/s (45.8kB/s)(4984KiB/111548msec) slat (usec): min=3315, max=57638, avg=15229.72, stdev=8068.63 clat (msec): min=25, max=1003, avg=432.95, stdev=111.91 lat (msec): min=35, max=1019, avg=448.19, stdev=113.20 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 288], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 472], 80.00th=[ 510], 90.00th=[ 575], 95.00th=[ 642], | 99.00th=[ 735], 99.50th=[ 810], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 96, per=24.80%, avg=44.40, stdev=19.76, samples=220 iops : min= 1, max= 24, avg=10.56, stdev= 4.95, samples=220 write: IOPS=11, BW=47.1KiB/s (48.2kB/s)(5256KiB/111548msec); 0 zone resets slat (usec): min=36, max=14742, avg=393.94, stdev=1037.99 clat (msec): min=137, max=2049, avg=930.67, stdev=258.27 lat (msec): min=137, max=2049, avg=931.06, stdev=258.22 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 567], 10.00th=[ 617], 20.00th=[ 701], | 30.00th=[ 776], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1586], 99.50th=[ 1871], 99.90th=[ 2039], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 111, per=24.69%, avg=46.17, stdev=19.78, samples=221 iops : min= 1, max= 27, avg=11.01, stdev= 4.94, samples=221 lat (msec) : 50=0.12%, 100=0.08%, 250=1.21%, 500=37.54%, 750=22.89% lat (msec) : 1000=18.44%, 2000=19.61%, >=2000=0.12% cpu : usr=0.12%, sys=0.76%, ctx=4626, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145581: Sat Sep 5 09:18:44 2026 read: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5060KiB/111589msec) slat (usec): min=3267, max=65160, avg=15610.70, stdev=8405.69 clat (msec): min=24, max=996, avg=431.14, stdev=107.75 lat (msec): min=31, max=1007, avg=446.76, stdev=109.27 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 472], 80.00th=[ 506], 90.00th=[ 567], 95.00th=[ 634], | 99.00th=[ 735], 99.50th=[ 760], 99.90th=[ 827], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 112, per=25.37%, avg=45.41, stdev=19.80, samples=218 iops : min= 1, max= 28, avg=10.82, stdev= 4.96, samples=218 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(5180KiB/111589msec); 0 zone resets slat (usec): min=36, max=34773, avg=435.33, stdev=1446.28 clat (msec): min=279, max=1991, avg=938.93, stdev=256.20 lat (msec): min=280, max=1991, avg=939.37, stdev=256.20 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1703], 99.50th=[ 1787], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 96, per=24.47%, avg=45.75, stdev=17.61, samples=220 iops : min= 1, max= 24, avg=10.90, stdev= 4.43, samples=220 lat (msec) : 50=0.04%, 100=0.20%, 250=0.70%, 500=39.41%, 750=21.84% lat (msec) : 1000=18.16%, 2000=19.65% cpu : usr=0.14%, sys=0.73%, ctx=5028, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=180KiB/s (184kB/s), 44.6KiB/s-45.3KiB/s (45.7kB/s-46.4kB/s), io=19.6MiB (20.5MB), run=111488-111592msec WRITE: bw=187KiB/s (192kB/s), 46.4KiB/s-47.2KiB/s (47.5kB/s-48.3kB/s), io=20.4MiB (21.4MB), run=111488-111592msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145600: Sat Sep 5 09:18:54 2026 read: IOPS=2, BW=89.7MiB/s (94.0MB/s)(40.0MiB/446msec) slat (nsec): min=31620k, max=31620k, avg=31619708.00, stdev= 0.00 clat (nsec): min=396379k, max=396379k, avg=396379432.00, stdev= 0.00 lat (nsec): min=428004k, max=428004k, avg=428003908.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.70%, ctx=4, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=89.7MiB/s (94.0MB/s), 89.7MiB/s-89.7MiB/s (94.0MB/s-94.0MB/s), io=40.0MiB (41.9MB), run=446-446msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout