== sanity test 398c: run fio to test AIO ================= 14:16:18 (1786040178) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.613756 s, 68.3 MB/s osc.lustre-OST0000-osc-ffff9941e597e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145295: Thu Aug 6 14:19:13 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/163068msec); 0 zone resets slat (usec): min=35, max=14344, avg=241.43, stdev=710.18 clat (msec): min=247, max=2434, avg=1018.54, stdev=314.90 lat (msec): min=247, max=2434, avg=1018.79, stdev=314.91 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 506], 10.00th=[ 651], 20.00th=[ 760], | 30.00th=[ 852], 40.00th=[ 927], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1368], 95.00th=[ 1485], | 99.00th=[ 2005], 99.50th=[ 2106], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 175, per=24.94%, avg=62.35, stdev=29.14, samples=322 iops : min= 1, max= 43, avg=15.14, stdev= 7.32, samples=322 lat (msec) : 250=0.12%, 500=4.73%, 750=13.75%, 1000=28.79%, 2000=51.64% lat (msec) : >=2000=0.98% cpu : usr=0.04%, sys=0.27%, ctx=1786, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145296: Thu Aug 6 14:19:13 2026 write: IOPS=15, BW=62.7KiB/s (64.2kB/s)(10.0MiB/163276msec); 0 zone resets slat (usec): min=36, max=15340, avg=295.37, stdev=969.56 clat (msec): min=309, max=2388, avg=1019.18, stdev=306.69 lat (msec): min=309, max=2389, avg=1019.48, stdev=306.72 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 514], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 844], 40.00th=[ 936], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1368], 95.00th=[ 1485], | 99.00th=[ 1972], 99.50th=[ 2089], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 223, per=25.28%, avg=63.20, stdev=29.91, samples=318 iops : min= 1, max= 55, avg=15.36, stdev= 7.55, samples=318 lat (msec) : 500=3.75%, 750=13.79%, 1000=29.18%, 2000=52.50%, >=2000=0.78% cpu : usr=0.04%, sys=0.25%, ctx=2000, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145297: Thu Aug 6 14:19:13 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/163145msec); 0 zone resets slat (usec): min=37, max=12195, avg=219.14, stdev=606.19 clat (msec): min=64, max=2229, avg=1018.53, stdev=309.89 lat (msec): min=64, max=2229, avg=1018.75, stdev=309.91 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 514], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 852], 40.00th=[ 927], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 1955], 99.50th=[ 2039], 99.90th=[ 2198], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 254, per=25.39%, avg=63.49, stdev=32.21, samples=317 iops : min= 1, max= 63, avg=15.43, stdev= 8.10, samples=317 lat (msec) : 100=0.04%, 500=4.18%, 750=14.53%, 1000=28.63%, 2000=51.84% lat (msec) : >=2000=0.78% cpu : usr=0.05%, sys=0.26%, ctx=1777, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145298: Thu Aug 6 14:19:13 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/162951msec); 0 zone resets slat (usec): min=35, max=13133, avg=217.28, stdev=598.63 clat (msec): min=59, max=2432, avg=1017.26, stdev=311.19 lat (msec): min=59, max=2433, avg=1017.48, stdev=311.18 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 514], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 852], 40.00th=[ 927], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1469], | 99.00th=[ 1972], 99.50th=[ 2089], 99.90th=[ 2400], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 207, per=24.99%, avg=62.47, stdev=29.06, samples=322 iops : min= 1, max= 51, avg=15.17, stdev= 7.29, samples=322 lat (msec) : 100=0.12%, 250=0.08%, 500=4.49%, 750=13.32%, 1000=29.77% lat (msec) : 2000=51.37%, >=2000=0.86% cpu : usr=0.04%, sys=0.26%, ctx=1833, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=251KiB/s (257kB/s), 62.7KiB/s-62.8KiB/s (64.2kB/s-64.3kB/s), io=40.0MiB (41.9MB), run=162951-163276msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145332: Thu Aug 6 14:21:05 2026 read: IOPS=12, BW=49.5KiB/s (50.7kB/s)(5048KiB/101975msec) slat (usec): min=3142, max=47418, avg=14190.00, stdev=7543.94 clat (msec): min=27, max=945, avg=386.57, stdev=93.62 lat (msec): min=40, max=964, avg=400.76, stdev=94.71 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 259], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 397], | 70.00th=[ 426], 80.00th=[ 456], 90.00th=[ 502], 95.00th=[ 542], | 99.00th=[ 651], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 119, per=25.29%, avg=49.56, stdev=21.93, samples=200 iops : min= 1, max= 29, avg=11.91, stdev= 5.56, samples=200 write: IOPS=12, BW=50.9KiB/s (52.1kB/s)(5192KiB/101975msec); 0 zone resets slat (usec): min=45, max=21579, avg=353.95, stdev=1114.99 clat (msec): min=289, max=1501, avg=864.34, stdev=191.42 lat (msec): min=292, max=1502, avg=864.69, stdev=191.38 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 810], 50.00th=[ 852], 60.00th=[ 894], | 70.00th=[ 944], 80.00th=[ 1020], 90.00th=[ 1133], 95.00th=[ 1200], | 99.00th=[ 1318], 99.50th=[ 1418], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 119, per=24.82%, avg=50.63, stdev=18.71, samples=201 iops : min= 1, max= 29, avg=12.17, stdev= 4.66, samples=201 lat (msec) : 50=0.12%, 100=0.12%, 250=1.64%, 500=43.52%, 750=17.27% lat (msec) : 1000=26.09%, 2000=11.25% cpu : usr=0.10%, sys=0.64%, ctx=4225, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145333: Thu Aug 6 14:21:05 2026 read: IOPS=12, BW=48.8KiB/s (49.0kB/s)(4976KiB/101971msec) slat (usec): min=2389, max=57041, avg=14552.54, stdev=8070.47 clat (msec): min=21, max=952, avg=384.43, stdev=92.44 lat (msec): min=27, max=985, avg=398.99, stdev=93.62 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 397], | 70.00th=[ 422], 80.00th=[ 456], 90.00th=[ 502], 95.00th=[ 535], | 99.00th=[ 625], 99.50th=[ 693], 99.90th=[ 802], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 151, per=24.90%, avg=48.81, stdev=22.08, samples=200 iops : min= 1, max= 37, avg=11.72, stdev= 5.53, samples=200 write: IOPS=12, BW=51.6KiB/s (52.9kB/s)(5264KiB/101971msec); 0 zone resets slat (usec): min=36, max=10266, avg=297.02, stdev=696.46 clat (msec): min=111, max=1495, avg=859.74, stdev=189.00 lat (msec): min=111, max=1495, avg=860.03, stdev=189.00 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 810], 50.00th=[ 844], 60.00th=[ 885], | 70.00th=[ 944], 80.00th=[ 1020], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1334], 99.50th=[ 1401], 99.90th=[ 1485], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 112, per=25.19%, avg=51.39, stdev=19.70, samples=201 iops : min= 1, max= 28, avg=12.36, stdev= 4.94, samples=201 lat (msec) : 50=0.23%, 100=0.16%, 250=1.64%, 500=42.81%, 750=17.58% lat (msec) : 1000=25.78%, 2000=11.80% cpu : usr=0.11%, sys=0.68%, ctx=4495, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145334: Thu Aug 6 14:21:05 2026 read: IOPS=12, BW=48.0KiB/s (50.1kB/s)(4984KiB/101765msec) slat (usec): min=58, max=53313, avg=14165.00, stdev=7777.84 clat (msec): min=30, max=1026, avg=384.08, stdev=94.35 lat (msec): min=40, max=1032, avg=398.25, stdev=95.31 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 249], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 376], 60.00th=[ 401], | 70.00th=[ 422], 80.00th=[ 447], 90.00th=[ 493], 95.00th=[ 542], | 99.00th=[ 625], 99.50th=[ 793], 99.90th=[ 944], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 119, per=24.99%, avg=48.99, stdev=21.53, samples=200 iops : min= 1, max= 29, avg=11.76, stdev= 5.40, samples=200 write: IOPS=12, BW=51.6KiB/s (52.9kB/s)(5256KiB/101765msec); 0 zone resets slat (usec): min=42, max=12572, avg=300.95, stdev=714.72 clat (msec): min=132, max=1441, avg=860.57, stdev=191.67 lat (msec): min=133, max=1441, avg=860.88, stdev=191.70 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 944], 80.00th=[ 1028], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1301], 99.50th=[ 1351], 99.90th=[ 1418], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 111, per=24.95%, avg=50.90, stdev=18.88, samples=202 iops : min= 1, max= 27, avg=12.24, stdev= 4.74, samples=202 lat (msec) : 50=0.16%, 100=0.12%, 250=2.38%, 500=42.85%, 750=16.76% lat (msec) : 1000=25.86%, 2000=11.88% cpu : usr=0.09%, sys=0.73%, ctx=4567, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145335: Thu Aug 6 14:21:05 2026 read: IOPS=12, BW=49.6KiB/s (50.8kB/s)(5060KiB/101975msec) slat (usec): min=2860, max=53505, avg=14155.29, stdev=7936.04 clat (msec): min=12, max=943, avg=386.45, stdev=93.79 lat (msec): min=22, max=961, avg=400.61, stdev=94.79 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 268], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 359], 50.00th=[ 376], 60.00th=[ 397], | 70.00th=[ 422], 80.00th=[ 451], 90.00th=[ 498], 95.00th=[ 542], | 99.00th=[ 693], 99.50th=[ 768], 99.90th=[ 927], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 111, per=25.13%, avg=49.26, stdev=20.74, samples=202 iops : min= 1, max= 27, avg=11.83, stdev= 5.22, samples=202 write: IOPS=12, BW=50.8KiB/s (52.0kB/s)(5180KiB/101975msec); 0 zone resets slat (usec): min=40, max=15007, avg=336.14, stdev=912.58 clat (msec): min=107, max=1584, avg=865.91, stdev=193.41 lat (msec): min=107, max=1584, avg=866.25, stdev=193.40 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 894], | 70.00th=[ 944], 80.00th=[ 1020], 90.00th=[ 1133], 95.00th=[ 1200], | 99.00th=[ 1334], 99.50th=[ 1368], 99.90th=[ 1485], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 104, per=24.72%, avg=50.43, stdev=19.46, samples=201 iops : min= 1, max= 26, avg=12.12, stdev= 4.88, samples=201 lat (msec) : 20=0.04%, 50=0.16%, 100=0.20%, 250=1.41%, 500=44.26% lat (msec) : 750=16.80%, 1000=25.74%, 2000=11.41% cpu : usr=0.08%, sys=0.67%, ctx=4197, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=197KiB/s (202kB/s), 48.8KiB/s-49.6KiB/s (49.0kB/s-50.8kB/s), io=19.6MiB (20.5MB), run=101765-101975msec WRITE: bw=205KiB/s (210kB/s), 50.8KiB/s-51.6KiB/s (52.0kB/s-52.9kB/s), io=20.4MiB (21.4MB), run=101765-101975msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145353: Thu Aug 6 14:21:15 2026 read: IOPS=1, BW=75.6MiB/s (79.3MB/s)(40.0MiB/529msec) slat (nsec): min=40307k, max=40307k, avg=40306858.00, stdev= 0.00 clat (nsec): min=468491k, max=468491k, avg=468490610.00, stdev= 0.00 lat (nsec): min=508837k, max=508837k, avg=508837051.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 468], 10.00th=[ 468], 20.00th=[ 468], | 30.00th=[ 468], 40.00th=[ 468], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.46%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=75.6MiB/s (79.3MB/s), 75.6MiB/s-75.6MiB/s (79.3MB/s-79.3MB/s), io=40.0MiB (41.9MB), run=529-529msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout