== sanity test 398c: run fio to test AIO ================= 15:45:15 (1786736715) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.496384 s, 84.5 MB/s osc.lustre-OST0000-osc-ffff8dda1a85b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144590: Fri Aug 14 15:47:47 2026 write: IOPS=17, BW=71.5KiB/s (73.2kB/s)(10.0MiB/143210msec); 0 zone resets slat (usec): min=37, max=24550, avg=245.83, stdev=923.28 clat (msec): min=281, max=2077, avg=894.45, stdev=222.28 lat (msec): min=282, max=2077, avg=894.70, stdev=222.29 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 523], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1519], 99.50th=[ 1720], 99.90th=[ 1921], 99.95th=[ 1972], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 144, per=25.03%, avg=71.58, stdev=30.12, samples=281 iops : min= 1, max= 36, avg=17.46, stdev= 7.59, samples=281 lat (msec) : 500=3.28%, 750=21.64%, 1000=45.39%, 2000=29.65%, >=2000=0.04% cpu : usr=0.05%, sys=0.28%, ctx=2381, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144591: Fri Aug 14 15:47:47 2026 write: IOPS=17, BW=71.6KiB/s (73.3kB/s)(10.0MiB/143003msec); 0 zone resets slat (usec): min=35, max=14796, avg=249.73, stdev=768.85 clat (msec): min=198, max=2038, avg=893.13, stdev=227.85 lat (msec): min=198, max=2038, avg=893.38, stdev=227.84 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 535], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1620], 99.50th=[ 1821], 99.90th=[ 2005], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 128, per=24.80%, avg=70.94, stdev=28.39, samples=284 iops : min= 1, max= 32, avg=17.30, stdev= 7.14, samples=284 lat (msec) : 250=0.31%, 500=3.12%, 750=22.27%, 1000=45.70%, 2000=28.44% lat (msec) : >=2000=0.16% cpu : usr=0.04%, sys=0.28%, ctx=1907, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144592: Fri Aug 14 15:47:47 2026 write: IOPS=17, BW=71.6KiB/s (73.3kB/s)(10.0MiB/142990msec); 0 zone resets slat (usec): min=37, max=17847, avg=243.40, stdev=731.76 clat (msec): min=105, max=2017, avg=893.11, stdev=224.82 lat (msec): min=111, max=2017, avg=893.35, stdev=224.82 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 527], 10.00th=[ 609], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1536], 99.50th=[ 1636], 99.90th=[ 1972], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 144, per=24.80%, avg=70.93, stdev=27.01, samples=284 iops : min= 1, max= 36, avg=17.30, stdev= 6.77, samples=284 lat (msec) : 250=0.43%, 500=3.20%, 750=21.45%, 1000=46.37%, 2000=28.48% lat (msec) : >=2000=0.08% cpu : usr=0.04%, sys=0.29%, ctx=2287, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144593: Fri Aug 14 15:47:47 2026 write: IOPS=17, BW=71.6KiB/s (73.3kB/s)(10.0MiB/143044msec); 0 zone resets slat (usec): min=36, max=25167, avg=248.68, stdev=844.93 clat (msec): min=173, max=1919, avg=893.41, stdev=222.92 lat (msec): min=173, max=1920, avg=893.66, stdev=222.91 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 527], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1070], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1536], 99.50th=[ 1687], 99.90th=[ 1787], 99.95th=[ 1871], | 99.99th=[ 1921] bw ( KiB/s): min= 15, max= 128, per=24.89%, avg=71.19, stdev=24.83, samples=283 iops : min= 3, max= 32, avg=17.36, stdev= 6.23, samples=283 lat (msec) : 250=0.08%, 500=3.36%, 750=21.91%, 1000=44.84%, 2000=29.80% cpu : usr=0.06%, sys=0.31%, ctx=2471, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=286KiB/s (293kB/s), 71.5KiB/s-71.6KiB/s (73.2kB/s-73.3kB/s), io=40.0MiB (41.9MB), run=142990-143210msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144623: Fri Aug 14 15:49:42 2026 read: IOPS=11, BW=47.1KiB/s (48.2kB/s)(5048KiB/107283msec) slat (usec): min=2643, max=87241, avg=14354.19, stdev=9511.88 clat (msec): min=32, max=1932, avg=416.41, stdev=173.84 lat (msec): min=40, max=1959, avg=430.77, stdev=177.17 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 211], 10.00th=[ 241], 20.00th=[ 296], | 30.00th=[ 330], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 430], | 70.00th=[ 464], 80.00th=[ 514], 90.00th=[ 600], 95.00th=[ 701], | 99.00th=[ 902], 99.50th=[ 1418], 99.90th=[ 1854], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 127, per=25.44%, avg=47.31, stdev=23.12, samples=209 iops : min= 1, max= 31, avg=11.33, stdev= 5.78, samples=209 write: IOPS=12, BW=48.4KiB/s (49.6kB/s)(5192KiB/107283msec); 0 zone resets slat (usec): min=43, max=9242, avg=286.64, stdev=628.25 clat (msec): min=288, max=2535, avg=899.36, stdev=336.50 lat (msec): min=288, max=2535, avg=899.66, stdev=336.52 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 451], 10.00th=[ 510], 20.00th=[ 609], | 30.00th=[ 701], 40.00th=[ 793], 50.00th=[ 885], 60.00th=[ 969], | 70.00th=[ 1036], 80.00th=[ 1116], 90.00th=[ 1217], 95.00th=[ 1435], | 99.00th=[ 2265], 99.50th=[ 2366], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 136, per=24.96%, avg=48.43, stdev=23.36, samples=210 iops : min= 1, max= 34, avg=11.60, stdev= 5.92, samples=210 lat (msec) : 50=0.20%, 100=0.12%, 250=5.08%, 500=37.23%, 750=23.01% lat (msec) : 1000=16.17%, 2000=17.15%, >=2000=1.05% cpu : usr=0.09%, sys=0.67%, ctx=4767, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144624: Fri Aug 14 15:49:42 2026 read: IOPS=11, BW=46.2KiB/s (47.3kB/s)(4976KiB/107679msec) slat (usec): min=2839, max=74370, avg=14013.27, stdev=8955.24 clat (msec): min=8, max=2125, avg=419.61, stdev=177.96 lat (msec): min=21, max=2134, avg=433.63, stdev=180.88 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 207], 10.00th=[ 247], 20.00th=[ 296], | 30.00th=[ 334], 40.00th=[ 363], 50.00th=[ 393], 60.00th=[ 426], | 70.00th=[ 468], 80.00th=[ 531], 90.00th=[ 600], 95.00th=[ 701], | 99.00th=[ 936], 99.50th=[ 1368], 99.90th=[ 1787], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 104, per=24.93%, avg=46.36, stdev=23.14, samples=210 iops : min= 1, max= 26, avg=11.09, stdev= 5.82, samples=210 write: IOPS=12, BW=48.9KiB/s (50.1kB/s)(5264KiB/107679msec); 0 zone resets slat (usec): min=37, max=13746, avg=307.33, stdev=747.90 clat (msec): min=275, max=2444, avg=896.62, stdev=330.51 lat (msec): min=275, max=2444, avg=896.93, stdev=330.53 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 439], 10.00th=[ 506], 20.00th=[ 625], | 30.00th=[ 709], 40.00th=[ 785], 50.00th=[ 877], 60.00th=[ 961], | 70.00th=[ 1036], 80.00th=[ 1116], 90.00th=[ 1234], 95.00th=[ 1469], | 99.00th=[ 2198], 99.50th=[ 2265], 99.90th=[ 2400], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=25.06%, avg=48.61, stdev=23.73, samples=212 iops : min= 1, max= 32, avg=11.65, stdev= 5.99, samples=212 lat (msec) : 10=0.04%, 50=0.27%, 100=0.20%, 250=4.57%, 500=36.29% lat (msec) : 750=23.75%, 1000=16.76%, 2000=17.46%, >=2000=0.66% cpu : usr=0.08%, sys=0.63%, ctx=5049, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144625: Fri Aug 14 15:49:42 2026 read: IOPS=11, BW=46.7KiB/s (47.8kB/s)(4984KiB/106808msec) slat (usec): min=63, max=82869, avg=14003.76, stdev=8910.79 clat (msec): min=25, max=1559, avg=411.71, stdev=158.17 lat (msec): min=34, max=1571, avg=425.72, stdev=161.16 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 211], 10.00th=[ 247], 20.00th=[ 292], | 30.00th=[ 326], 40.00th=[ 355], 50.00th=[ 388], 60.00th=[ 422], | 70.00th=[ 460], 80.00th=[ 510], 90.00th=[ 592], 95.00th=[ 676], | 99.00th=[ 919], 99.50th=[ 986], 99.90th=[ 1469], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 175, per=25.40%, avg=47.25, stdev=25.69, samples=207 iops : min= 1, max= 43, avg=11.30, stdev= 6.43, samples=207 write: IOPS=12, BW=49.2KiB/s (50.4kB/s)(5256KiB/106808msec); 0 zone resets slat (usec): min=47, max=11502, avg=294.85, stdev=685.37 clat (msec): min=174, max=2369, avg=895.31, stdev=316.27 lat (msec): min=175, max=2369, avg=895.60, stdev=316.31 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 435], 10.00th=[ 506], 20.00th=[ 617], | 30.00th=[ 709], 40.00th=[ 802], 50.00th=[ 902], 60.00th=[ 986], | 70.00th=[ 1045], 80.00th=[ 1116], 90.00th=[ 1217], 95.00th=[ 1385], | 99.00th=[ 2022], 99.50th=[ 2232], 99.90th=[ 2299], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 120, per=25.32%, avg=49.12, stdev=24.33, samples=209 iops : min= 1, max= 30, avg=11.78, stdev= 6.14, samples=209 lat (msec) : 50=0.16%, 100=0.08%, 250=5.04%, 500=37.38%, 750=21.76% lat (msec) : 1000=16.21%, 2000=18.79%, >=2000=0.59% cpu : usr=0.09%, sys=0.64%, ctx=4612, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144626: Fri Aug 14 15:49:42 2026 read: IOPS=11, BW=46.0KiB/s (48.1kB/s)(5060KiB/107678msec) slat (usec): min=2251, max=87687, avg=13789.11, stdev=8579.25 clat (msec): min=9, max=2233, avg=418.99, stdev=174.10 lat (msec): min=11, max=2256, avg=432.79, stdev=176.59 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 207], 10.00th=[ 241], 20.00th=[ 296], | 30.00th=[ 334], 40.00th=[ 363], 50.00th=[ 393], 60.00th=[ 426], | 70.00th=[ 468], 80.00th=[ 523], 90.00th=[ 600], 95.00th=[ 709], | 99.00th=[ 894], 99.50th=[ 1003], 99.90th=[ 1888], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 116, per=25.53%, avg=47.48, stdev=24.04, samples=209 iops : min= 1, max= 29, avg=11.36, stdev= 6.06, samples=209 write: IOPS=12, BW=48.1KiB/s (49.3kB/s)(5180KiB/107678msec); 0 zone resets slat (usec): min=49, max=23747, avg=309.00, stdev=1025.98 clat (msec): min=253, max=2594, avg=903.93, stdev=341.45 lat (msec): min=253, max=2594, avg=904.24, stdev=341.46 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 439], 10.00th=[ 498], 20.00th=[ 625], | 30.00th=[ 709], 40.00th=[ 802], 50.00th=[ 885], 60.00th=[ 978], | 70.00th=[ 1036], 80.00th=[ 1116], 90.00th=[ 1217], 95.00th=[ 1469], | 99.00th=[ 2265], 99.50th=[ 2433], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 120, per=24.86%, avg=48.24, stdev=24.66, samples=210 iops : min= 1, max= 30, avg=11.56, stdev= 6.20, samples=210 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=0.04%, 250=5.23% lat (msec) : 500=37.50%, 750=21.64%, 1000=17.11%, 2000=17.34%, >=2000=0.98% cpu : usr=0.08%, sys=0.69%, ctx=4932, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=186KiB/s (191kB/s), 46.2KiB/s-47.1KiB/s (47.3kB/s-48.2kB/s), io=19.6MiB (20.5MB), run=106808-107679msec WRITE: bw=194KiB/s (199kB/s), 48.1KiB/s-49.2KiB/s (49.3kB/s-50.4kB/s), io=20.4MiB (21.4MB), run=106808-107679msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144646: Fri Aug 14 15:49:57 2026 read: IOPS=1, BW=48.1MiB/s (50.4MB/s)(40.0MiB/832msec) slat (nsec): min=18962k, max=18962k, avg=18961985.00, stdev= 0.00 clat (nsec): min=793351k, max=793351k, avg=793350650.00, stdev= 0.00 lat (nsec): min=812316k, max=812316k, avg=812316468.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 793], 5.00th=[ 793], 10.00th=[ 793], 20.00th=[ 793], | 30.00th=[ 793], 40.00th=[ 793], 50.00th=[ 793], 60.00th=[ 793], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=0.84%, ctx=43, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=48.1MiB/s (50.4MB/s), 48.1MiB/s-48.1MiB/s (50.4MB/s-50.4MB/s), io=40.0MiB (41.9MB), run=832-832msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout