== sanity test 398c: run fio to test AIO ================= 13:32:41 (1790789561) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.516398 s, 81.2 MB/s osc.lustre-OST0000-osc-ffff889e5fff3800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145797: Wed Sep 30 13:35:56 2026 write: IOPS=13, BW=55.5KiB/s (56.8kB/s)(10.0MiB/184508msec); 0 zone resets slat (usec): min=28, max=26194, avg=349.11, stdev=1181.18 clat (msec): min=74, max=2484, avg=1151.86, stdev=306.57 lat (msec): min=75, max=2485, avg=1152.21, stdev=306.52 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 659], 10.00th=[ 768], 20.00th=[ 911], | 30.00th=[ 995], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1385], 90.00th=[ 1519], 95.00th=[ 1670], | 99.00th=[ 1972], 99.50th=[ 2072], 99.90th=[ 2433], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 135, per=25.18%, avg=55.64, stdev=26.83, samples=361 iops : min= 1, max= 33, avg=13.39, stdev= 6.71, samples=361 lat (msec) : 100=0.04%, 250=0.35%, 500=1.60%, 750=6.33%, 1000=21.99% lat (msec) : 2000=68.83%, >=2000=0.86% cpu : usr=0.05%, sys=0.28%, ctx=1862, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145798: Wed Sep 30 13:35:56 2026 write: IOPS=13, BW=55.5KiB/s (56.9kB/s)(10.0MiB/184379msec); 0 zone resets slat (usec): min=28, max=17989, avg=300.18, stdev=1028.16 clat (msec): min=89, max=2488, avg=1151.50, stdev=310.82 lat (msec): min=89, max=2488, avg=1151.80, stdev=310.78 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 667], 10.00th=[ 768], 20.00th=[ 902], | 30.00th=[ 986], 40.00th=[ 1070], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1284], 80.00th=[ 1401], 90.00th=[ 1502], 95.00th=[ 1670], | 99.00th=[ 1989], 99.50th=[ 2123], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=25.09%, avg=55.46, stdev=27.05, samples=362 iops : min= 1, max= 32, avg=13.34, stdev= 6.77, samples=362 lat (msec) : 100=0.04%, 250=0.62%, 500=1.21%, 750=6.80%, 1000=22.97% lat (msec) : 2000=67.46%, >=2000=0.90% cpu : usr=0.05%, sys=0.24%, ctx=1736, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145799: Wed Sep 30 13:35:56 2026 write: IOPS=13, BW=55.4KiB/s (56.8kB/s)(10.0MiB/184739msec); 0 zone resets slat (usec): min=30, max=20323, avg=296.61, stdev=960.05 clat (msec): min=99, max=2338, avg=1153.52, stdev=301.85 lat (msec): min=100, max=2338, avg=1153.82, stdev=301.85 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 667], 10.00th=[ 785], 20.00th=[ 911], | 30.00th=[ 986], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1284], 80.00th=[ 1401], 90.00th=[ 1519], 95.00th=[ 1653], | 99.00th=[ 2039], 99.50th=[ 2106], 99.90th=[ 2299], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 128, per=25.52%, avg=56.39, stdev=25.65, samples=356 iops : min= 1, max= 32, avg=13.58, stdev= 6.45, samples=356 lat (msec) : 100=0.04%, 500=0.66%, 750=6.88%, 1000=24.18%, 2000=67.23% lat (msec) : >=2000=1.02% cpu : usr=0.05%, sys=0.26%, ctx=1827, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145800: Wed Sep 30 13:35:56 2026 write: IOPS=13, BW=55.4KiB/s (56.7kB/s)(10.0MiB/184816msec); 0 zone resets slat (usec): min=29, max=18097, avg=303.49, stdev=985.10 clat (msec): min=259, max=2487, avg=1153.95, stdev=300.78 lat (msec): min=259, max=2487, avg=1154.26, stdev=300.78 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 693], 10.00th=[ 785], 20.00th=[ 902], | 30.00th=[ 995], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1519], 95.00th=[ 1636], | 99.00th=[ 2039], 99.50th=[ 2123], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=25.75%, avg=56.91, stdev=25.59, samples=352 iops : min= 1, max= 32, avg=13.71, stdev= 6.39, samples=352 lat (msec) : 500=0.94%, 750=7.38%, 1000=22.30%, 2000=68.36%, >=2000=1.02% cpu : usr=0.05%, sys=0.24%, ctx=1734, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=222KiB/s (227kB/s), 55.4KiB/s-55.5KiB/s (56.7kB/s-56.9kB/s), io=40.0MiB (41.9MB), run=184379-184816msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145840: Wed Sep 30 13:38:27 2026 read: IOPS=8, BW=35.8KiB/s (36.7kB/s)(5048KiB/140958msec) slat (usec): min=36, max=113551, avg=17795.08, stdev=10896.61 clat (msec): min=17, max=1593, avg=550.83, stdev=215.46 lat (msec): min=28, max=1624, avg=568.64, stdev=218.42 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 317], 10.00th=[ 359], 20.00th=[ 409], | 30.00th=[ 447], 40.00th=[ 477], 50.00th=[ 506], 60.00th=[ 542], | 70.00th=[ 584], 80.00th=[ 651], 90.00th=[ 810], 95.00th=[ 995], | 99.00th=[ 1435], 99.50th=[ 1536], 99.90th=[ 1569], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 116, per=25.49%, avg=35.95, stdev=19.22, samples=274 iops : min= 1, max= 29, avg= 8.46, stdev= 4.83, samples=274 write: IOPS=9, BW=36.8KiB/s (37.7kB/s)(5192KiB/140958msec); 0 zone resets slat (usec): min=30, max=24456, avg=391.59, stdev=1267.13 clat (msec): min=134, max=3310, avg=1183.22, stdev=411.39 lat (msec): min=134, max=3310, avg=1183.61, stdev=411.47 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 634], 10.00th=[ 726], 20.00th=[ 860], | 30.00th=[ 961], 40.00th=[ 1045], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1318], 80.00th=[ 1485], 90.00th=[ 1703], 95.00th=[ 1921], | 99.00th=[ 2668], 99.50th=[ 2869], 99.90th=[ 3104], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 7, max= 103, per=24.94%, avg=36.67, stdev=19.14, samples=275 iops : min= 1, max= 25, avg= 8.63, stdev= 4.84, samples=275 lat (msec) : 20=0.04%, 50=0.16%, 100=0.43%, 250=0.98%, 500=22.81% lat (msec) : 750=24.53%, 1000=15.35%, 2000=33.83%, >=2000=1.88% cpu : usr=0.10%, sys=0.55%, ctx=3896, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145841: Wed Sep 30 13:38:27 2026 read: IOPS=8, BW=35.0KiB/s (35.9kB/s)(4976KiB/142067msec) slat (usec): min=3648, max=85029, avg=17956.48, stdev=10433.20 clat (msec): min=37, max=1641, avg=554.43, stdev=208.17 lat (msec): min=44, max=1668, avg=572.39, stdev=210.54 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 326], 10.00th=[ 363], 20.00th=[ 409], | 30.00th=[ 443], 40.00th=[ 481], 50.00th=[ 506], 60.00th=[ 542], | 70.00th=[ 584], 80.00th=[ 651], 90.00th=[ 810], 95.00th=[ 1003], | 99.00th=[ 1385], 99.50th=[ 1485], 99.90th=[ 1569], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 95, per=25.11%, avg=35.41, stdev=18.62, samples=273 iops : min= 1, max= 23, avg= 8.32, stdev= 4.69, samples=273 write: IOPS=9, BW=37.1KiB/s (37.9kB/s)(5264KiB/142067msec); 0 zone resets slat (usec): min=28, max=13737, avg=378.88, stdev=1114.75 clat (msec): min=216, max=3310, avg=1183.87, stdev=412.25 lat (msec): min=216, max=3310, avg=1184.25, stdev=412.20 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 609], 10.00th=[ 718], 20.00th=[ 869], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1217], | 70.00th=[ 1318], 80.00th=[ 1469], 90.00th=[ 1720], 95.00th=[ 1921], | 99.00th=[ 2668], 99.50th=[ 2802], 99.90th=[ 3205], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 7, max= 96, per=25.61%, avg=37.65, stdev=20.32, samples=273 iops : min= 1, max= 24, avg= 8.87, stdev= 5.11, samples=273 lat (msec) : 50=0.04%, 100=0.23%, 250=0.31%, 500=23.16%, 750=24.22% lat (msec) : 1000=15.86%, 2000=34.30%, >=2000=1.88% cpu : usr=0.08%, sys=0.54%, ctx=3659, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145842: Wed Sep 30 13:38:27 2026 read: IOPS=8, BW=35.1KiB/s (35.9kB/s)(4984KiB/142061msec) slat (msec): min=3, max=107, avg=17.69, stdev=10.51 clat (msec): min=18, max=1659, avg=550.18, stdev=208.05 lat (msec): min=30, max=1706, avg=567.88, stdev=210.67 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 326], 10.00th=[ 368], 20.00th=[ 409], | 30.00th=[ 443], 40.00th=[ 472], 50.00th=[ 506], 60.00th=[ 542], | 70.00th=[ 584], 80.00th=[ 642], 90.00th=[ 818], 95.00th=[ 1011], | 99.00th=[ 1318], 99.50th=[ 1469], 99.90th=[ 1569], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 103, per=25.12%, avg=35.41, stdev=18.30, samples=275 iops : min= 1, max= 25, avg= 8.31, stdev= 4.59, samples=275 write: IOPS=9, BW=36.0KiB/s (37.9kB/s)(5256KiB/142061msec); 0 zone resets slat (usec): min=33, max=14757, avg=346.05, stdev=1045.68 clat (msec): min=378, max=2981, avg=1188.48, stdev=400.65 lat (msec): min=379, max=2981, avg=1188.82, stdev=400.65 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 659], 10.00th=[ 751], 20.00th=[ 877], | 30.00th=[ 961], 40.00th=[ 1036], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1318], 80.00th=[ 1452], 90.00th=[ 1720], 95.00th=[ 1921], | 99.00th=[ 2601], 99.50th=[ 2735], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 95, per=24.88%, avg=36.57, stdev=19.41, samples=279 iops : min= 1, max= 23, avg= 8.60, stdev= 4.90, samples=279 lat (msec) : 20=0.08%, 50=0.08%, 100=0.27%, 250=0.27%, 500=23.28% lat (msec) : 750=23.52%, 1000=16.64%, 2000=33.83%, >=2000=2.03% cpu : usr=0.10%, sys=0.57%, ctx=4322, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145843: Wed Sep 30 13:38:27 2026 read: IOPS=8, BW=35.8KiB/s (36.7kB/s)(5060KiB/141316msec) slat (usec): min=3154, max=73498, avg=17573.70, stdev=9696.60 clat (msec): min=16, max=1589, avg=551.21, stdev=207.31 lat (msec): min=23, max=1615, avg=568.80, stdev=209.26 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 330], 10.00th=[ 363], 20.00th=[ 414], | 30.00th=[ 443], 40.00th=[ 477], 50.00th=[ 506], 60.00th=[ 542], | 70.00th=[ 584], 80.00th=[ 642], 90.00th=[ 818], 95.00th=[ 1011], | 99.00th=[ 1267], 99.50th=[ 1435], 99.90th=[ 1552], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 88, per=26.18%, avg=36.92, stdev=18.34, samples=267 iops : min= 1, max= 22, avg= 8.69, stdev= 4.62, samples=267 write: IOPS=9, BW=36.7KiB/s (37.5kB/s)(5180KiB/141316msec); 0 zone resets slat (usec): min=28, max=13217, avg=398.84, stdev=1191.29 clat (msec): min=282, max=3114, avg=1187.51, stdev=407.34 lat (msec): min=282, max=3114, avg=1187.91, stdev=407.40 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 651], 10.00th=[ 735], 20.00th=[ 869], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1217], | 70.00th=[ 1334], 80.00th=[ 1469], 90.00th=[ 1687], 95.00th=[ 1938], | 99.00th=[ 2668], 99.50th=[ 2769], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 7, max= 96, per=25.15%, avg=36.97, stdev=19.42, samples=273 iops : min= 1, max= 24, avg= 8.70, stdev= 4.89, samples=273 lat (msec) : 20=0.08%, 50=0.12%, 100=0.35%, 250=0.35%, 500=22.70% lat (msec) : 750=25.20%, 1000=15.90%, 2000=33.12%, >=2000=2.19% cpu : usr=0.09%, sys=0.59%, ctx=4384, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=141KiB/s (145kB/s), 35.0KiB/s-35.8KiB/s (35.9kB/s-36.7kB/s), io=19.6MiB (20.5MB), run=140958-142067msec WRITE: bw=147KiB/s (151kB/s), 36.7KiB/s-37.1KiB/s (37.5kB/s-37.9kB/s), io=20.4MiB (21.4MB), run=140958-142067msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145868: Wed Sep 30 13:38:37 2026 read: IOPS=1, BW=78.6MiB/s (82.4MB/s)(40.0MiB/509msec) slat (nsec): min=46215k, max=46215k, avg=46215369.00, stdev= 0.00 clat (nsec): min=424593k, max=424593k, avg=424592961.00, stdev= 0.00 lat (nsec): min=470847k, max=470847k, avg=470846893.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.95%, ctx=44, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=78.6MiB/s (82.4MB/s), 78.6MiB/s-78.6MiB/s (82.4MB/s-82.4MB/s), io=40.0MiB (41.9MB), run=509-509msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout