== sanity test 398c: run fio to test AIO ================= 11:45:50 (1787845550) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.355221 s, 118 MB/s osc.lustre-OST0000-osc-ffff988303633000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146415: Thu Aug 27 11:48:32 2026 write: IOPS=16, BW=66.8KiB/s (68.4kB/s)(10.0MiB/153216msec); 0 zone resets slat (usec): min=29, max=9968, avg=231.41, stdev=670.11 clat (msec): min=165, max=1582, avg=956.47, stdev=198.08 lat (msec): min=166, max=1582, avg=956.71, stdev=198.08 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 802], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1062], 80.00th=[ 1116], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1418], 99.50th=[ 1502], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 247, per=25.19%, avg=67.01, stdev=30.60, samples=301 iops : min= 1, max= 61, avg=16.32, stdev= 7.65, samples=301 lat (msec) : 250=0.20%, 500=1.68%, 750=10.39%, 1000=45.00%, 2000=42.73% cpu : usr=0.05%, sys=0.22%, ctx=1755, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146416: Thu Aug 27 11:48:32 2026 write: IOPS=16, BW=66.8KiB/s (68.4kB/s)(10.0MiB/153216msec); 0 zone resets slat (usec): min=28, max=23411, avg=263.25, stdev=1003.10 clat (msec): min=103, max=1537, avg=956.85, stdev=200.19 lat (msec): min=103, max=1537, avg=957.12, stdev=200.19 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1062], 80.00th=[ 1116], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1418], 99.50th=[ 1452], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 128, per=25.27%, avg=67.22, stdev=25.95, samples=300 iops : min= 1, max= 32, avg=16.37, stdev= 6.50, samples=300 lat (msec) : 250=0.62%, 500=1.29%, 750=10.16%, 1000=45.78%, 2000=42.15% cpu : usr=0.05%, sys=0.24%, ctx=2049, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146417: Thu Aug 27 11:48:32 2026 write: IOPS=16, BW=66.7KiB/s (68.3kB/s)(10.0MiB/153494msec); 0 zone resets slat (usec): min=29, max=30565, avg=239.80, stdev=921.70 clat (msec): min=362, max=1563, avg=958.24, stdev=192.08 lat (msec): min=363, max=1563, avg=958.48, stdev=192.10 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 651], 10.00th=[ 735], 20.00th=[ 810], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 961], 60.00th=[ 1003], | 70.00th=[ 1053], 80.00th=[ 1116], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1418], 99.50th=[ 1485], 99.90th=[ 1536], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8, max= 128, per=25.19%, avg=67.01, stdev=27.10, samples=301 iops : min= 2, max= 32, avg=16.32, stdev= 6.79, samples=301 lat (msec) : 500=1.29%, 750=11.29%, 1000=46.41%, 2000=41.02% cpu : usr=0.05%, sys=0.24%, ctx=1811, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146418: Thu Aug 27 11:48:32 2026 write: IOPS=16, BW=66.7KiB/s (68.3kB/s)(10.0MiB/153437msec); 0 zone resets slat (usec): min=28, max=17770, avg=232.67, stdev=740.16 clat (msec): min=83, max=1578, avg=957.93, stdev=195.33 lat (msec): min=83, max=1581, avg=958.16, stdev=195.33 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 667], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1116], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1418], 99.50th=[ 1452], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 136, per=25.29%, avg=67.27, stdev=32.13, samples=300 iops : min= 1, max= 34, avg=16.39, stdev= 8.04, samples=300 lat (msec) : 100=0.04%, 500=1.45%, 750=11.13%, 1000=46.13%, 2000=41.25% cpu : usr=0.05%, sys=0.24%, ctx=1902, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=267KiB/s (273kB/s), 66.7KiB/s-66.8KiB/s (68.3kB/s-68.4kB/s), io=40.0MiB (41.9MB), run=153216-153494msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146449: Thu Aug 27 11:50:54 2026 read: IOPS=9, BW=37.6KiB/s (38.5kB/s)(5048KiB/134411msec) slat (usec): min=2798, max=76445, avg=16725.00, stdev=9740.75 clat (msec): min=12, max=1033, avg=511.64, stdev=142.12 lat (msec): min=28, max=1052, avg=528.38, stdev=143.94 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 313], 10.00th=[ 351], 20.00th=[ 397], | 30.00th=[ 435], 40.00th=[ 468], 50.00th=[ 498], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 709], 95.00th=[ 768], | 99.00th=[ 894], 99.50th=[ 936], 99.90th=[ 1028], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 5, max= 95, per=25.19%, avg=37.28, stdev=20.10, samples=264 iops : min= 1, max= 23, avg= 8.80, stdev= 5.07, samples=264 write: IOPS=9, BW=38.6KiB/s (39.6kB/s)(5192KiB/134411msec); 0 zone resets slat (usec): min=29, max=22806, avg=334.72, stdev=1220.34 clat (msec): min=375, max=2375, avg=1139.31, stdev=305.10 lat (msec): min=375, max=2375, avg=1139.64, stdev=305.13 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 709], 10.00th=[ 785], 20.00th=[ 877], | 30.00th=[ 961], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1401], 90.00th=[ 1552], 95.00th=[ 1687], | 99.00th=[ 2039], 99.50th=[ 2123], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 95, per=24.95%, avg=38.67, stdev=17.80, samples=262 iops : min= 1, max= 23, avg= 9.15, stdev= 4.49, samples=262 lat (msec) : 20=0.04%, 50=0.04%, 100=0.23%, 250=0.51%, 500=24.61% lat (msec) : 750=24.53%, 1000=17.07%, 2000=32.46%, >=2000=0.51% cpu : usr=0.10%, sys=0.54%, ctx=4548, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146450: Thu Aug 27 11:50:54 2026 read: IOPS=9, BW=36.9KiB/s (37.8kB/s)(4976KiB/134697msec) slat (usec): min=3142, max=80696, avg=16294.61, stdev=9199.49 clat (msec): min=15, max=1109, avg=504.45, stdev=130.67 lat (msec): min=35, max=1139, avg=520.75, stdev=131.73 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 317], 10.00th=[ 355], 20.00th=[ 401], | 30.00th=[ 439], 40.00th=[ 468], 50.00th=[ 493], 60.00th=[ 523], | 70.00th=[ 558], 80.00th=[ 609], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 852], 99.50th=[ 936], 99.90th=[ 1083], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 88, per=25.35%, avg=37.52, stdev=19.01, samples=258 iops : min= 1, max= 22, avg= 8.85, stdev= 4.82, samples=258 write: IOPS=9, BW=39.1KiB/s (40.0kB/s)(5264KiB/134697msec); 0 zone resets slat (usec): min=29, max=29566, avg=281.26, stdev=1208.63 clat (msec): min=261, max=2321, avg=1141.99, stdev=309.93 lat (msec): min=262, max=2321, avg=1142.28, stdev=309.99 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 684], 10.00th=[ 760], 20.00th=[ 885], | 30.00th=[ 961], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1401], 90.00th=[ 1552], 95.00th=[ 1687], | 99.00th=[ 2022], 99.50th=[ 2165], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 110, per=24.99%, avg=38.73, stdev=17.12, samples=264 iops : min= 1, max= 27, avg= 9.16, stdev= 4.33, samples=264 lat (msec) : 20=0.04%, 50=0.08%, 100=0.08%, 250=0.35%, 500=25.90% lat (msec) : 750=24.84%, 1000=14.92%, 2000=33.16%, >=2000=0.62% cpu : usr=0.10%, sys=0.56%, ctx=4306, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146451: Thu Aug 27 11:50:54 2026 read: IOPS=9, BW=37.1KiB/s (37.0kB/s)(4984KiB/134352msec) slat (usec): min=52, max=72991, avg=16834.02, stdev=9790.61 clat (msec): min=40, max=1203, avg=510.38, stdev=145.07 lat (msec): min=41, max=1219, avg=527.22, stdev=146.89 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 309], 10.00th=[ 342], 20.00th=[ 397], | 30.00th=[ 435], 40.00th=[ 468], 50.00th=[ 493], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 625], 90.00th=[ 701], 95.00th=[ 768], | 99.00th=[ 911], 99.50th=[ 1028], 99.90th=[ 1167], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 88, per=25.29%, avg=37.43, stdev=18.64, samples=260 iops : min= 1, max= 22, avg= 8.83, stdev= 4.71, samples=260 write: IOPS=9, BW=39.1KiB/s (40.1kB/s)(5256KiB/134352msec); 0 zone resets slat (usec): min=30, max=11324, avg=299.04, stdev=779.67 clat (msec): min=127, max=2310, avg=1134.38, stdev=310.36 lat (msec): min=127, max=2311, avg=1134.68, stdev=310.37 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 659], 10.00th=[ 768], 20.00th=[ 894], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1536], 95.00th=[ 1653], | 99.00th=[ 1938], 99.50th=[ 2198], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 103, per=24.85%, avg=38.52, stdev=19.00, samples=265 iops : min= 1, max= 25, avg= 9.11, stdev= 4.81, samples=265 lat (msec) : 50=0.08%, 100=0.20%, 250=0.86%, 500=24.65%, 750=24.53% lat (msec) : 1000=16.13%, 2000=33.16%, >=2000=0.39% cpu : usr=0.10%, sys=0.52%, ctx=4507, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146452: Thu Aug 27 11:50:54 2026 read: IOPS=9, BW=37.6KiB/s (38.5kB/s)(5060KiB/134526msec) slat (usec): min=3061, max=82015, avg=16237.05, stdev=9433.97 clat (msec): min=23, max=1084, avg=516.07, stdev=141.91 lat (msec): min=33, max=1110, avg=532.32, stdev=143.20 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 313], 10.00th=[ 355], 20.00th=[ 401], | 30.00th=[ 439], 40.00th=[ 477], 50.00th=[ 502], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 625], 90.00th=[ 709], 95.00th=[ 776], | 99.00th=[ 911], 99.50th=[ 953], 99.90th=[ 1011], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 96, per=26.14%, avg=38.69, stdev=18.17, samples=256 iops : min= 1, max= 24, avg= 9.15, stdev= 4.57, samples=256 write: IOPS=9, BW=38.5KiB/s (39.4kB/s)(5180KiB/134526msec); 0 zone resets slat (usec): min=26, max=20995, avg=289.82, stdev=976.92 clat (msec): min=343, max=2349, avg=1138.55, stdev=296.56 lat (msec): min=343, max=2349, avg=1138.85, stdev=296.57 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 709], 10.00th=[ 776], 20.00th=[ 894], | 30.00th=[ 969], 40.00th=[ 1045], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1519], 95.00th=[ 1653], | 99.00th=[ 1905], 99.50th=[ 2123], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 103, per=24.72%, avg=38.32, stdev=17.52, samples=263 iops : min= 1, max= 25, avg= 9.06, stdev= 4.42, samples=263 lat (msec) : 50=0.16%, 100=0.16%, 250=0.62%, 500=23.95%, 750=25.23% lat (msec) : 1000=16.45%, 2000=33.05%, >=2000=0.39% cpu : usr=0.08%, sys=0.55%, ctx=4621, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=149KiB/s (153kB/s), 36.9KiB/s-37.6KiB/s (37.8kB/s-38.5kB/s), io=19.6MiB (20.5MB), run=134352-134697msec WRITE: bw=155KiB/s (159kB/s), 38.5KiB/s-39.1KiB/s (39.4kB/s-40.1kB/s), io=20.4MiB (21.4MB), run=134352-134697msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146476: Thu Aug 27 11:51:04 2026 read: IOPS=2, BW=82.8MiB/s (86.8MB/s)(40.0MiB/483msec) slat (nsec): min=38432k, max=38432k, avg=38431693.00, stdev= 0.00 clat (nsec): min=413646k, max=413646k, avg=413646031.00, stdev= 0.00 lat (nsec): min=452082k, max=452082k, avg=452082410.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.05%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=82.8MiB/s (86.8MB/s), 82.8MiB/s-82.8MiB/s (86.8MB/s-86.8MB/s), io=40.0MiB (41.9MB), run=483-483msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout