== sanity test 398c: run fio to test AIO ================= 09:18:44 (1788441524) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.377387 s, 111 MB/s osc.lustre-OST0000-osc-ffff89b7101de000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145179: Thu Sep 3 09:21:38 2026 write: IOPS=15, BW=62.4KiB/s (63.9kB/s)(10.0MiB/164126msec); 0 zone resets slat (usec): min=23, max=16534, avg=235.84, stdev=868.60 clat (msec): min=231, max=2578, avg=1024.58, stdev=340.69 lat (msec): min=232, max=2578, avg=1024.82, stdev=340.69 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 558], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1687], | 99.00th=[ 1989], 99.50th=[ 2056], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 255, per=26.83%, avg=66.80, stdev=35.51, samples=301 iops : min= 1, max= 63, avg=16.14, stdev= 8.90, samples=301 lat (msec) : 250=0.62%, 500=3.09%, 750=16.02%, 1000=32.89%, 2000=46.45% lat (msec) : >=2000=0.94% cpu : usr=0.05%, sys=0.21%, ctx=1504, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145180: Thu Sep 3 09:21:38 2026 write: IOPS=15, BW=62.3KiB/s (63.8kB/s)(10.0MiB/164415msec); 0 zone resets slat (usec): min=28, max=31337, avg=281.01, stdev=1128.69 clat (msec): min=254, max=2588, avg=1026.48, stdev=340.28 lat (msec): min=254, max=2588, avg=1026.77, stdev=340.29 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 567], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 978], 60.00th=[ 1062], | 70.00th=[ 1150], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1687], | 99.00th=[ 2039], 99.50th=[ 2265], 99.90th=[ 2467], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 175, per=25.06%, avg=62.39, stdev=28.81, samples=322 iops : min= 1, max= 43, avg=15.04, stdev= 7.23, samples=322 lat (msec) : 500=3.28%, 750=15.51%, 1000=33.75%, 2000=46.17%, >=2000=1.29% cpu : usr=0.05%, sys=0.27%, ctx=1959, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145181: Thu Sep 3 09:21:38 2026 write: IOPS=15, BW=62.3KiB/s (63.8kB/s)(10.0MiB/164455msec); 0 zone resets slat (usec): min=24, max=26751, avg=276.56, stdev=1128.74 clat (msec): min=204, max=2588, avg=1026.48, stdev=339.42 lat (msec): min=204, max=2588, avg=1026.76, stdev=339.43 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 978], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1485], 95.00th=[ 1670], | 99.00th=[ 2005], 99.50th=[ 2198], 99.90th=[ 2500], 99.95th=[ 2567], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 223, per=25.79%, avg=64.21, stdev=30.63, samples=313 iops : min= 1, max= 55, avg=15.49, stdev= 7.66, samples=313 lat (msec) : 250=0.62%, 500=2.77%, 750=16.17%, 1000=32.66%, 2000=46.76% lat (msec) : >=2000=1.02% cpu : usr=0.05%, sys=0.25%, ctx=1855, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145182: Thu Sep 3 09:21:38 2026 write: IOPS=15, BW=62.5KiB/s (63.0kB/s)(10.0MiB/163881msec); 0 zone resets slat (usec): min=28, max=24031, avg=244.41, stdev=793.50 clat (msec): min=109, max=2587, avg=1023.60, stdev=338.98 lat (msec): min=110, max=2587, avg=1023.85, stdev=338.95 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 558], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 978], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1653], | 99.00th=[ 2005], 99.50th=[ 2123], 99.90th=[ 2467], 99.95th=[ 2534], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 193, per=25.04%, avg=62.35, stdev=30.72, samples=322 iops : min= 1, max= 48, avg=15.03, stdev= 7.69, samples=322 lat (msec) : 250=0.74%, 500=2.89%, 750=15.20%, 1000=32.89%, 2000=47.27% lat (msec) : >=2000=1.02% cpu : usr=0.04%, sys=0.23%, ctx=1945, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=249KiB/s (255kB/s), 62.3KiB/s-62.5KiB/s (63.8kB/s-63.0kB/s), io=40.0MiB (41.9MB), run=163881-164455msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145215: Thu Sep 3 09:23:52 2026 read: IOPS=9, BW=39.6KiB/s (40.6kB/s)(5048KiB/127381msec) slat (usec): min=34, max=73010, avg=16341.02, stdev=9211.15 clat (msec): min=19, max=1678, avg=499.00, stdev=144.86 lat (msec): min=25, max=1689, avg=515.36, stdev=146.38 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 305], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 460], 50.00th=[ 485], 60.00th=[ 518], | 70.00th=[ 558], 80.00th=[ 600], 90.00th=[ 676], 95.00th=[ 726], | 99.00th=[ 869], 99.50th=[ 1045], 99.90th=[ 1603], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 111, per=25.38%, avg=39.59, stdev=19.07, samples=249 iops : min= 1, max= 27, avg= 9.39, stdev= 4.79, samples=249 write: IOPS=10, BW=40.8KiB/s (41.7kB/s)(5192KiB/127381msec); 0 zone resets slat (usec): min=32, max=23366, avg=290.86, stdev=946.12 clat (msec): min=170, max=2122, avg=1067.43, stdev=300.44 lat (msec): min=170, max=2122, avg=1067.73, stdev=300.40 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 969], 50.00th=[ 1062], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1435], 95.00th=[ 1569], | 99.00th=[ 1955], 99.50th=[ 2022], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 112, per=24.79%, avg=40.41, stdev=19.87, samples=251 iops : min= 1, max= 28, avg= 9.59, stdev= 5.01, samples=251 lat (msec) : 20=0.04%, 50=0.31%, 100=0.16%, 250=0.70%, 500=26.64% lat (msec) : 750=27.07%, 1000=16.05%, 2000=28.67%, >=2000=0.35% cpu : usr=0.10%, sys=0.54%, ctx=4677, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145216: Thu Sep 3 09:23:52 2026 read: IOPS=9, BW=38.9KiB/s (39.8kB/s)(4976KiB/127934msec) slat (usec): min=3008, max=80473, avg=16947.34, stdev=9541.45 clat (msec): min=16, max=1241, avg=500.39, stdev=132.43 lat (msec): min=21, max=1246, avg=517.35, stdev=133.54 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 317], 10.00th=[ 359], 20.00th=[ 401], | 30.00th=[ 430], 40.00th=[ 460], 50.00th=[ 485], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 726], | 99.00th=[ 877], 99.50th=[ 902], 99.90th=[ 1167], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 88, per=24.91%, avg=38.86, stdev=18.96, samples=250 iops : min= 1, max= 22, avg= 9.21, stdev= 4.77, samples=250 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(5264KiB/127934msec); 0 zone resets slat (usec): min=34, max=25220, avg=317.16, stdev=1018.14 clat (msec): min=204, max=2167, avg=1063.25, stdev=291.19 lat (msec): min=204, max=2167, avg=1063.57, stdev=291.18 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 592], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1284], 90.00th=[ 1418], 95.00th=[ 1519], | 99.00th=[ 1921], 99.50th=[ 2089], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 88, per=24.94%, avg=40.65, stdev=18.72, samples=253 iops : min= 1, max= 22, avg= 9.66, stdev= 4.70, samples=253 lat (msec) : 20=0.08%, 50=0.12%, 250=0.55%, 500=27.46%, 750=24.73% lat (msec) : 1000=18.01%, 2000=28.67%, >=2000=0.39% cpu : usr=0.09%, sys=0.59%, ctx=4599, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145217: Thu Sep 3 09:23:52 2026 read: IOPS=9, BW=38.0KiB/s (39.9kB/s)(4984KiB/127936msec) slat (usec): min=3019, max=90457, avg=16498.62, stdev=9393.71 clat (msec): min=21, max=933, avg=491.76, stdev=127.62 lat (msec): min=45, max=972, avg=508.26, stdev=128.97 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 309], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 485], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 667], 95.00th=[ 709], | 99.00th=[ 869], 99.50th=[ 911], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 118, per=25.13%, avg=39.20, stdev=18.51, samples=249 iops : min= 1, max= 29, avg= 9.29, stdev= 4.66, samples=249 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(5256KiB/127936msec); 0 zone resets slat (usec): min=29, max=20375, avg=344.34, stdev=1283.98 clat (msec): min=402, max=2152, avg=1071.61, stdev=289.29 lat (msec): min=402, max=2152, avg=1071.96, stdev=289.32 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 634], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 902], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1301], 90.00th=[ 1435], 95.00th=[ 1552], | 99.00th=[ 1938], 99.50th=[ 2056], 99.90th=[ 2123], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 104, per=24.95%, avg=40.67, stdev=17.41, samples=252 iops : min= 1, max= 26, avg= 9.66, stdev= 4.40, samples=252 lat (msec) : 50=0.12%, 100=0.16%, 250=0.47%, 500=27.38%, 750=25.12% lat (msec) : 1000=17.11%, 2000=29.22%, >=2000=0.43% cpu : usr=0.09%, sys=0.57%, ctx=4168, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145218: Thu Sep 3 09:23:52 2026 read: IOPS=9, BW=39.0KiB/s (40.9kB/s)(5060KiB/126652msec) slat (usec): min=3455, max=73949, avg=16507.74, stdev=9531.08 clat (msec): min=29, max=1200, avg=489.97, stdev=127.12 lat (msec): min=50, max=1222, avg=506.48, stdev=128.59 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 321], 10.00th=[ 355], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 451], 50.00th=[ 477], 60.00th=[ 502], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 709], | 99.00th=[ 869], 99.50th=[ 911], 99.90th=[ 1116], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 111, per=25.95%, avg=40.48, stdev=18.43, samples=245 iops : min= 1, max= 27, avg= 9.61, stdev= 4.59, samples=245 write: IOPS=10, BW=40.9KiB/s (41.9kB/s)(5180KiB/126652msec); 0 zone resets slat (usec): min=27, max=15530, avg=333.78, stdev=992.60 clat (msec): min=409, max=2186, avg=1065.85, stdev=295.86 lat (msec): min=409, max=2186, avg=1066.18, stdev=295.82 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1301], 90.00th=[ 1418], 95.00th=[ 1536], | 99.00th=[ 1972], 99.50th=[ 2056], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 96, per=25.08%, avg=40.89, stdev=18.27, samples=247 iops : min= 1, max= 24, avg= 9.72, stdev= 4.62, samples=247 lat (msec) : 50=0.08%, 100=0.20%, 250=0.59%, 500=28.87%, 750=25.16% lat (msec) : 1000=16.56%, 2000=28.16%, >=2000=0.39% cpu : usr=0.11%, sys=0.59%, ctx=4581, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=157KiB/s (161kB/s), 38.9KiB/s-39.0KiB/s (39.8kB/s-40.9kB/s), io=19.6MiB (20.5MB), run=126652-127936msec WRITE: bw=163KiB/s (167kB/s), 40.8KiB/s-41.1KiB/s (41.7kB/s-42.1kB/s), io=20.4MiB (21.4MB), run=126652-127936msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145241: Thu Sep 3 09:24:00 2026 read: IOPS=1, BW=75.9MiB/s (79.6MB/s)(40.0MiB/527msec) slat (nsec): min=30515k, max=30515k, avg=30515316.00, stdev= 0.00 clat (nsec): min=480663k, max=480663k, avg=480662617.00, stdev= 0.00 lat (nsec): min=511183k, max=511183k, avg=511183453.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 481], 10.00th=[ 481], 20.00th=[ 481], | 30.00th=[ 481], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.28%, ctx=25, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=75.9MiB/s (79.6MB/s), 75.9MiB/s-75.9MiB/s (79.6MB/s-79.6MB/s), io=40.0MiB (41.9MB), run=527-527msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout