== sanity test 398c: run fio to test AIO ================= 02:28:40 (1786170520) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.677289 s, 61.9 MB/s osc.lustre-OST0000-osc-ffff963f78e18800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145810: Sat Aug 8 02:31:59 2026 write: IOPS=13, BW=55.1KiB/s (56.4kB/s)(10.0MiB/185828msec); 0 zone resets slat (usec): min=38, max=16430, avg=312.89, stdev=927.98 clat (msec): min=377, max=3156, avg=1159.82, stdev=340.54 lat (msec): min=377, max=3156, avg=1160.13, stdev=340.57 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 676], 10.00th=[ 785], 20.00th=[ 911], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1401], 90.00th=[ 1603], 95.00th=[ 1754], | 99.00th=[ 2106], 99.50th=[ 2467], 99.90th=[ 3138], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 7, max= 128, per=25.03%, avg=55.08, stdev=27.43, samples=365 iops : min= 1, max= 32, avg=13.26, stdev= 6.89, samples=365 lat (msec) : 500=0.23%, 750=7.30%, 1000=26.76%, 2000=63.87%, >=2000=1.84% cpu : usr=0.04%, sys=0.26%, ctx=1880, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145811: Sat Aug 8 02:31:59 2026 write: IOPS=13, BW=55.2KiB/s (56.6kB/s)(10.0MiB/185384msec); 0 zone resets slat (usec): min=38, max=28111, avg=352.21, stdev=1258.07 clat (msec): min=219, max=2990, avg=1156.79, stdev=342.99 lat (msec): min=219, max=2990, avg=1157.14, stdev=343.01 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 676], 10.00th=[ 785], 20.00th=[ 894], | 30.00th=[ 969], 40.00th=[ 1045], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1401], 90.00th=[ 1569], 95.00th=[ 1720], | 99.00th=[ 2299], 99.50th=[ 2467], 99.90th=[ 2937], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 134, per=25.97%, avg=57.13, stdev=28.03, samples=352 iops : min= 1, max= 33, avg=13.78, stdev= 7.03, samples=352 lat (msec) : 250=0.04%, 500=0.98%, 750=7.07%, 1000=25.98%, 2000=64.02% lat (msec) : >=2000=1.91% cpu : usr=0.05%, sys=0.29%, ctx=1830, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145812: Sat Aug 8 02:31:59 2026 write: IOPS=13, BW=55.4KiB/s (56.7kB/s)(10.0MiB/184784msec); 0 zone resets slat (usec): min=38, max=20538, avg=353.09, stdev=1163.75 clat (msec): min=88, max=3154, avg=1154.03, stdev=349.33 lat (msec): min=91, max=3155, avg=1154.38, stdev=349.36 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 651], 10.00th=[ 768], 20.00th=[ 894], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1586], 95.00th=[ 1737], | 99.00th=[ 2333], 99.50th=[ 2534], 99.90th=[ 3004], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 7, max= 135, per=25.17%, avg=55.37, stdev=24.42, samples=363 iops : min= 1, max= 33, avg=13.33, stdev= 6.14, samples=363 lat (msec) : 100=0.04%, 250=0.43%, 500=0.43%, 750=8.09%, 1000=25.16% lat (msec) : 2000=63.95%, >=2000=1.91% cpu : usr=0.04%, sys=0.30%, ctx=1974, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145813: Sat Aug 8 02:31:59 2026 write: IOPS=13, BW=55.1KiB/s (56.5kB/s)(10.0MiB/185737msec); 0 zone resets slat (usec): min=39, max=32780, avg=322.69, stdev=1290.58 clat (msec): min=429, max=2678, avg=1158.94, stdev=337.99 lat (msec): min=430, max=2679, avg=1159.26, stdev=338.03 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 693], 10.00th=[ 785], 20.00th=[ 894], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1401], 90.00th=[ 1603], 95.00th=[ 1770], | 99.00th=[ 2232], 99.50th=[ 2467], 99.90th=[ 2601], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 128, per=25.46%, avg=56.00, stdev=31.06, samples=359 iops : min= 1, max= 32, avg=13.49, stdev= 7.77, samples=359 lat (msec) : 500=0.66%, 750=7.19%, 1000=25.66%, 2000=64.45%, >=2000=2.03% cpu : usr=0.03%, sys=0.28%, ctx=1755, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=220KiB/s (226kB/s), 55.1KiB/s-55.4KiB/s (56.4kB/s-56.7kB/s), io=40.0MiB (41.9MB), run=184784-185828msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145849: Sat Aug 8 02:34:16 2026 read: IOPS=9, BW=39.8KiB/s (40.7kB/s)(5048KiB/126971msec) slat (usec): min=3790, max=70673, avg=18209.00, stdev=10072.52 clat (msec): min=30, max=1006, avg=496.62, stdev=132.61 lat (msec): min=45, max=1047, avg=514.84, stdev=134.14 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 313], 10.00th=[ 351], 20.00th=[ 388], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 485], 60.00th=[ 514], | 70.00th=[ 558], 80.00th=[ 609], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 844], 99.50th=[ 902], 99.90th=[ 961], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 7, max= 96, per=24.99%, avg=39.23, stdev=20.87, samples=251 iops : min= 1, max= 24, avg= 9.27, stdev= 5.25, samples=251 write: IOPS=10, BW=40.9KiB/s (41.9kB/s)(5192KiB/126971msec); 0 zone resets slat (usec): min=43, max=25407, avg=452.49, stdev=1363.76 clat (msec): min=396, max=2281, avg=1061.28, stdev=319.93 lat (msec): min=396, max=2281, avg=1061.74, stdev=319.88 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 802], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1502], 95.00th=[ 1720], | 99.00th=[ 2005], 99.50th=[ 2089], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 5, max= 103, per=25.02%, avg=40.79, stdev=19.76, samples=248 iops : min= 1, max= 25, avg= 9.66, stdev= 4.98, samples=248 lat (msec) : 50=0.12%, 100=0.27%, 250=0.43%, 500=26.72%, 750=27.34% lat (msec) : 1000=17.73%, 2000=26.88%, >=2000=0.51% cpu : usr=0.10%, sys=0.67%, ctx=4402, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145850: Sat Aug 8 02:34:16 2026 read: IOPS=9, BW=38.0KiB/s (39.9kB/s)(4976KiB/127741msec) slat (usec): min=2923, max=79613, avg=17832.21, stdev=9918.13 clat (msec): min=115, max=1139, avg=505.42, stdev=133.74 lat (msec): min=126, max=1165, avg=523.27, stdev=135.25 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 317], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 464], 50.00th=[ 493], 60.00th=[ 527], | 70.00th=[ 567], 80.00th=[ 609], 90.00th=[ 676], 95.00th=[ 735], | 99.00th=[ 885], 99.50th=[ 919], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 5, max= 104, per=24.58%, avg=38.59, stdev=19.61, samples=251 iops : min= 1, max= 26, avg= 9.11, stdev= 4.96, samples=251 write: IOPS=10, BW=41.2KiB/s (42.2kB/s)(5264KiB/127741msec); 0 zone resets slat (usec): min=47, max=16145, avg=367.27, stdev=1003.17 clat (msec): min=364, max=2265, avg=1055.28, stdev=320.90 lat (msec): min=364, max=2265, avg=1055.65, stdev=320.88 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 609], 10.00th=[ 676], 20.00th=[ 793], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1519], 95.00th=[ 1703], | 99.00th=[ 1972], 99.50th=[ 2106], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 96, per=25.00%, avg=40.75, stdev=18.47, samples=252 iops : min= 1, max= 24, avg= 9.65, stdev= 4.66, samples=252 lat (msec) : 250=0.55%, 500=25.62%, 750=28.40%, 1000=18.79%, 2000=26.21% lat (msec) : >=2000=0.43% cpu : usr=0.10%, sys=0.66%, ctx=4388, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145851: Sat Aug 8 02:34:16 2026 read: IOPS=9, BW=39.0KiB/s (39.0kB/s)(4984KiB/127735msec) slat (usec): min=2899, max=94320, avg=18144.01, stdev=10241.64 clat (msec): min=35, max=1134, avg=511.11, stdev=138.19 lat (msec): min=55, max=1155, avg=529.26, stdev=139.92 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 321], 10.00th=[ 355], 20.00th=[ 397], | 30.00th=[ 435], 40.00th=[ 464], 50.00th=[ 493], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 684], 95.00th=[ 751], | 99.00th=[ 944], 99.50th=[ 995], 99.90th=[ 1116], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 110, per=25.03%, avg=39.29, stdev=19.18, samples=248 iops : min= 1, max= 27, avg= 9.28, stdev= 4.84, samples=248 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(5256KiB/127735msec); 0 zone resets slat (usec): min=43, max=25134, avg=437.29, stdev=1293.54 clat (msec): min=380, max=2248, avg=1049.36, stdev=314.46 lat (msec): min=380, max=2249, avg=1049.80, stdev=314.45 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 776], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1485], 95.00th=[ 1687], | 99.00th=[ 1888], 99.50th=[ 1989], 99.90th=[ 2198], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 119, per=25.08%, avg=40.88, stdev=19.30, samples=250 iops : min= 1, max= 29, avg= 9.68, stdev= 4.83, samples=250 lat (msec) : 50=0.04%, 100=0.04%, 250=0.31%, 500=25.51%, 750=28.79% lat (msec) : 1000=18.16%, 2000=26.91%, >=2000=0.23% cpu : usr=0.08%, sys=0.74%, ctx=4658, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145852: Sat Aug 8 02:34:16 2026 read: IOPS=9, BW=39.0KiB/s (40.9kB/s)(5060KiB/126565msec) slat (usec): min=80, max=67148, avg=18357.38, stdev=10259.43 clat (msec): min=21, max=1560, avg=503.52, stdev=144.73 lat (msec): min=29, max=1571, avg=521.88, stdev=146.60 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 309], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 430], 40.00th=[ 460], 50.00th=[ 493], 60.00th=[ 523], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 676], 95.00th=[ 743], | 99.00th=[ 894], 99.50th=[ 969], 99.90th=[ 1469], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 101, per=25.74%, avg=40.41, stdev=18.89, samples=244 iops : min= 1, max= 25, avg= 9.56, stdev= 4.76, samples=244 write: IOPS=10, BW=40.9KiB/s (41.9kB/s)(5180KiB/126565msec); 0 zone resets slat (usec): min=46, max=28686, avg=441.41, stdev=1364.81 clat (msec): min=149, max=2222, avg=1052.47, stdev=327.25 lat (msec): min=151, max=2228, avg=1052.92, stdev=327.24 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 869], 40.00th=[ 944], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1502], 95.00th=[ 1720], | 99.00th=[ 1921], 99.50th=[ 2005], 99.90th=[ 2165], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 112, per=25.21%, avg=41.09, stdev=19.00, samples=245 iops : min= 1, max= 28, avg= 9.73, stdev= 4.77, samples=245 lat (msec) : 50=0.27%, 100=0.23%, 250=0.74%, 500=25.82%, 750=28.52% lat (msec) : 1000=17.42%, 2000=26.72%, >=2000=0.27% cpu : usr=0.10%, sys=0.74%, ctx=4595, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=157KiB/s (161kB/s), 38.0KiB/s-39.0KiB/s (39.9kB/s-40.9kB/s), io=19.6MiB (20.5MB), run=126565-127741msec WRITE: bw=164KiB/s (167kB/s), 40.9KiB/s-41.2KiB/s (41.9kB/s-42.2kB/s), io=20.4MiB (21.4MB), run=126565-127741msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145876: Sat Aug 8 02:34:27 2026 read: IOPS=1, BW=60.4MiB/s (63.4MB/s)(40.0MiB/662msec) slat (nsec): min=30379k, max=30379k, avg=30378615.00, stdev= 0.00 clat (nsec): min=594298k, max=594298k, avg=594298268.00, stdev= 0.00 lat (nsec): min=624682k, max=624682k, avg=624682083.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 592], 10.00th=[ 592], 20.00th=[ 592], | 30.00th=[ 592], 40.00th=[ 592], 50.00th=[ 592], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.36%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=60.4MiB/s (63.4MB/s), 60.4MiB/s-60.4MiB/s (63.4MB/s-63.4MB/s), io=40.0MiB (41.9MB), run=662-662msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout