== sanity test 398c: run fio to test AIO ================= 09:33:51 (1761312831) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.415872 s, 101 MB/s osc.lustre-OST0000-osc-ffff998445369800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=124500: Fri Oct 24 09:35:45 2025 write: IOPS=23, BW=95.5KiB/s (97.8kB/s)(10.0MiB/107193msec); 0 zone resets slat (usec): min=25, max=5772, avg=190.68, stdev=403.30 clat (msec): min=155, max=1371, avg=669.30, stdev=169.63 lat (msec): min=155, max=1371, avg=669.49, stdev=169.64 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 422], 10.00th=[ 464], 20.00th=[ 535], | 30.00th=[ 584], 40.00th=[ 625], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 743], 80.00th=[ 802], 90.00th=[ 894], 95.00th=[ 969], | 99.00th=[ 1083], 99.50th=[ 1133], 99.90th=[ 1250], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 15, max= 256, per=24.82%, avg=94.82, stdev=32.32, samples=213 iops : min= 3, max= 64, avg=23.28, stdev= 8.11, samples=213 lat (msec) : 250=1.25%, 500=13.63%, 750=56.41%, 1000=25.66%, 2000=3.05% cpu : usr=0.07%, sys=0.36%, ctx=1654, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=124501: Fri Oct 24 09:35:45 2025 write: IOPS=23, BW=95.6KiB/s (97.9kB/s)(10.0MiB/107136msec); 0 zone resets slat (usec): min=25, max=6147, avg=188.00, stdev=412.48 clat (msec): min=141, max=1337, avg=668.77, stdev=172.19 lat (msec): min=141, max=1337, avg=668.96, stdev=172.20 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 414], 10.00th=[ 456], 20.00th=[ 535], | 30.00th=[ 584], 40.00th=[ 625], 50.00th=[ 667], 60.00th=[ 709], | 70.00th=[ 751], 80.00th=[ 810], 90.00th=[ 885], 95.00th=[ 961], | 99.00th=[ 1083], 99.50th=[ 1116], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1334] bw ( KiB/s): min= 15, max= 256, per=24.82%, avg=94.82, stdev=32.08, samples=213 iops : min= 3, max= 64, avg=23.28, stdev= 8.09, samples=213 lat (msec) : 250=1.33%, 500=13.71%, 750=54.30%, 1000=27.62%, 2000=3.05% cpu : usr=0.06%, sys=0.37%, ctx=1750, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=124502: Fri Oct 24 09:35:45 2025 write: IOPS=23, BW=95.5KiB/s (97.8kB/s)(10.0MiB/107195msec); 0 zone resets slat (usec): min=25, max=5169, avg=182.43, stdev=358.48 clat (msec): min=71, max=1372, avg=669.33, stdev=170.98 lat (msec): min=73, max=1372, avg=669.51, stdev=170.98 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 414], 10.00th=[ 464], 20.00th=[ 535], | 30.00th=[ 584], 40.00th=[ 625], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 743], 80.00th=[ 802], 90.00th=[ 894], 95.00th=[ 953], | 99.00th=[ 1099], 99.50th=[ 1133], 99.90th=[ 1318], 99.95th=[ 1334], | 99.99th=[ 1368] bw ( KiB/s): min= 15, max= 248, per=24.82%, avg=94.81, stdev=32.64, samples=213 iops : min= 3, max= 62, avg=23.28, stdev= 8.23, samples=213 lat (msec) : 100=0.04%, 250=1.29%, 500=14.22%, 750=55.35%, 1000=25.94% lat (msec) : 2000=3.16% cpu : usr=0.05%, sys=0.35%, ctx=1763, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=124503: Fri Oct 24 09:35:45 2025 write: IOPS=23, BW=95.8KiB/s (98.1kB/s)(10.0MiB/106852msec); 0 zone resets slat (usec): min=26, max=14772, avg=204.08, stdev=532.28 clat (msec): min=61, max=1372, avg=667.32, stdev=175.39 lat (msec): min=61, max=1372, avg=667.52, stdev=175.42 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 409], 10.00th=[ 451], 20.00th=[ 531], | 30.00th=[ 584], 40.00th=[ 625], 50.00th=[ 667], 60.00th=[ 709], | 70.00th=[ 751], 80.00th=[ 802], 90.00th=[ 885], 95.00th=[ 953], | 99.00th=[ 1099], 99.50th=[ 1133], 99.90th=[ 1250], 99.95th=[ 1301], | 99.99th=[ 1368] bw ( KiB/s): min= 23, max= 314, per=24.93%, avg=95.23, stdev=32.91, samples=212 iops : min= 5, max= 78, avg=23.38, stdev= 8.28, samples=212 lat (msec) : 100=0.47%, 250=1.25%, 500=14.26%, 750=53.67%, 1000=26.99% lat (msec) : 2000=3.36% cpu : usr=0.06%, sys=0.36%, ctx=1802, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=382KiB/s (391kB/s), 95.5KiB/s-95.8KiB/s (97.8kB/s-98.1kB/s), io=40.0MiB (41.9MB), run=106852-107195msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=124528: Fri Oct 24 09:37:07 2025 read: IOPS=16, BW=65.3KiB/s (66.9kB/s)(5048KiB/77275msec) slat (usec): min=2115, max=31733, avg=7737.83, stdev=3674.51 clat (msec): min=15, max=788, avg=314.34, stdev=79.09 lat (msec): min=19, max=794, avg=322.09, stdev=79.67 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 197], 10.00th=[ 224], 20.00th=[ 255], | 30.00th=[ 275], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 330], | 70.00th=[ 347], 80.00th=[ 376], 90.00th=[ 409], 95.00th=[ 443], | 99.00th=[ 527], 99.50th=[ 542], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 16, max= 152, per=25.11%, avg=65.05, stdev=25.33, samples=153 iops : min= 4, max= 38, avg=15.79, stdev= 6.35, samples=153 write: IOPS=16, BW=67.2KiB/s (68.8kB/s)(5192KiB/77275msec); 0 zone resets slat (usec): min=28, max=7009, avg=218.24, stdev=467.70 clat (msec): min=44, max=1441, avg=637.58, stdev=179.33 lat (msec): min=45, max=1441, avg=637.80, stdev=179.34 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 372], 10.00th=[ 426], 20.00th=[ 485], | 30.00th=[ 531], 40.00th=[ 575], 50.00th=[ 617], 60.00th=[ 667], | 70.00th=[ 718], 80.00th=[ 785], 90.00th=[ 860], 95.00th=[ 953], | 99.00th=[ 1116], 99.50th=[ 1250], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 16, max= 120, per=24.66%, avg=66.59, stdev=23.87, samples=153 iops : min= 4, max= 30, avg=16.18, stdev= 6.01, samples=153 lat (msec) : 20=0.04%, 50=0.16%, 100=0.27%, 250=8.36%, 500=51.13% lat (msec) : 750=27.15%, 1000=11.56%, 2000=1.33% cpu : usr=0.12%, sys=0.84%, ctx=3219, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=124529: Fri Oct 24 09:37:07 2025 read: IOPS=16, BW=64.3KiB/s (65.9kB/s)(4976KiB/77370msec) slat (usec): min=1759, max=30887, avg=7666.62, stdev=3356.90 clat (msec): min=4, max=539, avg=314.32, stdev=71.39 lat (msec): min=10, max=547, avg=322.00, stdev=72.17 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 207], 10.00th=[ 228], 20.00th=[ 257], | 30.00th=[ 279], 40.00th=[ 296], 50.00th=[ 313], 60.00th=[ 330], | 70.00th=[ 347], 80.00th=[ 372], 90.00th=[ 405], 95.00th=[ 439], | 99.00th=[ 502], 99.50th=[ 523], 99.90th=[ 535], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 7, max= 136, per=24.57%, avg=63.63, stdev=27.36, samples=154 iops : min= 1, max= 34, avg=15.44, stdev= 6.88, samples=154 write: IOPS=17, BW=68.0KiB/s (69.7kB/s)(5264KiB/77370msec); 0 zone resets slat (usec): min=27, max=8731, avg=216.71, stdev=442.09 clat (msec): min=177, max=1322, avg=635.05, stdev=170.79 lat (msec): min=177, max=1322, avg=635.27, stdev=170.78 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 380], 10.00th=[ 430], 20.00th=[ 485], | 30.00th=[ 535], 40.00th=[ 575], 50.00th=[ 617], 60.00th=[ 667], | 70.00th=[ 726], 80.00th=[ 785], 90.00th=[ 860], 95.00th=[ 936], | 99.00th=[ 1062], 99.50th=[ 1116], 99.90th=[ 1284], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 15, max= 120, per=24.86%, avg=67.12, stdev=23.17, samples=154 iops : min= 3, max= 30, avg=16.31, stdev= 5.82, samples=154 lat (msec) : 10=0.04%, 20=0.08%, 50=0.04%, 100=0.16%, 250=8.32% lat (msec) : 500=51.09%, 750=27.42%, 1000=11.84%, 2000=1.02% cpu : usr=0.10%, sys=0.82%, ctx=3111, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=124530: Fri Oct 24 09:37:07 2025 read: IOPS=16, BW=64.6KiB/s (66.1kB/s)(4984KiB/77162msec) slat (usec): min=36, max=32230, avg=7671.69, stdev=3623.44 clat (msec): min=7, max=1131, avg=315.21, stdev=81.60 lat (msec): min=10, max=1137, avg=322.88, stdev=82.13 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 207], 10.00th=[ 230], 20.00th=[ 255], | 30.00th=[ 275], 40.00th=[ 296], 50.00th=[ 313], 60.00th=[ 330], | 70.00th=[ 347], 80.00th=[ 372], 90.00th=[ 409], 95.00th=[ 439], | 99.00th=[ 518], 99.50th=[ 535], 99.90th=[ 1070], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 151, per=24.81%, avg=64.27, stdev=26.41, samples=153 iops : min= 1, max= 37, avg=15.59, stdev= 6.64, samples=153 write: IOPS=17, BW=68.1KiB/s (69.8kB/s)(5256KiB/77162msec); 0 zone resets slat (usec): min=29, max=7962, avg=215.99, stdev=426.64 clat (msec): min=73, max=1496, avg=632.31, stdev=180.47 lat (msec): min=75, max=1496, avg=632.53, stdev=180.49 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 380], 10.00th=[ 422], 20.00th=[ 485], | 30.00th=[ 531], 40.00th=[ 567], 50.00th=[ 609], 60.00th=[ 659], | 70.00th=[ 709], 80.00th=[ 776], 90.00th=[ 877], 95.00th=[ 961], | 99.00th=[ 1070], 99.50th=[ 1301], 99.90th=[ 1401], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 128, per=24.94%, avg=67.35, stdev=21.78, samples=153 iops : min= 1, max= 32, avg=16.37, stdev= 5.45, samples=153 lat (msec) : 10=0.04%, 20=0.08%, 50=0.16%, 100=0.08%, 250=8.20% lat (msec) : 500=51.41%, 750=27.07%, 1000=11.88%, 2000=1.09% cpu : usr=0.11%, sys=0.81%, ctx=3022, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=124531: Fri Oct 24 09:37:07 2025 read: IOPS=16, BW=65.6KiB/s (67.2kB/s)(5060KiB/77157msec) slat (usec): min=1917, max=24148, avg=7769.58, stdev=3548.32 clat (msec): min=8, max=1001, avg=315.42, stdev=81.61 lat (msec): min=14, max=1007, avg=323.20, stdev=82.20 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 201], 10.00th=[ 226], 20.00th=[ 257], | 30.00th=[ 275], 40.00th=[ 296], 50.00th=[ 313], 60.00th=[ 330], | 70.00th=[ 347], 80.00th=[ 372], 90.00th=[ 405], 95.00th=[ 439], | 99.00th=[ 535], 99.50th=[ 567], 99.90th=[ 995], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 15, max= 143, per=25.14%, avg=65.11, stdev=24.44, samples=153 iops : min= 3, max= 35, avg=15.80, stdev= 6.15, samples=153 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(5180KiB/77157msec); 0 zone resets slat (usec): min=27, max=7104, avg=216.94, stdev=440.95 clat (msec): min=168, max=1439, avg=635.88, stdev=178.68 lat (msec): min=169, max=1439, avg=636.10, stdev=178.68 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 376], 10.00th=[ 426], 20.00th=[ 485], | 30.00th=[ 531], 40.00th=[ 567], 50.00th=[ 617], 60.00th=[ 667], | 70.00th=[ 726], 80.00th=[ 785], 90.00th=[ 869], 95.00th=[ 944], | 99.00th=[ 1133], 99.50th=[ 1217], 99.90th=[ 1334], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 16, max= 128, per=24.81%, avg=66.97, stdev=22.53, samples=152 iops : min= 4, max= 32, avg=16.27, stdev= 5.66, samples=152 lat (msec) : 10=0.04%, 50=0.16%, 100=0.27%, 250=8.36%, 500=52.11% lat (msec) : 750=26.13%, 1000=11.37%, 2000=1.56% cpu : usr=0.10%, sys=0.83%, ctx=3315, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=259KiB/s (266kB/s), 64.3KiB/s-65.6KiB/s (65.9kB/s-67.2kB/s), io=19.6MiB (20.5MB), run=77157-77370msec WRITE: bw=270KiB/s (277kB/s), 67.1KiB/s-68.1KiB/s (68.7kB/s-69.8kB/s), io=20.4MiB (21.4MB), run=77157-77370msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=124545: Fri Oct 24 09:37:13 2025 read: IOPS=2, BW=112MiB/s (118MB/s)(40.0MiB/356msec) slat (nsec): min=24326k, max=24326k, avg=24325617.00, stdev= 0.00 clat (nsec): min=320236k, max=320236k, avg=320235842.00, stdev= 0.00 lat (nsec): min=344566k, max=344566k, avg=344566476.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.51%, ctx=2, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=112MiB/s (118MB/s), 112MiB/s-112MiB/s (118MB/s-118MB/s), io=40.0MiB (41.9MB), run=356-356msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck