== sanity test 398c: run fio to test AIO ================= 12:35:32 (1786034132) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.832345 s, 50.4 MB/s osc.lustre-OST0000-osc-ffff8f75df5aa000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144185: Thu Aug 6 12:40:11 2026 write: IOPS=9, BW=38.7KiB/s (39.6kB/s)(10.0MiB/264704msec); 0 zone resets slat (usec): min=49, max=16765, avg=195.54, stdev=610.62 clat (msec): min=384, max=4663, avg=1653.07, stdev=668.16 lat (msec): min=384, max=4663, avg=1653.27, stdev=668.19 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 651], 10.00th=[ 768], 20.00th=[ 995], | 30.00th=[ 1234], 40.00th=[ 1452], 50.00th=[ 1670], 60.00th=[ 1838], | 70.00th=[ 2022], 80.00th=[ 2232], 90.00th=[ 2500], 95.00th=[ 2802], | 99.00th=[ 3239], 99.50th=[ 3675], 99.90th=[ 4597], 99.95th=[ 4597], | 99.99th=[ 4665] bw ( KiB/s): min= 7, max= 128, per=27.67%, avg=42.61, stdev=29.16, samples=471 iops : min= 1, max= 32, avg=10.21, stdev= 7.33, samples=471 lat (msec) : 500=1.17%, 750=8.01%, 1000=10.98%, 2000=48.87%, >=2000=30.98% cpu : usr=0.02%, sys=0.14%, ctx=1673, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144186: Thu Aug 6 12:40:11 2026 write: IOPS=9, BW=38.6KiB/s (39.5kB/s)(10.0MiB/265210msec); 0 zone resets slat (usec): min=49, max=49171, avg=234.42, stdev=1256.97 clat (msec): min=444, max=3848, avg=1654.91, stdev=663.24 lat (msec): min=444, max=3848, avg=1655.15, stdev=663.27 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 651], 10.00th=[ 751], 20.00th=[ 995], | 30.00th=[ 1234], 40.00th=[ 1485], 50.00th=[ 1687], 60.00th=[ 1854], | 70.00th=[ 2022], 80.00th=[ 2232], 90.00th=[ 2500], 95.00th=[ 2769], | 99.00th=[ 3272], 99.50th=[ 3574], 99.90th=[ 3842], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 7, max= 128, per=27.92%, avg=42.99, stdev=27.92, samples=467 iops : min= 1, max= 32, avg=10.30, stdev= 7.01, samples=467 lat (msec) : 500=0.47%, 750=9.69%, 1000=10.47%, 2000=48.59%, >=2000=30.78% cpu : usr=0.02%, sys=0.14%, ctx=1606, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144187: Thu Aug 6 12:40:11 2026 write: IOPS=9, BW=38.6KiB/s (39.6kB/s)(10.0MiB/265062msec); 0 zone resets slat (usec): min=50, max=33051, avg=234.38, stdev=1126.48 clat (msec): min=195, max=4248, avg=1654.17, stdev=666.12 lat (msec): min=195, max=4248, avg=1654.41, stdev=666.15 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 651], 10.00th=[ 751], 20.00th=[ 978], | 30.00th=[ 1234], 40.00th=[ 1469], 50.00th=[ 1670], 60.00th=[ 1838], | 70.00th=[ 2005], 80.00th=[ 2232], 90.00th=[ 2500], 95.00th=[ 2735], | 99.00th=[ 3272], 99.50th=[ 3440], 99.90th=[ 4144], 99.95th=[ 4245], | 99.99th=[ 4245] bw ( KiB/s): min= 7, max= 128, per=27.56%, avg=42.44, stdev=27.63, samples=473 iops : min= 1, max= 32, avg=10.16, stdev= 6.95, samples=473 lat (msec) : 250=0.04%, 500=1.21%, 750=8.67%, 1000=10.90%, 2000=48.67% lat (msec) : >=2000=30.51% cpu : usr=0.03%, sys=0.16%, ctx=1878, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144188: Thu Aug 6 12:40:11 2026 write: IOPS=9, BW=38.6KiB/s (39.5kB/s)(10.0MiB/265213msec); 0 zone resets slat (usec): min=50, max=18942, avg=236.48, stdev=928.44 clat (msec): min=436, max=3830, avg=1655.18, stdev=651.18 lat (msec): min=436, max=3830, avg=1655.42, stdev=651.19 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 651], 10.00th=[ 768], 20.00th=[ 1011], | 30.00th=[ 1267], 40.00th=[ 1469], 50.00th=[ 1670], 60.00th=[ 1854], | 70.00th=[ 2039], 80.00th=[ 2232], 90.00th=[ 2467], 95.00th=[ 2702], | 99.00th=[ 3205], 99.50th=[ 3507], 99.90th=[ 3842], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 7, max= 144, per=27.92%, avg=42.99, stdev=27.70, samples=467 iops : min= 1, max= 36, avg=10.30, stdev= 6.95, samples=467 lat (msec) : 500=0.51%, 750=8.63%, 1000=10.43%, 2000=48.79%, >=2000=31.64% cpu : usr=0.03%, sys=0.14%, ctx=1612, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=154KiB/s (158kB/s), 38.6KiB/s-38.7KiB/s (39.5kB/s-39.6kB/s), io=40.0MiB (41.9MB), run=264704-265213msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144237: Thu Aug 6 12:43:30 2026 read: IOPS=6, BW=26.9KiB/s (27.6kB/s)(5048KiB/187568msec) slat (msec): min=2, max=113, avg=20.30, stdev=14.59 clat (msec): min=18, max=1928, avg=745.36, stdev=294.78 lat (msec): min=26, max=1965, avg=765.67, stdev=298.63 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 313], 10.00th=[ 359], 20.00th=[ 472], | 30.00th=[ 575], 40.00th=[ 667], 50.00th=[ 735], 60.00th=[ 810], | 70.00th=[ 894], 80.00th=[ 995], 90.00th=[ 1133], 95.00th=[ 1250], | 99.00th=[ 1502], 99.50th=[ 1636], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 87, per=27.72%, avg=29.38, stdev=17.59, samples=332 iops : min= 1, max= 21, avg= 6.80, stdev= 4.42, samples=332 write: IOPS=6, BW=27.7KiB/s (28.3kB/s)(5192KiB/187568msec); 0 zone resets slat (usec): min=54, max=90701, avg=405.37, stdev=3010.08 clat (msec): min=381, max=3950, avg=1563.76, stdev=607.74 lat (msec): min=382, max=3950, avg=1564.17, stdev=607.60 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 718], 10.00th=[ 810], 20.00th=[ 995], | 30.00th=[ 1167], 40.00th=[ 1351], 50.00th=[ 1552], 60.00th=[ 1703], | 70.00th=[ 1871], 80.00th=[ 2039], 90.00th=[ 2366], 95.00th=[ 2702], | 99.00th=[ 3138], 99.50th=[ 3306], 99.90th=[ 3742], 99.95th=[ 3943], | 99.99th=[ 3943] bw ( KiB/s): min= 7, max= 88, per=25.83%, avg=28.67, stdev=17.81, samples=351 iops : min= 1, max= 22, avg= 6.62, stdev= 4.48, samples=351 lat (msec) : 20=0.04%, 50=0.04%, 100=0.04%, 250=0.86%, 500=10.55% lat (msec) : 750=17.07%, 1000=21.60%, 2000=38.52%, >=2000=11.29% cpu : usr=0.06%, sys=0.40%, ctx=4261, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144238: Thu Aug 6 12:43:30 2026 read: IOPS=6, BW=26.5KiB/s (27.1kB/s)(4976KiB/187675msec) slat (usec): min=1952, max=98062, avg=20642.38, stdev=15318.72 clat (msec): min=23, max=1703, avg=736.73, stdev=279.87 lat (msec): min=45, max=1719, avg=757.38, stdev=283.86 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 309], 10.00th=[ 372], 20.00th=[ 477], | 30.00th=[ 567], 40.00th=[ 651], 50.00th=[ 735], 60.00th=[ 802], | 70.00th=[ 885], 80.00th=[ 986], 90.00th=[ 1099], 95.00th=[ 1217], | 99.00th=[ 1401], 99.50th=[ 1452], 99.90th=[ 1670], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 96, per=26.28%, avg=27.85, stdev=17.74, samples=345 iops : min= 1, max= 24, avg= 6.42, stdev= 4.49, samples=345 write: IOPS=7, BW=28.0KiB/s (28.7kB/s)(5264KiB/187675msec); 0 zone resets slat (usec): min=54, max=61006, avg=412.84, stdev=2671.14 clat (msec): min=338, max=3872, avg=1562.59, stdev=611.91 lat (msec): min=338, max=3872, avg=1563.01, stdev=611.93 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 726], 10.00th=[ 818], 20.00th=[ 978], | 30.00th=[ 1183], 40.00th=[ 1334], 50.00th=[ 1536], 60.00th=[ 1687], | 70.00th=[ 1871], 80.00th=[ 2039], 90.00th=[ 2366], 95.00th=[ 2735], | 99.00th=[ 3171], 99.50th=[ 3306], 99.90th=[ 3876], 99.95th=[ 3876], | 99.99th=[ 3876] bw ( KiB/s): min= 7, max= 104, per=26.15%, avg=29.02, stdev=18.77, samples=351 iops : min= 1, max= 26, avg= 6.72, stdev= 4.73, samples=351 lat (msec) : 50=0.12%, 100=0.08%, 250=0.47%, 500=11.09%, 750=17.03% lat (msec) : 1000=21.41%, 2000=38.28%, >=2000=11.52% cpu : usr=0.04%, sys=0.42%, ctx=4174, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144239: Thu Aug 6 12:43:30 2026 read: IOPS=6, BW=26.6KiB/s (27.2kB/s)(4984KiB/187567msec) slat (usec): min=59, max=147358, avg=20172.76, stdev=14902.38 clat (msec): min=22, max=1941, avg=741.22, stdev=296.09 lat (msec): min=26, max=1965, avg=761.40, stdev=299.80 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 275], 10.00th=[ 351], 20.00th=[ 464], | 30.00th=[ 575], 40.00th=[ 667], 50.00th=[ 735], 60.00th=[ 818], | 70.00th=[ 902], 80.00th=[ 986], 90.00th=[ 1116], 95.00th=[ 1217], | 99.00th=[ 1502], 99.50th=[ 1703], 99.90th=[ 1838], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 128, per=26.86%, avg=28.47, stdev=17.71, samples=339 iops : min= 1, max= 32, avg= 6.58, stdev= 4.47, samples=339 write: IOPS=7, BW=28.0KiB/s (28.7kB/s)(5256KiB/187567msec); 0 zone resets slat (usec): min=52, max=54075, avg=388.15, stdev=2254.79 clat (msec): min=232, max=3816, avg=1560.31, stdev=618.94 lat (msec): min=235, max=3816, avg=1560.70, stdev=618.93 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 718], 10.00th=[ 810], 20.00th=[ 995], | 30.00th=[ 1167], 40.00th=[ 1318], 50.00th=[ 1502], 60.00th=[ 1687], | 70.00th=[ 1871], 80.00th=[ 2056], 90.00th=[ 2333], 95.00th=[ 2702], | 99.00th=[ 3205], 99.50th=[ 3339], 99.90th=[ 3775], 99.95th=[ 3809], | 99.99th=[ 3809] bw ( KiB/s): min= 7, max= 111, per=26.15%, avg=29.03, stdev=17.78, samples=349 iops : min= 1, max= 27, avg= 6.71, stdev= 4.47, samples=349 lat (msec) : 50=0.20%, 100=0.31%, 250=1.05%, 500=10.08%, 750=16.68% lat (msec) : 1000=21.76%, 2000=37.97%, >=2000=11.95% cpu : usr=0.05%, sys=0.40%, ctx=4198, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144240: Thu Aug 6 12:43:30 2026 read: IOPS=6, BW=27.1KiB/s (27.8kB/s)(5060KiB/186633msec) slat (msec): min=2, max=103, avg=20.71, stdev=14.94 clat (msec): min=25, max=1699, avg=719.22, stdev=283.16 lat (msec): min=34, max=1720, avg=739.93, stdev=287.84 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 300], 10.00th=[ 355], 20.00th=[ 451], | 30.00th=[ 535], 40.00th=[ 634], 50.00th=[ 726], 60.00th=[ 793], | 70.00th=[ 869], 80.00th=[ 961], 90.00th=[ 1099], 95.00th=[ 1200], | 99.00th=[ 1401], 99.50th=[ 1502], 99.90th=[ 1670], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 95, per=27.42%, avg=29.07, stdev=17.51, samples=338 iops : min= 1, max= 23, avg= 6.73, stdev= 4.42, samples=338 write: IOPS=6, BW=27.8KiB/s (28.4kB/s)(5180KiB/186633msec); 0 zone resets slat (usec): min=50, max=33388, avg=370.06, stdev=1672.38 clat (msec): min=418, max=3942, avg=1579.50, stdev=614.35 lat (msec): min=418, max=3942, avg=1579.88, stdev=614.31 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 718], 10.00th=[ 835], 20.00th=[ 1036], | 30.00th=[ 1183], 40.00th=[ 1334], 50.00th=[ 1552], 60.00th=[ 1703], | 70.00th=[ 1871], 80.00th=[ 2072], 90.00th=[ 2433], 95.00th=[ 2702], | 99.00th=[ 3239], 99.50th=[ 3306], 99.90th=[ 3842], 99.95th=[ 3943], | 99.99th=[ 3943] bw ( KiB/s): min= 7, max= 93, per=25.78%, avg=28.62, stdev=18.03, samples=349 iops : min= 1, max= 23, avg= 6.62, stdev= 4.55, samples=349 lat (msec) : 50=0.16%, 100=0.23%, 250=0.86%, 500=11.84%, 750=16.60% lat (msec) : 1000=20.94%, 2000=37.77%, >=2000=11.60% cpu : usr=0.06%, sys=0.39%, ctx=4292, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=107KiB/s (109kB/s), 26.5KiB/s-27.1KiB/s (27.1kB/s-27.8kB/s), io=19.6MiB (20.5MB), run=186633-187675msec WRITE: bw=111KiB/s (114kB/s), 27.7KiB/s-28.0KiB/s (28.3kB/s-28.7kB/s), io=20.4MiB (21.4MB), run=186633-187675msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144272: Thu Aug 6 12:43:45 2026 read: IOPS=1, BW=51.2MiB/s (53.6MB/s)(40.0MiB/782msec) slat (nsec): min=57240k, max=57240k, avg=57240410.00, stdev= 0.00 clat (nsec): min=704267k, max=704267k, avg=704267414.00, stdev= 0.00 lat (nsec): min=761512k, max=761512k, avg=761511589.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 701], 5.00th=[ 701], 10.00th=[ 701], 20.00th=[ 701], | 30.00th=[ 701], 40.00th=[ 701], 50.00th=[ 701], 60.00th=[ 701], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] lat (msec) : 750=100.00% cpu : usr=0.13%, sys=1.41%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.2MiB/s (53.6MB/s), 51.2MiB/s-51.2MiB/s (53.6MB/s-53.6MB/s), io=40.0MiB (41.9MB), run=782-782msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout