== sanity test 398c: run fio to test AIO ================= 02:50:42 (1785394242) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.515271 s, 81.4 MB/s osc.lustre-OST0000-osc-ffff9b63e6b6c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144459: Thu Jul 30 02:55:05 2026 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(10.0MiB/249135msec); 0 zone resets slat (usec): min=36, max=29858, avg=219.25, stdev=1018.37 clat (msec): min=262, max=3569, avg=1555.71, stdev=586.72 lat (msec): min=263, max=3569, avg=1555.93, stdev=586.72 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 651], 10.00th=[ 751], 20.00th=[ 978], | 30.00th=[ 1267], 40.00th=[ 1418], 50.00th=[ 1552], 60.00th=[ 1687], | 70.00th=[ 1838], 80.00th=[ 2005], 90.00th=[ 2333], 95.00th=[ 2635], | 99.00th=[ 3104], 99.50th=[ 3306], 99.90th=[ 3373], 99.95th=[ 3406], | 99.99th=[ 3574] bw ( KiB/s): min= 7, max= 144, per=27.32%, avg=44.80, stdev=32.27, samples=448 iops : min= 1, max= 36, avg=10.73, stdev= 8.10, samples=448 lat (msec) : 500=1.52%, 750=8.95%, 1000=9.61%, 2000=59.45%, >=2000=20.47% cpu : usr=0.02%, sys=0.16%, ctx=1444, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144460: Thu Jul 30 02:55:05 2026 write: IOPS=10, BW=41.3KiB/s (42.3kB/s)(10.0MiB/248166msec); 0 zone resets slat (usec): min=38, max=16448, avg=233.43, stdev=867.76 clat (msec): min=149, max=3710, avg=1549.84, stdev=594.52 lat (msec): min=150, max=3710, avg=1550.07, stdev=594.54 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 642], 10.00th=[ 751], 20.00th=[ 961], | 30.00th=[ 1234], 40.00th=[ 1401], 50.00th=[ 1552], 60.00th=[ 1703], | 70.00th=[ 1838], 80.00th=[ 2022], 90.00th=[ 2299], 95.00th=[ 2601], | 99.00th=[ 3037], 99.50th=[ 3171], 99.90th=[ 3540], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 7, max= 138, per=26.35%, avg=43.22, stdev=25.76, samples=464 iops : min= 1, max= 34, avg=10.33, stdev= 6.49, samples=464 lat (msec) : 250=0.35%, 500=1.37%, 750=8.12%, 1000=11.52%, 2000=57.23% lat (msec) : >=2000=21.41% cpu : usr=0.03%, sys=0.17%, ctx=1833, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144461: Thu Jul 30 02:55:05 2026 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(10.0MiB/249129msec); 0 zone resets slat (usec): min=40, max=14402, avg=220.23, stdev=847.69 clat (msec): min=351, max=3575, avg=1555.48, stdev=584.81 lat (msec): min=351, max=3575, avg=1555.71, stdev=584.83 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 651], 10.00th=[ 768], 20.00th=[ 978], | 30.00th=[ 1234], 40.00th=[ 1418], 50.00th=[ 1569], 60.00th=[ 1720], | 70.00th=[ 1854], 80.00th=[ 2005], 90.00th=[ 2299], 95.00th=[ 2601], | 99.00th=[ 3104], 99.50th=[ 3205], 99.90th=[ 3574], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 7, max= 176, per=27.88%, avg=45.73, stdev=31.84, samples=439 iops : min= 1, max= 44, avg=10.96, stdev= 8.02, samples=439 lat (msec) : 500=1.68%, 750=7.46%, 1000=11.52%, 2000=58.98%, >=2000=20.35% cpu : usr=0.03%, sys=0.15%, ctx=1555, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144462: Thu Jul 30 02:55:05 2026 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(10.0MiB/249076msec); 0 zone resets slat (usec): min=44, max=17942, avg=241.07, stdev=1030.73 clat (msec): min=246, max=3576, avg=1554.98, stdev=590.22 lat (msec): min=246, max=3576, avg=1555.22, stdev=590.24 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 651], 10.00th=[ 751], 20.00th=[ 978], | 30.00th=[ 1250], 40.00th=[ 1401], 50.00th=[ 1536], 60.00th=[ 1703], | 70.00th=[ 1854], 80.00th=[ 2039], 90.00th=[ 2333], 95.00th=[ 2567], | 99.00th=[ 3037], 99.50th=[ 3306], 99.90th=[ 3406], 99.95th=[ 3406], | 99.99th=[ 3574] bw ( KiB/s): min= 7, max= 136, per=28.72%, avg=47.11, stdev=31.74, samples=426 iops : min= 1, max= 34, avg=11.30, stdev= 7.99, samples=426 lat (msec) : 250=0.62%, 500=0.78%, 750=8.36%, 1000=10.94%, 2000=57.34% lat (msec) : >=2000=21.95% cpu : usr=0.02%, sys=0.15%, ctx=1548, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=164KiB/s (168kB/s), 41.1KiB/s-41.3KiB/s (42.1kB/s-42.3kB/s), io=40.0MiB (41.9MB), run=248166-249135msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144509: Thu Jul 30 02:57:53 2026 read: IOPS=8, BW=32.5KiB/s (33.2kB/s)(5048KiB/155504msec) slat (usec): min=68, max=120690, avg=17819.13, stdev=12762.86 clat (msec): min=19, max=1683, avg=616.91, stdev=233.63 lat (msec): min=25, max=1703, avg=634.74, stdev=236.62 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 271], 10.00th=[ 321], 20.00th=[ 430], | 30.00th=[ 485], 40.00th=[ 558], 50.00th=[ 609], 60.00th=[ 667], | 70.00th=[ 718], 80.00th=[ 785], 90.00th=[ 902], 95.00th=[ 1020], | 99.00th=[ 1267], 99.50th=[ 1301], 99.90th=[ 1620], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 111, per=26.09%, avg=33.40, stdev=19.85, samples=296 iops : min= 1, max= 27, avg= 7.90, stdev= 5.02, samples=296 write: IOPS=8, BW=33.4KiB/s (34.2kB/s)(5192KiB/155504msec); 0 zone resets slat (usec): min=54, max=19867, avg=267.58, stdev=973.00 clat (msec): min=218, max=3254, avg=1298.15, stdev=507.09 lat (msec): min=218, max=3254, avg=1298.41, stdev=507.07 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 567], 10.00th=[ 659], 20.00th=[ 852], | 30.00th=[ 986], 40.00th=[ 1150], 50.00th=[ 1267], 60.00th=[ 1401], | 70.00th=[ 1536], 80.00th=[ 1670], 90.00th=[ 1938], 95.00th=[ 2198], | 99.00th=[ 2735], 99.50th=[ 2869], 99.90th=[ 3239], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 7, max= 119, per=25.46%, avg=34.11, stdev=20.71, samples=296 iops : min= 1, max= 29, avg= 8.08, stdev= 5.21, samples=296 lat (msec) : 20=0.04%, 50=0.04%, 100=0.35%, 250=1.56%, 500=15.16% lat (msec) : 750=27.11%, 1000=18.05%, 2000=33.95%, >=2000=3.75% cpu : usr=0.06%, sys=0.39%, ctx=4143, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144510: Thu Jul 30 02:57:53 2026 read: IOPS=7, BW=31.9KiB/s (32.7kB/s)(4976KiB/155809msec) slat (usec): min=2520, max=90185, avg=18157.29, stdev=11215.62 clat (msec): min=12, max=1728, avg=610.20, stdev=226.17 lat (msec): min=25, max=1783, avg=628.36, stdev=229.13 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 262], 10.00th=[ 317], 20.00th=[ 418], | 30.00th=[ 485], 40.00th=[ 550], 50.00th=[ 600], 60.00th=[ 667], | 70.00th=[ 726], 80.00th=[ 776], 90.00th=[ 877], 95.00th=[ 1003], | 99.00th=[ 1250], 99.50th=[ 1267], 99.90th=[ 1334], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 120, per=25.96%, avg=33.23, stdev=19.37, samples=292 iops : min= 1, max= 30, avg= 7.85, stdev= 4.90, samples=292 write: IOPS=8, BW=33.8KiB/s (34.6kB/s)(5264KiB/155809msec); 0 zone resets slat (usec): min=49, max=47529, avg=293.29, stdev=1630.97 clat (msec): min=307, max=3257, avg=1297.18, stdev=496.05 lat (msec): min=308, max=3257, avg=1297.47, stdev=496.19 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 844], | 30.00th=[ 1003], 40.00th=[ 1133], 50.00th=[ 1284], 60.00th=[ 1418], | 70.00th=[ 1536], 80.00th=[ 1687], 90.00th=[ 1921], 95.00th=[ 2140], | 99.00th=[ 2735], 99.50th=[ 2869], 99.90th=[ 3037], 99.95th=[ 3272], | 99.99th=[ 3272] bw ( KiB/s): min= 7, max= 120, per=25.50%, avg=34.17, stdev=20.78, samples=301 iops : min= 1, max= 30, avg= 8.10, stdev= 5.25, samples=301 lat (msec) : 20=0.08%, 50=0.04%, 100=0.12%, 250=1.76%, 500=15.20% lat (msec) : 750=26.91%, 1000=17.42%, 2000=34.45%, >=2000=4.02% cpu : usr=0.06%, sys=0.39%, ctx=4247, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144511: Thu Jul 30 02:57:53 2026 read: IOPS=7, BW=31.0KiB/s (32.8kB/s)(4984KiB/155808msec) slat (usec): min=2003, max=87024, avg=18040.18, stdev=12103.22 clat (msec): min=47, max=1621, avg=604.13, stdev=225.20 lat (msec): min=56, max=1662, avg=622.18, stdev=228.70 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 266], 10.00th=[ 321], 20.00th=[ 418], | 30.00th=[ 468], 40.00th=[ 531], 50.00th=[ 592], 60.00th=[ 651], | 70.00th=[ 718], 80.00th=[ 776], 90.00th=[ 894], 95.00th=[ 986], | 99.00th=[ 1200], 99.50th=[ 1318], 99.90th=[ 1603], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 120, per=26.41%, avg=33.81, stdev=20.56, samples=288 iops : min= 1, max= 30, avg= 8.01, stdev= 5.20, samples=288 write: IOPS=8, BW=33.7KiB/s (34.5kB/s)(5256KiB/155808msec); 0 zone resets slat (usec): min=41, max=32520, avg=303.34, stdev=1539.80 clat (msec): min=172, max=3173, avg=1303.66, stdev=509.32 lat (msec): min=172, max=3173, avg=1303.96, stdev=509.37 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 584], 10.00th=[ 659], 20.00th=[ 844], | 30.00th=[ 1003], 40.00th=[ 1150], 50.00th=[ 1284], 60.00th=[ 1401], | 70.00th=[ 1552], 80.00th=[ 1703], 90.00th=[ 1938], 95.00th=[ 2165], | 99.00th=[ 2802], 99.50th=[ 2937], 99.90th=[ 3138], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 7, max= 120, per=25.43%, avg=34.07, stdev=19.72, samples=301 iops : min= 1, max= 30, avg= 8.07, stdev= 4.97, samples=301 lat (msec) : 50=0.04%, 100=0.16%, 250=2.03%, 500=16.56%, 750=25.23% lat (msec) : 1000=17.77%, 2000=33.91%, >=2000=4.30% cpu : usr=0.05%, sys=0.41%, ctx=4107, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144512: Thu Jul 30 02:57:53 2026 read: IOPS=8, BW=32.6KiB/s (33.4kB/s)(5060KiB/155322msec) slat (usec): min=2306, max=93053, avg=18256.52, stdev=11686.23 clat (msec): min=7, max=1745, avg=605.11, stdev=239.36 lat (msec): min=19, max=1774, avg=623.37, stdev=242.14 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 259], 10.00th=[ 317], 20.00th=[ 401], | 30.00th=[ 477], 40.00th=[ 527], 50.00th=[ 592], 60.00th=[ 651], | 70.00th=[ 709], 80.00th=[ 776], 90.00th=[ 911], 95.00th=[ 1020], | 99.00th=[ 1301], 99.50th=[ 1385], 99.90th=[ 1636], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 111, per=26.41%, avg=33.81, stdev=20.32, samples=293 iops : min= 1, max= 27, avg= 8.00, stdev= 5.13, samples=293 write: IOPS=8, BW=33.3KiB/s (34.1kB/s)(5180KiB/155322msec); 0 zone resets slat (usec): min=54, max=28767, avg=267.88, stdev=1082.10 clat (msec): min=308, max=3259, avg=1305.87, stdev=498.79 lat (msec): min=308, max=3259, avg=1306.14, stdev=498.81 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 852], | 30.00th=[ 995], 40.00th=[ 1150], 50.00th=[ 1284], 60.00th=[ 1401], | 70.00th=[ 1536], 80.00th=[ 1703], 90.00th=[ 1972], 95.00th=[ 2198], | 99.00th=[ 2668], 99.50th=[ 2769], 99.90th=[ 2903], 99.95th=[ 3272], | 99.99th=[ 3272] bw ( KiB/s): min= 7, max= 120, per=25.69%, avg=34.42, stdev=19.70, samples=293 iops : min= 1, max= 30, avg= 8.15, stdev= 4.97, samples=293 lat (msec) : 10=0.04%, 50=0.12%, 100=0.16%, 250=1.72%, 500=16.48% lat (msec) : 750=26.56%, 1000=16.88%, 2000=33.55%, >=2000=4.49% cpu : usr=0.06%, sys=0.41%, ctx=4216, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=129KiB/s (132kB/s), 31.9KiB/s-32.6KiB/s (32.7kB/s-33.4kB/s), io=19.6MiB (20.5MB), run=155322-155809msec WRITE: bw=134KiB/s (137kB/s), 33.3KiB/s-33.8KiB/s (34.1kB/s-34.6kB/s), io=20.4MiB (21.4MB), run=155322-155809msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144540: Thu Jul 30 02:58:05 2026 read: IOPS=1, BW=51.7MiB/s (54.3MB/s)(40.0MiB/773msec) slat (nsec): min=65274k, max=65274k, avg=65273621.00, stdev= 0.00 clat (nsec): min=678054k, max=678054k, avg=678054471.00, stdev= 0.00 lat (nsec): min=743332k, max=743332k, avg=743332421.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.30%, ctx=9, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.7MiB/s (54.3MB/s), 51.7MiB/s-51.7MiB/s (54.3MB/s-54.3MB/s), io=40.0MiB (41.9MB), run=773-773msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout