== sanity test 398c: run fio to test AIO ================= 18:09:20 (1781215760) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.530476 s, 79.1 MB/s osc.lustre-OST0000-osc-ffff8bbfd802a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145002: Thu Jun 11 18:12:15 2026 write: IOPS=15, BW=63.3KiB/s (64.8kB/s)(10.0MiB/161816msec); 0 zone resets slat (usec): min=38, max=17729, avg=253.70, stdev=883.13 clat (msec): min=157, max=2564, avg=1010.42, stdev=301.72 lat (msec): min=157, max=2565, avg=1010.68, stdev=301.70 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1351], 95.00th=[ 1569], | 99.00th=[ 2089], 99.50th=[ 2165], 99.90th=[ 2467], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 176, per=25.08%, avg=63.19, stdev=28.70, samples=319 iops : min= 1, max= 44, avg=15.35, stdev= 7.24, samples=319 lat (msec) : 250=0.12%, 500=1.91%, 750=12.19%, 1000=40.94%, 2000=43.28% lat (msec) : >=2000=1.56% cpu : usr=0.03%, sys=0.29%, ctx=1724, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145003: Thu Jun 11 18:12:15 2026 write: IOPS=15, BW=63.2KiB/s (64.7kB/s)(10.0MiB/162124msec); 0 zone resets slat (usec): min=38, max=16420, avg=256.80, stdev=786.67 clat (msec): min=276, max=2212, avg=1011.45, stdev=285.27 lat (msec): min=276, max=2213, avg=1011.71, stdev=285.32 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 609], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1351], 95.00th=[ 1519], | 99.00th=[ 1989], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 200, per=25.07%, avg=63.19, stdev=28.76, samples=319 iops : min= 1, max= 50, avg=15.35, stdev= 7.22, samples=319 lat (msec) : 500=2.15%, 750=10.82%, 1000=41.95%, 2000=44.18%, >=2000=0.90% cpu : usr=0.04%, sys=0.27%, ctx=1685, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145004: Thu Jun 11 18:12:15 2026 write: IOPS=15, BW=63.2KiB/s (64.7kB/s)(10.0MiB/162125msec); 0 zone resets slat (usec): min=37, max=22931, avg=291.89, stdev=1062.27 clat (msec): min=222, max=2213, avg=1011.55, stdev=295.13 lat (msec): min=222, max=2213, avg=1011.85, stdev=295.14 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1586], | 99.00th=[ 2005], 99.50th=[ 2106], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 136, per=25.57%, avg=64.43, stdev=30.71, samples=313 iops : min= 1, max= 34, avg=15.67, stdev= 7.73, samples=313 lat (msec) : 250=0.62%, 500=1.37%, 750=12.30%, 1000=39.22%, 2000=45.47% lat (msec) : >=2000=1.02% cpu : usr=0.05%, sys=0.27%, ctx=1760, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145005: Thu Jun 11 18:12:15 2026 write: IOPS=15, BW=63.2KiB/s (64.7kB/s)(10.0MiB/162124msec); 0 zone resets slat (usec): min=39, max=13409, avg=276.95, stdev=812.31 clat (msec): min=87, max=2294, avg=1011.39, stdev=286.38 lat (msec): min=89, max=2294, avg=1011.67, stdev=286.41 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1536], | 99.00th=[ 2022], 99.50th=[ 2072], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 160, per=25.00%, avg=63.01, stdev=28.28, samples=320 iops : min= 1, max= 40, avg=15.30, stdev= 7.11, samples=320 lat (msec) : 100=0.04%, 250=0.04%, 500=1.91%, 750=11.05%, 1000=40.35% lat (msec) : 2000=45.47%, >=2000=1.13% cpu : usr=0.05%, sys=0.27%, ctx=1992, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=253KiB/s (259kB/s), 63.2KiB/s-63.3KiB/s (64.7kB/s-64.8kB/s), io=40.0MiB (41.9MB), run=161816-162125msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145038: Thu Jun 11 18:14:19 2026 read: IOPS=11, BW=44.5KiB/s (45.5kB/s)(5048KiB/113505msec) slat (usec): min=3053, max=63113, avg=15833.97, stdev=8823.26 clat (msec): min=12, max=863, avg=438.85, stdev=127.05 lat (msec): min=22, max=882, avg=454.69, stdev=128.57 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 268], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 422], 60.00th=[ 451], | 70.00th=[ 485], 80.00th=[ 535], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 818], 99.50th=[ 852], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 7, max= 126, per=25.16%, avg=44.28, stdev=22.79, samples=223 iops : min= 1, max= 31, avg=10.57, stdev= 5.73, samples=223 write: IOPS=11, BW=45.7KiB/s (46.8kB/s)(5192KiB/113505msec); 0 zone resets slat (usec): min=47, max=18244, avg=328.13, stdev=958.93 clat (msec): min=329, max=1756, avg=952.96, stdev=256.66 lat (msec): min=329, max=1758, avg=953.29, stdev=256.70 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 818], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1603], 99.50th=[ 1653], 99.90th=[ 1720], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 5, max= 119, per=24.54%, avg=45.15, stdev=19.53, samples=225 iops : min= 1, max= 29, avg=10.80, stdev= 4.87, samples=225 lat (msec) : 20=0.12%, 50=0.08%, 100=0.04%, 250=1.68%, 500=35.78% lat (msec) : 750=21.88%, 1000=20.16%, 2000=20.27% cpu : usr=0.09%, sys=0.68%, ctx=4347, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145039: Thu Jun 11 18:14:19 2026 read: IOPS=10, BW=43.8KiB/s (44.9kB/s)(4976KiB/113504msec) slat (usec): min=2970, max=71047, avg=15907.97, stdev=9040.10 clat (msec): min=10, max=938, avg=436.38, stdev=129.17 lat (msec): min=16, max=944, avg=452.29, stdev=130.78 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 268], 10.00th=[ 296], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 481], 80.00th=[ 535], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 793], 99.50th=[ 852], 99.90th=[ 877], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 166, per=24.79%, avg=43.63, stdev=23.58, samples=223 iops : min= 1, max= 41, avg=10.42, stdev= 5.91, samples=223 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(5264KiB/113504msec); 0 zone resets slat (usec): min=53, max=7505, avg=290.32, stdev=551.73 clat (msec): min=225, max=1782, avg=949.31, stdev=256.30 lat (msec): min=225, max=1782, avg=949.60, stdev=256.34 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1418], | 99.00th=[ 1636], 99.50th=[ 1687], 99.90th=[ 1754], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 103, per=24.98%, avg=45.96, stdev=19.86, samples=224 iops : min= 1, max= 25, avg=11.00, stdev= 4.94, samples=224 lat (msec) : 20=0.08%, 50=0.16%, 100=0.31%, 250=1.45%, 500=34.73% lat (msec) : 750=22.38%, 1000=20.04%, 2000=20.86% cpu : usr=0.10%, sys=0.68%, ctx=4791, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145040: Thu Jun 11 18:14:19 2026 read: IOPS=10, BW=43.9KiB/s (45.0kB/s)(4984KiB/113403msec) slat (usec): min=52, max=74916, avg=15945.91, stdev=9351.59 clat (msec): min=25, max=884, avg=443.70, stdev=127.39 lat (msec): min=35, max=894, avg=459.65, stdev=129.23 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 268], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 776], 99.50th=[ 810], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 126, per=24.83%, avg=43.70, stdev=21.31, samples=224 iops : min= 1, max= 31, avg=10.43, stdev= 5.36, samples=224 write: IOPS=11, BW=46.3KiB/s (47.5kB/s)(5256KiB/113403msec); 0 zone resets slat (usec): min=47, max=19565, avg=353.94, stdev=1055.41 clat (msec): min=226, max=1795, avg=943.59, stdev=258.64 lat (msec): min=227, max=1796, avg=943.95, stdev=258.75 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1536], 99.50th=[ 1653], 99.90th=[ 1787], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 95, per=25.01%, avg=46.02, stdev=18.95, samples=222 iops : min= 1, max= 23, avg=11.01, stdev= 4.76, samples=222 lat (msec) : 50=0.20%, 100=0.08%, 250=1.64%, 500=34.34%, 750=23.83% lat (msec) : 1000=20.51%, 2000=19.41% cpu : usr=0.08%, sys=0.66%, ctx=4726, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145041: Thu Jun 11 18:14:19 2026 read: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5060KiB/113467msec) slat (usec): min=3295, max=73117, avg=16120.16, stdev=9548.70 clat (msec): min=11, max=1245, avg=448.70, stdev=136.05 lat (msec): min=16, max=1251, avg=464.83, stdev=137.80 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 493], 80.00th=[ 542], 90.00th=[ 651], 95.00th=[ 709], | 99.00th=[ 835], 99.50th=[ 860], 99.90th=[ 1234], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 126, per=25.36%, avg=44.64, stdev=19.60, samples=222 iops : min= 1, max= 31, avg=10.67, stdev= 4.91, samples=222 write: IOPS=11, BW=45.7KiB/s (46.7kB/s)(5180KiB/113467msec); 0 zone resets slat (usec): min=55, max=24295, avg=347.42, stdev=955.51 clat (msec): min=224, max=1731, avg=943.45, stdev=255.31 lat (msec): min=224, max=1732, avg=943.80, stdev=255.33 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 575], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1620], 99.50th=[ 1636], 99.90th=[ 1703], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 95, per=24.89%, avg=45.80, stdev=18.57, samples=221 iops : min= 1, max= 23, avg=10.95, stdev= 4.68, samples=221 lat (msec) : 20=0.08%, 50=0.04%, 100=0.20%, 250=0.98%, 500=35.39% lat (msec) : 750=24.02%, 1000=19.65%, 2000=19.65% cpu : usr=0.09%, sys=0.67%, ctx=4901, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177KiB/s (181kB/s), 43.8KiB/s-44.6KiB/s (44.9kB/s-45.7kB/s), io=19.6MiB (20.5MB), run=113403-113505msec WRITE: bw=184KiB/s (188kB/s), 45.7KiB/s-46.4KiB/s (46.7kB/s-47.5kB/s), io=20.4MiB (21.4MB), run=113403-113505msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145062: Thu Jun 11 18:14:30 2026 read: IOPS=1, BW=74.1MiB/s (77.7MB/s)(40.0MiB/540msec) slat (nsec): min=41552k, max=41552k, avg=41552150.00, stdev= 0.00 clat (nsec): min=478791k, max=478791k, avg=478791430.00, stdev= 0.00 lat (nsec): min=520349k, max=520349k, avg=520349142.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 481], 10.00th=[ 481], 20.00th=[ 481], | 30.00th=[ 481], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] lat (msec) : 500=100.00% cpu : usr=0.56%, sys=3.34%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.1MiB/s (77.7MB/s), 74.1MiB/s-74.1MiB/s (77.7MB/s-77.7MB/s), io=40.0MiB (41.9MB), run=540-540msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout