== sanity test 398c: run fio to test AIO ================= 18:51:51 (1788216711) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.436078 s, 96.2 MB/s osc.lustre-OST0000-osc-ffff9a67c3ee7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145405: Mon Aug 31 18:54:50 2026 write: IOPS=15, BW=61.2KiB/s (62.7kB/s)(10.0MiB/167321msec); 0 zone resets slat (usec): min=23, max=19665, avg=250.92, stdev=1105.20 clat (msec): min=222, max=2205, avg=1044.51, stdev=311.86 lat (msec): min=222, max=2205, avg=1044.76, stdev=311.91 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1435], 95.00th=[ 1603], | 99.00th=[ 1921], 99.50th=[ 2022], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 254, per=24.94%, avg=60.86, stdev=29.72, samples=331 iops : min= 1, max= 63, avg=14.78, stdev= 7.47, samples=331 lat (msec) : 250=0.82%, 500=3.09%, 750=10.55%, 1000=31.88%, 2000=53.12% lat (msec) : >=2000=0.55% cpu : usr=0.05%, sys=0.23%, ctx=2003, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145406: Mon Aug 31 18:54:50 2026 write: IOPS=15, BW=61.1KiB/s (62.6kB/s)(10.0MiB/167536msec); 0 zone resets slat (usec): min=23, max=23300, avg=270.10, stdev=1139.06 clat (msec): min=202, max=2203, avg=1046.06, stdev=308.21 lat (msec): min=210, max=2204, avg=1046.33, stdev=308.22 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 558], 10.00th=[ 676], 20.00th=[ 802], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1905], 99.50th=[ 2039], 99.90th=[ 2140], 99.95th=[ 2165], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 254, per=25.02%, avg=61.04, stdev=31.21, samples=330 iops : min= 1, max= 63, avg=14.83, stdev= 7.86, samples=330 lat (msec) : 250=0.27%, 500=3.52%, 750=10.12%, 1000=30.51%, 2000=54.92% lat (msec) : >=2000=0.66% cpu : usr=0.04%, sys=0.24%, ctx=2019, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145407: Mon Aug 31 18:54:50 2026 write: IOPS=15, BW=61.2KiB/s (62.7kB/s)(10.0MiB/167230msec); 0 zone resets slat (usec): min=23, max=18028, avg=232.06, stdev=856.62 clat (msec): min=127, max=2192, avg=1044.45, stdev=311.37 lat (msec): min=127, max=2195, avg=1044.68, stdev=311.36 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 575], 10.00th=[ 684], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1636], | 99.00th=[ 1854], 99.50th=[ 2039], 99.90th=[ 2106], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 190, per=25.32%, avg=61.78, stdev=31.57, samples=326 iops : min= 1, max= 47, avg=15.01, stdev= 7.95, samples=326 lat (msec) : 250=0.59%, 500=3.28%, 750=10.59%, 1000=31.56%, 2000=53.48% lat (msec) : >=2000=0.51% cpu : usr=0.04%, sys=0.21%, ctx=1857, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145408: Mon Aug 31 18:54:50 2026 write: IOPS=15, BW=61.2KiB/s (62.6kB/s)(10.0MiB/167380msec); 0 zone resets slat (usec): min=22, max=17235, avg=220.88, stdev=893.26 clat (msec): min=201, max=2205, avg=1045.20, stdev=305.80 lat (msec): min=211, max=2205, avg=1045.42, stdev=305.80 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 575], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1435], 95.00th=[ 1603], | 99.00th=[ 1854], 99.50th=[ 1989], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 254, per=24.85%, avg=60.64, stdev=29.24, samples=332 iops : min= 1, max= 63, avg=14.73, stdev= 7.36, samples=332 lat (msec) : 250=0.35%, 500=3.32%, 750=11.05%, 1000=30.35%, 2000=54.45% lat (msec) : >=2000=0.47% cpu : usr=0.04%, sys=0.21%, ctx=1820, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=244KiB/s (250kB/s), 61.1KiB/s-61.2KiB/s (62.6kB/s-62.7kB/s), io=40.0MiB (41.9MB), run=167230-167536msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145442: Mon Aug 31 18:56:45 2026 read: IOPS=12, BW=48.4KiB/s (49.6kB/s)(5048KiB/104218msec) slat (usec): min=51, max=49201, avg=13765.52, stdev=7770.88 clat (msec): min=11, max=1011, avg=415.18, stdev=110.57 lat (msec): min=12, max=1019, avg=428.95, stdev=111.50 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 355], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 426], | 70.00th=[ 460], 80.00th=[ 493], 90.00th=[ 558], 95.00th=[ 617], | 99.00th=[ 709], 99.50th=[ 751], 99.90th=[ 944], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 7, max= 128, per=25.19%, avg=48.12, stdev=22.45, samples=206 iops : min= 1, max= 32, avg=11.58, stdev= 5.65, samples=206 write: IOPS=12, BW=49.8KiB/s (51.0kB/s)(5192KiB/104218msec); 0 zone resets slat (usec): min=25, max=9464, avg=243.59, stdev=669.91 clat (msec): min=133, max=1727, avg=866.39, stdev=250.79 lat (msec): min=133, max=1727, avg=866.63, stdev=250.83 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 489], 10.00th=[ 567], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 911], | 70.00th=[ 986], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1301], | 99.00th=[ 1552], 99.50th=[ 1636], 99.90th=[ 1703], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 111, per=24.78%, avg=49.32, stdev=22.09, samples=206 iops : min= 1, max= 27, avg=11.88, stdev= 5.54, samples=206 lat (msec) : 20=0.08%, 50=0.20%, 100=0.16%, 250=0.78%, 500=41.76% lat (msec) : 750=23.71%, 1000=18.91%, 2000=14.41% cpu : usr=0.10%, sys=0.62%, ctx=4805, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145443: Mon Aug 31 18:56:45 2026 read: IOPS=11, BW=47.4KiB/s (48.6kB/s)(4976KiB/104885msec) slat (usec): min=2798, max=64139, avg=14013.10, stdev=7739.16 clat (msec): min=29, max=882, avg=416.26, stdev=107.35 lat (msec): min=35, max=925, avg=430.28, stdev=108.69 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 355], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 426], | 70.00th=[ 456], 80.00th=[ 493], 90.00th=[ 567], 95.00th=[ 625], | 99.00th=[ 709], 99.50th=[ 718], 99.90th=[ 810], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 112, per=24.64%, avg=47.05, stdev=22.78, samples=207 iops : min= 1, max= 28, avg=11.31, stdev= 5.76, samples=207 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(5264KiB/104885msec); 0 zone resets slat (usec): min=26, max=27476, avg=243.05, stdev=1038.85 clat (msec): min=337, max=1704, avg=866.00, stdev=254.80 lat (msec): min=337, max=1704, avg=866.24, stdev=254.83 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 493], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 718], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 902], | 70.00th=[ 986], 80.00th=[ 1083], 90.00th=[ 1217], 95.00th=[ 1318], | 99.00th=[ 1536], 99.50th=[ 1636], 99.90th=[ 1687], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 8, max= 112, per=25.13%, avg=50.00, stdev=20.82, samples=207 iops : min= 2, max= 28, avg=12.05, stdev= 5.23, samples=207 lat (msec) : 50=0.16%, 100=0.08%, 250=0.82%, 500=41.17%, 750=24.92% lat (msec) : 1000=18.59%, 2000=14.26% cpu : usr=0.10%, sys=0.58%, ctx=4682, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145444: Mon Aug 31 18:56:45 2026 read: IOPS=11, BW=47.5KiB/s (48.7kB/s)(4984KiB/104881msec) slat (usec): min=2688, max=52943, avg=14328.41, stdev=7722.41 clat (msec): min=10, max=897, avg=413.84, stdev=104.93 lat (msec): min=25, max=923, avg=428.17, stdev=106.13 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 355], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 430], | 70.00th=[ 456], 80.00th=[ 493], 90.00th=[ 558], 95.00th=[ 600], | 99.00th=[ 701], 99.50th=[ 718], 99.90th=[ 835], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 136, per=24.75%, avg=47.28, stdev=24.08, samples=206 iops : min= 1, max= 34, avg=11.37, stdev= 6.04, samples=206 write: IOPS=12, BW=50.1KiB/s (51.3kB/s)(5256KiB/104881msec); 0 zone resets slat (usec): min=26, max=15336, avg=254.60, stdev=867.97 clat (msec): min=115, max=1752, avg=868.37, stdev=262.03 lat (msec): min=115, max=1752, avg=868.62, stdev=262.05 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 477], 10.00th=[ 567], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 911], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1217], 95.00th=[ 1318], | 99.00th=[ 1603], 99.50th=[ 1670], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 112, per=24.78%, avg=49.30, stdev=20.16, samples=207 iops : min= 1, max= 28, avg=11.87, stdev= 5.09, samples=207 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=1.56%, 500=40.62% lat (msec) : 750=23.59%, 1000=18.16%, 2000=15.78% cpu : usr=0.09%, sys=0.66%, ctx=4799, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145445: Mon Aug 31 18:56:45 2026 read: IOPS=12, BW=48.6KiB/s (49.7kB/s)(5060KiB/104217msec) slat (usec): min=2476, max=51127, avg=14279.80, stdev=7709.47 clat (msec): min=28, max=855, avg=415.65, stdev=110.18 lat (msec): min=40, max=896, avg=429.94, stdev=111.83 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 271], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 372], 50.00th=[ 401], 60.00th=[ 426], | 70.00th=[ 456], 80.00th=[ 498], 90.00th=[ 567], 95.00th=[ 625], | 99.00th=[ 718], 99.50th=[ 776], 99.90th=[ 852], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 7, max= 112, per=25.40%, avg=48.52, stdev=20.90, samples=205 iops : min= 1, max= 28, avg=11.68, stdev= 5.26, samples=205 write: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5180KiB/104217msec); 0 zone resets slat (usec): min=33, max=16780, avg=240.31, stdev=743.84 clat (msec): min=338, max=1737, avg=864.22, stdev=247.66 lat (msec): min=339, max=1737, avg=864.46, stdev=247.71 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 506], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 768], 50.00th=[ 835], 60.00th=[ 902], | 70.00th=[ 995], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1301], | 99.00th=[ 1552], 99.50th=[ 1603], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 112, per=24.75%, avg=49.24, stdev=21.08, samples=206 iops : min= 1, max= 28, avg=11.86, stdev= 5.31, samples=206 lat (msec) : 50=0.08%, 100=0.20%, 250=0.94%, 500=40.94%, 750=25.51% lat (msec) : 1000=17.66%, 2000=14.69% cpu : usr=0.10%, sys=0.64%, ctx=4768, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=191KiB/s (196kB/s), 47.4KiB/s-48.6KiB/s (48.6kB/s-49.7kB/s), io=19.6MiB (20.5MB), run=104217-104885msec WRITE: bw=199KiB/s (204kB/s), 49.7KiB/s-50.2KiB/s (50.9kB/s-51.4kB/s), io=20.4MiB (21.4MB), run=104217-104885msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145464: Mon Aug 31 18:56:57 2026 read: IOPS=1, BW=75.0MiB/s (78.7MB/s)(40.0MiB/533msec) slat (nsec): min=27747k, max=27747k, avg=27746522.00, stdev= 0.00 clat (nsec): min=483533k, max=483533k, avg=483533293.00, stdev= 0.00 lat (nsec): min=511285k, max=511285k, avg=511285115.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.44%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=75.0MiB/s (78.7MB/s), 75.0MiB/s-75.0MiB/s (78.7MB/s-78.7MB/s), io=40.0MiB (41.9MB), run=533-533msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout