== sanity test 398c: run fio to test AIO ================= 03:46:11 (1787384771) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.624152 s, 67.2 MB/s osc.lustre-OST0000-osc-ffff902044e09800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146296: Sat Aug 22 03:49:15 2026 write: IOPS=14, BW=59.3KiB/s (60.8kB/s)(10.0MiB/172598msec); 0 zone resets slat (usec): min=52, max=25065, avg=390.79, stdev=1214.75 clat (msec): min=403, max=1902, avg=1077.31, stdev=260.43 lat (msec): min=403, max=1902, avg=1077.70, stdev=260.44 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 625], 10.00th=[ 743], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1284], 90.00th=[ 1401], 95.00th=[ 1502], | 99.00th=[ 1687], 99.50th=[ 1770], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 128, per=25.23%, avg=59.80, stdev=28.59, samples=336 iops : min= 1, max= 32, avg=14.40, stdev= 7.16, samples=336 lat (msec) : 500=2.27%, 750=8.05%, 1000=26.45%, 2000=63.24% cpu : usr=0.04%, sys=0.29%, ctx=1728, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146297: Sat Aug 22 03:49:15 2026 write: IOPS=14, BW=59.3KiB/s (60.7kB/s)(10.0MiB/172747msec); 0 zone resets slat (usec): min=38, max=28505, avg=380.65, stdev=1116.34 clat (msec): min=402, max=1928, avg=1078.11, stdev=263.76 lat (msec): min=403, max=1928, avg=1078.49, stdev=263.78 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 634], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1070], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1301], 90.00th=[ 1418], 95.00th=[ 1502], | 99.00th=[ 1687], 99.50th=[ 1787], 99.90th=[ 1905], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 128, per=25.38%, avg=60.16, stdev=28.30, samples=334 iops : min= 1, max= 32, avg=14.50, stdev= 7.10, samples=334 lat (msec) : 500=1.91%, 750=7.89%, 1000=28.36%, 2000=61.84% cpu : usr=0.06%, sys=0.31%, ctx=2119, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146298: Sat Aug 22 03:49:15 2026 write: IOPS=14, BW=59.5KiB/s (60.9kB/s)(10.0MiB/172173msec); 0 zone resets slat (usec): min=51, max=21715, avg=418.21, stdev=1287.01 clat (msec): min=89, max=1927, avg=1075.26, stdev=268.73 lat (msec): min=90, max=1928, avg=1075.68, stdev=268.72 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 651], 10.00th=[ 751], 20.00th=[ 877], | 30.00th=[ 936], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1284], 90.00th=[ 1418], 95.00th=[ 1519], | 99.00th=[ 1670], 99.50th=[ 1754], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 154, per=24.85%, avg=58.90, stdev=26.88, samples=341 iops : min= 1, max= 38, avg=14.19, stdev= 6.71, samples=341 lat (msec) : 100=0.20%, 250=0.35%, 500=1.99%, 750=7.30%, 1000=29.14% lat (msec) : 2000=61.02% cpu : usr=0.06%, sys=0.32%, ctx=2173, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146299: Sat Aug 22 03:49:15 2026 write: IOPS=14, BW=59.3KiB/s (60.7kB/s)(10.0MiB/172606msec); 0 zone resets slat (usec): min=51, max=17037, avg=365.71, stdev=984.08 clat (msec): min=168, max=1919, avg=1077.39, stdev=263.79 lat (msec): min=170, max=1919, avg=1077.76, stdev=263.78 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 642], 10.00th=[ 743], 20.00th=[ 885], | 30.00th=[ 953], 40.00th=[ 1011], 50.00th=[ 1070], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1284], 90.00th=[ 1401], 95.00th=[ 1502], | 99.00th=[ 1687], 99.50th=[ 1770], 99.90th=[ 1905], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 128, per=25.38%, avg=60.15, stdev=27.87, samples=334 iops : min= 1, max= 32, avg=14.50, stdev= 6.98, samples=334 lat (msec) : 250=0.62%, 500=1.37%, 750=8.12%, 1000=27.73%, 2000=62.15% cpu : usr=0.05%, sys=0.30%, ctx=1957, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=237KiB/s (243kB/s), 59.3KiB/s-59.5KiB/s (60.7kB/s-60.9kB/s), io=40.0MiB (41.9MB), run=172173-172747msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146333: Sat Aug 22 03:51:25 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5048KiB/122115msec) slat (usec): min=3299, max=55767, avg=15980.42, stdev=8412.24 clat (msec): min=23, max=1069, avg=476.05, stdev=142.96 lat (msec): min=29, max=1084, avg=492.04, stdev=144.59 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 271], 10.00th=[ 317], 20.00th=[ 355], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 659], 95.00th=[ 735], | 99.00th=[ 869], 99.50th=[ 894], 99.90th=[ 978], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7, max= 110, per=25.29%, avg=41.48, stdev=19.53, samples=238 iops : min= 1, max= 27, avg= 9.84, stdev= 4.90, samples=238 write: IOPS=10, BW=42.5KiB/s (43.5kB/s)(5192KiB/122115msec); 0 zone resets slat (usec): min=53, max=13256, avg=468.87, stdev=1064.09 clat (msec): min=288, max=2008, avg=1023.84, stdev=304.32 lat (msec): min=288, max=2009, avg=1024.31, stdev=304.26 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 609], 10.00th=[ 684], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1062], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1620], | 99.00th=[ 1854], 99.50th=[ 1888], 99.90th=[ 1955], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 104, per=24.67%, avg=42.18, stdev=19.71, samples=240 iops : min= 1, max= 26, avg=10.03, stdev= 4.99, samples=240 lat (msec) : 50=0.20%, 100=0.20%, 250=1.21%, 500=28.75%, 750=26.37% lat (msec) : 1000=20.12%, 2000=23.12%, >=2000=0.04% cpu : usr=0.09%, sys=0.72%, ctx=4430, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146334: Sat Aug 22 03:51:25 2026 read: IOPS=10, BW=40.8KiB/s (41.8kB/s)(4976KiB/121882msec) slat (usec): min=86, max=75534, avg=16090.59, stdev=9013.43 clat (msec): min=51, max=1101, avg=470.48, stdev=139.38 lat (msec): min=58, max=1133, avg=486.59, stdev=141.13 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 271], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 575], 90.00th=[ 659], 95.00th=[ 735], | 99.00th=[ 827], 99.50th=[ 919], 99.90th=[ 1083], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 112, per=24.56%, avg=40.27, stdev=20.64, samples=241 iops : min= 1, max= 28, avg= 9.54, stdev= 5.17, samples=241 write: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5264KiB/121882msec); 0 zone resets slat (usec): min=56, max=14204, avg=441.27, stdev=1030.24 clat (msec): min=162, max=2006, avg=1020.50, stdev=306.74 lat (msec): min=162, max=2006, avg=1020.95, stdev=306.80 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 600], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1053], | 70.00th=[ 1150], 80.00th=[ 1301], 90.00th=[ 1435], 95.00th=[ 1620], | 99.00th=[ 1804], 99.50th=[ 1854], 99.90th=[ 1938], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 111, per=25.50%, avg=43.60, stdev=19.87, samples=236 iops : min= 1, max= 27, avg=10.37, stdev= 4.99, samples=236 lat (msec) : 100=0.16%, 250=2.07%, 500=29.30%, 750=25.00%, 1000=19.88% lat (msec) : 2000=23.55%, >=2000=0.04% cpu : usr=0.10%, sys=0.73%, ctx=4488, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146335: Sat Aug 22 03:51:25 2026 read: IOPS=10, BW=40.8KiB/s (41.8kB/s)(4984KiB/122119msec) slat (usec): min=3682, max=70844, avg=16224.62, stdev=8469.48 clat (msec): min=15, max=975, avg=469.70, stdev=136.98 lat (msec): min=27, max=993, avg=485.94, stdev=138.70 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 460], 60.00th=[ 493], | 70.00th=[ 535], 80.00th=[ 575], 90.00th=[ 651], 95.00th=[ 709], | 99.00th=[ 827], 99.50th=[ 885], 99.90th=[ 969], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 127, per=24.96%, avg=40.94, stdev=21.24, samples=239 iops : min= 1, max= 31, avg= 9.71, stdev= 5.32, samples=239 write: IOPS=10, BW=43.0KiB/s (44.1kB/s)(5256KiB/122119msec); 0 zone resets slat (usec): min=56, max=19526, avg=476.03, stdev=1261.72 clat (msec): min=122, max=2029, avg=1022.84, stdev=301.38 lat (msec): min=124, max=2029, avg=1023.32, stdev=301.42 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 617], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1062], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1435], 95.00th=[ 1620], | 99.00th=[ 1821], 99.50th=[ 1888], 99.90th=[ 1938], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 103, per=24.70%, avg=42.24, stdev=20.05, samples=242 iops : min= 1, max= 25, avg=10.03, stdev= 5.01, samples=242 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=1.25%, 500=29.02% lat (msec) : 750=25.98%, 1000=19.45%, 2000=23.98%, >=2000=0.04% cpu : usr=0.10%, sys=0.71%, ctx=4571, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146336: Sat Aug 22 03:51:25 2026 read: IOPS=10, BW=41.5KiB/s (42.5kB/s)(5060KiB/122012msec) slat (usec): min=3537, max=64304, avg=15971.39, stdev=8497.83 clat (msec): min=31, max=1132, avg=473.04, stdev=140.44 lat (msec): min=48, max=1139, avg=489.01, stdev=142.28 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 363], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 464], 60.00th=[ 493], | 70.00th=[ 531], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 735], | 99.00th=[ 885], 99.50th=[ 936], 99.90th=[ 1062], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 119, per=25.65%, avg=42.07, stdev=20.73, samples=235 iops : min= 1, max= 29, avg= 9.98, stdev= 5.16, samples=235 write: IOPS=10, BW=42.5KiB/s (43.5kB/s)(5180KiB/122012msec); 0 zone resets slat (usec): min=56, max=20074, avg=559.43, stdev=1586.61 clat (msec): min=226, max=2065, avg=1026.59, stdev=302.34 lat (msec): min=227, max=2065, avg=1027.15, stdev=302.38 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 617], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1070], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1435], 95.00th=[ 1603], | 99.00th=[ 1804], 99.50th=[ 1888], 99.90th=[ 1989], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 120, per=24.65%, avg=42.15, stdev=19.99, samples=240 iops : min= 1, max= 30, avg=10.02, stdev= 5.05, samples=240 lat (msec) : 50=0.12%, 100=0.27%, 250=1.45%, 500=29.22%, 750=25.86% lat (msec) : 1000=19.22%, 2000=23.83%, >=2000=0.04% cpu : usr=0.09%, sys=0.73%, ctx=4763, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=164KiB/s (168kB/s), 40.8KiB/s-41.5KiB/s (41.8kB/s-42.5kB/s), io=19.6MiB (20.5MB), run=121882-122119msec WRITE: bw=171KiB/s (175kB/s), 42.5KiB/s-43.2KiB/s (43.5kB/s-44.2kB/s), io=20.4MiB (21.4MB), run=121882-122119msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146357: Sat Aug 22 03:51:35 2026 read: IOPS=1, BW=65.0MiB/s (68.2MB/s)(40.0MiB/615msec) slat (nsec): min=35037k, max=35037k, avg=35036558.00, stdev= 0.00 clat (nsec): min=556223k, max=556223k, avg=556222595.00, stdev= 0.00 lat (nsec): min=591263k, max=591263k, avg=591263487.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] lat (msec) : 750=100.00% cpu : usr=0.16%, sys=2.28%, ctx=8, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.0MiB/s (68.2MB/s), 65.0MiB/s-65.0MiB/s (68.2MB/s-68.2MB/s), io=40.0MiB (41.9MB), run=615-615msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout