== sanity test 398c: run fio to test AIO ================= 22:53:39 (1776999219) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.382193 s, 110 MB/s osc.lustre-OST0000-osc-ffff8db2e8f53800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149477: Thu Apr 23 22:56:37 2026 write: IOPS=15, BW=60.0KiB/s (62.5kB/s)(10.0MiB/167883msec); 0 zone resets slat (usec): min=44, max=14007, avg=333.31, stdev=882.58 clat (msec): min=116, max=1722, avg=1048.12, stdev=243.14 lat (msec): min=116, max=1722, avg=1048.46, stdev=243.15 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 667], 10.00th=[ 776], 20.00th=[ 860], | 30.00th=[ 919], 40.00th=[ 986], 50.00th=[ 1045], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1351], 95.00th=[ 1435], | 99.00th=[ 1620], 99.50th=[ 1653], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 151, per=24.95%, avg=60.63, stdev=30.84, samples=332 iops : min= 1, max= 37, avg=14.69, stdev= 7.72, samples=332 lat (msec) : 250=0.66%, 500=2.07%, 750=5.78%, 1000=33.16%, 2000=58.32% cpu : usr=0.03%, sys=0.27%, ctx=2173, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149478: Thu Apr 23 22:56:37 2026 write: IOPS=15, BW=61.2KiB/s (62.7kB/s)(10.0MiB/167312msec); 0 zone resets slat (usec): min=40, max=8684, avg=264.50, stdev=619.09 clat (msec): min=131, max=1754, avg=1044.98, stdev=252.42 lat (msec): min=131, max=1754, avg=1045.24, stdev=252.39 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 667], 10.00th=[ 760], 20.00th=[ 869], | 30.00th=[ 927], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1620], 99.50th=[ 1636], 99.90th=[ 1720], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 196, per=25.10%, avg=60.99, stdev=31.66, samples=330 iops : min= 1, max= 49, avg=14.78, stdev= 7.92, samples=330 lat (msec) : 250=0.82%, 500=2.07%, 750=6.45%, 1000=33.79%, 2000=56.88% cpu : usr=0.04%, sys=0.27%, ctx=1964, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149479: Thu Apr 23 22:56:37 2026 write: IOPS=15, BW=61.1KiB/s (62.5kB/s)(10.0MiB/167646msec); 0 zone resets slat (usec): min=50, max=15379, avg=308.51, stdev=905.19 clat (msec): min=164, max=1743, avg=1046.42, stdev=248.13 lat (msec): min=164, max=1743, avg=1046.73, stdev=248.13 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 667], 10.00th=[ 743], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 995], 50.00th=[ 1045], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1351], 95.00th=[ 1435], | 99.00th=[ 1636], 99.50th=[ 1670], 99.90th=[ 1703], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 255, per=25.58%, avg=62.16, stdev=31.58, samples=324 iops : min= 1, max= 63, avg=15.07, stdev= 7.91, samples=324 lat (msec) : 250=0.62%, 500=1.88%, 750=8.28%, 1000=30.20%, 2000=59.02% cpu : usr=0.04%, sys=0.25%, ctx=1976, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149480: Thu Apr 23 22:56:37 2026 write: IOPS=15, BW=60.0KiB/s (62.5kB/s)(10.0MiB/167877msec); 0 zone resets slat (usec): min=47, max=22652, avg=311.18, stdev=912.58 clat (msec): min=165, max=1742, avg=1047.78, stdev=243.27 lat (msec): min=165, max=1742, avg=1048.10, stdev=243.28 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 676], 10.00th=[ 760], 20.00th=[ 852], | 30.00th=[ 927], 40.00th=[ 986], 50.00th=[ 1045], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1351], 95.00th=[ 1435], | 99.00th=[ 1636], 99.50th=[ 1670], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 151, per=24.96%, avg=60.65, stdev=26.52, samples=332 iops : min= 1, max= 37, avg=14.69, stdev= 6.66, samples=332 lat (msec) : 250=0.23%, 500=2.30%, 750=6.99%, 1000=32.27%, 2000=58.20% cpu : usr=0.04%, sys=0.25%, ctx=1946, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=244KiB/s (250kB/s), 60.0KiB/s-61.2KiB/s (62.5kB/s-62.7kB/s), io=40.0MiB (41.9MB), run=167312-167883msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149514: Thu Apr 23 22:58:54 2026 read: IOPS=9, BW=39.7KiB/s (40.7kB/s)(5048KiB/127151msec) slat (usec): min=3220, max=61503, avg=15255.39, stdev=7742.04 clat (msec): min=31, max=1166, avg=496.49, stdev=130.56 lat (msec): min=46, max=1171, avg=511.76, stdev=131.21 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 321], 10.00th=[ 359], 20.00th=[ 397], | 30.00th=[ 430], 40.00th=[ 451], 50.00th=[ 481], 60.00th=[ 510], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 651], 95.00th=[ 718], | 99.00th=[ 885], 99.50th=[ 1070], 99.90th=[ 1150], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 111, per=25.45%, avg=39.96, stdev=19.77, samples=247 iops : min= 1, max= 27, avg= 9.55, stdev= 4.93, samples=247 write: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5192KiB/127151msec); 0 zone resets slat (usec): min=57, max=27114, avg=383.28, stdev=1198.00 clat (msec): min=412, max=2121, avg=1066.00, stdev=271.79 lat (msec): min=412, max=2121, avg=1066.38, stdev=271.80 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 659], 10.00th=[ 735], 20.00th=[ 852], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1401], 95.00th=[ 1569], | 99.00th=[ 1871], 99.50th=[ 1921], 99.90th=[ 1989], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 119, per=25.02%, avg=41.03, stdev=19.53, samples=248 iops : min= 1, max= 29, avg= 9.81, stdev= 4.94, samples=248 lat (msec) : 50=0.08%, 100=0.08%, 250=0.47%, 500=27.73%, 750=24.92% lat (msec) : 1000=18.55%, 2000=28.12%, >=2000=0.04% cpu : usr=0.07%, sys=0.64%, ctx=4826, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149515: Thu Apr 23 22:58:54 2026 read: IOPS=9, BW=39.1KiB/s (40.1kB/s)(4976KiB/127193msec) slat (usec): min=3240, max=58333, avg=15803.64, stdev=8277.53 clat (msec): min=10, max=1219, avg=492.50, stdev=128.59 lat (msec): min=25, max=1246, avg=508.31, stdev=129.34 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 321], 10.00th=[ 355], 20.00th=[ 397], | 30.00th=[ 426], 40.00th=[ 451], 50.00th=[ 477], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 718], | 99.00th=[ 844], 99.50th=[ 869], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 112, per=25.09%, avg=39.39, stdev=18.58, samples=247 iops : min= 1, max= 28, avg= 9.40, stdev= 4.65, samples=247 write: IOPS=10, BW=41.4KiB/s (42.4kB/s)(5264KiB/127193msec); 0 zone resets slat (usec): min=53, max=40931, avg=406.72, stdev=1576.48 clat (msec): min=441, max=2061, avg=1062.02, stdev=280.26 lat (msec): min=442, max=2061, avg=1062.43, stdev=280.22 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 651], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 961], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1301], 90.00th=[ 1435], 95.00th=[ 1569], | 99.00th=[ 1804], 99.50th=[ 1871], 99.90th=[ 1972], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 95, per=25.28%, avg=41.45, stdev=18.88, samples=249 iops : min= 1, max= 23, avg= 9.92, stdev= 4.76, samples=249 lat (msec) : 20=0.08%, 50=0.04%, 100=0.12%, 250=0.62%, 500=27.30% lat (msec) : 750=25.04%, 1000=19.45%, 2000=27.30%, >=2000=0.04% cpu : usr=0.08%, sys=0.62%, ctx=4726, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149516: Thu Apr 23 22:58:54 2026 read: IOPS=9, BW=39.2KiB/s (40.1kB/s)(4984KiB/127153msec) slat (usec): min=2799, max=64770, avg=15749.07, stdev=8383.40 clat (msec): min=19, max=1213, avg=493.46, stdev=136.97 lat (msec): min=28, max=1228, avg=509.22, stdev=137.70 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 296], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 422], 40.00th=[ 451], 50.00th=[ 481], 60.00th=[ 510], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 877], 99.50th=[ 1045], 99.90th=[ 1183], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 103, per=25.13%, avg=39.46, stdev=19.87, samples=248 iops : min= 1, max= 25, avg= 9.42, stdev= 4.96, samples=248 write: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5256KiB/127153msec); 0 zone resets slat (usec): min=53, max=20102, avg=374.78, stdev=1096.63 clat (msec): min=462, max=2123, avg=1061.79, stdev=274.20 lat (msec): min=462, max=2129, avg=1062.17, stdev=274.23 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 667], 10.00th=[ 735], 20.00th=[ 835], | 30.00th=[ 911], 40.00th=[ 961], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1418], 95.00th=[ 1569], | 99.00th=[ 1787], 99.50th=[ 1905], 99.90th=[ 2056], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 101, per=25.04%, avg=41.07, stdev=18.53, samples=250 iops : min= 1, max= 25, avg= 9.83, stdev= 4.71, samples=250 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=0.74%, 500=27.19% lat (msec) : 750=24.53%, 1000=18.95%, 2000=28.20%, >=2000=0.12% cpu : usr=0.09%, sys=0.61%, ctx=4446, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149517: Thu Apr 23 22:58:54 2026 read: IOPS=10, BW=40.0KiB/s (40.0kB/s)(5060KiB/126442msec) slat (usec): min=60, max=74964, avg=15315.37, stdev=8310.98 clat (msec): min=22, max=1307, avg=497.66, stdev=140.14 lat (msec): min=41, max=1315, avg=512.99, stdev=141.20 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 317], 10.00th=[ 355], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 485], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 718], | 99.00th=[ 869], 99.50th=[ 1099], 99.90th=[ 1284], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 110, per=25.63%, avg=40.23, stdev=19.68, samples=246 iops : min= 1, max= 27, avg= 9.61, stdev= 4.93, samples=246 write: IOPS=10, BW=40.0KiB/s (41.9kB/s)(5180KiB/126442msec); 0 zone resets slat (usec): min=59, max=14958, avg=376.83, stdev=920.28 clat (msec): min=219, max=2029, avg=1059.56, stdev=273.58 lat (msec): min=219, max=2030, avg=1059.94, stdev=273.57 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 667], 10.00th=[ 735], 20.00th=[ 835], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1418], 95.00th=[ 1569], | 99.00th=[ 1821], 99.50th=[ 1888], 99.90th=[ 1938], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 96, per=24.85%, avg=40.76, stdev=19.06, samples=249 iops : min= 1, max= 24, avg= 9.75, stdev= 4.84, samples=249 lat (msec) : 50=0.08%, 100=0.31%, 250=0.62%, 500=26.80%, 750=25.20% lat (msec) : 1000=18.95%, 2000=28.01%, >=2000=0.04% cpu : usr=0.11%, sys=0.62%, ctx=4951, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=158KiB/s (162kB/s), 39.1KiB/s-40.0KiB/s (40.1kB/s-40.0kB/s), io=19.6MiB (20.5MB), run=126442-127193msec WRITE: bw=164KiB/s (168kB/s), 40.8KiB/s-41.4KiB/s (41.8kB/s-42.4kB/s), io=20.4MiB (21.4MB), run=126442-127193msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149542: Thu Apr 23 22:59:05 2026 read: IOPS=2, BW=90.9MiB/s (95.3MB/s)(40.0MiB/440msec) slat (nsec): min=27211k, max=27211k, avg=27210756.00, stdev= 0.00 clat (nsec): min=399588k, max=399588k, avg=399587861.00, stdev= 0.00 lat (nsec): min=426803k, max=426803k, avg=426803461.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] lat (msec) : 500=100.00% cpu : usr=0.46%, sys=3.87%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s (95.3MB/s-95.3MB/s), io=40.0MiB (41.9MB), run=440-440msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout