== sanity test 398c: run fio to test AIO ================= 05:09:20 (1787389760) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.743583 s, 56.4 MB/s osc.lustre-OST0000-osc-ffff955c06524800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145460: Sat Aug 22 05:12:55 2026 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(10.0MiB/204036msec); 0 zone resets slat (usec): min=38, max=31069, avg=425.35, stdev=1475.76 clat (msec): min=350, max=2668, avg=1273.58, stdev=415.56 lat (msec): min=351, max=2668, avg=1274.01, stdev=415.62 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 642], 10.00th=[ 793], 20.00th=[ 919], | 30.00th=[ 1020], 40.00th=[ 1133], 50.00th=[ 1234], 60.00th=[ 1334], | 70.00th=[ 1469], 80.00th=[ 1620], 90.00th=[ 1804], 95.00th=[ 2039], | 99.00th=[ 2400], 99.50th=[ 2467], 99.90th=[ 2635], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 144, per=26.59%, avg=53.18, stdev=27.95, samples=377 iops : min= 1, max= 36, avg=12.75, stdev= 7.02, samples=377 lat (msec) : 500=2.34%, 750=5.82%, 1000=20.43%, 2000=65.66%, >=2000=5.74% cpu : usr=0.05%, sys=0.28%, ctx=1984, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145461: Sat Aug 22 05:12:56 2026 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(10.0MiB/204040msec); 0 zone resets slat (usec): min=38, max=23405, avg=406.15, stdev=1157.04 clat (msec): min=335, max=2664, avg=1273.39, stdev=407.49 lat (msec): min=335, max=2665, avg=1273.80, stdev=407.50 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 667], 10.00th=[ 810], 20.00th=[ 927], | 30.00th=[ 1028], 40.00th=[ 1133], 50.00th=[ 1250], 60.00th=[ 1351], | 70.00th=[ 1469], 80.00th=[ 1603], 90.00th=[ 1787], 95.00th=[ 2039], | 99.00th=[ 2366], 99.50th=[ 2534], 99.90th=[ 2668], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 128, per=26.51%, avg=53.02, stdev=27.85, samples=378 iops : min= 1, max= 32, avg=12.70, stdev= 6.98, samples=378 lat (msec) : 500=2.34%, 750=5.62%, 1000=19.38%, 2000=67.30%, >=2000=5.35% cpu : usr=0.05%, sys=0.30%, ctx=2017, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145462: Sat Aug 22 05:12:56 2026 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(10.0MiB/203838msec); 0 zone resets slat (usec): min=37, max=19397, avg=413.19, stdev=1291.87 clat (msec): min=51, max=2664, avg=1272.80, stdev=414.58 lat (msec): min=52, max=2664, avg=1273.22, stdev=414.61 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 625], 10.00th=[ 802], 20.00th=[ 919], | 30.00th=[ 1020], 40.00th=[ 1133], 50.00th=[ 1250], 60.00th=[ 1368], | 70.00th=[ 1469], 80.00th=[ 1603], 90.00th=[ 1787], 95.00th=[ 2039], | 99.00th=[ 2366], 99.50th=[ 2467], 99.90th=[ 2668], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 143, per=26.58%, avg=53.17, stdev=28.19, samples=377 iops : min= 1, max= 35, avg=12.73, stdev= 7.12, samples=377 lat (msec) : 100=0.04%, 250=0.27%, 500=2.15%, 750=5.55%, 1000=21.09% lat (msec) : 2000=65.35%, >=2000=5.55% cpu : usr=0.04%, sys=0.30%, ctx=2058, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145463: Sat Aug 22 05:12:56 2026 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(10.0MiB/203989msec); 0 zone resets slat (usec): min=35, max=25684, avg=400.83, stdev=1285.59 clat (msec): min=389, max=2659, avg=1272.85, stdev=409.19 lat (msec): min=389, max=2659, avg=1273.25, stdev=409.22 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 667], 10.00th=[ 793], 20.00th=[ 911], | 30.00th=[ 1020], 40.00th=[ 1133], 50.00th=[ 1250], 60.00th=[ 1334], | 70.00th=[ 1469], 80.00th=[ 1620], 90.00th=[ 1804], 95.00th=[ 2022], | 99.00th=[ 2299], 99.50th=[ 2467], 99.90th=[ 2668], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 128, per=25.70%, avg=51.40, stdev=29.61, samples=390 iops : min= 1, max= 32, avg=12.29, stdev= 7.47, samples=390 lat (msec) : 500=2.19%, 750=5.23%, 1000=21.33%, 2000=65.74%, >=2000=5.51% cpu : usr=0.04%, sys=0.28%, ctx=2004, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=201KiB/s (206kB/s), 50.2KiB/s-50.2KiB/s (51.4kB/s-51.4kB/s), io=40.0MiB (41.9MB), run=203838-204040msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145503: Sat Aug 22 05:15:27 2026 read: IOPS=9, BW=36.8KiB/s (37.7kB/s)(5048KiB/137023msec) slat (usec): min=2716, max=91474, avg=18413.67, stdev=10397.83 clat (msec): min=23, max=1185, avg=529.05, stdev=163.62 lat (msec): min=43, max=1204, avg=547.47, stdev=165.97 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 309], 10.00th=[ 355], 20.00th=[ 405], | 30.00th=[ 443], 40.00th=[ 468], 50.00th=[ 502], 60.00th=[ 542], | 70.00th=[ 584], 80.00th=[ 651], 90.00th=[ 751], 95.00th=[ 852], | 99.00th=[ 1036], 99.50th=[ 1053], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 112, per=25.31%, avg=36.95, stdev=18.74, samples=266 iops : min= 1, max= 28, avg= 8.72, stdev= 4.71, samples=266 write: IOPS=9, BW=37.9KiB/s (38.8kB/s)(5192KiB/137023msec); 0 zone resets slat (usec): min=55, max=27279, avg=369.71, stdev=1090.71 clat (msec): min=334, max=2383, avg=1153.29, stdev=358.39 lat (msec): min=334, max=2383, avg=1153.66, stdev=358.36 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 584], 10.00th=[ 701], 20.00th=[ 852], | 30.00th=[ 961], 40.00th=[ 1062], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1318], 80.00th=[ 1418], 90.00th=[ 1620], 95.00th=[ 1838], | 99.00th=[ 2072], 99.50th=[ 2140], 99.90th=[ 2366], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 103, per=25.08%, avg=38.12, stdev=18.61, samples=266 iops : min= 1, max= 25, avg= 9.01, stdev= 4.68, samples=266 lat (msec) : 50=0.08%, 100=0.04%, 250=0.82%, 500=24.88%, 750=25.04% lat (msec) : 1000=14.65%, 2000=33.59%, >=2000=0.90% cpu : usr=0.08%, sys=0.61%, ctx=4103, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145504: Sat Aug 22 05:15:27 2026 read: IOPS=9, BW=36.4KiB/s (37.3kB/s)(4976KiB/136697msec) slat (usec): min=79, max=100019, avg=18379.96, stdev=10760.28 clat (msec): min=19, max=1168, avg=521.51, stdev=168.05 lat (msec): min=20, max=1186, avg=539.90, stdev=170.45 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 284], 10.00th=[ 338], 20.00th=[ 397], | 30.00th=[ 443], 40.00th=[ 472], 50.00th=[ 502], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 634], 90.00th=[ 751], 95.00th=[ 827], | 99.00th=[ 1045], 99.50th=[ 1099], 99.90th=[ 1150], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 103, per=25.26%, avg=36.89, stdev=18.47, samples=263 iops : min= 1, max= 25, avg= 8.71, stdev= 4.64, samples=263 write: IOPS=9, BW=38.5KiB/s (39.4kB/s)(5264KiB/136697msec); 0 zone resets slat (usec): min=56, max=18504, avg=402.16, stdev=1150.43 clat (msec): min=180, max=2396, avg=1149.99, stdev=359.21 lat (msec): min=180, max=2396, avg=1150.40, stdev=359.18 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 592], 10.00th=[ 709], 20.00th=[ 844], | 30.00th=[ 961], 40.00th=[ 1053], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1620], 95.00th=[ 1804], | 99.00th=[ 2106], 99.50th=[ 2198], 99.90th=[ 2265], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 111, per=25.19%, avg=38.29, stdev=19.91, samples=268 iops : min= 1, max= 27, avg= 9.05, stdev= 5.06, samples=268 lat (msec) : 20=0.04%, 50=0.16%, 100=0.35%, 250=0.98%, 500=23.95% lat (msec) : 750=24.61%, 1000=15.47%, 2000=33.48%, >=2000=0.98% cpu : usr=0.09%, sys=0.64%, ctx=4528, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145505: Sat Aug 22 05:15:27 2026 read: IOPS=9, BW=36.4KiB/s (37.3kB/s)(4984KiB/136789msec) slat (usec): min=2659, max=91575, avg=18548.83, stdev=10803.45 clat (msec): min=23, max=1165, avg=525.17, stdev=163.88 lat (msec): min=30, max=1192, avg=543.72, stdev=166.32 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 300], 10.00th=[ 351], 20.00th=[ 405], | 30.00th=[ 443], 40.00th=[ 472], 50.00th=[ 502], 60.00th=[ 542], | 70.00th=[ 575], 80.00th=[ 642], 90.00th=[ 743], 95.00th=[ 852], | 99.00th=[ 1011], 99.50th=[ 1062], 99.90th=[ 1133], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 95, per=25.38%, avg=37.05, stdev=18.62, samples=263 iops : min= 1, max= 23, avg= 8.74, stdev= 4.69, samples=263 write: IOPS=9, BW=38.4KiB/s (39.3kB/s)(5256KiB/136789msec); 0 zone resets slat (usec): min=37, max=27972, avg=495.01, stdev=1525.96 clat (msec): min=324, max=2373, avg=1146.85, stdev=368.72 lat (msec): min=324, max=2373, avg=1147.35, stdev=368.70 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 575], 10.00th=[ 684], 20.00th=[ 844], | 30.00th=[ 953], 40.00th=[ 1045], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1636], 95.00th=[ 1838], | 99.00th=[ 2123], 99.50th=[ 2198], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 111, per=25.14%, avg=38.21, stdev=18.98, samples=267 iops : min= 1, max= 27, avg= 9.03, stdev= 4.78, samples=267 lat (msec) : 50=0.16%, 100=0.20%, 250=0.90%, 500=24.69%, 750=25.08% lat (msec) : 1000=15.39%, 2000=32.30%, >=2000=1.29% cpu : usr=0.09%, sys=0.68%, ctx=4669, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145506: Sat Aug 22 05:15:27 2026 read: IOPS=9, BW=36.0KiB/s (37.9kB/s)(5060KiB/136877msec) slat (usec): min=2776, max=90244, avg=17846.71, stdev=10467.45 clat (msec): min=21, max=1204, avg=526.68, stdev=164.68 lat (msec): min=32, max=1236, avg=544.55, stdev=167.38 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 292], 10.00th=[ 351], 20.00th=[ 405], | 30.00th=[ 439], 40.00th=[ 472], 50.00th=[ 502], 60.00th=[ 531], | 70.00th=[ 575], 80.00th=[ 651], 90.00th=[ 760], 95.00th=[ 844], | 99.00th=[ 1011], 99.50th=[ 1053], 99.90th=[ 1183], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 95, per=25.70%, avg=37.52, stdev=18.49, samples=262 iops : min= 1, max= 23, avg= 8.82, stdev= 4.66, samples=262 write: IOPS=9, BW=37.8KiB/s (38.8kB/s)(5180KiB/136877msec); 0 zone resets slat (usec): min=38, max=14936, avg=387.17, stdev=913.95 clat (msec): min=321, max=2394, avg=1156.04, stdev=366.04 lat (msec): min=322, max=2395, avg=1156.44, stdev=366.02 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 844], | 30.00th=[ 978], 40.00th=[ 1070], 50.00th=[ 1133], 60.00th=[ 1234], | 70.00th=[ 1318], 80.00th=[ 1435], 90.00th=[ 1620], 95.00th=[ 1854], | 99.00th=[ 2106], 99.50th=[ 2165], 99.90th=[ 2265], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 143, per=25.10%, avg=38.15, stdev=19.21, samples=264 iops : min= 1, max= 35, avg= 8.98, stdev= 4.85, samples=264 lat (msec) : 50=0.20%, 100=0.08%, 250=0.86%, 500=24.65%, 750=25.12% lat (msec) : 1000=14.57%, 2000=33.09%, >=2000=1.45% cpu : usr=0.09%, sys=0.63%, ctx=4500, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=146KiB/s (150kB/s), 36.4KiB/s-36.0KiB/s (37.3kB/s-37.9kB/s), io=19.6MiB (20.5MB), run=136697-137023msec WRITE: bw=152KiB/s (156kB/s), 37.8KiB/s-38.5KiB/s (38.8kB/s-39.4kB/s), io=20.4MiB (21.4MB), run=136697-137023msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145531: Sat Aug 22 05:15:41 2026 read: IOPS=1, BW=69.6MiB/s (72.9MB/s)(40.0MiB/575msec) slat (nsec): min=39045k, max=39045k, avg=39045085.00, stdev= 0.00 clat (nsec): min=506321k, max=506321k, avg=506320659.00, stdev= 0.00 lat (nsec): min=545401k, max=545401k, avg=545400747.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.74%, ctx=31, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=69.6MiB/s (72.9MB/s), 69.6MiB/s-69.6MiB/s (72.9MB/s-72.9MB/s), io=40.0MiB (41.9MB), run=575-575msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout