== sanity test 398c: run fio to test AIO ================= 08:42:47 (1789130567) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.435775 s, 96.2 MB/s osc.lustre-OST0000-osc-ffff9d4d908b5800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145837: Fri Sep 11 08:45:35 2026 write: IOPS=16, BW=65.7KiB/s (67.3kB/s)(10.0MiB/155918msec); 0 zone resets slat (usec): min=23, max=18683, avg=258.03, stdev=940.92 clat (msec): min=234, max=2138, avg=973.27, stdev=263.42 lat (msec): min=235, max=2138, avg=973.53, stdev=263.40 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 609], 10.00th=[ 676], 20.00th=[ 776], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1469], | 99.00th=[ 1737], 99.50th=[ 2005], 99.90th=[ 2106], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 158, per=25.77%, avg=67.52, stdev=29.76, samples=298 iops : min= 1, max= 39, avg=16.34, stdev= 7.45, samples=298 lat (msec) : 250=0.62%, 500=2.19%, 750=14.61%, 1000=41.99%, 2000=40.12% lat (msec) : >=2000=0.47% cpu : usr=0.06%, sys=0.24%, ctx=1651, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145838: Fri Sep 11 08:45:35 2026 write: IOPS=16, BW=65.8KiB/s (67.4kB/s)(10.0MiB/155645msec); 0 zone resets slat (usec): min=21, max=30737, avg=290.81, stdev=1193.54 clat (msec): min=210, max=2518, avg=971.83, stdev=265.59 lat (msec): min=210, max=2519, avg=972.12, stdev=265.56 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 609], 10.00th=[ 684], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1435], | 99.00th=[ 1804], 99.50th=[ 1955], 99.90th=[ 2165], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 150, per=25.17%, avg=65.95, stdev=32.03, samples=305 iops : min= 1, max= 37, avg=15.94, stdev= 7.99, samples=305 lat (msec) : 250=0.62%, 500=2.23%, 750=14.38%, 1000=43.05%, 2000=39.26% lat (msec) : >=2000=0.47% cpu : usr=0.06%, sys=0.27%, ctx=1693, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145839: Fri Sep 11 08:45:35 2026 write: IOPS=16, BW=65.6KiB/s (67.2kB/s)(10.0MiB/156071msec); 0 zone resets slat (usec): min=24, max=15008, avg=275.31, stdev=942.57 clat (msec): min=348, max=2117, avg=974.08, stdev=254.85 lat (msec): min=348, max=2117, avg=974.35, stdev=254.84 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1167], 90.00th=[ 1284], 95.00th=[ 1435], | 99.00th=[ 1720], 99.50th=[ 2005], 99.90th=[ 2106], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 8, max= 182, per=25.03%, avg=65.57, stdev=29.58, samples=307 iops : min= 2, max= 45, avg=15.85, stdev= 7.38, samples=307 lat (msec) : 500=1.99%, 750=13.87%, 1000=43.63%, 2000=40.08%, >=2000=0.43% cpu : usr=0.05%, sys=0.30%, ctx=2024, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145840: Fri Sep 11 08:45:35 2026 write: IOPS=16, BW=65.6KiB/s (67.2kB/s)(10.0MiB/156074msec); 0 zone resets slat (usec): min=23, max=23540, avg=270.58, stdev=1025.66 clat (msec): min=93, max=2131, avg=974.09, stdev=263.04 lat (msec): min=100, max=2131, avg=974.37, stdev=263.01 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1452], | 99.00th=[ 1888], 99.50th=[ 2039], 99.90th=[ 2089], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 223, per=26.05%, avg=68.24, stdev=32.51, samples=295 iops : min= 1, max= 55, avg=16.53, stdev= 8.11, samples=295 lat (msec) : 100=0.04%, 500=2.66%, 750=14.14%, 1000=42.85%, 2000=39.77% lat (msec) : >=2000=0.55% cpu : usr=0.06%, sys=0.25%, ctx=1651, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=262KiB/s (269kB/s), 65.6KiB/s-65.8KiB/s (67.2kB/s-67.4kB/s), io=40.0MiB (41.9MB), run=155645-156074msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145873: Fri Sep 11 08:47:40 2026 read: IOPS=10, BW=43.6KiB/s (44.7kB/s)(5048KiB/115690msec) slat (usec): min=3115, max=65691, avg=16157.09, stdev=9418.19 clat (msec): min=11, max=1029, avg=447.82, stdev=136.91 lat (msec): min=20, max=1041, avg=463.99, stdev=139.21 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 426], 60.00th=[ 464], | 70.00th=[ 498], 80.00th=[ 558], 90.00th=[ 634], 95.00th=[ 718], | 99.00th=[ 802], 99.50th=[ 852], 99.90th=[ 978], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 96, per=25.28%, avg=43.73, stdev=21.02, samples=226 iops : min= 1, max= 24, avg=10.38, stdev= 5.31, samples=226 write: IOPS=11, BW=44.9KiB/s (45.0kB/s)(5192KiB/115690msec); 0 zone resets slat (usec): min=32, max=13541, avg=339.75, stdev=1041.61 clat (msec): min=346, max=2309, avg=970.93, stdev=295.75 lat (msec): min=346, max=2310, avg=971.27, stdev=295.83 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 550], 10.00th=[ 634], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1519], | 99.00th=[ 1854], 99.50th=[ 2140], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 111, per=24.89%, avg=44.81, stdev=21.51, samples=226 iops : min= 1, max= 27, avg=10.65, stdev= 5.43, samples=226 lat (msec) : 20=0.04%, 50=0.08%, 100=0.23%, 250=1.09%, 500=34.57% lat (msec) : 750=22.81%, 1000=21.13%, 2000=19.73%, >=2000=0.31% cpu : usr=0.13%, sys=0.76%, ctx=4691, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145874: Fri Sep 11 08:47:40 2026 read: IOPS=10, BW=42.9KiB/s (43.0kB/s)(4976KiB/115860msec) slat (usec): min=2871, max=81158, avg=16677.22, stdev=9242.17 clat (msec): min=23, max=975, avg=447.92, stdev=140.79 lat (msec): min=34, max=995, avg=464.61, stdev=142.78 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 268], 10.00th=[ 296], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 456], | 70.00th=[ 493], 80.00th=[ 558], 90.00th=[ 651], 95.00th=[ 735], | 99.00th=[ 844], 99.50th=[ 885], 99.90th=[ 927], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 127, per=25.16%, avg=43.53, stdev=20.99, samples=223 iops : min= 1, max= 31, avg=10.33, stdev= 5.28, samples=223 write: IOPS=11, BW=45.4KiB/s (46.5kB/s)(5264KiB/115860msec); 0 zone resets slat (usec): min=28, max=17799, avg=327.10, stdev=957.83 clat (msec): min=171, max=2219, avg=966.85, stdev=289.38 lat (msec): min=171, max=2219, avg=967.18, stdev=289.44 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 542], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1183], 90.00th=[ 1351], 95.00th=[ 1519], | 99.00th=[ 1821], 99.50th=[ 1888], 99.90th=[ 2198], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 119, per=25.06%, avg=45.11, stdev=22.47, samples=228 iops : min= 1, max= 29, avg=10.72, stdev= 5.66, samples=228 lat (msec) : 50=0.23%, 100=0.20%, 250=1.17%, 500=34.45%, 750=21.95% lat (msec) : 1000=21.60%, 2000=20.16%, >=2000=0.23% cpu : usr=0.13%, sys=0.75%, ctx=4725, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145875: Fri Sep 11 08:47:40 2026 read: IOPS=10, BW=43.2KiB/s (44.2kB/s)(4984KiB/115360msec) slat (usec): min=36, max=66191, avg=16342.86, stdev=9125.82 clat (msec): min=19, max=1016, avg=440.17, stdev=141.13 lat (msec): min=27, max=1040, avg=456.52, stdev=143.57 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 262], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 481], 80.00th=[ 550], 90.00th=[ 634], 95.00th=[ 718], | 99.00th=[ 827], 99.50th=[ 877], 99.90th=[ 995], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 104, per=25.02%, avg=43.28, stdev=21.70, samples=226 iops : min= 1, max= 26, avg=10.27, stdev= 5.47, samples=226 write: IOPS=11, BW=45.6KiB/s (46.7kB/s)(5256KiB/115360msec); 0 zone resets slat (usec): min=27, max=16752, avg=371.18, stdev=1147.11 clat (msec): min=191, max=2269, avg=970.25, stdev=300.17 lat (msec): min=191, max=2270, avg=970.63, stdev=300.19 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1183], 90.00th=[ 1368], 95.00th=[ 1519], | 99.00th=[ 1854], 99.50th=[ 2056], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 111, per=25.02%, avg=45.03, stdev=20.89, samples=227 iops : min= 1, max= 27, avg=10.70, stdev= 5.28, samples=227 lat (msec) : 20=0.04%, 50=0.27%, 100=0.31%, 250=1.68%, 500=34.45% lat (msec) : 750=21.72%, 1000=20.66%, 2000=20.59%, >=2000=0.27% cpu : usr=0.12%, sys=0.75%, ctx=4550, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145876: Fri Sep 11 08:47:40 2026 read: IOPS=10, BW=43.7KiB/s (44.8kB/s)(5060KiB/115778msec) slat (usec): min=3128, max=54876, avg=16079.07, stdev=8567.24 clat (msec): min=27, max=1007, avg=448.26, stdev=132.58 lat (msec): min=33, max=1020, avg=464.35, stdev=134.29 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 498], 80.00th=[ 550], 90.00th=[ 642], 95.00th=[ 709], | 99.00th=[ 802], 99.50th=[ 852], 99.90th=[ 995], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 7, max= 108, per=25.28%, avg=43.73, stdev=20.76, samples=226 iops : min= 1, max= 27, avg=10.38, stdev= 5.22, samples=226 write: IOPS=11, BW=44.7KiB/s (45.8kB/s)(5180KiB/115778msec); 0 zone resets slat (usec): min=35, max=20015, avg=377.29, stdev=1179.31 clat (msec): min=249, max=2263, avg=973.40, stdev=294.77 lat (msec): min=249, max=2263, avg=973.78, stdev=294.76 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 542], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1502], | 99.00th=[ 1888], 99.50th=[ 1938], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 102, per=24.65%, avg=44.37, stdev=20.18, samples=228 iops : min= 1, max= 25, avg=10.54, stdev= 5.08, samples=228 lat (msec) : 50=0.08%, 100=0.08%, 250=1.17%, 500=35.31%, 750=22.46% lat (msec) : 1000=20.23%, 2000=20.43%, >=2000=0.23% cpu : usr=0.12%, sys=0.69%, ctx=4449, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=173KiB/s (177kB/s), 42.9KiB/s-43.7KiB/s (43.0kB/s-44.8kB/s), io=19.6MiB (20.5MB), run=115360-115860msec WRITE: bw=180KiB/s (185kB/s), 44.7KiB/s-45.6KiB/s (45.8kB/s-46.7kB/s), io=20.4MiB (21.4MB), run=115360-115860msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145896: Fri Sep 11 08:47:53 2026 read: IOPS=1, BW=79.2MiB/s (83.1MB/s)(40.0MiB/505msec) slat (nsec): min=39840k, max=39840k, avg=39839945.00, stdev= 0.00 clat (nsec): min=443379k, max=443379k, avg=443379420.00, stdev= 0.00 lat (nsec): min=483265k, max=483265k, avg=483265338.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 443], 10.00th=[ 443], 20.00th=[ 443], | 30.00th=[ 443], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.98%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=79.2MiB/s (83.1MB/s), 79.2MiB/s-79.2MiB/s (83.1MB/s-83.1MB/s), io=40.0MiB (41.9MB), run=505-505msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout