== sanity test 398c: run fio to test AIO ================= 01:13:04 (1788757984) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.556093 s, 75.4 MB/s osc.lustre-OST0000-osc-ffff8ff0d6685800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145893: Mon Sep 7 01:16:15 2026 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(10.0MiB/179237msec); 0 zone resets slat (usec): min=23, max=17657, avg=242.13, stdev=1029.92 clat (msec): min=301, max=2208, avg=1119.11, stdev=292.57 lat (msec): min=301, max=2208, avg=1119.35, stdev=292.55 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 659], 10.00th=[ 785], 20.00th=[ 894], | 30.00th=[ 961], 40.00th=[ 1028], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1519], 95.00th=[ 1670], | 99.00th=[ 1955], 99.50th=[ 2089], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 128, per=26.04%, avg=59.37, stdev=27.86, samples=338 iops : min= 1, max= 32, avg=14.32, stdev= 6.99, samples=338 lat (msec) : 500=0.20%, 750=7.81%, 1000=27.54%, 2000=63.63%, >=2000=0.82% cpu : usr=0.04%, sys=0.21%, ctx=1787, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145894: Mon Sep 7 01:16:15 2026 write: IOPS=14, BW=57.2KiB/s (58.5kB/s)(10.0MiB/179093msec); 0 zone resets slat (usec): min=25, max=23539, avg=265.66, stdev=1088.27 clat (msec): min=237, max=2237, avg=1118.59, stdev=289.96 lat (msec): min=238, max=2238, avg=1118.86, stdev=289.98 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 684], 10.00th=[ 793], 20.00th=[ 902], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1099], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1519], 95.00th=[ 1636], | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 128, per=25.70%, avg=58.59, stdev=30.13, samples=343 iops : min= 1, max= 32, avg=14.12, stdev= 7.54, samples=343 lat (msec) : 250=0.62%, 500=0.66%, 750=5.59%, 1000=27.38%, 2000=64.69% lat (msec) : >=2000=1.05% cpu : usr=0.05%, sys=0.23%, ctx=1656, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145895: Mon Sep 7 01:16:15 2026 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(10.0MiB/179207msec); 0 zone resets slat (usec): min=24, max=21820, avg=267.86, stdev=1014.80 clat (msec): min=120, max=2203, avg=1118.65, stdev=292.80 lat (msec): min=121, max=2203, avg=1118.92, stdev=292.81 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 693], 10.00th=[ 793], 20.00th=[ 894], | 30.00th=[ 961], 40.00th=[ 1028], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1536], 95.00th=[ 1653], | 99.00th=[ 1905], 99.50th=[ 2039], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 135, per=25.04%, avg=57.10, stdev=29.59, samples=352 iops : min= 1, max= 33, avg=13.75, stdev= 7.39, samples=352 lat (msec) : 250=0.12%, 500=0.86%, 750=6.17%, 1000=29.41%, 2000=62.89% lat (msec) : >=2000=0.55% cpu : usr=0.05%, sys=0.23%, ctx=1896, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145896: Mon Sep 7 01:16:15 2026 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(10.0MiB/179209msec); 0 zone resets slat (usec): min=23, max=24231, avg=269.31, stdev=1086.12 clat (msec): min=300, max=2237, avg=1118.75, stdev=284.61 lat (msec): min=300, max=2237, avg=1119.02, stdev=284.63 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 709], 10.00th=[ 793], 20.00th=[ 902], | 30.00th=[ 969], 40.00th=[ 1028], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1519], 95.00th=[ 1636], | 99.00th=[ 1938], 99.50th=[ 2022], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 128, per=25.31%, avg=57.71, stdev=28.38, samples=348 iops : min= 1, max= 32, avg=13.90, stdev= 7.13, samples=348 lat (msec) : 500=0.86%, 750=5.90%, 1000=28.48%, 2000=64.14%, >=2000=0.62% cpu : usr=0.05%, sys=0.21%, ctx=1874, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=229KiB/s (234kB/s), 57.1KiB/s-57.2KiB/s (58.5kB/s-58.5kB/s), io=40.0MiB (41.9MB), run=179093-179237msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145931: Mon Sep 7 01:18:23 2026 read: IOPS=10, BW=42.0KiB/s (43.0kB/s)(5048KiB/117510msec) slat (usec): min=41, max=78947, avg=15771.26, stdev=9593.38 clat (msec): min=15, max=1584, avg=457.72, stdev=146.30 lat (msec): min=21, max=1596, avg=473.50, stdev=148.54 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 268], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 558], 90.00th=[ 634], 95.00th=[ 701], | 99.00th=[ 844], 99.50th=[ 902], 99.90th=[ 1552], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 120, per=25.21%, avg=42.60, stdev=21.18, samples=232 iops : min= 1, max= 30, avg=10.14, stdev= 5.32, samples=232 write: IOPS=11, BW=44.2KiB/s (45.2kB/s)(5192KiB/117510msec); 0 zone resets slat (usec): min=27, max=22083, avg=321.58, stdev=1033.88 clat (msec): min=101, max=2200, avg=986.48, stdev=299.13 lat (msec): min=101, max=2200, avg=986.81, stdev=299.13 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 506], 10.00th=[ 609], 20.00th=[ 751], | 30.00th=[ 844], 40.00th=[ 919], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1536], | 99.00th=[ 1754], 99.50th=[ 1821], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 135, per=25.03%, avg=44.05, stdev=20.99, samples=230 iops : min= 1, max= 33, avg=10.50, stdev= 5.25, samples=230 lat (msec) : 20=0.04%, 50=0.16%, 100=0.23%, 250=1.64%, 500=34.26% lat (msec) : 750=21.41%, 1000=19.30%, 2000=22.81%, >=2000=0.16% cpu : usr=0.10%, sys=0.62%, ctx=4716, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145932: Mon Sep 7 01:18:23 2026 read: IOPS=10, BW=42.0KiB/s (43.0kB/s)(4976KiB/118400msec) slat (usec): min=2901, max=73291, avg=15607.23, stdev=9160.09 clat (msec): min=19, max=1585, avg=454.05, stdev=143.26 lat (msec): min=35, max=1617, avg=469.67, stdev=145.34 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 268], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 502], 80.00th=[ 558], 90.00th=[ 634], 95.00th=[ 701], | 99.00th=[ 835], 99.50th=[ 927], 99.90th=[ 1267], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 6, max= 120, per=24.95%, avg=42.17, stdev=20.91, samples=231 iops : min= 1, max= 30, avg=10.03, stdev= 5.27, samples=231 write: IOPS=11, BW=44.5KiB/s (45.5kB/s)(5264KiB/118400msec); 0 zone resets slat (usec): min=27, max=14482, avg=315.53, stdev=966.06 clat (msec): min=271, max=2071, avg=992.59, stdev=290.32 lat (msec): min=271, max=2072, avg=992.91, stdev=290.28 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 506], 10.00th=[ 634], 20.00th=[ 760], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1502], | 99.00th=[ 1770], 99.50th=[ 1854], 99.90th=[ 1921], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 119, per=24.97%, avg=43.94, stdev=21.28, samples=234 iops : min= 1, max= 29, avg=10.47, stdev= 5.33, samples=234 lat (msec) : 20=0.04%, 50=0.23%, 100=0.23%, 250=1.33%, 500=34.45% lat (msec) : 750=20.27%, 1000=19.45%, 2000=23.95%, >=2000=0.04% cpu : usr=0.10%, sys=0.65%, ctx=4495, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145933: Mon Sep 7 01:18:23 2026 read: IOPS=10, BW=42.1KiB/s (43.1kB/s)(4984KiB/118522msec) slat (usec): min=2936, max=71798, avg=15774.69, stdev=9290.25 clat (msec): min=18, max=951, avg=462.04, stdev=137.97 lat (msec): min=23, max=987, avg=477.82, stdev=139.87 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 275], 10.00th=[ 317], 20.00th=[ 355], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 477], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 651], 95.00th=[ 718], | 99.00th=[ 835], 99.50th=[ 885], 99.90th=[ 919], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 6, max= 111, per=25.14%, avg=42.49, stdev=21.25, samples=230 iops : min= 1, max= 27, avg=10.12, stdev= 5.34, samples=230 write: IOPS=11, BW=44.3KiB/s (45.4kB/s)(5256KiB/118522msec); 0 zone resets slat (usec): min=25, max=23838, avg=292.35, stdev=1096.57 clat (msec): min=114, max=1914, avg=987.01, stdev=278.37 lat (msec): min=114, max=1914, avg=987.30, stdev=278.37 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 527], 10.00th=[ 651], 20.00th=[ 768], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1485], | 99.00th=[ 1737], 99.50th=[ 1770], 99.90th=[ 1888], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 119, per=24.91%, avg=43.84, stdev=19.21, samples=234 iops : min= 1, max= 29, avg=10.45, stdev= 4.84, samples=234 lat (msec) : 20=0.04%, 50=0.08%, 100=0.20%, 250=1.29%, 500=33.01% lat (msec) : 750=22.15%, 1000=19.34%, 2000=23.91% cpu : usr=0.10%, sys=0.68%, ctx=4595, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145934: Mon Sep 7 01:18:23 2026 read: IOPS=10, BW=42.9KiB/s (43.9kB/s)(5060KiB/118083msec) slat (usec): min=2575, max=70553, avg=15784.26, stdev=9074.08 clat (msec): min=9, max=1732, avg=455.13, stdev=138.46 lat (msec): min=24, max=1741, avg=470.92, stdev=140.20 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 502], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 684], | 99.00th=[ 793], 99.50th=[ 827], 99.90th=[ 1620], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 119, per=25.56%, avg=43.19, stdev=19.96, samples=230 iops : min= 1, max= 29, avg=10.29, stdev= 5.00, samples=230 write: IOPS=10, BW=43.9KiB/s (44.9kB/s)(5180KiB/118083msec); 0 zone resets slat (usec): min=34, max=24323, avg=359.75, stdev=1321.38 clat (msec): min=150, max=2573, avg=995.58, stdev=304.20 lat (msec): min=150, max=2574, avg=995.94, stdev=304.19 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 542], 10.00th=[ 659], 20.00th=[ 768], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1552], | 99.00th=[ 1804], 99.50th=[ 2265], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 96, per=24.85%, avg=43.73, stdev=19.36, samples=231 iops : min= 1, max= 24, avg=10.42, stdev= 4.88, samples=231 lat (msec) : 10=0.04%, 50=0.16%, 100=0.08%, 250=1.17%, 500=34.61% lat (msec) : 750=21.84%, 1000=18.91%, 2000=22.89%, >=2000=0.31% cpu : usr=0.10%, sys=0.69%, ctx=4744, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=169KiB/s (173kB/s), 42.0KiB/s-42.0KiB/s (43.0kB/s-43.0kB/s), io=19.6MiB (20.5MB), run=117510-118522msec WRITE: bw=176KiB/s (181kB/s), 43.9KiB/s-44.5KiB/s (44.9kB/s-45.5kB/s), io=20.4MiB (21.4MB), run=117510-118522msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145956: Mon Sep 7 01:18:32 2026 read: IOPS=1, BW=55.5MiB/s (58.2MB/s)(40.0MiB/721msec) slat (nsec): min=24734k, max=24734k, avg=24733523.00, stdev= 0.00 clat (nsec): min=681692k, max=681692k, avg=681692056.00, stdev= 0.00 lat (nsec): min=706430k, max=706430k, avg=706430497.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 684], 5.00th=[ 684], 10.00th=[ 684], 20.00th=[ 684], | 30.00th=[ 684], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.53%, ctx=32, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=55.5MiB/s (58.2MB/s), 55.5MiB/s-55.5MiB/s (58.2MB/s-58.2MB/s), io=40.0MiB (41.9MB), run=721-721msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout