== sanity test 398c: run fio to test AIO ================= 22:44:26 (1789613066) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.649713 s, 64.6 MB/s osc.lustre-OST0000-osc-ffff95c231be1000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=175200: Wed Sep 16 22:47:23 2026 write: IOPS=15, BW=61.3KiB/s (62.8kB/s)(10.0MiB/166952msec); 0 zone resets slat (usec): min=22, max=23553, avg=175.85, stdev=751.21 clat (msec): min=206, max=2606, avg=1042.13, stdev=345.84 lat (msec): min=206, max=2607, avg=1042.30, stdev=345.87 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 995], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1519], 95.00th=[ 1670], | 99.00th=[ 2072], 99.50th=[ 2198], 99.90th=[ 2299], 99.95th=[ 2433], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 255, per=25.86%, avg=63.36, stdev=33.58, samples=318 iops : min= 1, max= 63, avg=15.42, stdev= 8.40, samples=318 lat (msec) : 250=0.62%, 500=3.71%, 750=10.86%, 1000=35.82%, 2000=47.62% lat (msec) : >=2000=1.37% cpu : usr=0.04%, sys=0.19%, ctx=1783, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=175201: Wed Sep 16 22:47:23 2026 write: IOPS=15, BW=61.5KiB/s (62.0kB/s)(10.0MiB/166471msec); 0 zone resets slat (usec): min=21, max=14234, avg=172.65, stdev=620.29 clat (msec): min=202, max=2303, avg=1038.90, stdev=349.00 lat (msec): min=202, max=2304, avg=1039.07, stdev=349.03 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 558], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 919], 50.00th=[ 986], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1301], 90.00th=[ 1502], 95.00th=[ 1720], | 99.00th=[ 2056], 99.50th=[ 2198], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 255, per=25.30%, avg=61.98, stdev=29.01, samples=325 iops : min= 1, max= 63, avg=15.07, stdev= 7.26, samples=325 lat (msec) : 250=1.25%, 500=2.97%, 750=11.76%, 1000=35.74%, 2000=46.80% lat (msec) : >=2000=1.48% cpu : usr=0.05%, sys=0.20%, ctx=1825, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=175202: Wed Sep 16 22:47:23 2026 write: IOPS=15, BW=61.4KiB/s (62.9kB/s)(10.0MiB/166766msec); 0 zone resets slat (usec): min=22, max=13716, avg=195.18, stdev=680.96 clat (msec): min=195, max=2417, avg=1040.74, stdev=358.40 lat (msec): min=195, max=2417, avg=1040.94, stdev=358.42 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 776], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 986], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1301], 90.00th=[ 1502], 95.00th=[ 1703], | 99.00th=[ 2165], 99.50th=[ 2333], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 255, per=25.38%, avg=62.19, stdev=30.67, samples=324 iops : min= 1, max= 63, avg=15.12, stdev= 7.70, samples=324 lat (msec) : 250=1.25%, 500=3.16%, 750=12.50%, 1000=35.23%, 2000=45.86% lat (msec) : >=2000=1.99% cpu : usr=0.04%, sys=0.19%, ctx=1865, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=175203: Wed Sep 16 22:47:23 2026 write: IOPS=15, BW=61.9KiB/s (63.4kB/s)(10.0MiB/165384msec); 0 zone resets slat (usec): min=22, max=9904, avg=174.43, stdev=536.42 clat (msec): min=116, max=2472, avg=1032.86, stdev=353.06 lat (msec): min=116, max=2473, avg=1033.04, stdev=353.07 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 535], 10.00th=[ 659], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 978], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1301], 90.00th=[ 1502], 95.00th=[ 1703], | 99.00th=[ 2056], 99.50th=[ 2165], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 228, per=25.50%, avg=62.48, stdev=31.92, samples=322 iops : min= 1, max= 57, avg=15.20, stdev= 8.03, samples=322 lat (msec) : 250=1.76%, 500=2.85%, 750=12.38%, 1000=35.51%, 2000=46.09% lat (msec) : >=2000=1.41% cpu : usr=0.04%, sys=0.20%, ctx=1755, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=245KiB/s (251kB/s), 61.3KiB/s-61.9KiB/s (62.8kB/s-63.4kB/s), io=40.0MiB (41.9MB), run=165384-166952msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=175237: Wed Sep 16 22:49:32 2026 read: IOPS=10, BW=42.3KiB/s (43.3kB/s)(5048KiB/119314msec) slat (usec): min=3007, max=53117, avg=15288.12, stdev=8937.57 clat (msec): min=38, max=1036, avg=467.28, stdev=147.79 lat (msec): min=49, max=1062, avg=482.58, stdev=150.30 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 667], 95.00th=[ 760], | 99.00th=[ 902], 99.50th=[ 978], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 120, per=24.86%, avg=41.76, stdev=21.29, samples=237 iops : min= 1, max= 30, avg= 9.98, stdev= 5.37, samples=237 write: IOPS=10, BW=43.5KiB/s (44.6kB/s)(5192KiB/119314msec); 0 zone resets slat (usec): min=24, max=12185, avg=238.19, stdev=645.81 clat (msec): min=264, max=2196, avg=998.24, stdev=299.39 lat (msec): min=264, max=2196, avg=998.48, stdev=299.42 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1401], 95.00th=[ 1519], | 99.00th=[ 2022], 99.50th=[ 2056], 99.90th=[ 2089], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 104, per=24.64%, avg=42.87, stdev=19.70, samples=237 iops : min= 1, max= 26, avg=10.26, stdev= 4.94, samples=237 lat (msec) : 50=0.04%, 100=0.16%, 250=1.25%, 500=32.85%, 750=21.68% lat (msec) : 1000=22.07%, 2000=21.41%, >=2000=0.55% cpu : usr=0.10%, sys=0.63%, ctx=4652, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175238: Wed Sep 16 22:49:32 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(4976KiB/119358msec) slat (usec): min=37, max=63309, avg=15232.23, stdev=8884.00 clat (msec): min=18, max=1000, avg=471.60, stdev=147.54 lat (msec): min=38, max=1014, avg=486.83, stdev=149.86 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 292], 10.00th=[ 321], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 439], 60.00th=[ 472], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 684], 95.00th=[ 768], | 99.00th=[ 902], 99.50th=[ 936], 99.90th=[ 978], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 111, per=24.80%, avg=41.66, stdev=19.30, samples=234 iops : min= 1, max= 27, avg= 9.95, stdev= 4.81, samples=234 write: IOPS=11, BW=44.1KiB/s (45.2kB/s)(5264KiB/119358msec); 0 zone resets slat (usec): min=25, max=19148, avg=274.11, stdev=940.99 clat (msec): min=156, max=2256, avg=989.87, stdev=305.93 lat (msec): min=156, max=2256, avg=990.15, stdev=305.98 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 558], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 2005], 99.50th=[ 2140], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 101, per=25.10%, avg=43.68, stdev=20.25, samples=236 iops : min= 1, max= 25, avg=10.47, stdev= 5.14, samples=236 lat (msec) : 20=0.04%, 50=0.12%, 100=0.04%, 250=1.48%, 500=32.23% lat (msec) : 750=21.84%, 1000=22.23%, 2000=21.48%, >=2000=0.55% cpu : usr=0.11%, sys=0.61%, ctx=4132, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175239: Wed Sep 16 22:49:32 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(4984KiB/119393msec) slat (usec): min=2876, max=67477, avg=15230.48, stdev=8824.83 clat (msec): min=21, max=1076, avg=466.28, stdev=147.32 lat (msec): min=34, max=1091, avg=481.51, stdev=149.20 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 518], 80.00th=[ 567], 90.00th=[ 659], 95.00th=[ 760], | 99.00th=[ 911], 99.50th=[ 927], 99.90th=[ 1053], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 117, per=24.96%, avg=41.93, stdev=20.49, samples=234 iops : min= 1, max= 29, avg=10.03, stdev= 5.18, samples=234 write: IOPS=11, BW=44.0KiB/s (45.1kB/s)(5256KiB/119393msec); 0 zone resets slat (usec): min=25, max=23142, avg=292.74, stdev=1073.20 clat (msec): min=194, max=2189, avg=994.86, stdev=298.25 lat (msec): min=195, max=2189, avg=995.16, stdev=298.30 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 760], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1536], | 99.00th=[ 1888], 99.50th=[ 2039], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 111, per=24.95%, avg=43.42, stdev=19.84, samples=236 iops : min= 1, max= 27, avg=10.40, stdev= 4.99, samples=236 lat (msec) : 50=0.16%, 100=0.23%, 250=1.37%, 500=32.30%, 750=21.17% lat (msec) : 1000=22.73%, 2000=21.64%, >=2000=0.39% cpu : usr=0.09%, sys=0.63%, ctx=4307, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175240: Wed Sep 16 22:49:32 2026 read: IOPS=10, BW=42.4KiB/s (43.4kB/s)(5060KiB/119388msec) slat (usec): min=2943, max=75280, avg=15482.77, stdev=9153.88 clat (msec): min=52, max=1000, avg=466.43, stdev=138.35 lat (msec): min=60, max=1024, avg=481.92, stdev=140.30 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 292], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 477], | 70.00th=[ 510], 80.00th=[ 558], 90.00th=[ 651], 95.00th=[ 743], | 99.00th=[ 911], 99.50th=[ 961], 99.90th=[ 995], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 111, per=25.22%, avg=42.38, stdev=19.55, samples=234 iops : min= 1, max= 27, avg=10.12, stdev= 4.95, samples=234 write: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5180KiB/119388msec); 0 zone resets slat (usec): min=23, max=16029, avg=257.91, stdev=855.43 clat (msec): min=213, max=2342, avg=1001.43, stdev=305.10 lat (msec): min=213, max=2342, avg=1001.69, stdev=305.14 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1401], 95.00th=[ 1552], | 99.00th=[ 1888], 99.50th=[ 2056], 99.90th=[ 2198], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 96, per=24.82%, avg=43.18, stdev=18.16, samples=235 iops : min= 1, max= 24, avg=10.34, stdev= 4.58, samples=235 lat (msec) : 100=0.12%, 250=1.37%, 500=33.28%, 750=21.76%, 1000=22.19% lat (msec) : 2000=20.86%, >=2000=0.43% cpu : usr=0.08%, sys=0.55%, ctx=4276, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=168KiB/s (172kB/s), 41.7KiB/s-42.4KiB/s (42.7kB/s-43.4kB/s), io=19.6MiB (20.5MB), run=119314-119393msec WRITE: bw=175KiB/s (179kB/s), 43.4KiB/s-44.1KiB/s (44.4kB/s-45.2kB/s), io=20.4MiB (21.4MB), run=119314-119393msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=175270: Wed Sep 16 22:49:41 2026 read: IOPS=1, BW=75.9MiB/s (79.6MB/s)(40.0MiB/527msec) slat (nsec): min=42172k, max=42172k, avg=42171959.00, stdev= 0.00 clat (nsec): min=471999k, max=471999k, avg=471998693.00, stdev= 0.00 lat (nsec): min=514176k, max=514176k, avg=514175637.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 472], 10.00th=[ 472], 20.00th=[ 472], | 30.00th=[ 472], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.90%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=75.9MiB/s (79.6MB/s), 75.9MiB/s-75.9MiB/s (79.6MB/s-79.6MB/s), io=40.0MiB (41.9MB), run=527-527msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout