== sanity test 398c: run fio to test AIO ================= 04:00:32 (1788595232) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.621231 s, 67.5 MB/s osc.lustre-OST0000-osc-ffff91f40a957000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=174973: Sat Sep 5 04:03:50 2026 write: IOPS=13, BW=55.1KiB/s (56.4kB/s)(10.0MiB/185870msec); 0 zone resets slat (usec): min=28, max=19161, avg=287.73, stdev=980.66 clat (msec): min=142, max=2306, avg=1160.83, stdev=290.52 lat (msec): min=142, max=2306, avg=1161.12, stdev=290.50 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 793], 10.00th=[ 860], 20.00th=[ 944], | 30.00th=[ 1003], 40.00th=[ 1070], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1267], 80.00th=[ 1351], 90.00th=[ 1502], 95.00th=[ 1703], | 99.00th=[ 2039], 99.50th=[ 2165], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 147, per=25.17%, avg=55.38, stdev=23.24, samples=362 iops : min= 1, max= 36, avg=13.35, stdev= 5.83, samples=362 lat (msec) : 250=0.51%, 500=1.21%, 750=1.52%, 1000=25.74%, 2000=69.65% lat (msec) : >=2000=1.37% cpu : usr=0.05%, sys=0.25%, ctx=2004, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=174974: Sat Sep 5 04:03:50 2026 write: IOPS=13, BW=55.1KiB/s (56.4kB/s)(10.0MiB/185860msec); 0 zone resets slat (usec): min=29, max=21344, avg=286.78, stdev=907.53 clat (msec): min=204, max=2307, avg=1160.68, stdev=282.23 lat (msec): min=205, max=2307, avg=1160.97, stdev=282.22 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 793], 10.00th=[ 852], 20.00th=[ 944], | 30.00th=[ 1011], 40.00th=[ 1083], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1267], 80.00th=[ 1351], 90.00th=[ 1502], 95.00th=[ 1653], | 99.00th=[ 2072], 99.50th=[ 2198], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 177, per=25.38%, avg=55.83, stdev=27.55, samples=359 iops : min= 1, max= 44, avg=13.46, stdev= 6.92, samples=359 lat (msec) : 250=0.62%, 500=0.62%, 750=1.76%, 1000=25.39%, 2000=69.96% lat (msec) : >=2000=1.64% cpu : usr=0.04%, sys=0.23%, ctx=1944, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=174975: Sat Sep 5 04:03:50 2026 write: IOPS=13, BW=55.1KiB/s (56.4kB/s)(10.0MiB/185856msec); 0 zone resets slat (usec): min=29, max=25501, avg=265.85, stdev=1116.84 clat (msec): min=149, max=2214, avg=1161.11, stdev=284.52 lat (msec): min=150, max=2214, avg=1161.38, stdev=284.50 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 793], 10.00th=[ 869], 20.00th=[ 944], | 30.00th=[ 1003], 40.00th=[ 1070], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1267], 80.00th=[ 1351], 90.00th=[ 1519], 95.00th=[ 1670], | 99.00th=[ 2072], 99.50th=[ 2140], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 128, per=27.25%, avg=59.94, stdev=31.19, samples=335 iops : min= 1, max= 32, avg=14.50, stdev= 7.82, samples=335 lat (msec) : 250=0.04%, 500=1.33%, 750=1.84%, 1000=26.21%, 2000=69.26% lat (msec) : >=2000=1.33% cpu : usr=0.05%, sys=0.21%, ctx=1710, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=174976: Sat Sep 5 04:03:50 2026 write: IOPS=13, BW=55.1KiB/s (56.4kB/s)(10.0MiB/185850msec); 0 zone resets slat (usec): min=28, max=18099, avg=292.19, stdev=1069.56 clat (msec): min=204, max=2277, avg=1160.73, stdev=282.43 lat (msec): min=204, max=2277, avg=1161.02, stdev=282.44 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 785], 10.00th=[ 860], 20.00th=[ 944], | 30.00th=[ 1011], 40.00th=[ 1070], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1267], 80.00th=[ 1351], 90.00th=[ 1502], 95.00th=[ 1670], | 99.00th=[ 2039], 99.50th=[ 2165], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 135, per=25.05%, avg=55.10, stdev=22.76, samples=364 iops : min= 1, max= 33, avg=13.28, stdev= 5.70, samples=364 lat (msec) : 250=0.31%, 500=0.94%, 750=2.15%, 1000=25.23%, 2000=69.92% lat (msec) : >=2000=1.45% cpu : usr=0.04%, sys=0.22%, ctx=1911, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=220KiB/s (226kB/s), 55.1KiB/s-55.1KiB/s (56.4kB/s-56.4kB/s), io=40.0MiB (41.9MB), run=185850-185870msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=175023: Sat Sep 5 04:05:56 2026 read: IOPS=10, BW=43.1KiB/s (44.1kB/s)(5048KiB/117251msec) slat (usec): min=33, max=61403, avg=15506.51, stdev=8543.85 clat (msec): min=25, max=1434, avg=457.28, stdev=127.83 lat (msec): min=36, max=1457, avg=472.80, stdev=129.47 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 363], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 447], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 827], 99.50th=[ 978], 99.90th=[ 1351], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 103, per=25.28%, avg=42.98, stdev=19.92, samples=230 iops : min= 1, max= 25, avg=10.27, stdev= 4.99, samples=230 write: IOPS=11, BW=44.3KiB/s (45.3kB/s)(5192KiB/117251msec); 0 zone resets slat (usec): min=38, max=19199, avg=350.35, stdev=1053.71 clat (msec): min=124, max=2208, avg=984.29, stdev=282.19 lat (msec): min=124, max=2208, avg=984.65, stdev=282.16 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 558], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1653], 99.50th=[ 1754], 99.90th=[ 2056], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 111, per=24.90%, avg=44.08, stdev=21.05, samples=230 iops : min= 1, max= 27, avg=10.55, stdev= 5.26, samples=230 lat (msec) : 50=0.08%, 100=0.16%, 250=1.02%, 500=33.87%, 750=23.87% lat (msec) : 1000=17.89%, 2000=23.01%, >=2000=0.12% cpu : usr=0.10%, sys=0.67%, ctx=4713, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175024: Sat Sep 5 04:05:56 2026 read: IOPS=10, BW=42.3KiB/s (43.3kB/s)(4976KiB/117733msec) slat (usec): min=3220, max=53642, avg=15722.61, stdev=8088.46 clat (msec): min=30, max=1136, avg=455.90, stdev=127.12 lat (msec): min=43, max=1179, avg=471.64, stdev=128.74 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 288], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 667], | 99.00th=[ 818], 99.50th=[ 919], 99.90th=[ 1099], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 112, per=25.24%, avg=42.91, stdev=20.90, samples=227 iops : min= 1, max= 28, avg=10.25, stdev= 5.24, samples=227 write: IOPS=11, BW=44.7KiB/s (45.8kB/s)(5264KiB/117733msec); 0 zone resets slat (usec): min=32, max=27218, avg=343.43, stdev=1249.91 clat (msec): min=146, max=1703, avg=983.45, stdev=273.21 lat (msec): min=146, max=1703, avg=983.79, stdev=273.19 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 575], 10.00th=[ 634], 20.00th=[ 735], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1452], | 99.00th=[ 1603], 99.50th=[ 1636], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 96, per=25.18%, avg=44.57, stdev=18.74, samples=231 iops : min= 1, max= 24, avg=10.67, stdev= 4.69, samples=231 lat (msec) : 50=0.16%, 100=0.23%, 250=0.86%, 500=33.05%, 750=24.26% lat (msec) : 1000=17.66%, 2000=23.79% cpu : usr=0.09%, sys=0.64%, ctx=4924, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175025: Sat Sep 5 04:05:56 2026 read: IOPS=10, BW=42.3KiB/s (43.3kB/s)(4984KiB/117805msec) slat (usec): min=2692, max=53994, avg=15765.94, stdev=8292.26 clat (msec): min=5, max=1064, avg=450.62, stdev=119.15 lat (msec): min=17, max=1084, avg=466.40, stdev=120.89 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 439], 60.00th=[ 464], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 793], 99.50th=[ 877], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 104, per=24.82%, avg=42.19, stdev=21.57, samples=232 iops : min= 1, max= 26, avg=10.07, stdev= 5.40, samples=232 write: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5256KiB/117805msec); 0 zone resets slat (usec): min=33, max=14920, avg=378.90, stdev=1114.67 clat (msec): min=145, max=1750, avg=989.87, stdev=273.58 lat (msec): min=145, max=1750, avg=990.25, stdev=273.54 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 1620], 99.50th=[ 1653], 99.90th=[ 1720], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 103, per=24.83%, avg=43.96, stdev=19.13, samples=233 iops : min= 1, max= 25, avg=10.52, stdev= 4.80, samples=233 lat (msec) : 10=0.04%, 50=0.12%, 100=0.31%, 250=0.78%, 500=34.77% lat (msec) : 750=22.19%, 1000=17.85%, 2000=23.95% cpu : usr=0.10%, sys=0.72%, ctx=4698, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175026: Sat Sep 5 04:05:56 2026 read: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5060KiB/117250msec) slat (usec): min=3517, max=83520, avg=15030.96, stdev=7977.17 clat (msec): min=12, max=1667, avg=456.18, stdev=140.14 lat (msec): min=22, max=1687, avg=471.22, stdev=141.16 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 288], 10.00th=[ 326], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 885], 99.50th=[ 1053], 99.90th=[ 1653], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 119, per=25.58%, avg=43.48, stdev=19.14, samples=228 iops : min= 1, max= 29, avg=10.39, stdev= 4.81, samples=228 write: IOPS=11, BW=44.2KiB/s (45.2kB/s)(5180KiB/117250msec); 0 zone resets slat (usec): min=35, max=8932, avg=315.75, stdev=727.61 clat (msec): min=208, max=2414, avg=985.25, stdev=272.78 lat (msec): min=208, max=2414, avg=985.57, stdev=272.80 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 743], | 30.00th=[ 844], 40.00th=[ 919], 50.00th=[ 986], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1620], 99.50th=[ 1754], 99.90th=[ 2299], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 95, per=24.76%, avg=43.82, stdev=18.72, samples=231 iops : min= 1, max= 23, avg=10.48, stdev= 4.64, samples=231 lat (msec) : 20=0.12%, 50=0.08%, 100=0.23%, 250=0.94%, 500=33.75% lat (msec) : 750=23.59%, 1000=17.46%, 2000=23.71%, >=2000=0.12% cpu : usr=0.11%, sys=0.68%, ctx=4841, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=170KiB/s (174kB/s), 42.3KiB/s-43.2KiB/s (43.3kB/s-44.2kB/s), io=19.6MiB (20.5MB), run=117250-117805msec WRITE: bw=177KiB/s (182kB/s), 44.2KiB/s-44.7KiB/s (45.2kB/s-45.8kB/s), io=20.4MiB (21.4MB), run=117250-117805msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=175049: Sat Sep 5 04:06:04 2026 read: IOPS=1, BW=63.4MiB/s (66.5MB/s)(40.0MiB/631msec) slat (nsec): min=47599k, max=47599k, avg=47598660.00, stdev= 0.00 clat (nsec): min=560505k, max=560505k, avg=560505375.00, stdev= 0.00 lat (nsec): min=608109k, max=608109k, avg=608108838.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.59%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.4MiB/s (66.5MB/s), 63.4MiB/s-63.4MiB/s (66.5MB/s-66.5MB/s), io=40.0MiB (41.9MB), run=631-631msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout