== sanity test 398c: run fio to test AIO ================= 11:56:30 (1786636590) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.668754 s, 62.7 MB/s osc.lustre-OST0000-osc-ffff97c664397000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145614: Thu Aug 13 11:59:11 2026 write: IOPS=17, BW=68.3KiB/s (69.9kB/s)(10.0MiB/149915msec); 0 zone resets slat (usec): min=37, max=23195, avg=327.41, stdev=1028.21 clat (msec): min=143, max=2139, avg=935.42, stdev=306.99 lat (msec): min=143, max=2139, avg=935.75, stdev=307.00 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 531], 10.00th=[ 617], 20.00th=[ 693], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 944], | 70.00th=[ 1028], 80.00th=[ 1133], 90.00th=[ 1368], 95.00th=[ 1536], | 99.00th=[ 1871], 99.50th=[ 2005], 99.90th=[ 2123], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 255, per=25.08%, avg=68.47, stdev=28.63, samples=294 iops : min= 1, max= 63, avg=16.60, stdev= 7.19, samples=294 lat (msec) : 250=0.04%, 500=4.30%, 750=23.63%, 1000=39.22%, 2000=32.23% lat (msec) : >=2000=0.59% cpu : usr=0.06%, sys=0.34%, ctx=1961, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145615: Thu Aug 13 11:59:11 2026 write: IOPS=17, BW=68.4KiB/s (70.1kB/s)(10.0MiB/149679msec); 0 zone resets slat (usec): min=36, max=12912, avg=315.21, stdev=852.12 clat (msec): min=248, max=2160, avg=934.24, stdev=303.38 lat (msec): min=248, max=2160, avg=934.56, stdev=303.40 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 514], 10.00th=[ 617], 20.00th=[ 701], | 30.00th=[ 768], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 944], | 70.00th=[ 1028], 80.00th=[ 1150], 90.00th=[ 1351], 95.00th=[ 1552], | 99.00th=[ 1888], 99.50th=[ 1955], 99.90th=[ 2089], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 231, per=25.08%, avg=68.46, stdev=30.21, samples=294 iops : min= 1, max= 57, avg=16.60, stdev= 7.58, samples=294 lat (msec) : 250=0.04%, 500=4.77%, 750=22.54%, 1000=39.65%, 2000=32.73% lat (msec) : >=2000=0.27% cpu : usr=0.05%, sys=0.38%, ctx=2251, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145616: Thu Aug 13 11:59:11 2026 write: IOPS=17, BW=68.5KiB/s (70.2kB/s)(10.0MiB/149470msec); 0 zone resets slat (usec): min=37, max=21249, avg=320.35, stdev=1073.90 clat (msec): min=127, max=2082, avg=933.41, stdev=306.71 lat (msec): min=128, max=2082, avg=933.73, stdev=306.73 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 514], 10.00th=[ 600], 20.00th=[ 693], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 953], | 70.00th=[ 1028], 80.00th=[ 1150], 90.00th=[ 1351], 95.00th=[ 1536], | 99.00th=[ 1838], 99.50th=[ 1938], 99.90th=[ 2005], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 151, per=25.16%, avg=68.68, stdev=32.01, samples=293 iops : min= 1, max= 37, avg=16.66, stdev= 8.03, samples=293 lat (msec) : 250=0.43%, 500=3.98%, 750=23.55%, 1000=39.80%, 2000=32.07% lat (msec) : >=2000=0.16% cpu : usr=0.06%, sys=0.32%, ctx=2015, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145617: Thu Aug 13 11:59:11 2026 write: IOPS=17, BW=68.3KiB/s (69.9kB/s)(10.0MiB/149918msec); 0 zone resets slat (usec): min=36, max=16781, avg=327.07, stdev=989.84 clat (msec): min=353, max=2059, avg=935.69, stdev=303.44 lat (msec): min=353, max=2059, avg=936.02, stdev=303.45 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 535], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 944], | 70.00th=[ 1028], 80.00th=[ 1133], 90.00th=[ 1368], 95.00th=[ 1519], | 99.00th=[ 1888], 99.50th=[ 2005], 99.90th=[ 2022], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 255, per=25.33%, avg=69.15, stdev=32.53, samples=291 iops : min= 1, max= 63, avg=16.77, stdev= 8.18, samples=291 lat (msec) : 500=4.14%, 750=23.91%, 1000=38.59%, 2000=32.81%, >=2000=0.55% cpu : usr=0.07%, sys=0.31%, ctx=1805, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=273KiB/s (280kB/s), 68.3KiB/s-68.5KiB/s (69.9kB/s-70.2kB/s), io=40.0MiB (41.9MB), run=149470-149918msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145650: Thu Aug 13 12:01:13 2026 read: IOPS=11, BW=44.0KiB/s (45.1kB/s)(5048KiB/114655msec) slat (usec): min=2917, max=70044, avg=16813.61, stdev=8957.61 clat (msec): min=12, max=987, avg=435.46, stdev=127.38 lat (msec): min=20, max=1009, avg=452.30, stdev=129.13 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 443], | 70.00th=[ 477], 80.00th=[ 514], 90.00th=[ 592], 95.00th=[ 684], | 99.00th=[ 852], 99.50th=[ 936], 99.90th=[ 969], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 127, per=25.34%, avg=44.09, stdev=22.35, samples=224 iops : min= 1, max= 31, avg=10.48, stdev= 5.61, samples=224 write: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5192KiB/114655msec); 0 zone resets slat (usec): min=42, max=26445, avg=498.84, stdev=1612.65 clat (msec): min=196, max=2608, avg=969.60, stdev=293.85 lat (msec): min=197, max=2608, avg=970.10, stdev=293.84 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1334], 95.00th=[ 1519], | 99.00th=[ 1972], 99.50th=[ 2072], 99.90th=[ 2534], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 96, per=24.61%, avg=44.79, stdev=19.70, samples=226 iops : min= 1, max= 24, avg=10.65, stdev= 4.91, samples=226 lat (msec) : 20=0.04%, 50=0.12%, 100=0.12%, 250=1.05%, 500=37.70% lat (msec) : 750=19.69%, 1000=22.46%, 2000=18.48%, >=2000=0.35% cpu : usr=0.12%, sys=0.78%, ctx=4714, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145651: Thu Aug 13 12:01:13 2026 read: IOPS=10, BW=43.4KiB/s (44.4kB/s)(4976KiB/114754msec) slat (usec): min=3396, max=65374, avg=16545.04, stdev=8956.77 clat (msec): min=30, max=1058, avg=437.74, stdev=129.38 lat (msec): min=43, max=1089, avg=454.30, stdev=130.95 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 523], 90.00th=[ 609], 95.00th=[ 684], | 99.00th=[ 827], 99.50th=[ 927], 99.90th=[ 978], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 111, per=24.93%, avg=43.37, stdev=20.15, samples=224 iops : min= 1, max= 27, avg=10.30, stdev= 5.04, samples=224 write: IOPS=11, BW=45.9KiB/s (46.0kB/s)(5264KiB/114754msec); 0 zone resets slat (usec): min=40, max=20310, avg=504.73, stdev=1330.72 clat (msec): min=146, max=2564, avg=962.09, stdev=300.04 lat (msec): min=146, max=2564, avg=962.60, stdev=300.06 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 558], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1301], 95.00th=[ 1519], | 99.00th=[ 1989], 99.50th=[ 2299], 99.90th=[ 2534], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 120, per=25.22%, avg=45.90, stdev=21.31, samples=224 iops : min= 1, max= 30, avg=10.93, stdev= 5.37, samples=224 lat (msec) : 50=0.08%, 100=0.23%, 250=1.09%, 500=36.91%, 750=19.96% lat (msec) : 1000=22.27%, 2000=18.95%, >=2000=0.51% cpu : usr=0.10%, sys=0.84%, ctx=4717, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145652: Thu Aug 13 12:01:13 2026 read: IOPS=10, BW=43.5KiB/s (44.5kB/s)(4984KiB/114581msec) slat (usec): min=82, max=85610, avg=16710.93, stdev=9313.80 clat (msec): min=26, max=994, avg=434.46, stdev=126.12 lat (msec): min=41, max=1017, avg=451.18, stdev=128.03 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 338], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 472], 80.00th=[ 510], 90.00th=[ 584], 95.00th=[ 667], | 99.00th=[ 860], 99.50th=[ 936], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 119, per=25.20%, avg=43.85, stdev=21.75, samples=223 iops : min= 1, max= 29, avg=10.43, stdev= 5.43, samples=223 write: IOPS=11, BW=45.9KiB/s (46.0kB/s)(5256KiB/114581msec); 0 zone resets slat (usec): min=36, max=14628, avg=456.92, stdev=1014.40 clat (msec): min=203, max=2648, avg=965.94, stdev=295.52 lat (msec): min=203, max=2648, avg=966.40, stdev=295.50 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 575], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 978], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1318], 95.00th=[ 1519], | 99.00th=[ 2005], 99.50th=[ 2106], 99.90th=[ 2500], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 103, per=24.83%, avg=45.19, stdev=21.17, samples=226 iops : min= 1, max= 25, avg=10.76, stdev= 5.31, samples=226 lat (msec) : 50=0.16%, 100=0.23%, 250=0.90%, 500=37.58%, 750=18.16% lat (msec) : 1000=24.14%, 2000=18.28%, >=2000=0.55% cpu : usr=0.13%, sys=0.83%, ctx=4674, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145653: Thu Aug 13 12:01:13 2026 read: IOPS=11, BW=44.1KiB/s (45.2kB/s)(5060KiB/114669msec) slat (usec): min=3340, max=74332, avg=16801.94, stdev=9399.95 clat (msec): min=43, max=1060, avg=441.39, stdev=128.84 lat (msec): min=51, max=1091, avg=458.20, stdev=130.85 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 523], 90.00th=[ 609], 95.00th=[ 693], | 99.00th=[ 852], 99.50th=[ 936], 99.90th=[ 1003], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 103, per=25.60%, avg=44.54, stdev=19.86, samples=222 iops : min= 1, max= 25, avg=10.59, stdev= 4.97, samples=222 write: IOPS=11, BW=45.2KiB/s (46.3kB/s)(5180KiB/114669msec); 0 zone resets slat (usec): min=55, max=25582, avg=465.20, stdev=1164.67 clat (msec): min=344, max=2489, avg=965.20, stdev=290.52 lat (msec): min=344, max=2489, avg=965.67, stdev=290.53 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 575], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 919], 60.00th=[ 978], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1502], | 99.00th=[ 1989], 99.50th=[ 2089], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 104, per=24.79%, avg=45.12, stdev=20.30, samples=223 iops : min= 1, max= 26, avg=10.74, stdev= 5.08, samples=223 lat (msec) : 50=0.12%, 100=0.16%, 250=0.78%, 500=36.84%, 750=19.96% lat (msec) : 1000=23.55%, 2000=18.12%, >=2000=0.47% cpu : usr=0.10%, sys=0.84%, ctx=4569, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=175KiB/s (179kB/s), 43.4KiB/s-44.1KiB/s (44.4kB/s-45.2kB/s), io=19.6MiB (20.5MB), run=114581-114754msec WRITE: bw=182KiB/s (186kB/s), 45.2KiB/s-45.9KiB/s (46.3kB/s-46.0kB/s), io=20.4MiB (21.4MB), run=114581-114754msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145686: Thu Aug 13 12:01:24 2026 read: IOPS=1, BW=51.9MiB/s (54.5MB/s)(40.0MiB/770msec) slat (nsec): min=53512k, max=53512k, avg=53511908.00, stdev= 0.00 clat (nsec): min=682215k, max=682215k, avg=682214684.00, stdev= 0.00 lat (nsec): min=735732k, max=735732k, avg=735731548.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 684], 5.00th=[ 684], 10.00th=[ 684], 20.00th=[ 684], | 30.00th=[ 684], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.30%, ctx=40, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.9MiB/s (54.5MB/s), 51.9MiB/s-51.9MiB/s (54.5MB/s-54.5MB/s), io=40.0MiB (41.9MB), run=770-770msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout