== sanity test 398c: run fio to test AIO ================= 05:57:23 (1788688643) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.76652 s, 54.7 MB/s osc.lustre-OST0000-osc-ffff941f44f4a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146292: Sun Sep 6 06:00:15 2026 write: IOPS=16, BW=65.1KiB/s (66.7kB/s)(10.0MiB/157317msec); 0 zone resets slat (usec): min=23, max=27104, avg=283.67, stdev=1016.77 clat (msec): min=190, max=1823, avg=982.35, stdev=210.45 lat (msec): min=190, max=1823, avg=982.63, stdev=210.48 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 651], 10.00th=[ 743], 20.00th=[ 835], | 30.00th=[ 885], 40.00th=[ 936], 50.00th=[ 978], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1536], 99.50th=[ 1603], 99.90th=[ 1770], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 128, per=25.41%, avg=65.80, stdev=31.86, samples=306 iops : min= 1, max= 32, avg=15.94, stdev= 7.98, samples=306 lat (msec) : 250=0.62%, 500=1.33%, 750=8.95%, 1000=44.80%, 2000=44.30% cpu : usr=0.05%, sys=0.26%, ctx=1938, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146293: Sun Sep 6 06:00:15 2026 write: IOPS=16, BW=64.9KiB/s (66.5kB/s)(10.0MiB/157704msec); 0 zone resets slat (usec): min=29, max=14998, avg=276.78, stdev=897.30 clat (msec): min=162, max=1901, avg=984.22, stdev=209.83 lat (msec): min=163, max=1901, avg=984.50, stdev=209.82 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 676], 10.00th=[ 743], 20.00th=[ 818], | 30.00th=[ 877], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1011], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1334], | 99.00th=[ 1552], 99.50th=[ 1687], 99.90th=[ 1888], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 128, per=24.99%, avg=64.73, stdev=23.91, samples=311 iops : min= 1, max= 32, avg=15.66, stdev= 6.02, samples=311 lat (msec) : 250=0.04%, 500=1.29%, 750=9.41%, 1000=45.55%, 2000=43.71% cpu : usr=0.06%, sys=0.26%, ctx=1873, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146294: Sun Sep 6 06:00:15 2026 write: IOPS=16, BW=64.9KiB/s (66.5kB/s)(10.0MiB/157688msec); 0 zone resets slat (usec): min=29, max=15523, avg=289.25, stdev=963.69 clat (msec): min=330, max=1972, avg=983.94, stdev=200.85 lat (msec): min=330, max=1973, avg=984.24, stdev=200.86 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 676], 10.00th=[ 751], 20.00th=[ 835], | 30.00th=[ 877], 40.00th=[ 927], 50.00th=[ 969], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1234], 95.00th=[ 1318], | 99.00th=[ 1502], 99.50th=[ 1636], 99.90th=[ 1821], 99.95th=[ 1838], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 128, per=25.08%, avg=64.95, stdev=24.03, samples=310 iops : min= 1, max= 32, avg=15.72, stdev= 6.06, samples=310 lat (msec) : 500=1.29%, 750=8.28%, 1000=46.80%, 2000=43.63% cpu : usr=0.06%, sys=0.28%, ctx=2066, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146295: Sun Sep 6 06:00:15 2026 write: IOPS=16, BW=64.0KiB/s (66.5kB/s)(10.0MiB/157611msec); 0 zone resets slat (usec): min=30, max=15283, avg=316.57, stdev=949.59 clat (msec): min=349, max=1973, avg=983.53, stdev=201.67 lat (msec): min=357, max=1974, avg=983.85, stdev=201.63 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 676], 10.00th=[ 751], 20.00th=[ 827], | 30.00th=[ 885], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1234], 95.00th=[ 1318], | 99.00th=[ 1502], 99.50th=[ 1653], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 128, per=24.84%, avg=64.32, stdev=27.51, samples=313 iops : min= 1, max= 32, avg=15.56, stdev= 6.96, samples=313 lat (msec) : 500=1.41%, 750=8.32%, 1000=46.60%, 2000=43.67% cpu : usr=0.06%, sys=0.30%, ctx=2193, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=260KiB/s (266kB/s), 64.9KiB/s-65.1KiB/s (66.5kB/s-66.7kB/s), io=40.0MiB (41.9MB), run=157317-157704msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146327: Sun Sep 6 06:02:24 2026 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5048KiB/120398msec) slat (usec): min=3240, max=75360, avg=15727.63, stdev=9267.70 clat (msec): min=10, max=1161, avg=466.98, stdev=139.06 lat (msec): min=36, max=1179, avg=482.72, stdev=140.65 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 284], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 514], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 953], 99.50th=[ 1045], 99.90th=[ 1133], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 103, per=25.09%, avg=41.65, stdev=20.30, samples=237 iops : min= 1, max= 25, avg= 9.86, stdev= 5.13, samples=237 write: IOPS=10, BW=43.1KiB/s (44.2kB/s)(5192KiB/120398msec); 0 zone resets slat (usec): min=36, max=16597, avg=307.30, stdev=987.49 clat (msec): min=252, max=2250, avg=1011.65, stdev=309.06 lat (msec): min=252, max=2250, avg=1011.96, stdev=309.04 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 818], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1569], | 99.00th=[ 1938], 99.50th=[ 2022], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 118, per=24.91%, avg=43.09, stdev=20.76, samples=235 iops : min= 1, max= 29, avg=10.23, stdev= 5.20, samples=235 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=0.74%, 500=32.23% lat (msec) : 750=25.23%, 1000=17.50%, 2000=23.71%, >=2000=0.31% cpu : usr=0.12%, sys=0.60%, ctx=4436, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146328: Sun Sep 6 06:02:24 2026 read: IOPS=10, BW=41.2KiB/s (42.2kB/s)(4976KiB/120676msec) slat (usec): min=3075, max=74957, avg=15780.01, stdev=8892.49 clat (msec): min=47, max=1094, avg=466.23, stdev=138.47 lat (msec): min=54, max=1120, avg=482.02, stdev=139.82 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 477], | 70.00th=[ 518], 80.00th=[ 567], 90.00th=[ 634], 95.00th=[ 718], | 99.00th=[ 936], 99.50th=[ 995], 99.90th=[ 1083], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 120, per=24.64%, avg=40.90, stdev=21.53, samples=238 iops : min= 1, max= 30, avg= 9.68, stdev= 5.43, samples=238 write: IOPS=10, BW=43.6KiB/s (44.7kB/s)(5264KiB/120676msec); 0 zone resets slat (usec): min=33, max=17142, avg=390.58, stdev=1140.82 clat (msec): min=72, max=2248, avg=1008.98, stdev=315.31 lat (msec): min=72, max=2248, avg=1009.38, stdev=315.27 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 567], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1062], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1418], 95.00th=[ 1552], | 99.00th=[ 1821], 99.50th=[ 2165], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 126, per=24.82%, avg=42.95, stdev=19.94, samples=239 iops : min= 1, max= 31, avg=10.19, stdev= 4.98, samples=239 lat (msec) : 50=0.08%, 100=0.27%, 250=0.94%, 500=32.11%, 750=24.30% lat (msec) : 1000=18.09%, 2000=23.83%, >=2000=0.39% cpu : usr=0.12%, sys=0.69%, ctx=4803, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146329: Sun Sep 6 06:02:24 2026 read: IOPS=10, BW=41.4KiB/s (42.4kB/s)(4984KiB/120397msec) slat (usec): min=47, max=59807, avg=16006.69, stdev=8746.39 clat (msec): min=41, max=1072, avg=470.51, stdev=139.50 lat (msec): min=59, max=1108, avg=486.52, stdev=140.92 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 363], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 527], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 726], | 99.00th=[ 919], 99.50th=[ 969], 99.90th=[ 1053], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7, max= 111, per=24.72%, avg=41.03, stdev=20.73, samples=238 iops : min= 1, max= 27, avg= 9.71, stdev= 5.23, samples=238 write: IOPS=10, BW=43.7KiB/s (44.7kB/s)(5256KiB/120397msec); 0 zone resets slat (usec): min=36, max=20294, avg=345.24, stdev=1114.25 clat (msec): min=213, max=2229, avg=1003.40, stdev=304.00 lat (msec): min=213, max=2229, avg=1003.74, stdev=303.99 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 584], 10.00th=[ 659], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1536], | 99.00th=[ 1821], 99.50th=[ 2056], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 104, per=24.83%, avg=42.95, stdev=19.73, samples=238 iops : min= 1, max= 26, avg=10.19, stdev= 4.95, samples=238 lat (msec) : 50=0.04%, 100=0.20%, 250=1.17%, 500=30.82%, 750=25.31% lat (msec) : 1000=19.18%, 2000=23.01%, >=2000=0.27% cpu : usr=0.10%, sys=0.60%, ctx=4334, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146330: Sun Sep 6 06:02:24 2026 read: IOPS=10, BW=41.0KiB/s (43.0kB/s)(5060KiB/120494msec) slat (usec): min=2405, max=81062, avg=15583.25, stdev=8746.83 clat (msec): min=21, max=1085, avg=463.74, stdev=132.17 lat (msec): min=34, max=1115, avg=479.33, stdev=134.14 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 296], 10.00th=[ 326], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 518], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 927], 99.50th=[ 986], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 111, per=25.51%, avg=42.35, stdev=20.95, samples=234 iops : min= 1, max= 27, avg=10.04, stdev= 5.27, samples=234 write: IOPS=10, BW=42.0KiB/s (44.0kB/s)(5180KiB/120494msec); 0 zone resets slat (usec): min=32, max=8752, avg=317.74, stdev=708.78 clat (msec): min=204, max=2313, avg=1017.82, stdev=307.42 lat (msec): min=204, max=2314, avg=1018.14, stdev=307.43 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 584], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 919], 50.00th=[ 986], 60.00th=[ 1062], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1552], | 99.00th=[ 1905], 99.50th=[ 2039], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 88, per=24.49%, avg=42.37, stdev=18.13, samples=238 iops : min= 1, max= 22, avg=10.05, stdev= 4.60, samples=238 lat (msec) : 50=0.12%, 100=0.04%, 250=0.66%, 500=32.46%, 750=24.80% lat (msec) : 1000=17.30%, 2000=24.34%, >=2000=0.27% cpu : usr=0.10%, sys=0.66%, ctx=4545, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=166KiB/s (170kB/s), 41.2KiB/s-41.0KiB/s (42.2kB/s-43.0kB/s), io=19.6MiB (20.5MB), run=120397-120676msec WRITE: bw=173KiB/s (177kB/s), 42.0KiB/s-43.7KiB/s (44.0kB/s-44.7kB/s), io=20.4MiB (21.4MB), run=120397-120676msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146364: Sun Sep 6 06:02:33 2026 read: IOPS=1, BW=49.0MiB/s (51.4MB/s)(40.0MiB/816msec) slat (nsec): min=52739k, max=52739k, avg=52739306.00, stdev= 0.00 clat (nsec): min=734757k, max=734757k, avg=734756713.00, stdev= 0.00 lat (nsec): min=787500k, max=787500k, avg=787500294.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 735], 5.00th=[ 735], 10.00th=[ 735], 20.00th=[ 735], | 30.00th=[ 735], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 735], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] lat (msec) : 750=100.00% cpu : usr=0.37%, sys=2.33%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=49.0MiB/s (51.4MB/s), 49.0MiB/s-49.0MiB/s (51.4MB/s-51.4MB/s), io=40.0MiB (41.9MB), run=816-816msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout