== sanity test 398c: run fio to test AIO ================= 19:40:04 (1788565204) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.775583 s, 54.1 MB/s osc.lustre-OST0000-osc-ffff9fc9c87a9000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146142: Fri Sep 4 19:43:45 2026 write: IOPS=12, BW=48.9KiB/s (50.1kB/s)(10.0MiB/209464msec); 0 zone resets slat (usec): min=29, max=31073, avg=215.05, stdev=969.89 clat (msec): min=92, max=2894, avg=1308.57, stdev=437.48 lat (msec): min=92, max=2894, avg=1308.79, stdev=437.47 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 726], 10.00th=[ 844], 20.00th=[ 944], | 30.00th=[ 1045], 40.00th=[ 1133], 50.00th=[ 1250], 60.00th=[ 1351], | 70.00th=[ 1502], 80.00th=[ 1670], 90.00th=[ 1905], 95.00th=[ 2165], | 99.00th=[ 2534], 99.50th=[ 2601], 99.90th=[ 2802], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 199, per=27.13%, avg=52.91, stdev=30.65, samples=380 iops : min= 1, max= 49, avg=12.77, stdev= 7.70, samples=380 lat (msec) : 100=0.04%, 250=0.04%, 500=1.84%, 750=3.40%, 1000=21.05% lat (msec) : 2000=65.86%, >=2000=7.77% cpu : usr=0.02%, sys=0.16%, ctx=1643, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146143: Fri Sep 4 19:43:45 2026 write: IOPS=12, BW=48.9KiB/s (50.1kB/s)(10.0MiB/209323msec); 0 zone resets slat (usec): min=30, max=21861, avg=214.87, stdev=1034.56 clat (msec): min=212, max=2888, avg=1307.53, stdev=441.36 lat (msec): min=212, max=2888, avg=1307.75, stdev=441.37 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 718], 10.00th=[ 818], 20.00th=[ 953], | 30.00th=[ 1045], 40.00th=[ 1133], 50.00th=[ 1250], 60.00th=[ 1351], | 70.00th=[ 1502], 80.00th=[ 1653], 90.00th=[ 1871], 95.00th=[ 2198], | 99.00th=[ 2500], 99.50th=[ 2601], 99.90th=[ 2735], 99.95th=[ 2735], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 256, per=25.74%, avg=50.20, stdev=30.02, samples=400 iops : min= 1, max= 64, avg=12.07, stdev= 7.56, samples=400 lat (msec) : 250=0.78%, 500=1.72%, 750=4.14%, 1000=18.55%, 2000=67.58% lat (msec) : >=2000=7.23% cpu : usr=0.04%, sys=0.16%, ctx=1574, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146144: Fri Sep 4 19:43:45 2026 write: IOPS=12, BW=48.9KiB/s (50.0kB/s)(10.0MiB/209524msec); 0 zone resets slat (usec): min=29, max=28838, avg=216.73, stdev=993.37 clat (msec): min=373, max=2894, avg=1308.49, stdev=438.89 lat (msec): min=373, max=2894, avg=1308.71, stdev=438.88 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 726], 10.00th=[ 818], 20.00th=[ 944], | 30.00th=[ 1036], 40.00th=[ 1133], 50.00th=[ 1250], 60.00th=[ 1351], | 70.00th=[ 1485], 80.00th=[ 1653], 90.00th=[ 1888], 95.00th=[ 2165], | 99.00th=[ 2500], 99.50th=[ 2635], 99.90th=[ 2869], 99.95th=[ 2869], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 255, per=25.44%, avg=49.61, stdev=29.13, samples=405 iops : min= 1, max= 63, avg=11.94, stdev= 7.30, samples=405 lat (msec) : 500=1.95%, 750=4.22%, 1000=19.69%, 2000=66.33%, >=2000=7.81% cpu : usr=0.03%, sys=0.16%, ctx=1756, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146145: Fri Sep 4 19:43:45 2026 write: IOPS=12, BW=48.8KiB/s (49.0kB/s)(10.0MiB/209810msec); 0 zone resets slat (usec): min=24, max=17928, avg=190.54, stdev=852.54 clat (msec): min=398, max=2962, avg=1310.27, stdev=441.30 lat (msec): min=398, max=2962, avg=1310.46, stdev=441.32 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 718], 10.00th=[ 810], 20.00th=[ 944], | 30.00th=[ 1045], 40.00th=[ 1150], 50.00th=[ 1250], 60.00th=[ 1368], | 70.00th=[ 1502], 80.00th=[ 1653], 90.00th=[ 1888], 95.00th=[ 2165], | 99.00th=[ 2500], 99.50th=[ 2735], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 255, per=26.03%, avg=50.77, stdev=30.75, samples=396 iops : min= 1, max= 63, avg=12.23, stdev= 7.71, samples=396 lat (msec) : 500=1.88%, 750=5.00%, 1000=19.96%, 2000=65.47%, >=2000=7.70% cpu : usr=0.04%, sys=0.15%, ctx=1700, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=195KiB/s (200kB/s), 48.8KiB/s-48.9KiB/s (49.0kB/s-50.1kB/s), io=40.0MiB (41.9MB), run=209323-209810msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146186: Fri Sep 4 19:45:52 2026 read: IOPS=10, BW=43.5KiB/s (44.6kB/s)(5048KiB/116014msec) slat (usec): min=2412, max=67165, avg=13436.36, stdev=8422.58 clat (msec): min=25, max=1140, avg=453.64, stdev=152.87 lat (msec): min=34, max=1171, avg=467.08, stdev=155.35 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 268], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 397], 50.00th=[ 426], 60.00th=[ 460], | 70.00th=[ 502], 80.00th=[ 558], 90.00th=[ 651], 95.00th=[ 751], | 99.00th=[ 936], 99.50th=[ 1062], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 144, per=25.25%, avg=43.43, stdev=20.99, samples=228 iops : min= 1, max= 36, avg=10.38, stdev= 5.33, samples=228 write: IOPS=11, BW=44.8KiB/s (45.8kB/s)(5192KiB/116014msec); 0 zone resets slat (usec): min=31, max=17874, avg=247.86, stdev=911.52 clat (msec): min=251, max=2699, avg=973.04, stdev=318.40 lat (msec): min=251, max=2699, avg=973.29, stdev=318.42 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 575], 10.00th=[ 651], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1351], 95.00th=[ 1569], | 99.00th=[ 2056], 99.50th=[ 2400], 99.90th=[ 2668], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 111, per=24.77%, avg=44.34, stdev=22.74, samples=229 iops : min= 1, max= 27, avg=10.60, stdev= 5.71, samples=229 lat (msec) : 50=0.08%, 100=0.12%, 250=1.33%, 500=33.79%, 750=23.05% lat (msec) : 1000=21.29%, 2000=19.84%, >=2000=0.51% cpu : usr=0.09%, sys=0.49%, ctx=4636, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146187: Fri Sep 4 19:45:52 2026 read: IOPS=10, BW=42.8KiB/s (43.9kB/s)(4976KiB/116176msec) slat (usec): min=2968, max=71835, avg=14122.52, stdev=8578.76 clat (msec): min=72, max=1285, avg=450.46, stdev=151.06 lat (msec): min=88, max=1309, avg=464.59, stdev=153.40 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 271], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 422], 60.00th=[ 456], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 651], 95.00th=[ 751], | 99.00th=[ 919], 99.50th=[ 1083], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 7, max= 104, per=24.81%, avg=42.67, stdev=20.86, samples=228 iops : min= 1, max= 26, avg=10.18, stdev= 5.32, samples=228 write: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5264KiB/116176msec); 0 zone resets slat (usec): min=30, max=15131, avg=211.51, stdev=664.60 clat (msec): min=157, max=2864, avg=971.31, stdev=325.86 lat (msec): min=157, max=2864, avg=971.53, stdev=325.85 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 542], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1183], 90.00th=[ 1351], 95.00th=[ 1552], | 99.00th=[ 2140], 99.50th=[ 2299], 99.90th=[ 2601], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 119, per=24.96%, avg=44.68, stdev=21.20, samples=231 iops : min= 1, max= 29, avg=10.69, stdev= 5.37, samples=231 lat (msec) : 100=0.12%, 250=1.41%, 500=34.41%, 750=22.73%, 1000=20.66% lat (msec) : 2000=20.04%, >=2000=0.62% cpu : usr=0.07%, sys=0.52%, ctx=4624, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146188: Fri Sep 4 19:45:52 2026 read: IOPS=10, BW=42.9KiB/s (43.0kB/s)(4984KiB/116053msec) slat (usec): min=63, max=76497, avg=13922.86, stdev=8693.54 clat (msec): min=24, max=1328, avg=454.36, stdev=154.17 lat (msec): min=38, max=1348, avg=468.29, stdev=156.79 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 262], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 368], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 464], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 651], 95.00th=[ 760], | 99.00th=[ 927], 99.50th=[ 978], 99.90th=[ 1167], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 112, per=25.16%, avg=43.27, stdev=20.75, samples=226 iops : min= 1, max= 28, avg=10.34, stdev= 5.25, samples=226 write: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5256KiB/116053msec); 0 zone resets slat (usec): min=35, max=16412, avg=245.89, stdev=860.35 clat (msec): min=120, max=2731, avg=968.31, stdev=324.37 lat (msec): min=122, max=2731, avg=968.56, stdev=324.41 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 978], | 70.00th=[ 1070], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1552], | 99.00th=[ 2089], 99.50th=[ 2333], 99.90th=[ 2635], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min= 7, max= 103, per=24.94%, avg=44.64, stdev=20.75, samples=230 iops : min= 1, max= 25, avg=10.68, stdev= 5.26, samples=230 lat (msec) : 50=0.16%, 100=0.27%, 250=1.56%, 500=33.09%, 750=23.01% lat (msec) : 1000=22.19%, 2000=19.06%, >=2000=0.66% cpu : usr=0.08%, sys=0.50%, ctx=4694, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146189: Fri Sep 4 19:45:52 2026 read: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5060KiB/116134msec) slat (usec): min=2337, max=66513, avg=14054.47, stdev=8536.51 clat (msec): min=20, max=969, avg=453.08, stdev=144.99 lat (msec): min=26, max=986, avg=467.14, stdev=147.73 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 330], | 30.00th=[ 368], 40.00th=[ 401], 50.00th=[ 435], 60.00th=[ 460], | 70.00th=[ 498], 80.00th=[ 550], 90.00th=[ 651], 95.00th=[ 760], | 99.00th=[ 885], 99.50th=[ 919], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 96, per=25.50%, avg=43.87, stdev=19.31, samples=226 iops : min= 1, max= 24, avg=10.48, stdev= 4.91, samples=226 write: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5180KiB/116134msec); 0 zone resets slat (usec): min=35, max=26108, avg=251.08, stdev=1044.66 clat (msec): min=289, max=2894, avg=975.96, stdev=322.62 lat (msec): min=289, max=2894, avg=976.22, stdev=322.67 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 726], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1569], | 99.00th=[ 2039], 99.50th=[ 2198], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 88, per=24.86%, avg=44.50, stdev=19.32, samples=228 iops : min= 1, max= 22, avg=10.64, stdev= 4.88, samples=228 lat (msec) : 50=0.12%, 100=0.12%, 250=1.37%, 500=34.14%, 750=22.93% lat (msec) : 1000=21.84%, 2000=18.87%, >=2000=0.62% cpu : usr=0.10%, sys=0.53%, ctx=4639, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=173KiB/s (177kB/s), 42.8KiB/s-43.6KiB/s (43.9kB/s-44.6kB/s), io=19.6MiB (20.5MB), run=116014-116176msec WRITE: bw=180KiB/s (184kB/s), 44.6KiB/s-45.3KiB/s (45.7kB/s-46.4kB/s), io=20.4MiB (21.4MB), run=116014-116176msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146210: Fri Sep 4 19:46:01 2026 read: IOPS=1, BW=50.1MiB/s (52.5MB/s)(40.0MiB/799msec) slat (nsec): min=77957k, max=77957k, avg=77957067.00, stdev= 0.00 clat (nsec): min=658610k, max=658610k, avg=658609931.00, stdev= 0.00 lat (nsec): min=736572k, max=736572k, avg=736571843.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 659], 5.00th=[ 659], 10.00th=[ 659], 20.00th=[ 659], | 30.00th=[ 659], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.50%, ctx=38, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=50.1MiB/s (52.5MB/s), 50.1MiB/s-50.1MiB/s (52.5MB/s-52.5MB/s), io=40.0MiB (41.9MB), run=799-799msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout