== sanity test 398c: run fio to test AIO ================= 18:04:06 (1787609046) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.537448 s, 78.0 MB/s osc.lustre-OST0000-osc-ffff958eeb329000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145603: Mon Aug 24 18:06:53 2026 write: IOPS=16, BW=65.7KiB/s (67.2kB/s)(10.0MiB/155953msec); 0 zone resets slat (usec): min=22, max=27883, avg=293.47, stdev=1312.55 clat (msec): min=323, max=2490, avg=974.15, stdev=275.36 lat (msec): min=323, max=2490, avg=974.45, stdev=275.40 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 558], 10.00th=[ 659], 20.00th=[ 768], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1435], | 99.00th=[ 1770], 99.50th=[ 2106], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=25.17%, avg=65.94, stdev=28.39, samples=305 iops : min= 1, max= 32, avg=16.00, stdev= 7.13, samples=305 lat (msec) : 500=3.59%, 750=13.87%, 1000=40.78%, 2000=41.13%, >=2000=0.62% cpu : usr=0.07%, sys=0.30%, ctx=2028, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145604: Mon Aug 24 18:06:53 2026 write: IOPS=16, BW=65.6KiB/s (67.2kB/s)(10.0MiB/155980msec); 0 zone resets slat (usec): min=21, max=20073, avg=259.54, stdev=1034.62 clat (msec): min=322, max=2512, avg=973.95, stdev=279.31 lat (msec): min=322, max=2512, avg=974.22, stdev=279.31 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 567], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1804], 99.50th=[ 2056], 99.90th=[ 2400], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 143, per=25.67%, avg=67.25, stdev=30.84, samples=299 iops : min= 1, max= 35, avg=16.33, stdev= 7.75, samples=299 lat (msec) : 500=3.71%, 750=14.65%, 1000=39.18%, 2000=41.91%, >=2000=0.55% cpu : usr=0.06%, sys=0.25%, ctx=1762, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145605: Mon Aug 24 18:06:53 2026 write: IOPS=16, BW=65.6KiB/s (67.2kB/s)(10.0MiB/156127msec); 0 zone resets slat (usec): min=22, max=43151, avg=284.14, stdev=1458.12 clat (msec): min=337, max=2321, avg=974.74, stdev=275.74 lat (msec): min=337, max=2321, avg=975.03, stdev=275.75 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 531], 10.00th=[ 651], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1821], 99.50th=[ 2039], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 128, per=25.94%, avg=67.96, stdev=32.05, samples=296 iops : min= 1, max= 32, avg=16.51, stdev= 8.00, samples=296 lat (msec) : 500=3.16%, 750=14.30%, 1000=40.51%, 2000=41.48%, >=2000=0.55% cpu : usr=0.06%, sys=0.28%, ctx=2021, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145606: Mon Aug 24 18:06:53 2026 write: IOPS=16, BW=65.7KiB/s (67.3kB/s)(10.0MiB/155837msec); 0 zone resets slat (usec): min=23, max=37820, avg=277.30, stdev=1380.77 clat (msec): min=94, max=2491, avg=973.44, stdev=277.69 lat (msec): min=95, max=2492, avg=973.71, stdev=277.66 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 558], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1469], | 99.00th=[ 1888], 99.50th=[ 2005], 99.90th=[ 2140], 99.95th=[ 2333], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 223, per=25.28%, avg=66.22, stdev=34.22, samples=303 iops : min= 1, max= 55, avg=16.07, stdev= 8.57, samples=303 lat (msec) : 100=0.08%, 250=0.43%, 500=3.48%, 750=12.15%, 1000=42.34% lat (msec) : 2000=40.98%, >=2000=0.55% cpu : usr=0.06%, sys=0.28%, ctx=1885, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=262KiB/s (269kB/s), 65.6KiB/s-65.7KiB/s (67.2kB/s-67.3kB/s), io=40.0MiB (41.9MB), run=155837-156127msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145638: Mon Aug 24 18:09:00 2026 read: IOPS=10, BW=42.5KiB/s (43.6kB/s)(5048KiB/118684msec) slat (usec): min=2766, max=75497, avg=15945.57, stdev=9745.44 clat (msec): min=8, max=1433, avg=463.04, stdev=179.01 lat (msec): min=19, max=1493, avg=478.99, stdev=182.14 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 255], 10.00th=[ 292], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 422], 60.00th=[ 456], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 701], 95.00th=[ 793], | 99.00th=[ 1070], 99.50th=[ 1217], 99.90th=[ 1334], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 111, per=25.45%, avg=42.75, stdev=20.36, samples=231 iops : min= 1, max= 27, avg=10.22, stdev= 5.13, samples=231 write: IOPS=10, BW=43.7KiB/s (44.8kB/s)(5192KiB/118684msec); 0 zone resets slat (usec): min=35, max=25496, avg=407.90, stdev=1483.37 clat (msec): min=362, max=2711, avg=993.75, stdev=339.84 lat (msec): min=362, max=2711, avg=994.16, stdev=339.95 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 531], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 793], 40.00th=[ 869], 50.00th=[ 944], 60.00th=[ 1020], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1401], 95.00th=[ 1636], | 99.00th=[ 2140], 99.50th=[ 2333], 99.90th=[ 2567], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 119, per=25.42%, avg=44.24, stdev=21.28, samples=229 iops : min= 1, max= 29, avg=10.59, stdev= 5.36, samples=229 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.20%, 250=1.72% lat (msec) : 500=33.12%, 750=22.66%, 1000=20.04%, 2000=21.37%, >=2000=0.70% cpu : usr=0.12%, sys=0.74%, ctx=4629, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145639: Mon Aug 24 18:09:00 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(4976KiB/119447msec) slat (usec): min=70, max=92947, avg=16501.36, stdev=10720.86 clat (msec): min=20, max=1494, avg=467.94, stdev=175.63 lat (msec): min=21, max=1534, avg=484.46, stdev=178.70 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 262], 10.00th=[ 292], 20.00th=[ 334], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 430], 60.00th=[ 468], | 70.00th=[ 527], 80.00th=[ 584], 90.00th=[ 709], 95.00th=[ 793], | 99.00th=[ 1070], 99.50th=[ 1116], 99.90th=[ 1334], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 128, per=25.03%, avg=42.04, stdev=21.57, samples=231 iops : min= 1, max= 32, avg=10.01, stdev= 5.45, samples=231 write: IOPS=11, BW=44.1KiB/s (45.1kB/s)(5264KiB/119447msec); 0 zone resets slat (usec): min=29, max=18760, avg=377.94, stdev=1277.54 clat (msec): min=97, max=2607, avg=993.08, stdev=344.25 lat (msec): min=99, max=2608, avg=993.46, stdev=344.31 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 502], 10.00th=[ 617], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1401], 95.00th=[ 1620], | 99.00th=[ 2089], 99.50th=[ 2265], 99.90th=[ 2500], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 133, per=25.58%, avg=44.52, stdev=21.61, samples=232 iops : min= 1, max= 33, avg=10.63, stdev= 5.46, samples=232 lat (msec) : 50=0.12%, 100=0.35%, 250=1.68%, 500=32.62%, 750=22.11% lat (msec) : 1000=19.69%, 2000=22.70%, >=2000=0.74% cpu : usr=0.11%, sys=0.70%, ctx=4657, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145640: Mon Aug 24 18:09:00 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(4984KiB/119433msec) slat (msec): min=3, max=134, avg=16.47, stdev=10.60 clat (msec): min=8, max=1454, avg=460.08, stdev=168.42 lat (msec): min=15, max=1468, avg=476.56, stdev=171.29 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 266], 10.00th=[ 296], 20.00th=[ 338], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 502], 80.00th=[ 567], 90.00th=[ 701], 95.00th=[ 776], | 99.00th=[ 1020], 99.50th=[ 1150], 99.90th=[ 1318], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 7, max= 142, per=25.27%, avg=42.46, stdev=21.69, samples=230 iops : min= 1, max= 35, avg=10.14, stdev= 5.48, samples=230 write: IOPS=11, BW=44.0KiB/s (45.1kB/s)(5256KiB/119433msec); 0 zone resets slat (usec): min=29, max=41666, avg=399.62, stdev=1554.76 clat (msec): min=309, max=2752, avg=999.15, stdev=332.35 lat (msec): min=309, max=2752, avg=999.56, stdev=332.46 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 558], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1636], | 99.00th=[ 2039], 99.50th=[ 2198], 99.90th=[ 2702], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 103, per=25.16%, avg=43.78, stdev=19.84, samples=234 iops : min= 1, max= 25, avg=10.48, stdev= 4.97, samples=234 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.08%, 250=1.41% lat (msec) : 500=34.02%, 750=21.21%, 1000=20.04%, 2000=22.50%, >=2000=0.62% cpu : usr=0.13%, sys=0.72%, ctx=4793, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145641: Mon Aug 24 18:09:00 2026 read: IOPS=10, BW=42.5KiB/s (43.5kB/s)(5060KiB/119081msec) slat (usec): min=3097, max=85033, avg=16357.69, stdev=10217.83 clat (msec): min=17, max=1578, avg=463.85, stdev=174.66 lat (msec): min=27, max=1605, avg=480.22, stdev=177.58 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 268], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 422], 60.00th=[ 456], | 70.00th=[ 506], 80.00th=[ 575], 90.00th=[ 709], 95.00th=[ 793], | 99.00th=[ 1011], 99.50th=[ 1150], 99.90th=[ 1368], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 112, per=25.58%, avg=42.98, stdev=21.39, samples=231 iops : min= 1, max= 28, avg=10.25, stdev= 5.40, samples=231 write: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5180KiB/119081msec); 0 zone resets slat (usec): min=28, max=18541, avg=362.66, stdev=1224.72 clat (msec): min=231, max=2699, avg=999.11, stdev=353.85 lat (msec): min=232, max=2699, avg=999.48, stdev=353.90 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 535], 10.00th=[ 609], 20.00th=[ 718], | 30.00th=[ 802], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1036], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1653], | 99.00th=[ 2299], 99.50th=[ 2400], 99.90th=[ 2668], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 96, per=24.93%, avg=43.38, stdev=20.18, samples=233 iops : min= 1, max= 24, avg=10.35, stdev= 5.10, samples=233 lat (msec) : 20=0.04%, 50=0.08%, 100=0.04%, 250=1.52%, 500=34.26% lat (msec) : 750=21.25%, 1000=19.73%, 2000=22.15%, >=2000=0.94% cpu : usr=0.13%, sys=0.76%, ctx=4824, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=168KiB/s (172kB/s), 41.7KiB/s-42.5KiB/s (42.7kB/s-43.6kB/s), io=19.6MiB (20.5MB), run=118684-119447msec WRITE: bw=175KiB/s (179kB/s), 43.5KiB/s-44.1KiB/s (44.5kB/s-45.1kB/s), io=20.4MiB (21.4MB), run=118684-119447msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145663: Mon Aug 24 18:09:10 2026 read: IOPS=2, BW=95.2MiB/s (99.9MB/s)(40.0MiB/420msec) slat (nsec): min=26402k, max=26402k, avg=26401635.00, stdev= 0.00 clat (nsec): min=377608k, max=377608k, avg=377608217.00, stdev= 0.00 lat (nsec): min=404016k, max=404016k, avg=404015928.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.34%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=95.2MiB/s (99.9MB/s), 95.2MiB/s-95.2MiB/s (99.9MB/s-99.9MB/s), io=40.0MiB (41.9MB), run=420-420msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout