== sanity test 398c: run fio to test AIO ================= 11:55:06 (1772470506) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.448179 s, 93.6 MB/s osc.lustre-OST0000-osc-ffff98d4d2399800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148520: Mon Mar 2 11:57:42 2026 write: IOPS=17, BW=69.2KiB/s (70.8kB/s)(10.0MiB/148051msec); 0 zone resets slat (usec): min=22, max=20905, avg=208.13, stdev=835.45 clat (msec): min=123, max=1698, avg=924.68, stdev=242.56 lat (msec): min=123, max=1699, avg=924.89, stdev=242.53 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 527], 10.00th=[ 667], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 911], 60.00th=[ 953], | 70.00th=[ 1020], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1401], | 99.00th=[ 1586], 99.50th=[ 1620], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 193, per=24.89%, avg=68.45, stdev=28.30, samples=294 iops : min= 1, max= 48, avg=16.68, stdev= 7.08, samples=294 lat (msec) : 250=0.66%, 500=3.48%, 750=15.00%, 1000=49.10%, 2000=31.76% cpu : usr=0.07%, sys=0.25%, ctx=2207, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148521: Mon Mar 2 11:57:42 2026 write: IOPS=17, BW=68.9KiB/s (70.5kB/s)(10.0MiB/148638msec); 0 zone resets slat (usec): min=21, max=102514, avg=227.27, stdev=2153.06 clat (msec): min=335, max=1767, avg=927.46, stdev=229.13 lat (msec): min=336, max=1767, avg=927.69, stdev=229.03 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 869], 50.00th=[ 902], 60.00th=[ 953], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1368], | 99.00th=[ 1586], 99.50th=[ 1620], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 128, per=24.92%, avg=68.53, stdev=30.43, samples=294 iops : min= 1, max= 32, avg=16.70, stdev= 7.61, samples=294 lat (msec) : 500=3.55%, 750=15.31%, 1000=50.59%, 2000=30.55% cpu : usr=0.04%, sys=0.25%, ctx=2178, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148522: Mon Mar 2 11:57:42 2026 write: IOPS=17, BW=68.0KiB/s (70.7kB/s)(10.0MiB/148418msec); 0 zone resets slat (usec): min=21, max=16141, avg=183.72, stdev=759.41 clat (msec): min=330, max=1682, avg=926.85, stdev=232.94 lat (msec): min=330, max=1682, avg=927.04, stdev=232.95 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 567], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 902], 60.00th=[ 961], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1385], | 99.00th=[ 1569], 99.50th=[ 1603], 99.90th=[ 1670], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 15, max= 248, per=25.09%, avg=68.99, stdev=33.34, samples=292 iops : min= 3, max= 62, avg=16.82, stdev= 8.38, samples=292 lat (msec) : 500=3.98%, 750=15.20%, 1000=48.28%, 2000=32.54% cpu : usr=0.05%, sys=0.21%, ctx=1528, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148523: Mon Mar 2 11:57:42 2026 write: IOPS=17, BW=68.9KiB/s (70.5kB/s)(10.0MiB/148641msec); 0 zone resets slat (usec): min=21, max=24798, avg=208.75, stdev=961.21 clat (msec): min=372, max=1768, avg=927.41, stdev=226.64 lat (msec): min=372, max=1768, avg=927.62, stdev=226.60 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 902], 60.00th=[ 953], | 70.00th=[ 1011], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1368], | 99.00th=[ 1536], 99.50th=[ 1586], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 207, per=25.17%, avg=69.22, stdev=29.10, samples=291 iops : min= 1, max= 51, avg=16.87, stdev= 7.30, samples=291 lat (msec) : 500=3.20%, 750=15.62%, 1000=49.38%, 2000=31.80% cpu : usr=0.05%, sys=0.23%, ctx=1895, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=276KiB/s (282kB/s), 68.9KiB/s-69.2KiB/s (70.5kB/s-70.8kB/s), io=40.0MiB (41.9MB), run=148051-148641msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148555: Mon Mar 2 11:59:33 2026 read: IOPS=12, BW=50.3KiB/s (51.5kB/s)(5048KiB/100430msec) slat (usec): min=2479, max=63737, avg=13292.49, stdev=7455.62 clat (msec): min=7, max=757, avg=388.54, stdev=101.18 lat (msec): min=16, max=773, avg=401.84, stdev=102.30 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 232], 10.00th=[ 275], 20.00th=[ 313], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 468], 90.00th=[ 518], 95.00th=[ 558], | 99.00th=[ 659], 99.50th=[ 709], 99.90th=[ 743], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 7, max= 144, per=25.16%, avg=50.07, stdev=22.24, samples=197 iops : min= 1, max= 36, avg=12.04, stdev= 5.62, samples=197 write: IOPS=12, BW=51.7KiB/s (52.9kB/s)(5192KiB/100430msec); 0 zone resets slat (usec): min=23, max=17059, avg=233.75, stdev=724.09 clat (msec): min=236, max=1573, avg=843.49, stdev=221.48 lat (msec): min=236, max=1573, avg=843.73, stdev=221.50 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 510], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1116], 95.00th=[ 1217], | 99.00th=[ 1418], 99.50th=[ 1502], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8, max= 134, per=24.85%, avg=51.44, stdev=20.90, samples=198 iops : min= 2, max= 33, avg=12.38, stdev= 5.20, samples=198 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.27%, 250=3.05% lat (msec) : 500=41.21%, 750=21.29%, 1000=21.72%, 2000=12.34% cpu : usr=0.10%, sys=0.63%, ctx=4675, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148556: Mon Mar 2 11:59:33 2026 read: IOPS=12, BW=49.5KiB/s (50.7kB/s)(4976KiB/100592msec) slat (usec): min=2350, max=53343, avg=13399.29, stdev=7720.24 clat (msec): min=14, max=769, avg=389.81, stdev=99.10 lat (msec): min=26, max=794, avg=403.22, stdev=100.59 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 247], 10.00th=[ 279], 20.00th=[ 313], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 405], | 70.00th=[ 435], 80.00th=[ 468], 90.00th=[ 523], 95.00th=[ 567], | 99.00th=[ 651], 99.50th=[ 684], 99.90th=[ 735], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 7, max= 104, per=24.50%, avg=48.75, stdev=21.59, samples=199 iops : min= 1, max= 26, avg=11.71, stdev= 5.43, samples=199 write: IOPS=13, BW=52.3KiB/s (53.6kB/s)(5264KiB/100592msec); 0 zone resets slat (usec): min=25, max=31005, avg=304.53, stdev=1206.85 clat (msec): min=266, max=1527, avg=839.04, stdev=225.40 lat (msec): min=266, max=1527, avg=839.35, stdev=225.38 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 464], 10.00th=[ 558], 20.00th=[ 642], | 30.00th=[ 718], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1020], 90.00th=[ 1133], 95.00th=[ 1234], | 99.00th=[ 1418], 99.50th=[ 1469], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 8, max= 127, per=25.14%, avg=52.03, stdev=22.38, samples=198 iops : min= 2, max= 31, avg=12.53, stdev= 5.62, samples=198 lat (msec) : 20=0.04%, 50=0.20%, 100=0.16%, 250=2.23%, 500=42.66% lat (msec) : 750=20.82%, 1000=22.15%, 2000=11.76% cpu : usr=0.11%, sys=0.68%, ctx=4789, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148557: Mon Mar 2 11:59:33 2026 read: IOPS=12, BW=49.6KiB/s (50.8kB/s)(4984KiB/100433msec) slat (usec): min=2679, max=60766, avg=13630.48, stdev=7987.62 clat (msec): min=27, max=742, avg=383.24, stdev=103.10 lat (msec): min=39, max=792, avg=396.88, stdev=104.49 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 228], 10.00th=[ 266], 20.00th=[ 305], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 401], | 70.00th=[ 430], 80.00th=[ 460], 90.00th=[ 518], 95.00th=[ 567], | 99.00th=[ 651], 99.50th=[ 684], 99.90th=[ 726], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 7, max= 111, per=24.95%, avg=49.66, stdev=22.67, samples=197 iops : min= 1, max= 27, avg=11.93, stdev= 5.67, samples=197 write: IOPS=13, BW=52.3KiB/s (53.6kB/s)(5256KiB/100433msec); 0 zone resets slat (usec): min=31, max=17604, avg=276.50, stdev=1048.35 clat (msec): min=262, max=1571, avg=843.60, stdev=218.98 lat (msec): min=262, max=1573, avg=843.88, stdev=218.96 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 477], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1020], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1385], 99.50th=[ 1401], 99.90th=[ 1502], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 183, per=25.05%, avg=51.86, stdev=21.52, samples=198 iops : min= 1, max= 45, avg=12.48, stdev= 5.38, samples=198 lat (msec) : 50=0.23%, 100=0.43%, 250=3.24%, 500=41.72%, 750=19.84% lat (msec) : 1000=22.42%, 2000=12.11% cpu : usr=0.12%, sys=0.69%, ctx=4934, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148558: Mon Mar 2 11:59:33 2026 read: IOPS=12, BW=50.6KiB/s (51.8kB/s)(5060KiB/99961msec) slat (usec): min=52, max=53849, avg=13279.10, stdev=7267.59 clat (msec): min=28, max=1064, avg=390.15, stdev=110.62 lat (msec): min=29, max=1075, avg=403.44, stdev=111.86 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 230], 10.00th=[ 268], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 439], 80.00th=[ 468], 90.00th=[ 527], 95.00th=[ 575], | 99.00th=[ 659], 99.50th=[ 743], 99.90th=[ 1053], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 128, per=25.30%, avg=50.35, stdev=22.14, samples=197 iops : min= 1, max= 32, avg=12.11, stdev= 5.57, samples=197 write: IOPS=12, BW=51.8KiB/s (53.1kB/s)(5180KiB/99961msec); 0 zone resets slat (usec): min=33, max=13954, avg=256.87, stdev=836.29 clat (msec): min=176, max=1573, avg=839.63, stdev=225.22 lat (msec): min=176, max=1573, avg=839.88, stdev=225.21 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 489], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 718], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 911], | 70.00th=[ 969], 80.00th=[ 1011], 90.00th=[ 1116], 95.00th=[ 1217], | 99.00th=[ 1401], 99.50th=[ 1469], 99.90th=[ 1519], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8, max= 127, per=24.78%, avg=51.28, stdev=20.77, samples=197 iops : min= 2, max= 31, avg=12.34, stdev= 5.21, samples=197 lat (msec) : 50=0.27%, 100=0.23%, 250=3.48%, 500=41.17%, 750=21.91% lat (msec) : 1000=21.45%, 2000=11.48% cpu : usr=0.11%, sys=0.64%, ctx=4914, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=199KiB/s (204kB/s), 49.5KiB/s-50.6KiB/s (50.7kB/s-51.8kB/s), io=19.6MiB (20.5MB), run=99961-100592msec WRITE: bw=208KiB/s (213kB/s), 51.7KiB/s-52.3KiB/s (52.9kB/s-53.6kB/s), io=20.4MiB (21.4MB), run=99961-100592msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148576: Mon Mar 2 11:59:43 2026 read: IOPS=1, BW=65.1MiB/s (68.3MB/s)(40.0MiB/614msec) slat (nsec): min=36743k, max=36743k, avg=36742646.00, stdev= 0.00 clat (nsec): min=552742k, max=552742k, avg=552742074.00, stdev= 0.00 lat (nsec): min=589489k, max=589489k, avg=589489230.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.63%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.1MiB/s (68.3MB/s), 65.1MiB/s-65.1MiB/s (68.3MB/s-68.3MB/s), io=40.0MiB (41.9MB), run=614-614msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout