== sanity test 398c: run fio to test AIO ================= 23:01:34 (1790737294) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.555928 s, 75.4 MB/s osc.lustre-OST0000-osc-ffff9315d298d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147476: Tue Sep 29 23:04:03 2026 write: IOPS=18, BW=73.4KiB/s (75.1kB/s)(10.0MiB/139584msec); 0 zone resets slat (usec): min=28, max=13329, avg=170.77, stdev=554.31 clat (msec): min=141, max=2298, avg=871.28, stdev=282.92 lat (msec): min=142, max=2298, avg=871.45, stdev=282.93 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 489], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 961], 80.00th=[ 1062], 90.00th=[ 1234], 95.00th=[ 1401], | 99.00th=[ 1687], 99.50th=[ 2022], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 255, per=24.85%, avg=72.82, stdev=35.19, samples=277 iops : min= 1, max= 63, avg=17.75, stdev= 8.83, samples=277 lat (msec) : 250=2.50%, 500=2.66%, 750=28.48%, 1000=41.33%, 2000=24.45% lat (msec) : >=2000=0.59% cpu : usr=0.04%, sys=0.21%, ctx=1688, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147477: Tue Sep 29 23:04:03 2026 write: IOPS=18, BW=73.3KiB/s (75.1kB/s)(10.0MiB/139644msec); 0 zone resets slat (usec): min=23, max=18698, avg=165.54, stdev=606.04 clat (msec): min=99, max=2178, avg=872.24, stdev=284.87 lat (msec): min=99, max=2178, avg=872.41, stdev=284.87 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 502], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1062], 90.00th=[ 1217], 95.00th=[ 1401], | 99.00th=[ 1703], 99.50th=[ 2072], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 15, max= 255, per=24.94%, avg=73.07, stdev=32.42, samples=276 iops : min= 3, max= 63, avg=17.82, stdev= 8.12, samples=276 lat (msec) : 100=0.04%, 250=2.03%, 500=2.85%, 750=29.18%, 1000=40.82% lat (msec) : 2000=24.53%, >=2000=0.55% cpu : usr=0.04%, sys=0.21%, ctx=1798, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147478: Tue Sep 29 23:04:03 2026 write: IOPS=18, BW=73.3KiB/s (75.1kB/s)(10.0MiB/139640msec); 0 zone resets slat (usec): min=26, max=14671, avg=157.61, stdev=540.72 clat (msec): min=151, max=2323, avg=871.82, stdev=282.41 lat (msec): min=151, max=2323, avg=871.98, stdev=282.41 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 502], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1053], 90.00th=[ 1217], 95.00th=[ 1385], | 99.00th=[ 1703], 99.50th=[ 1871], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2333] bw ( KiB/s): min= 8, max= 255, per=24.85%, avg=72.81, stdev=33.06, samples=277 iops : min= 2, max= 63, avg=17.75, stdev= 8.27, samples=277 lat (msec) : 250=1.88%, 500=3.01%, 750=30.16%, 1000=39.53%, 2000=25.20% lat (msec) : >=2000=0.23% cpu : usr=0.05%, sys=0.18%, ctx=1775, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147479: Tue Sep 29 23:04:03 2026 write: IOPS=18, BW=73.4KiB/s (75.1kB/s)(10.0MiB/139592msec); 0 zone resets slat (usec): min=26, max=7444, avg=165.23, stdev=471.89 clat (msec): min=152, max=2322, avg=871.35, stdev=285.01 lat (msec): min=152, max=2322, avg=871.52, stdev=285.01 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 502], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 961], 80.00th=[ 1045], 90.00th=[ 1234], 95.00th=[ 1418], | 99.00th=[ 1687], 99.50th=[ 1905], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 255, per=24.85%, avg=72.81, stdev=31.96, samples=277 iops : min= 1, max= 63, avg=17.75, stdev= 8.00, samples=277 lat (msec) : 250=1.91%, 500=3.05%, 750=27.89%, 1000=42.15%, 2000=24.57% lat (msec) : >=2000=0.43% cpu : usr=0.04%, sys=0.22%, ctx=1856, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=293KiB/s (300kB/s), 73.3KiB/s-73.4KiB/s (75.1kB/s-75.1kB/s), io=40.0MiB (41.9MB), run=139584-139644msec 4% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147509: Tue Sep 29 23:05:47 2026 read: IOPS=13, BW=53.3KiB/s (54.6kB/s)(5048KiB/94756msec) slat (usec): min=2499, max=51404, avg=12314.89, stdev=6824.25 clat (msec): min=10, max=625, avg=363.46, stdev=89.48 lat (msec): min=23, max=660, avg=375.78, stdev=90.33 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 230], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 409], 80.00th=[ 439], 90.00th=[ 481], 95.00th=[ 518], | 99.00th=[ 567], 99.50th=[ 600], 99.90th=[ 617], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 7, max= 127, per=25.04%, avg=52.84, stdev=23.40, samples=188 iops : min= 1, max= 31, avg=12.75, stdev= 5.89, samples=188 write: IOPS=13, BW=54.8KiB/s (56.1kB/s)(5192KiB/94756msec); 0 zone resets slat (usec): min=31, max=18997, avg=215.25, stdev=722.94 clat (msec): min=214, max=1787, avg=799.91, stdev=216.84 lat (msec): min=214, max=1787, avg=800.12, stdev=216.87 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 468], 10.00th=[ 527], 20.00th=[ 609], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 978], 90.00th=[ 1053], 95.00th=[ 1116], | 99.00th=[ 1385], 99.50th=[ 1770], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 120, per=24.61%, avg=54.14, stdev=21.51, samples=188 iops : min= 1, max= 30, avg=13.09, stdev= 5.40, samples=188 lat (msec) : 20=0.08%, 50=0.20%, 100=0.12%, 250=3.67%, 500=45.35% lat (msec) : 750=20.47%, 1000=22.07%, 2000=8.05% cpu : usr=0.09%, sys=0.54%, ctx=4441, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147510: Tue Sep 29 23:05:47 2026 read: IOPS=13, BW=52.5KiB/s (53.8kB/s)(4976KiB/94790msec) slat (usec): min=2866, max=66632, avg=12286.09, stdev=7228.00 clat (msec): min=11, max=672, avg=358.09, stdev=84.93 lat (msec): min=21, max=696, avg=370.38, stdev=85.90 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 236], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 372], | 70.00th=[ 401], 80.00th=[ 426], 90.00th=[ 472], 95.00th=[ 510], | 99.00th=[ 558], 99.50th=[ 575], 99.90th=[ 642], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 7, max= 136, per=24.73%, avg=52.18, stdev=24.02, samples=187 iops : min= 1, max= 34, avg=12.59, stdev= 6.06, samples=187 write: IOPS=13, BW=55.5KiB/s (56.9kB/s)(5264KiB/94790msec); 0 zone resets slat (usec): min=33, max=11636, avg=220.44, stdev=672.67 clat (msec): min=264, max=1769, avg=799.91, stdev=218.04 lat (msec): min=265, max=1769, avg=800.13, stdev=218.04 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 477], 10.00th=[ 542], 20.00th=[ 609], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 969], 90.00th=[ 1070], 95.00th=[ 1150], | 99.00th=[ 1385], 99.50th=[ 1536], 99.90th=[ 1737], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 8, max= 111, per=24.94%, avg=54.87, stdev=19.25, samples=188 iops : min= 2, max= 27, avg=13.26, stdev= 4.82, samples=188 lat (msec) : 20=0.04%, 50=0.04%, 100=0.23%, 250=3.28%, 500=45.47% lat (msec) : 750=21.33%, 1000=21.17%, 2000=8.44% cpu : usr=0.08%, sys=0.62%, ctx=4880, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147511: Tue Sep 29 23:05:47 2026 read: IOPS=13, BW=52.6KiB/s (53.9kB/s)(4984KiB/94715msec) slat (usec): min=37, max=70038, avg=12111.98, stdev=7065.85 clat (msec): min=38, max=751, avg=362.28, stdev=87.66 lat (msec): min=47, max=764, avg=374.40, stdev=88.82 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 236], 10.00th=[ 264], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 439], 90.00th=[ 481], 95.00th=[ 514], | 99.00th=[ 558], 99.50th=[ 584], 99.90th=[ 617], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 7, max= 127, per=24.76%, avg=52.25, stdev=23.71, samples=188 iops : min= 1, max= 31, avg=12.61, stdev= 5.94, samples=188 write: IOPS=13, BW=55.5KiB/s (56.8kB/s)(5256KiB/94715msec); 0 zone resets slat (usec): min=30, max=14284, avg=232.76, stdev=713.67 clat (msec): min=167, max=1715, avg=797.30, stdev=212.93 lat (msec): min=168, max=1715, avg=797.53, stdev=212.91 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 468], 10.00th=[ 531], 20.00th=[ 609], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 978], 90.00th=[ 1062], 95.00th=[ 1133], | 99.00th=[ 1368], 99.50th=[ 1418], 99.90th=[ 1586], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 15, max= 104, per=24.85%, avg=54.67, stdev=18.40, samples=188 iops : min= 3, max= 26, avg=13.22, stdev= 4.67, samples=188 lat (msec) : 50=0.12%, 100=0.20%, 250=3.24%, 500=45.43%, 750=20.62% lat (msec) : 1000=22.15%, 2000=8.24% cpu : usr=0.10%, sys=0.60%, ctx=4197, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147512: Tue Sep 29 23:05:47 2026 read: IOPS=13, BW=53.5KiB/s (54.7kB/s)(5060KiB/94660msec) slat (usec): min=2727, max=69135, avg=12218.92, stdev=6961.83 clat (msec): min=13, max=891, avg=358.86, stdev=94.83 lat (msec): min=23, max=907, avg=371.08, stdev=95.90 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 224], 10.00th=[ 255], 20.00th=[ 292], | 30.00th=[ 313], 40.00th=[ 330], 50.00th=[ 359], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 435], 90.00th=[ 481], 95.00th=[ 514], | 99.00th=[ 575], 99.50th=[ 617], 99.90th=[ 810], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 136, per=25.38%, avg=53.54, stdev=23.55, samples=186 iops : min= 1, max= 34, avg=12.93, stdev= 5.95, samples=186 write: IOPS=13, BW=54.7KiB/s (56.0kB/s)(5180KiB/94660msec); 0 zone resets slat (usec): min=33, max=6685, avg=203.04, stdev=472.22 clat (msec): min=175, max=1761, avg=803.77, stdev=217.62 lat (msec): min=175, max=1761, avg=803.97, stdev=217.62 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 468], 10.00th=[ 535], 20.00th=[ 625], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 911], 80.00th=[ 969], 90.00th=[ 1083], 95.00th=[ 1183], | 99.00th=[ 1401], 99.50th=[ 1552], 99.90th=[ 1754], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 125, per=24.55%, avg=54.00, stdev=22.34, samples=188 iops : min= 1, max= 31, avg=13.05, stdev= 5.61, samples=188 lat (msec) : 20=0.04%, 50=0.27%, 100=0.47%, 250=3.59%, 500=45.35% lat (msec) : 750=21.02%, 1000=21.29%, 2000=7.97% cpu : usr=0.08%, sys=0.57%, ctx=4864, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=212KiB/s (217kB/s), 52.5KiB/s-53.5KiB/s (53.8kB/s-54.7kB/s), io=19.6MiB (20.5MB), run=94660-94790msec WRITE: bw=220KiB/s (226kB/s), 54.7KiB/s-55.5KiB/s (56.0kB/s-56.9kB/s), io=20.4MiB (21.4MB), run=94660-94790msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147530: Tue Sep 29 23:05:57 2026 read: IOPS=1, BW=76.8MiB/s (80.5MB/s)(40.0MiB/521msec) slat (nsec): min=38510k, max=38510k, avg=38509500.00, stdev= 0.00 clat (nsec): min=454470k, max=454470k, avg=454469916.00, stdev= 0.00 lat (nsec): min=492984k, max=492984k, avg=492984044.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.50%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=76.8MiB/s (80.5MB/s), 76.8MiB/s-76.8MiB/s (80.5MB/s-80.5MB/s), io=40.0MiB (41.9MB), run=521-521msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout