== sanity test 398c: run fio to test AIO ================= 17:56:48 (1776203808) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.61355 s, 68.4 MB/s osc.lustre-OST0000-osc-ffff9480e0c61800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=122814: Tue Apr 14 17:59:19 2026 write: IOPS=18, BW=72.9KiB/s (74.7kB/s)(10.0MiB/140384msec); 0 zone resets slat (usec): min=26, max=14337, avg=317.92, stdev=916.34 clat (msec): min=202, max=1819, avg=876.10, stdev=213.74 lat (msec): min=206, max=1820, avg=876.42, stdev=213.70 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 802], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1267], | 99.00th=[ 1502], 99.50th=[ 1552], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 128, per=24.91%, avg=72.49, stdev=32.69, samples=278 iops : min= 1, max= 32, avg=17.61, stdev= 8.23, samples=278 lat (msec) : 250=0.04%, 500=1.45%, 750=27.30%, 1000=46.99%, 2000=24.22% cpu : usr=0.07%, sys=0.32%, ctx=1712, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122815: Tue Apr 14 17:59:19 2026 write: IOPS=18, BW=72.9KiB/s (74.7kB/s)(10.0MiB/140431msec); 0 zone resets slat (usec): min=27, max=19714, avg=322.64, stdev=1001.88 clat (msec): min=302, max=1799, avg=876.25, stdev=210.01 lat (msec): min=302, max=1799, avg=876.57, stdev=210.02 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1485], 99.50th=[ 1586], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 128, per=24.99%, avg=72.73, stdev=28.74, samples=277 iops : min= 1, max= 32, avg=17.67, stdev= 7.22, samples=277 lat (msec) : 500=1.80%, 750=25.31%, 1000=50.20%, 2000=22.70% cpu : usr=0.07%, sys=0.35%, ctx=1932, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122816: Tue Apr 14 17:59:19 2026 write: IOPS=18, BW=73.1KiB/s (74.9kB/s)(10.0MiB/140050msec); 0 zone resets slat (usec): min=23, max=17176, avg=291.77, stdev=904.31 clat (msec): min=219, max=1853, avg=874.53, stdev=218.31 lat (msec): min=219, max=1853, avg=874.82, stdev=218.29 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 567], 10.00th=[ 609], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1519], 99.50th=[ 1620], 99.90th=[ 1804], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 128, per=24.89%, avg=72.43, stdev=29.88, samples=278 iops : min= 1, max= 32, avg=17.59, stdev= 7.48, samples=278 lat (msec) : 250=0.62%, 500=1.21%, 750=26.05%, 1000=48.28%, 2000=23.83% cpu : usr=0.07%, sys=0.31%, ctx=1684, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122817: Tue Apr 14 17:59:19 2026 write: IOPS=18, BW=72.9KiB/s (74.7kB/s)(10.0MiB/140387msec); 0 zone resets slat (usec): min=22, max=15464, avg=304.25, stdev=827.60 clat (msec): min=177, max=1758, avg=876.09, stdev=215.14 lat (msec): min=179, max=1758, avg=876.40, stdev=215.10 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 558], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 902], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1485], 99.50th=[ 1586], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 167, per=24.99%, avg=72.72, stdev=25.65, samples=277 iops : min= 1, max= 41, avg=17.67, stdev= 6.45, samples=277 lat (msec) : 250=0.04%, 500=1.56%, 750=26.80%, 1000=45.90%, 2000=25.70% cpu : usr=0.07%, sys=0.34%, ctx=1779, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=292KiB/s (299kB/s), 72.9KiB/s-73.1KiB/s (74.7kB/s-74.9kB/s), io=40.0MiB (41.9MB), run=140050-140431msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=122847: Tue Apr 14 18:01:05 2026 read: IOPS=12, BW=51.9KiB/s (53.2kB/s)(5048KiB/97201msec) slat (usec): min=2003, max=76473, avg=11860.38, stdev=6911.99 clat (msec): min=6, max=1217, avg=398.31, stdev=112.38 lat (msec): min=18, max=1234, avg=410.18, stdev=113.64 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 255], 10.00th=[ 275], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 355], 50.00th=[ 380], 60.00th=[ 409], | 70.00th=[ 443], 80.00th=[ 481], 90.00th=[ 542], 95.00th=[ 575], | 99.00th=[ 726], 99.50th=[ 785], 99.90th=[ 1083], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 151, per=25.32%, avg=51.92, stdev=24.56, samples=191 iops : min= 1, max= 37, avg=12.49, stdev= 6.16, samples=191 write: IOPS=13, BW=53.4KiB/s (54.7kB/s)(5192KiB/97201msec); 0 zone resets slat (usec): min=26, max=14451, avg=361.04, stdev=1009.71 clat (msec): min=197, max=1884, avg=796.97, stdev=254.58 lat (msec): min=198, max=1885, avg=797.34, stdev=254.56 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 447], 10.00th=[ 502], 20.00th=[ 575], | 30.00th=[ 651], 40.00th=[ 709], 50.00th=[ 776], 60.00th=[ 835], | 70.00th=[ 894], 80.00th=[ 978], 90.00th=[ 1099], 95.00th=[ 1234], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1871], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 120, per=24.86%, avg=53.20, stdev=22.81, samples=191 iops : min= 1, max= 30, avg=12.81, stdev= 5.70, samples=191 lat (msec) : 10=0.04%, 50=0.08%, 100=0.04%, 250=1.95%, 500=44.06% lat (msec) : 750=26.13%, 1000=18.40%, 2000=9.30% cpu : usr=0.12%, sys=0.84%, ctx=3261, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122848: Tue Apr 14 18:01:05 2026 read: IOPS=12, BW=51.0KiB/s (52.2kB/s)(4976KiB/97540msec) slat (usec): min=2122, max=59140, avg=11654.62, stdev=6900.84 clat (msec): min=58, max=905, avg=401.68, stdev=109.01 lat (msec): min=69, max=924, avg=413.35, stdev=110.61 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 253], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 388], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 592], | 99.00th=[ 726], 99.50th=[ 751], 99.90th=[ 835], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 136, per=24.64%, avg=50.51, stdev=22.58, samples=193 iops : min= 1, max= 34, avg=12.13, stdev= 5.66, samples=193 write: IOPS=13, BW=53.0KiB/s (55.3kB/s)(5264KiB/97540msec); 0 zone resets slat (usec): min=27, max=14816, avg=400.92, stdev=1155.06 clat (msec): min=289, max=1691, avg=793.48, stdev=247.51 lat (msec): min=289, max=1691, avg=793.88, stdev=247.55 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 460], 10.00th=[ 518], 20.00th=[ 575], | 30.00th=[ 634], 40.00th=[ 701], 50.00th=[ 768], 60.00th=[ 835], | 70.00th=[ 902], 80.00th=[ 978], 90.00th=[ 1099], 95.00th=[ 1200], | 99.00th=[ 1569], 99.50th=[ 1636], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 158, per=25.02%, avg=53.54, stdev=23.51, samples=193 iops : min= 1, max= 39, avg=12.90, stdev= 5.87, samples=193 lat (msec) : 100=0.16%, 250=2.11%, 500=42.27%, 750=27.93%, 1000=18.12% lat (msec) : 2000=9.41% cpu : usr=0.13%, sys=0.85%, ctx=3236, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122849: Tue Apr 14 18:01:05 2026 read: IOPS=12, BW=51.1KiB/s (52.4kB/s)(4984KiB/97465msec) slat (usec): min=52, max=80485, avg=12133.59, stdev=7260.51 clat (msec): min=46, max=1132, avg=394.86, stdev=105.83 lat (msec): min=67, max=1142, avg=407.01, stdev=107.23 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 251], 10.00th=[ 271], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 355], 50.00th=[ 380], 60.00th=[ 409], | 70.00th=[ 443], 80.00th=[ 481], 90.00th=[ 523], 95.00th=[ 575], | 99.00th=[ 684], 99.50th=[ 709], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 151, per=24.87%, avg=50.98, stdev=26.37, samples=192 iops : min= 1, max= 37, avg=12.26, stdev= 6.61, samples=192 write: IOPS=13, BW=53.9KiB/s (55.2kB/s)(5256KiB/97465msec); 0 zone resets slat (usec): min=28, max=18273, avg=430.06, stdev=1309.28 clat (msec): min=65, max=1683, avg=799.53, stdev=248.57 lat (msec): min=67, max=1683, avg=799.96, stdev=248.60 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 456], 10.00th=[ 506], 20.00th=[ 584], | 30.00th=[ 651], 40.00th=[ 718], 50.00th=[ 776], 60.00th=[ 844], | 70.00th=[ 911], 80.00th=[ 986], 90.00th=[ 1099], 95.00th=[ 1217], | 99.00th=[ 1569], 99.50th=[ 1620], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 104, per=24.88%, avg=53.24, stdev=21.50, samples=193 iops : min= 1, max= 26, avg=12.81, stdev= 5.35, samples=193 lat (msec) : 50=0.04%, 100=0.08%, 250=2.23%, 500=44.06%, 750=25.55% lat (msec) : 1000=18.28%, 2000=9.77% cpu : usr=0.15%, sys=0.82%, ctx=3422, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122850: Tue Apr 14 18:01:05 2026 read: IOPS=13, BW=52.7KiB/s (54.0kB/s)(5060KiB/95938msec) slat (usec): min=35, max=49697, avg=12102.66, stdev=6929.66 clat (msec): min=23, max=1276, avg=396.39, stdev=116.70 lat (msec): min=30, max=1293, avg=408.50, stdev=118.00 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 249], 10.00th=[ 271], 20.00th=[ 313], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 380], 60.00th=[ 409], | 70.00th=[ 443], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 575], | 99.00th=[ 693], 99.50th=[ 869], 99.90th=[ 1234], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 7, max= 112, per=25.74%, avg=52.77, stdev=23.03, samples=188 iops : min= 1, max= 28, avg=12.69, stdev= 5.77, samples=188 write: IOPS=13, BW=53.0KiB/s (55.3kB/s)(5180KiB/95938msec); 0 zone resets slat (usec): min=27, max=26899, avg=373.48, stdev=1184.83 clat (msec): min=66, max=1685, avg=784.89, stdev=249.31 lat (msec): min=66, max=1685, avg=785.26, stdev=249.34 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 435], 10.00th=[ 502], 20.00th=[ 567], | 30.00th=[ 634], 40.00th=[ 693], 50.00th=[ 768], 60.00th=[ 835], | 70.00th=[ 902], 80.00th=[ 978], 90.00th=[ 1083], 95.00th=[ 1217], | 99.00th=[ 1569], 99.50th=[ 1586], 99.90th=[ 1670], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 112, per=25.09%, avg=53.70, stdev=21.63, samples=189 iops : min= 1, max= 28, avg=12.93, stdev= 5.41, samples=189 lat (msec) : 50=0.27%, 100=0.20%, 250=2.38%, 500=43.48%, 750=26.88% lat (msec) : 1000=17.73%, 2000=9.06% cpu : usr=0.15%, sys=0.82%, ctx=3337, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=206KiB/s (211kB/s), 51.0KiB/s-52.7KiB/s (52.2kB/s-54.0kB/s), io=19.6MiB (20.5MB), run=95938-97540msec WRITE: bw=214KiB/s (219kB/s), 53.4KiB/s-53.0KiB/s (54.7kB/s-55.3kB/s), io=20.4MiB (21.4MB), run=95938-97540msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=122881: Tue Apr 14 18:01:17 2026 read: IOPS=1, BW=61.9MiB/s (64.9MB/s)(40.0MiB/646msec) slat (nsec): min=78622k, max=78622k, avg=78622166.00, stdev= 0.00 clat (nsec): min=542989k, max=542989k, avg=542989268.00, stdev= 0.00 lat (nsec): min=621615k, max=621615k, avg=621615497.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.95%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=61.9MiB/s (64.9MB/s), 61.9MiB/s-61.9MiB/s (64.9MB/s-64.9MB/s), io=40.0MiB (41.9MB), run=646-646msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck