== sanity test 398c: run fio to test AIO ================= 15:15:50 (1776194150) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.524876 s, 79.9 MB/s osc.lustre-OST0000-osc-ffff9daa45309800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=125453: Tue Apr 14 15:18:24 2026 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144375msec); 0 zone resets slat (usec): min=20, max=20193, avg=223.62, stdev=968.42 clat (msec): min=79, max=2294, avg=901.70, stdev=273.40 lat (msec): min=80, max=2295, avg=901.92, stdev=273.44 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 510], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1217], 95.00th=[ 1351], | 99.00th=[ 1821], 99.50th=[ 2005], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 231, per=24.98%, avg=70.70, stdev=36.26, samples=285 iops : min= 1, max= 57, avg=17.24, stdev= 9.09, samples=285 lat (msec) : 100=0.04%, 250=0.51%, 500=4.41%, 750=20.51%, 1000=45.23% lat (msec) : 2000=28.79%, >=2000=0.51% cpu : usr=0.06%, sys=0.28%, ctx=1713, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125454: Tue Apr 14 15:18:24 2026 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144506msec); 0 zone resets slat (usec): min=20, max=21996, avg=203.61, stdev=988.54 clat (msec): min=104, max=2173, avg=902.17, stdev=269.54 lat (msec): min=104, max=2173, avg=902.37, stdev=269.56 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 510], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1385], | 99.00th=[ 1804], 99.50th=[ 1989], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8, max= 231, per=25.09%, avg=71.01, stdev=30.67, samples=284 iops : min= 2, max= 57, avg=17.31, stdev= 7.68, samples=284 lat (msec) : 250=0.31%, 500=4.49%, 750=20.59%, 1000=45.35%, 2000=28.79% lat (msec) : >=2000=0.47% cpu : usr=0.05%, sys=0.23%, ctx=1511, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125455: Tue Apr 14 15:18:24 2026 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144507msec); 0 zone resets slat (usec): min=21, max=21993, avg=214.79, stdev=952.68 clat (msec): min=60, max=2226, avg=902.16, stdev=272.30 lat (msec): min=61, max=2226, avg=902.38, stdev=272.32 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 506], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 776], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 919], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1888], 99.50th=[ 2056], 99.90th=[ 2140], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 8, max= 239, per=24.89%, avg=70.43, stdev=32.37, samples=286 iops : min= 2, max= 59, avg=17.17, stdev= 8.10, samples=286 lat (msec) : 100=0.08%, 250=0.39%, 500=4.34%, 750=21.60%, 1000=44.80% lat (msec) : 2000=28.12%, >=2000=0.66% cpu : usr=0.04%, sys=0.26%, ctx=1618, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125456: Tue Apr 14 15:18:24 2026 write: IOPS=17, BW=70.8KiB/s (72.5kB/s)(10.0MiB/144612msec); 0 zone resets slat (usec): min=20, max=15143, avg=205.69, stdev=895.47 clat (msec): min=161, max=2231, avg=902.65, stdev=266.19 lat (msec): min=161, max=2231, avg=902.86, stdev=266.20 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 502], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 995], 80.00th=[ 1070], 90.00th=[ 1217], 95.00th=[ 1351], | 99.00th=[ 1787], 99.50th=[ 1989], 99.90th=[ 2165], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 8, max= 255, per=25.36%, avg=71.77, stdev=33.54, samples=280 iops : min= 2, max= 63, avg=17.50, stdev= 8.40, samples=280 lat (msec) : 250=0.04%, 500=4.80%, 750=20.23%, 1000=45.31%, 2000=29.22% lat (msec) : >=2000=0.39% cpu : usr=0.04%, sys=0.24%, ctx=1380, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=283KiB/s (290kB/s), 70.8KiB/s-70.9KiB/s (72.5kB/s-72.6kB/s), io=40.0MiB (41.9MB), run=144375-144612msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=125488: Tue Apr 14 15:20:11 2026 read: IOPS=12, BW=51.0KiB/s (52.3kB/s)(5048KiB/98899msec) slat (usec): min=2168, max=50649, avg=12330.52, stdev=8118.79 clat (msec): min=35, max=992, avg=400.33, stdev=114.54 lat (msec): min=43, max=1003, avg=412.66, stdev=115.99 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 249], 10.00th=[ 275], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 443], 80.00th=[ 493], 90.00th=[ 567], 95.00th=[ 625], | 99.00th=[ 735], 99.50th=[ 751], 99.90th=[ 785], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 135, per=24.96%, avg=50.41, stdev=23.52, samples=196 iops : min= 1, max= 33, avg=12.11, stdev= 5.93, samples=196 write: IOPS=13, BW=52.5KiB/s (53.8kB/s)(5192KiB/98899msec); 0 zone resets slat (usec): min=24, max=23068, avg=304.21, stdev=1316.18 clat (msec): min=248, max=1969, avg=815.52, stdev=272.06 lat (msec): min=248, max=1970, avg=815.82, stdev=272.04 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 409], 10.00th=[ 477], 20.00th=[ 592], | 30.00th=[ 667], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 852], | 70.00th=[ 927], 80.00th=[ 1020], 90.00th=[ 1167], 95.00th=[ 1318], | 99.00th=[ 1636], 99.50th=[ 1770], 99.90th=[ 1921], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 143, per=24.77%, avg=52.27, stdev=22.71, samples=195 iops : min= 1, max= 35, avg=12.57, stdev= 5.72, samples=195 lat (msec) : 50=0.04%, 100=0.12%, 250=2.42%, 500=43.79%, 750=24.84% lat (msec) : 1000=18.24%, 2000=10.55% cpu : usr=0.10%, sys=0.71%, ctx=3368, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125489: Tue Apr 14 15:20:11 2026 read: IOPS=12, BW=50.3KiB/s (51.5kB/s)(4976KiB/98944msec) slat (usec): min=2165, max=79639, avg=11886.35, stdev=8245.77 clat (msec): min=12, max=1028, avg=394.45, stdev=114.55 lat (msec): min=26, max=1043, avg=406.34, stdev=116.21 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 245], 10.00th=[ 275], 20.00th=[ 309], | 30.00th=[ 330], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 405], | 70.00th=[ 430], 80.00th=[ 472], 90.00th=[ 550], 95.00th=[ 617], | 99.00th=[ 743], 99.50th=[ 760], 99.90th=[ 1003], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 143, per=24.72%, avg=49.93, stdev=24.92, samples=194 iops : min= 1, max= 35, avg=12.00, stdev= 6.27, samples=194 write: IOPS=13, BW=53.2KiB/s (54.5kB/s)(5264KiB/98944msec); 0 zone resets slat (usec): min=27, max=16631, avg=258.13, stdev=1043.35 clat (msec): min=159, max=1892, avg=816.91, stdev=272.95 lat (msec): min=159, max=1892, avg=817.18, stdev=272.93 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 405], 10.00th=[ 477], 20.00th=[ 584], | 30.00th=[ 676], 40.00th=[ 735], 50.00th=[ 793], 60.00th=[ 860], | 70.00th=[ 936], 80.00th=[ 1028], 90.00th=[ 1167], 95.00th=[ 1284], | 99.00th=[ 1620], 99.50th=[ 1703], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 15, max= 119, per=24.99%, avg=52.73, stdev=20.52, samples=195 iops : min= 3, max= 29, avg=12.70, stdev= 5.17, samples=195 lat (msec) : 20=0.08%, 50=0.16%, 100=0.04%, 250=2.62%, 500=44.34% lat (msec) : 750=23.55%, 1000=17.89%, 2000=11.33% cpu : usr=0.10%, sys=0.65%, ctx=3119, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125490: Tue Apr 14 15:20:11 2026 read: IOPS=12, BW=50.4KiB/s (51.6kB/s)(4984KiB/98837msec) slat (usec): min=1869, max=55964, avg=11998.52, stdev=8054.23 clat (msec): min=9, max=1032, avg=400.87, stdev=118.55 lat (msec): min=13, max=1038, avg=412.88, stdev=119.76 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 253], 10.00th=[ 279], 20.00th=[ 313], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 405], | 70.00th=[ 439], 80.00th=[ 485], 90.00th=[ 558], 95.00th=[ 634], | 99.00th=[ 743], 99.50th=[ 776], 99.90th=[ 1003], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 111, per=24.86%, avg=50.22, stdev=21.96, samples=195 iops : min= 1, max= 27, avg=12.06, stdev= 5.50, samples=195 write: IOPS=13, BW=53.2KiB/s (54.5kB/s)(5256KiB/98837msec); 0 zone resets slat (usec): min=24, max=24519, avg=256.53, stdev=1139.40 clat (msec): min=242, max=1958, avg=809.67, stdev=267.23 lat (msec): min=242, max=1958, avg=809.93, stdev=267.20 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 422], 10.00th=[ 489], 20.00th=[ 584], | 30.00th=[ 651], 40.00th=[ 726], 50.00th=[ 785], 60.00th=[ 844], | 70.00th=[ 911], 80.00th=[ 1011], 90.00th=[ 1150], 95.00th=[ 1301], | 99.00th=[ 1636], 99.50th=[ 1703], 99.90th=[ 1871], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 119, per=24.99%, avg=52.72, stdev=22.60, samples=195 iops : min= 1, max= 29, avg=12.69, stdev= 5.69, samples=195 lat (msec) : 10=0.04%, 100=0.16%, 250=2.11%, 500=43.24%, 750=25.35% lat (msec) : 1000=18.44%, 2000=10.66% cpu : usr=0.11%, sys=0.68%, ctx=3345, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125491: Tue Apr 14 15:20:11 2026 read: IOPS=12, BW=51.4KiB/s (52.7kB/s)(5060KiB/98374msec) slat (usec): min=64, max=97760, avg=12233.20, stdev=8482.89 clat (msec): min=18, max=1000, avg=397.69, stdev=118.29 lat (msec): min=19, max=1031, avg=409.93, stdev=119.97 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 245], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 330], 40.00th=[ 355], 50.00th=[ 380], 60.00th=[ 409], | 70.00th=[ 439], 80.00th=[ 485], 90.00th=[ 558], 95.00th=[ 625], | 99.00th=[ 718], 99.50th=[ 776], 99.90th=[ 877], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 119, per=25.72%, avg=51.95, stdev=22.93, samples=191 iops : min= 1, max= 29, avg=12.49, stdev= 5.78, samples=191 write: IOPS=13, BW=52.7KiB/s (53.9kB/s)(5180KiB/98374msec); 0 zone resets slat (usec): min=26, max=25503, avg=274.11, stdev=1284.50 clat (msec): min=97, max=1921, avg=813.72, stdev=268.68 lat (msec): min=97, max=1921, avg=814.00, stdev=268.69 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 405], 10.00th=[ 481], 20.00th=[ 584], | 30.00th=[ 676], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 852], | 70.00th=[ 936], 80.00th=[ 1020], 90.00th=[ 1150], 95.00th=[ 1301], | 99.00th=[ 1620], 99.50th=[ 1703], 99.90th=[ 1905], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 135, per=24.79%, avg=52.31, stdev=22.76, samples=194 iops : min= 1, max= 33, avg=12.59, stdev= 5.72, samples=194 lat (msec) : 20=0.16%, 50=0.20%, 100=0.12%, 250=2.89%, 500=43.01% lat (msec) : 750=23.55%, 1000=19.06%, 2000=11.02% cpu : usr=0.12%, sys=0.71%, ctx=3383, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=203KiB/s (208kB/s), 50.3KiB/s-51.4KiB/s (51.5kB/s-52.7kB/s), io=19.6MiB (20.5MB), run=98374-98944msec WRITE: bw=211KiB/s (216kB/s), 52.5KiB/s-53.2KiB/s (53.8kB/s-54.5kB/s), io=20.4MiB (21.4MB), run=98374-98944msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=125518: Tue Apr 14 15:20:21 2026 read: IOPS=1, BW=75.5MiB/s (79.1MB/s)(40.0MiB/530msec) slat (nsec): min=38997k, max=38997k, avg=38997064.00, stdev= 0.00 clat (nsec): min=457859k, max=457859k, avg=457859069.00, stdev= 0.00 lat (nsec): min=496869k, max=496869k, avg=496868895.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.84%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=75.5MiB/s (79.1MB/s), 75.5MiB/s-75.5MiB/s (79.1MB/s-79.1MB/s), io=40.0MiB (41.9MB), run=530-530msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck