== sanity test 398c: run fio to test AIO ================= 07:47:24 (1787140044) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.533546 s, 78.6 MB/s osc.lustre-OST0000-osc-ffff8a89868c2000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145576: Wed Aug 19 07:50:26 2026 write: IOPS=14, BW=59.8KiB/s (61.2kB/s)(10.0MiB/171369msec); 0 zone resets slat (usec): min=51, max=17926, avg=421.96, stdev=1240.70 clat (msec): min=119, max=2276, avg=1070.11, stdev=270.97 lat (msec): min=120, max=2276, avg=1070.53, stdev=270.96 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 676], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1070], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1536], | 99.00th=[ 1888], 99.50th=[ 2072], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 140, per=24.98%, avg=59.46, stdev=27.75, samples=338 iops : min= 1, max= 35, avg=14.35, stdev= 6.96, samples=338 lat (msec) : 250=0.62%, 500=0.62%, 750=8.83%, 1000=30.16%, 2000=59.06% lat (msec) : >=2000=0.70% cpu : usr=0.05%, sys=0.35%, ctx=2189, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145577: Wed Aug 19 07:50:26 2026 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(10.0MiB/171603msec); 0 zone resets slat (usec): min=51, max=21049, avg=351.09, stdev=1077.22 clat (msec): min=182, max=2273, avg=1071.03, stdev=266.99 lat (msec): min=182, max=2273, avg=1071.38, stdev=266.93 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 684], 10.00th=[ 768], 20.00th=[ 860], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1062], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1267], 90.00th=[ 1385], 95.00th=[ 1536], | 99.00th=[ 1888], 99.50th=[ 2072], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 136, per=25.06%, avg=59.65, stdev=27.60, samples=337 iops : min= 1, max= 34, avg=14.40, stdev= 6.92, samples=337 lat (msec) : 250=0.08%, 500=1.09%, 750=7.38%, 1000=31.33%, 2000=59.45% lat (msec) : >=2000=0.66% cpu : usr=0.05%, sys=0.28%, ctx=1765, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145578: Wed Aug 19 07:50:26 2026 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(10.0MiB/171654msec); 0 zone resets slat (usec): min=51, max=22568, avg=316.50, stdev=862.30 clat (msec): min=295, max=2273, avg=1071.31, stdev=264.94 lat (msec): min=299, max=2273, avg=1071.63, stdev=264.92 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 667], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 927], 40.00th=[ 995], 50.00th=[ 1062], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1267], 90.00th=[ 1368], 95.00th=[ 1536], | 99.00th=[ 1871], 99.50th=[ 2022], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 128, per=24.77%, avg=58.94, stdev=27.49, samples=341 iops : min= 1, max= 32, avg=14.23, stdev= 6.92, samples=341 lat (msec) : 500=0.86%, 750=8.98%, 1000=31.52%, 2000=58.01%, >=2000=0.62% cpu : usr=0.04%, sys=0.29%, ctx=1715, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145579: Wed Aug 19 07:50:26 2026 write: IOPS=14, BW=59.6KiB/s (61.1kB/s)(10.0MiB/171719msec); 0 zone resets slat (usec): min=47, max=22520, avg=423.06, stdev=1338.71 clat (msec): min=348, max=2275, avg=1071.70, stdev=260.42 lat (msec): min=349, max=2275, avg=1072.12, stdev=260.38 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 667], 10.00th=[ 760], 20.00th=[ 869], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1062], 60.00th=[ 1116], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1385], 95.00th=[ 1536], | 99.00th=[ 1905], 99.50th=[ 2005], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 128, per=24.98%, avg=59.44, stdev=25.87, samples=338 iops : min= 1, max= 32, avg=14.35, stdev= 6.51, samples=338 lat (msec) : 500=0.70%, 750=8.44%, 1000=30.74%, 2000=59.61%, >=2000=0.51% cpu : usr=0.05%, sys=0.32%, ctx=2142, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=239KiB/s (244kB/s), 59.6KiB/s-59.8KiB/s (61.1kB/s-61.2kB/s), io=40.0MiB (41.9MB), run=171369-171719msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145614: Wed Aug 19 07:52:37 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(5048KiB/120995msec) slat (usec): min=2727, max=54351, avg=15834.77, stdev=7833.03 clat (msec): min=11, max=879, avg=476.51, stdev=107.42 lat (msec): min=29, max=897, avg=492.36, stdev=108.08 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 326], 10.00th=[ 359], 20.00th=[ 388], | 30.00th=[ 422], 40.00th=[ 447], 50.00th=[ 472], 60.00th=[ 498], | 70.00th=[ 523], 80.00th=[ 558], 90.00th=[ 617], 95.00th=[ 659], | 99.00th=[ 768], 99.50th=[ 776], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 95, per=25.32%, avg=41.78, stdev=18.10, samples=236 iops : min= 1, max= 23, avg= 9.92, stdev= 4.56, samples=236 write: IOPS=10, BW=42.9KiB/s (43.9kB/s)(5192KiB/120995msec); 0 zone resets slat (usec): min=57, max=9855, avg=381.06, stdev=797.22 clat (msec): min=435, max=1742, avg=1009.07, stdev=237.03 lat (msec): min=436, max=1742, avg=1009.46, stdev=237.01 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 617], 10.00th=[ 701], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 944], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1586], 99.50th=[ 1636], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 104, per=24.80%, avg=42.66, stdev=17.95, samples=238 iops : min= 1, max= 26, avg=10.15, stdev= 4.53, samples=238 lat (msec) : 20=0.04%, 50=0.04%, 100=0.31%, 250=0.23%, 500=29.96% lat (msec) : 750=25.43%, 1000=19.26%, 2000=24.73% cpu : usr=0.08%, sys=0.67%, ctx=4483, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145615: Wed Aug 19 07:52:37 2026 read: IOPS=10, BW=41.1KiB/s (42.1kB/s)(4976KiB/121073msec) slat (usec): min=3172, max=80971, avg=16014.90, stdev=8570.99 clat (msec): min=19, max=874, avg=466.41, stdev=106.78 lat (msec): min=28, max=892, avg=482.43, stdev=107.64 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 317], 10.00th=[ 347], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 489], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 592], 95.00th=[ 651], | 99.00th=[ 760], 99.50th=[ 785], 99.90th=[ 852], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 111, per=24.70%, avg=40.75, stdev=20.25, samples=238 iops : min= 1, max= 27, avg= 9.67, stdev= 5.09, samples=238 write: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5264KiB/121073msec); 0 zone resets slat (usec): min=59, max=14179, avg=420.34, stdev=1020.99 clat (msec): min=445, max=1785, avg=1012.22, stdev=238.26 lat (msec): min=446, max=1785, avg=1012.64, stdev=238.21 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 625], 10.00th=[ 684], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1569], 99.50th=[ 1603], 99.90th=[ 1770], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 119, per=25.17%, avg=43.29, stdev=18.10, samples=238 iops : min= 1, max= 29, avg=10.30, stdev= 4.55, samples=238 lat (msec) : 20=0.04%, 50=0.16%, 100=0.16%, 250=0.23%, 500=31.56% lat (msec) : 750=23.01%, 1000=19.22%, 2000=25.62% cpu : usr=0.11%, sys=0.68%, ctx=4384, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145616: Wed Aug 19 07:52:37 2026 read: IOPS=10, BW=41.2KiB/s (42.2kB/s)(4984KiB/120997msec) slat (usec): min=3621, max=60068, avg=15863.80, stdev=8188.26 clat (msec): min=11, max=893, avg=471.96, stdev=112.59 lat (msec): min=18, max=913, avg=487.83, stdev=113.31 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 309], 10.00th=[ 347], 20.00th=[ 384], | 30.00th=[ 418], 40.00th=[ 447], 50.00th=[ 468], 60.00th=[ 493], | 70.00th=[ 527], 80.00th=[ 558], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 751], 99.50th=[ 785], 99.90th=[ 877], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 96, per=25.08%, avg=41.38, stdev=18.85, samples=236 iops : min= 1, max= 24, avg= 9.83, stdev= 4.74, samples=236 write: IOPS=10, BW=43.4KiB/s (44.5kB/s)(5256KiB/120997msec); 0 zone resets slat (usec): min=53, max=14140, avg=416.31, stdev=1030.25 clat (msec): min=323, max=1716, avg=1007.79, stdev=236.92 lat (msec): min=323, max=1716, avg=1008.21, stdev=236.91 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 995], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1586], 99.50th=[ 1636], 99.90th=[ 1670], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 103, per=24.94%, avg=42.89, stdev=19.02, samples=239 iops : min= 1, max= 25, avg=10.20, stdev= 4.75, samples=239 lat (msec) : 20=0.12%, 50=0.12%, 100=0.27%, 250=0.35%, 500=29.80% lat (msec) : 750=24.57%, 1000=19.57%, 2000=25.20% cpu : usr=0.08%, sys=0.64%, ctx=4083, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145617: Wed Aug 19 07:52:37 2026 read: IOPS=10, BW=42.1KiB/s (43.1kB/s)(5060KiB/120083msec) slat (usec): min=101, max=66631, avg=15172.37, stdev=7651.65 clat (msec): min=9, max=1235, avg=468.96, stdev=116.92 lat (msec): min=13, max=1244, avg=484.14, stdev=117.95 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 309], 10.00th=[ 342], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 443], 50.00th=[ 468], 60.00th=[ 493], | 70.00th=[ 518], 80.00th=[ 558], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 760], 99.50th=[ 776], 99.90th=[ 1200], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 103, per=25.47%, avg=42.03, stdev=16.58, samples=236 iops : min= 1, max= 25, avg= 9.99, stdev= 4.16, samples=236 write: IOPS=10, BW=43.1KiB/s (44.2kB/s)(5180KiB/120083msec); 0 zone resets slat (usec): min=55, max=22199, avg=419.36, stdev=1049.39 clat (msec): min=160, max=2261, avg=1009.24, stdev=257.74 lat (msec): min=161, max=2261, avg=1009.67, stdev=257.71 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1687], 99.50th=[ 2106], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 103, per=24.74%, avg=42.56, stdev=19.62, samples=237 iops : min= 1, max= 25, avg=10.12, stdev= 4.92, samples=237 lat (msec) : 10=0.04%, 20=0.04%, 50=0.27%, 100=0.20%, 250=0.51% lat (msec) : 500=30.51%, 750=24.34%, 1000=20.31%, 2000=23.52%, >=2000=0.27% cpu : usr=0.10%, sys=0.67%, ctx=4393, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=166KiB/s (170kB/s), 41.1KiB/s-42.1KiB/s (42.1kB/s-43.1kB/s), io=19.6MiB (20.5MB), run=120083-121073msec WRITE: bw=173KiB/s (177kB/s), 42.9KiB/s-43.5KiB/s (43.9kB/s-44.5kB/s), io=20.4MiB (21.4MB), run=120083-121073msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145638: Wed Aug 19 07:52:47 2026 read: IOPS=1, BW=76.3MiB/s (80.0MB/s)(40.0MiB/524msec) slat (nsec): min=35034k, max=35034k, avg=35034438.00, stdev= 0.00 clat (nsec): min=464098k, max=464098k, avg=464097989.00, stdev= 0.00 lat (nsec): min=499138k, max=499138k, avg=499137520.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.49%, ctx=42, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=76.3MiB/s (80.0MB/s), 76.3MiB/s-76.3MiB/s (80.0MB/s-80.0MB/s), io=40.0MiB (41.9MB), run=524-524msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout