== sanity test 398c: run fio to test AIO ================= 10:09:56 (1783951796) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.683177 s, 61.4 MB/s osc.lustre-OST0000-osc-ffff9549675f7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146355: Mon Jul 13 10:13:09 2026 write: IOPS=13, BW=55.0KiB/s (57.3kB/s)(10.0MiB/182888msec); 0 zone resets slat (usec): min=40, max=18944, avg=365.14, stdev=1059.97 clat (msec): min=182, max=2871, avg=1141.61, stdev=333.74 lat (msec): min=184, max=2871, avg=1141.99, stdev=333.70 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 701], 10.00th=[ 785], 20.00th=[ 877], | 30.00th=[ 961], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1250], 80.00th=[ 1351], 90.00th=[ 1502], 95.00th=[ 1737], | 99.00th=[ 2299], 99.50th=[ 2467], 99.90th=[ 2836], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 136, per=25.08%, avg=55.93, stdev=27.76, samples=360 iops : min= 1, max= 34, avg=13.55, stdev= 6.97, samples=360 lat (msec) : 250=0.04%, 500=1.33%, 750=6.02%, 1000=28.28%, 2000=61.84% lat (msec) : >=2000=2.50% cpu : usr=0.04%, sys=0.26%, ctx=2012, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146356: Mon Jul 13 10:13:09 2026 write: IOPS=14, BW=56.1KiB/s (57.4kB/s)(10.0MiB/182565msec); 0 zone resets slat (usec): min=41, max=13551, avg=334.91, stdev=869.75 clat (msec): min=168, max=2752, avg=1140.05, stdev=335.01 lat (msec): min=168, max=2752, avg=1140.38, stdev=335.00 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 701], 10.00th=[ 793], 20.00th=[ 894], | 30.00th=[ 969], 40.00th=[ 1045], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1502], 95.00th=[ 1754], | 99.00th=[ 2366], 99.50th=[ 2567], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 136, per=26.26%, avg=58.55, stdev=27.69, samples=344 iops : min= 1, max= 34, avg=14.20, stdev= 6.94, samples=344 lat (msec) : 250=0.04%, 500=1.52%, 750=5.78%, 1000=26.88%, 2000=62.93% lat (msec) : >=2000=2.85% cpu : usr=0.04%, sys=0.28%, ctx=2131, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146357: Mon Jul 13 10:13:09 2026 write: IOPS=13, BW=55.0KiB/s (57.3kB/s)(10.0MiB/182887msec); 0 zone resets slat (usec): min=38, max=16109, avg=320.68, stdev=937.04 clat (msec): min=348, max=2752, avg=1141.47, stdev=334.67 lat (msec): min=349, max=2752, avg=1141.80, stdev=334.64 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 701], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 978], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1250], 80.00th=[ 1351], 90.00th=[ 1502], 95.00th=[ 1754], | 99.00th=[ 2366], 99.50th=[ 2601], 99.90th=[ 2702], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 128, per=25.58%, avg=57.04, stdev=28.52, samples=353 iops : min= 1, max= 32, avg=13.83, stdev= 7.15, samples=353 lat (msec) : 500=1.25%, 750=6.48%, 1000=25.90%, 2000=63.91%, >=2000=2.46% cpu : usr=0.04%, sys=0.26%, ctx=1865, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146358: Mon Jul 13 10:13:09 2026 write: IOPS=13, BW=55.0KiB/s (57.3kB/s)(10.0MiB/182930msec); 0 zone resets slat (usec): min=38, max=13547, avg=346.25, stdev=1003.94 clat (msec): min=250, max=2628, avg=1141.84, stdev=331.19 lat (msec): min=250, max=2628, avg=1142.19, stdev=331.17 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 701], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 969], 40.00th=[ 1036], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1368], 90.00th=[ 1552], 95.00th=[ 1770], | 99.00th=[ 2123], 99.50th=[ 2366], 99.90th=[ 2567], 99.95th=[ 2601], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 136, per=26.73%, avg=59.60, stdev=33.24, samples=338 iops : min= 1, max= 34, avg=14.47, stdev= 8.33, samples=338 lat (msec) : 500=1.48%, 750=5.98%, 1000=26.95%, 2000=63.09%, >=2000=2.50% cpu : usr=0.04%, sys=0.24%, ctx=1790, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=224KiB/s (229kB/s), 55.0KiB/s-56.1KiB/s (57.3kB/s-57.4kB/s), io=40.0MiB (41.9MB), run=182565-182930msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146396: Mon Jul 13 10:15:19 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(5048KiB/120736msec) slat (usec): min=2780, max=69289, avg=16070.16, stdev=8548.06 clat (msec): min=11, max=983, avg=471.99, stdev=121.21 lat (msec): min=16, max=1003, avg=488.08, stdev=122.55 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 493], | 70.00th=[ 531], 80.00th=[ 567], 90.00th=[ 634], 95.00th=[ 684], | 99.00th=[ 785], 99.50th=[ 844], 99.90th=[ 936], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 104, per=24.99%, avg=41.48, stdev=19.76, samples=238 iops : min= 1, max= 26, avg= 9.83, stdev= 4.98, samples=238 write: IOPS=10, BW=43.0KiB/s (44.0kB/s)(5192KiB/120736msec); 0 zone resets slat (usec): min=45, max=20940, avg=471.98, stdev=1273.75 clat (msec): min=431, max=1777, avg=1010.11, stdev=231.58 lat (msec): min=431, max=1777, avg=1010.58, stdev=231.61 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 659], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 995], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1653], 99.50th=[ 1737], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 108, per=24.78%, avg=42.87, stdev=18.30, samples=236 iops : min= 1, max= 27, avg=10.18, stdev= 4.61, samples=236 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=0.51%, 500=30.31% lat (msec) : 750=23.91%, 1000=20.27%, 2000=24.73% cpu : usr=0.09%, sys=0.69%, ctx=4503, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146397: Mon Jul 13 10:15:19 2026 read: IOPS=10, BW=41.2KiB/s (42.2kB/s)(4976KiB/120686msec) slat (usec): min=3105, max=57876, avg=15865.11, stdev=8495.22 clat (msec): min=27, max=1015, avg=469.59, stdev=126.70 lat (msec): min=39, max=1019, avg=485.47, stdev=127.83 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 296], 10.00th=[ 326], 20.00th=[ 368], | 30.00th=[ 397], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 493], | 70.00th=[ 531], 80.00th=[ 567], 90.00th=[ 634], 95.00th=[ 693], | 99.00th=[ 793], 99.50th=[ 844], 99.90th=[ 986], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 119, per=24.81%, avg=41.18, stdev=19.53, samples=236 iops : min= 1, max= 29, avg= 9.75, stdev= 4.93, samples=236 write: IOPS=10, BW=43.6KiB/s (44.7kB/s)(5264KiB/120686msec); 0 zone resets slat (usec): min=55, max=15084, avg=398.82, stdev=924.97 clat (msec): min=449, max=1954, avg=1005.00, stdev=235.95 lat (msec): min=449, max=1954, avg=1005.40, stdev=235.94 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 995], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1653], 99.50th=[ 1754], 99.90th=[ 1921], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 104, per=25.17%, avg=43.54, stdev=20.87, samples=236 iops : min= 1, max= 26, avg=10.34, stdev= 5.25, samples=236 lat (msec) : 50=0.12%, 100=0.23%, 250=0.94%, 500=29.61%, 750=23.32% lat (msec) : 1000=20.27%, 2000=25.51% cpu : usr=0.11%, sys=0.66%, ctx=4470, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146398: Mon Jul 13 10:15:19 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(4984KiB/120682msec) slat (usec): min=3741, max=67132, avg=16053.50, stdev=8473.44 clat (msec): min=21, max=985, avg=464.28, stdev=117.11 lat (msec): min=36, max=1011, avg=480.35, stdev=118.61 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 296], 10.00th=[ 326], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 676], | 99.00th=[ 760], 99.50th=[ 785], 99.90th=[ 844], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 104, per=24.60%, avg=40.84, stdev=19.92, samples=239 iops : min= 1, max= 26, avg= 9.67, stdev= 5.00, samples=239 write: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5256KiB/120682msec); 0 zone resets slat (usec): min=57, max=17517, avg=428.58, stdev=1169.39 clat (msec): min=422, max=1971, avg=1010.78, stdev=234.85 lat (msec): min=422, max=1971, avg=1011.22, stdev=234.86 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 651], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1053], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1653], 99.50th=[ 1770], 99.90th=[ 1921], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 96, per=24.93%, avg=43.13, stdev=19.74, samples=237 iops : min= 1, max= 24, avg=10.24, stdev= 4.98, samples=237 lat (msec) : 50=0.08%, 100=0.12%, 250=0.51%, 500=31.84%, 750=22.19% lat (msec) : 1000=19.61%, 2000=25.66% cpu : usr=0.09%, sys=0.68%, ctx=4510, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146399: Mon Jul 13 10:15:19 2026 read: IOPS=10, BW=42.2KiB/s (43.2kB/s)(5060KiB/119884msec) slat (usec): min=63, max=52543, avg=15764.70, stdev=8534.38 clat (msec): min=12, max=1261, avg=466.71, stdev=132.54 lat (msec): min=13, max=1290, avg=482.49, stdev=134.22 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 288], 10.00th=[ 326], 20.00th=[ 368], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 634], 95.00th=[ 693], | 99.00th=[ 810], 99.50th=[ 827], 99.90th=[ 1217], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 7, max= 103, per=25.28%, avg=41.97, stdev=21.09, samples=235 iops : min= 1, max= 25, avg= 9.94, stdev= 5.31, samples=235 write: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5180KiB/119884msec); 0 zone resets slat (usec): min=44, max=14038, avg=472.46, stdev=1124.60 clat (msec): min=133, max=1973, avg=1008.38, stdev=234.49 lat (msec): min=133, max=1974, avg=1008.85, stdev=234.51 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 659], 10.00th=[ 735], 20.00th=[ 818], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 995], 60.00th=[ 1053], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1653], 99.50th=[ 1687], 99.90th=[ 1888], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 111, per=24.66%, avg=42.66, stdev=18.67, samples=237 iops : min= 1, max= 27, avg=10.12, stdev= 4.70, samples=237 lat (msec) : 20=0.08%, 50=0.20%, 100=0.23%, 250=1.41%, 500=30.31% lat (msec) : 750=22.07%, 1000=20.66%, 2000=25.04% cpu : usr=0.09%, sys=0.68%, ctx=4448, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=166KiB/s (170kB/s), 41.2KiB/s-42.2KiB/s (42.2kB/s-43.2kB/s), io=19.6MiB (20.5MB), run=119884-120736msec WRITE: bw=173KiB/s (177kB/s), 43.0KiB/s-43.6KiB/s (44.0kB/s-44.7kB/s), io=20.4MiB (21.4MB), run=119884-120736msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146421: Mon Jul 13 10:15:29 2026 read: IOPS=1, BW=60.1MiB/s (62.0MB/s)(40.0MiB/666msec) slat (nsec): min=61171k, max=61171k, avg=61171437.00, stdev= 0.00 clat (nsec): min=578615k, max=578615k, avg=578615174.00, stdev= 0.00 lat (nsec): min=639791k, max=639791k, avg=639791058.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.35%, ctx=9, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=60.1MiB/s (62.0MB/s), 60.1MiB/s-60.1MiB/s (62.0MB/s-62.0MB/s), io=40.0MiB (41.9MB), run=666-666msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout