== sanity test 398c: run fio to test AIO ================= 09:30:45 (1788874245) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.651747 s, 64.4 MB/s osc.lustre-OST0000-osc-ffff9f13eb181800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146340: Tue Sep 8 09:33:20 2026 write: IOPS=17, BW=71.7KiB/s (73.4kB/s)(10.0MiB/142836msec); 0 zone resets slat (usec): min=23, max=17723, avg=258.16, stdev=869.92 clat (msec): min=203, max=1844, avg=892.06, stdev=233.26 lat (msec): min=203, max=1845, avg=892.33, stdev=233.26 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 575], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1552], 99.50th=[ 1703], 99.90th=[ 1821], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 128, per=24.98%, avg=71.45, stdev=28.72, samples=282 iops : min= 1, max= 32, avg=17.38, stdev= 7.24, samples=282 lat (msec) : 250=0.62%, 500=1.13%, 750=28.95%, 1000=40.16%, 2000=29.14% cpu : usr=0.08%, sys=0.30%, ctx=2172, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146341: Tue Sep 8 09:33:20 2026 write: IOPS=17, BW=71.6KiB/s (73.3kB/s)(10.0MiB/143076msec); 0 zone resets slat (usec): min=23, max=17782, avg=239.41, stdev=766.27 clat (msec): min=269, max=1803, avg=892.93, stdev=232.46 lat (msec): min=269, max=1803, avg=893.18, stdev=232.45 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 575], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1603], 99.50th=[ 1720], 99.90th=[ 1787], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 128, per=25.46%, avg=72.82, stdev=31.30, samples=276 iops : min= 1, max= 32, avg=17.72, stdev= 7.82, samples=276 lat (msec) : 500=1.41%, 750=28.44%, 1000=39.77%, 2000=30.39% cpu : usr=0.05%, sys=0.29%, ctx=2208, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146342: Tue Sep 8 09:33:20 2026 write: IOPS=17, BW=71.6KiB/s (73.4kB/s)(10.0MiB/142919msec); 0 zone resets slat (usec): min=22, max=16107, avg=256.48, stdev=907.47 clat (msec): min=177, max=1763, avg=892.09, stdev=231.14 lat (msec): min=178, max=1763, avg=892.35, stdev=231.15 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1502], 99.50th=[ 1620], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 128, per=25.82%, avg=73.84, stdev=28.84, samples=273 iops : min= 1, max= 32, avg=17.97, stdev= 7.27, samples=273 lat (msec) : 250=0.62%, 500=1.45%, 750=27.58%, 1000=40.04%, 2000=30.31% cpu : usr=0.06%, sys=0.29%, ctx=1834, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146343: Tue Sep 8 09:33:20 2026 write: IOPS=17, BW=71.6KiB/s (73.3kB/s)(10.0MiB/143027msec); 0 zone resets slat (usec): min=23, max=18767, avg=247.87, stdev=852.87 clat (msec): min=209, max=1781, avg=892.71, stdev=230.35 lat (msec): min=209, max=1781, avg=892.96, stdev=230.36 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1183], 95.00th=[ 1301], | 99.00th=[ 1586], 99.50th=[ 1653], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 128, per=25.32%, avg=72.43, stdev=29.31, samples=278 iops : min= 1, max= 32, avg=17.62, stdev= 7.37, samples=278 lat (msec) : 250=0.27%, 500=0.94%, 750=28.67%, 1000=40.31%, 2000=29.80% cpu : usr=0.06%, sys=0.28%, ctx=1695, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=286KiB/s (293kB/s), 71.6KiB/s-71.7KiB/s (73.3kB/s-73.4kB/s), io=40.0MiB (41.9MB), run=142836-143076msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146374: Tue Sep 8 09:35:19 2026 read: IOPS=11, BW=46.8KiB/s (47.9kB/s)(5048KiB/107809msec) slat (usec): min=2370, max=64486, avg=14935.21, stdev=8081.15 clat (msec): min=23, max=1087, avg=413.66, stdev=127.02 lat (msec): min=31, max=1102, avg=428.61, stdev=128.80 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 239], 10.00th=[ 275], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 401], 60.00th=[ 426], | 70.00th=[ 468], 80.00th=[ 510], 90.00th=[ 575], 95.00th=[ 625], | 99.00th=[ 827], 99.50th=[ 911], 99.90th=[ 1003], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 119, per=25.19%, avg=46.86, stdev=22.80, samples=211 iops : min= 1, max= 29, avg=11.20, stdev= 5.70, samples=211 write: IOPS=12, BW=48.2KiB/s (49.3kB/s)(5192KiB/107809msec); 0 zone resets slat (usec): min=24, max=19596, avg=371.03, stdev=1274.27 clat (msec): min=312, max=1888, avg=908.42, stdev=271.33 lat (msec): min=313, max=1888, avg=908.79, stdev=271.36 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 506], 10.00th=[ 575], 20.00th=[ 676], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 885], 60.00th=[ 953], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1284], 95.00th=[ 1418], | 99.00th=[ 1653], 99.50th=[ 1687], 99.90th=[ 1871], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 136, per=24.97%, avg=48.18, stdev=22.50, samples=211 iops : min= 1, max= 34, avg=11.54, stdev= 5.67, samples=211 lat (msec) : 50=0.04%, 100=0.31%, 250=2.70%, 500=37.66%, 750=22.23% lat (msec) : 1000=20.00%, 2000=17.07% cpu : usr=0.12%, sys=0.80%, ctx=4878, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146375: Tue Sep 8 09:35:19 2026 read: IOPS=11, BW=46.3KiB/s (47.4kB/s)(4976KiB/107416msec) slat (usec): min=2880, max=62544, avg=14741.58, stdev=8297.17 clat (msec): min=22, max=991, avg=413.83, stdev=126.29 lat (msec): min=31, max=1023, avg=428.58, stdev=128.35 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 234], 10.00th=[ 264], 20.00th=[ 313], | 30.00th=[ 347], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 468], 80.00th=[ 510], 90.00th=[ 575], 95.00th=[ 642], | 99.00th=[ 785], 99.50th=[ 818], 99.90th=[ 944], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 112, per=24.92%, avg=46.34, stdev=21.79, samples=210 iops : min= 1, max= 28, avg=11.07, stdev= 5.47, samples=210 write: IOPS=12, BW=49.0KiB/s (50.2kB/s)(5264KiB/107416msec); 0 zone resets slat (usec): min=29, max=19444, avg=335.05, stdev=1053.70 clat (msec): min=327, max=1844, avg=898.07, stdev=263.31 lat (msec): min=327, max=1844, avg=898.41, stdev=263.32 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 498], 10.00th=[ 584], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 810], 50.00th=[ 877], 60.00th=[ 944], | 70.00th=[ 1011], 80.00th=[ 1083], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1620], 99.50th=[ 1687], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 112, per=25.35%, avg=48.93, stdev=22.51, samples=211 iops : min= 1, max= 28, avg=11.73, stdev= 5.66, samples=211 lat (msec) : 50=0.20%, 100=0.08%, 250=3.52%, 500=37.03%, 750=23.36% lat (msec) : 1000=19.10%, 2000=16.72% cpu : usr=0.11%, sys=0.81%, ctx=4746, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146376: Tue Sep 8 09:35:19 2026 read: IOPS=11, BW=46.2KiB/s (47.3kB/s)(4984KiB/107807msec) slat (usec): min=2566, max=64162, avg=15109.31, stdev=8728.58 clat (msec): min=22, max=1052, avg=420.82, stdev=131.77 lat (msec): min=28, max=1076, avg=435.94, stdev=133.87 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 239], 10.00th=[ 275], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 443], | 70.00th=[ 477], 80.00th=[ 523], 90.00th=[ 592], 95.00th=[ 651], | 99.00th=[ 818], 99.50th=[ 919], 99.90th=[ 1003], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 7, max= 119, per=25.05%, avg=46.60, stdev=21.59, samples=210 iops : min= 1, max= 29, avg=11.13, stdev= 5.41, samples=210 write: IOPS=12, BW=48.8KiB/s (49.9kB/s)(5256KiB/107807msec); 0 zone resets slat (usec): min=28, max=18453, avg=298.28, stdev=947.32 clat (msec): min=316, max=1937, avg=895.86, stdev=262.07 lat (msec): min=316, max=1937, avg=896.16, stdev=262.04 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 498], 10.00th=[ 575], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 810], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1653], 99.50th=[ 1737], 99.90th=[ 1888], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 112, per=25.11%, avg=48.46, stdev=23.43, samples=212 iops : min= 1, max= 28, avg=11.60, stdev= 5.87, samples=212 lat (msec) : 50=0.16%, 100=0.27%, 250=2.70%, 500=35.90%, 750=25.04% lat (msec) : 1000=20.00%, 2000=15.94% cpu : usr=0.15%, sys=0.82%, ctx=4528, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146377: Tue Sep 8 09:35:19 2026 read: IOPS=11, BW=47.2KiB/s (48.4kB/s)(5060KiB/107116msec) slat (usec): min=50, max=58654, avg=14614.58, stdev=8207.76 clat (msec): min=22, max=1050, avg=420.54, stdev=137.84 lat (msec): min=26, max=1078, avg=435.16, stdev=139.53 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 243], 10.00th=[ 271], 20.00th=[ 313], | 30.00th=[ 347], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 472], 80.00th=[ 510], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 927], 99.50th=[ 978], 99.90th=[ 1045], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 7, max= 119, per=25.23%, avg=46.93, stdev=22.20, samples=211 iops : min= 1, max= 29, avg=11.22, stdev= 5.56, samples=211 write: IOPS=12, BW=48.4KiB/s (49.5kB/s)(5180KiB/107116msec); 0 zone resets slat (usec): min=23, max=23926, avg=360.01, stdev=1486.19 clat (msec): min=171, max=1837, avg=896.96, stdev=268.68 lat (msec): min=171, max=1837, avg=897.33, stdev=268.69 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 489], 10.00th=[ 575], 20.00th=[ 667], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 877], 60.00th=[ 944], | 70.00th=[ 1020], 80.00th=[ 1099], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1670], 99.50th=[ 1720], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 112, per=24.81%, avg=47.88, stdev=21.32, samples=212 iops : min= 1, max= 28, avg=11.46, stdev= 5.37, samples=212 lat (msec) : 50=0.20%, 100=0.23%, 250=2.93%, 500=37.66%, 750=22.62% lat (msec) : 1000=19.65%, 2000=16.72% cpu : usr=0.12%, sys=0.80%, ctx=4665, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=186KiB/s (191kB/s), 46.2KiB/s-47.2KiB/s (47.3kB/s-48.4kB/s), io=19.6MiB (20.5MB), run=107116-107809msec WRITE: bw=194KiB/s (198kB/s), 48.2KiB/s-49.0KiB/s (49.3kB/s-50.2kB/s), io=20.4MiB (21.4MB), run=107116-107809msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146396: Tue Sep 8 09:35:29 2026 read: IOPS=2, BW=80.0MiB/s (84.9MB/s)(40.0MiB/494msec) slat (nsec): min=50467k, max=50467k, avg=50467424.00, stdev= 0.00 clat (nsec): min=429674k, max=429674k, avg=429674020.00, stdev= 0.00 lat (nsec): min=480147k, max=480147k, avg=480146822.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.84%, ctx=43, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=80.0MiB/s (84.9MB/s), 80.0MiB/s-80.0MiB/s (84.9MB/s-84.9MB/s), io=40.0MiB (41.9MB), run=494-494msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout