== sanity test 398c: run fio to test AIO ================= 09:32:23 (1788874343) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.491979 s, 85.3 MB/s osc.lustre-OST0000-osc-ffff8f280a6c2000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144723: Tue Sep 8 09:35:57 2026 write: IOPS=12, BW=49.0KiB/s (51.2kB/s)(10.0MiB/204818msec); 0 zone resets slat (usec): min=40, max=21764, avg=389.81, stdev=1206.12 clat (msec): min=323, max=3222, avg=1278.80, stdev=379.84 lat (msec): min=330, max=3222, avg=1279.20, stdev=379.80 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 726], 10.00th=[ 852], 20.00th=[ 961], | 30.00th=[ 1036], 40.00th=[ 1133], 50.00th=[ 1234], 60.00th=[ 1351], | 70.00th=[ 1452], 80.00th=[ 1586], 90.00th=[ 1737], 95.00th=[ 1888], | 99.00th=[ 2467], 99.50th=[ 2668], 99.90th=[ 3138], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 7, max= 128, per=26.27%, avg=52.28, stdev=28.48, samples=384 iops : min= 1, max= 32, avg=12.55, stdev= 7.14, samples=384 lat (msec) : 500=0.08%, 750=5.43%, 1000=19.57%, 2000=72.03%, >=2000=2.89% cpu : usr=0.04%, sys=0.26%, ctx=1767, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144724: Tue Sep 8 09:35:57 2026 write: IOPS=12, BW=50.0KiB/s (51.2kB/s)(10.0MiB/204696msec); 0 zone resets slat (usec): min=50, max=18057, avg=403.09, stdev=1221.46 clat (msec): min=366, max=3134, avg=1277.75, stdev=378.08 lat (msec): min=367, max=3134, avg=1278.16, stdev=378.08 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 718], 10.00th=[ 835], 20.00th=[ 961], | 30.00th=[ 1045], 40.00th=[ 1150], 50.00th=[ 1250], 60.00th=[ 1351], | 70.00th=[ 1469], 80.00th=[ 1603], 90.00th=[ 1737], 95.00th=[ 1871], | 99.00th=[ 2433], 99.50th=[ 2567], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 128, per=25.21%, avg=50.17, stdev=26.50, samples=400 iops : min= 1, max= 32, avg=12.03, stdev= 6.63, samples=400 lat (msec) : 500=0.78%, 750=5.04%, 1000=19.45%, 2000=71.68%, >=2000=3.05% cpu : usr=0.03%, sys=0.26%, ctx=2028, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144725: Tue Sep 8 09:35:57 2026 write: IOPS=12, BW=50.1KiB/s (51.3kB/s)(10.0MiB/204471msec); 0 zone resets slat (usec): min=46, max=22583, avg=412.87, stdev=1272.74 clat (msec): min=175, max=3031, avg=1276.97, stdev=384.68 lat (msec): min=175, max=3031, avg=1277.39, stdev=384.68 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 709], 10.00th=[ 827], 20.00th=[ 961], | 30.00th=[ 1053], 40.00th=[ 1150], 50.00th=[ 1250], 60.00th=[ 1351], | 70.00th=[ 1469], 80.00th=[ 1586], 90.00th=[ 1754], 95.00th=[ 1871], | 99.00th=[ 2366], 99.50th=[ 2601], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 135, per=25.18%, avg=50.12, stdev=24.12, samples=400 iops : min= 1, max= 33, avg=12.01, stdev= 6.03, samples=400 lat (msec) : 250=0.39%, 500=0.59%, 750=5.43%, 1000=18.05%, 2000=72.23% lat (msec) : >=2000=3.32% cpu : usr=0.05%, sys=0.28%, ctx=1962, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144726: Tue Sep 8 09:35:57 2026 write: IOPS=12, BW=50.0KiB/s (51.2kB/s)(10.0MiB/204700msec); 0 zone resets slat (usec): min=48, max=22778, avg=394.28, stdev=1265.13 clat (msec): min=175, max=2976, avg=1278.11, stdev=373.91 lat (msec): min=177, max=2976, avg=1278.51, stdev=373.90 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 718], 10.00th=[ 844], 20.00th=[ 969], | 30.00th=[ 1053], 40.00th=[ 1150], 50.00th=[ 1250], 60.00th=[ 1351], | 70.00th=[ 1469], 80.00th=[ 1586], 90.00th=[ 1737], 95.00th=[ 1854], | 99.00th=[ 2366], 99.50th=[ 2769], 99.90th=[ 2937], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 143, per=25.27%, avg=50.30, stdev=26.01, samples=399 iops : min= 1, max= 35, avg=12.06, stdev= 6.51, samples=399 lat (msec) : 250=0.04%, 500=0.62%, 750=5.20%, 1000=18.24%, 2000=73.20% lat (msec) : >=2000=2.70% cpu : usr=0.05%, sys=0.28%, ctx=1927, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=200KiB/s (205kB/s), 49.0KiB/s-50.1KiB/s (51.2kB/s-51.3kB/s), io=40.0MiB (41.9MB), run=204471-204818msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144765: Tue Sep 8 09:38:10 2026 read: IOPS=10, BW=41.1KiB/s (42.1kB/s)(5048KiB/122860msec) slat (usec): min=3090, max=71675, avg=15929.30, stdev=8532.18 clat (msec): min=16, max=1383, avg=493.58, stdev=148.43 lat (msec): min=27, max=1402, avg=509.51, stdev=149.87 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 305], 10.00th=[ 342], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 439], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 542], 80.00th=[ 600], 90.00th=[ 684], 95.00th=[ 776], | 99.00th=[ 919], 99.50th=[ 986], 99.90th=[ 1368], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 7, max= 111, per=25.31%, avg=41.26, stdev=20.88, samples=239 iops : min= 1, max= 27, avg= 9.77, stdev= 5.27, samples=239 write: IOPS=10, BW=42.3KiB/s (43.3kB/s)(5192KiB/122860msec); 0 zone resets slat (usec): min=52, max=24723, avg=496.90, stdev=1344.97 clat (msec): min=235, max=2090, avg=1015.56, stdev=270.59 lat (msec): min=235, max=2090, avg=1016.06, stdev=270.59 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 617], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1770], 99.50th=[ 1972], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 102, per=24.99%, avg=42.23, stdev=18.92, samples=240 iops : min= 1, max= 25, avg=10.01, stdev= 4.76, samples=240 lat (msec) : 20=0.08%, 50=0.12%, 100=0.12%, 250=0.47%, 500=29.57% lat (msec) : 750=23.01%, 1000=22.07%, 2000=24.34%, >=2000=0.23% cpu : usr=0.09%, sys=0.66%, ctx=4346, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144766: Tue Sep 8 09:38:10 2026 read: IOPS=10, BW=40.5KiB/s (41.4kB/s)(4976KiB/123015msec) slat (usec): min=3165, max=70204, avg=16038.30, stdev=8586.28 clat (msec): min=27, max=1072, avg=488.08, stdev=140.41 lat (msec): min=34, max=1092, avg=504.13, stdev=142.03 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 300], 10.00th=[ 342], 20.00th=[ 384], | 30.00th=[ 414], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 498], | 70.00th=[ 535], 80.00th=[ 584], 90.00th=[ 676], 95.00th=[ 768], | 99.00th=[ 902], 99.50th=[ 953], 99.90th=[ 1045], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7, max= 96, per=24.85%, avg=40.51, stdev=19.86, samples=240 iops : min= 1, max= 24, avg= 9.58, stdev= 5.01, samples=240 write: IOPS=10, BW=42.8KiB/s (43.8kB/s)(5264KiB/123015msec); 0 zone resets slat (usec): min=52, max=13295, avg=445.47, stdev=1031.92 clat (msec): min=353, max=2109, avg=1016.52, stdev=272.31 lat (msec): min=354, max=2109, avg=1016.97, stdev=272.28 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1754], 99.50th=[ 1972], 99.90th=[ 2089], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 111, per=25.12%, avg=42.45, stdev=19.30, samples=242 iops : min= 1, max= 27, avg=10.07, stdev= 4.84, samples=242 lat (msec) : 50=0.08%, 100=0.12%, 250=0.55%, 500=30.66%, 750=21.99% lat (msec) : 1000=21.21%, 2000=25.23%, >=2000=0.16% cpu : usr=0.09%, sys=0.66%, ctx=4278, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144767: Tue Sep 8 09:38:10 2026 read: IOPS=10, BW=40.6KiB/s (41.6kB/s)(4984KiB/122677msec) slat (usec): min=92, max=74035, avg=16201.55, stdev=8780.70 clat (msec): min=18, max=1203, avg=492.76, stdev=146.49 lat (msec): min=26, max=1222, avg=508.97, stdev=148.29 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 300], 10.00th=[ 338], 20.00th=[ 384], | 30.00th=[ 418], 40.00th=[ 443], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 676], 95.00th=[ 776], | 99.00th=[ 894], 99.50th=[ 1011], 99.90th=[ 1183], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 111, per=24.84%, avg=40.49, stdev=21.16, samples=241 iops : min= 1, max= 27, avg= 9.58, stdev= 5.33, samples=241 write: IOPS=10, BW=42.8KiB/s (43.9kB/s)(5256KiB/122677msec); 0 zone resets slat (usec): min=41, max=22125, avg=447.22, stdev=1083.60 clat (msec): min=98, max=2084, avg=1009.65, stdev=273.39 lat (msec): min=99, max=2085, avg=1010.10, stdev=273.39 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1770], 99.50th=[ 1854], 99.90th=[ 2039], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 95, per=25.22%, avg=42.62, stdev=17.76, samples=240 iops : min= 1, max= 23, avg=10.10, stdev= 4.49, samples=240 lat (msec) : 20=0.04%, 50=0.16%, 100=0.27%, 250=1.05%, 500=28.59% lat (msec) : 750=22.73%, 1000=22.23%, 2000=24.84%, >=2000=0.08% cpu : usr=0.10%, sys=0.68%, ctx=4279, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144768: Tue Sep 8 09:38:10 2026 read: IOPS=10, BW=41.1KiB/s (42.1kB/s)(5060KiB/123006msec) slat (usec): min=2700, max=62992, avg=16095.25, stdev=8516.27 clat (msec): min=12, max=1156, avg=492.39, stdev=147.34 lat (msec): min=28, max=1179, avg=508.49, stdev=148.92 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 296], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 443], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 785], | 99.00th=[ 911], 99.50th=[ 961], 99.90th=[ 1062], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 111, per=25.30%, avg=41.24, stdev=18.31, samples=240 iops : min= 1, max= 27, avg= 9.76, stdev= 4.63, samples=240 write: IOPS=10, BW=42.1KiB/s (43.1kB/s)(5180KiB/123006msec); 0 zone resets slat (usec): min=54, max=27544, avg=514.63, stdev=1507.84 clat (msec): min=317, max=2105, avg=1019.57, stdev=273.75 lat (msec): min=317, max=2105, avg=1020.08, stdev=273.65 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1770], 99.50th=[ 1955], 99.90th=[ 2072], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 88, per=24.80%, avg=41.90, stdev=18.50, samples=241 iops : min= 1, max= 22, avg= 9.93, stdev= 4.66, samples=241 lat (msec) : 20=0.04%, 50=0.12%, 100=0.20%, 250=0.62%, 500=29.26% lat (msec) : 750=22.70%, 1000=22.38%, 2000=24.57%, >=2000=0.12% cpu : usr=0.11%, sys=0.63%, ctx=4344, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=163KiB/s (167kB/s), 40.5KiB/s-41.1KiB/s (41.4kB/s-42.1kB/s), io=19.6MiB (20.5MB), run=122677-123015msec WRITE: bw=170KiB/s (174kB/s), 42.1KiB/s-42.8KiB/s (43.1kB/s-43.9kB/s), io=20.4MiB (21.4MB), run=122677-123015msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144793: Tue Sep 8 09:38:20 2026 read: IOPS=1, BW=60.9MiB/s (63.8MB/s)(40.0MiB/657msec) slat (nsec): min=35862k, max=35862k, avg=35861842.00, stdev= 0.00 clat (nsec): min=575300k, max=575300k, avg=575300304.00, stdev= 0.00 lat (nsec): min=611167k, max=611167k, avg=611167354.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.29%, ctx=10, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=60.9MiB/s (63.8MB/s), 60.9MiB/s-60.9MiB/s (63.8MB/s-63.8MB/s), io=40.0MiB (41.9MB), run=657-657msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout