== sanity test 398c: run fio to test AIO ================= 16:47:27 (1776199647) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.581741 s, 72.1 MB/s osc.lustre-OST0000-osc-ffff9d2fc353b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=123104: Tue Apr 14 16:50:34 2026 write: IOPS=14, BW=58.4KiB/s (59.8kB/s)(10.0MiB/175287msec); 0 zone resets slat (usec): min=27, max=20746, avg=358.48, stdev=1183.11 clat (msec): min=140, max=2278, avg=1094.65, stdev=270.95 lat (msec): min=140, max=2278, avg=1095.01, stdev=270.95 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 676], 10.00th=[ 768], 20.00th=[ 877], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1452], 95.00th=[ 1552], | 99.00th=[ 1804], 99.50th=[ 1921], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 128, per=25.19%, avg=58.44, stdev=26.11, samples=344 iops : min= 1, max= 32, avg=14.07, stdev= 6.52, samples=344 lat (msec) : 250=0.47%, 500=0.43%, 750=7.85%, 1000=29.69%, 2000=61.33% lat (msec) : >=2000=0.23% cpu : usr=0.06%, sys=0.28%, ctx=1791, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123105: Tue Apr 14 16:50:34 2026 write: IOPS=14, BW=58.1KiB/s (59.5kB/s)(10.0MiB/176359msec); 0 zone resets slat (usec): min=19, max=18951, avg=386.53, stdev=1263.14 clat (msec): min=333, max=2026, avg=1100.92, stdev=260.05 lat (msec): min=334, max=2026, avg=1101.30, stdev=260.05 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 701], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 969], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1452], 95.00th=[ 1552], | 99.00th=[ 1737], 99.50th=[ 1838], 99.90th=[ 1972], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 127, per=24.83%, avg=57.61, stdev=25.21, samples=348 iops : min= 1, max= 31, avg=13.87, stdev= 6.29, samples=348 lat (msec) : 500=0.86%, 750=6.95%, 1000=28.09%, 2000=64.02%, >=2000=0.08% cpu : usr=0.05%, sys=0.26%, ctx=1573, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123106: Tue Apr 14 16:50:34 2026 write: IOPS=14, BW=58.1KiB/s (59.5kB/s)(10.0MiB/176316msec); 0 zone resets slat (usec): min=26, max=22129, avg=372.69, stdev=1345.00 clat (msec): min=389, max=2042, avg=1100.69, stdev=267.68 lat (msec): min=389, max=2042, avg=1101.06, stdev=267.67 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 667], 10.00th=[ 768], 20.00th=[ 885], | 30.00th=[ 969], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1552], | 99.00th=[ 1754], 99.50th=[ 1921], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 128, per=25.04%, avg=58.10, stdev=27.61, samples=346 iops : min= 1, max= 32, avg=13.98, stdev= 6.90, samples=346 lat (msec) : 500=0.55%, 750=8.40%, 1000=26.80%, 2000=64.22%, >=2000=0.04% cpu : usr=0.06%, sys=0.25%, ctx=1557, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123107: Tue Apr 14 16:50:34 2026 write: IOPS=14, BW=58.1KiB/s (59.5kB/s)(10.0MiB/176206msec); 0 zone resets slat (usec): min=26, max=21181, avg=374.15, stdev=1260.74 clat (msec): min=334, max=2175, avg=1099.68, stdev=263.93 lat (msec): min=334, max=2175, avg=1100.06, stdev=263.94 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 667], 10.00th=[ 776], 20.00th=[ 885], | 30.00th=[ 961], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1452], 95.00th=[ 1569], | 99.00th=[ 1787], 99.50th=[ 1854], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 128, per=25.29%, avg=58.68, stdev=28.57, samples=342 iops : min= 1, max= 32, avg=14.13, stdev= 7.15, samples=342 lat (msec) : 500=1.29%, 750=7.11%, 1000=27.85%, 2000=63.59%, >=2000=0.16% cpu : usr=0.04%, sys=0.29%, ctx=1737, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=232KiB/s (238kB/s), 58.1KiB/s-58.4KiB/s (59.5kB/s-59.8kB/s), io=40.0MiB (41.9MB), run=175287-176359msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=123141: Tue Apr 14 16:52:40 2026 read: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5048KiB/116121msec) slat (usec): min=2231, max=61353, avg=14005.64, stdev=8212.86 clat (msec): min=4, max=1048, avg=464.76, stdev=111.40 lat (msec): min=34, max=1056, avg=478.78, stdev=112.47 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 300], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 735], 99.50th=[ 844], 99.90th=[ 986], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 7, max= 136, per=25.32%, avg=43.55, stdev=20.31, samples=226 iops : min= 1, max= 34, avg=10.35, stdev= 5.11, samples=226 write: IOPS=11, BW=44.7KiB/s (45.8kB/s)(5192KiB/116121msec); 0 zone resets slat (usec): min=23, max=23748, avg=463.54, stdev=1465.05 clat (msec): min=254, max=1843, avg=963.64, stdev=255.89 lat (msec): min=254, max=1843, avg=964.11, stdev=255.84 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 558], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1435], | 99.00th=[ 1586], 99.50th=[ 1670], 99.90th=[ 1821], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 8, max= 103, per=24.52%, avg=43.89, stdev=19.95, samples=230 iops : min= 2, max= 25, avg=10.43, stdev= 4.99, samples=230 lat (msec) : 10=0.04%, 50=0.04%, 100=0.08%, 250=0.78%, 500=32.70% lat (msec) : 750=25.90%, 1000=18.91%, 2000=21.56% cpu : usr=0.12%, sys=0.74%, ctx=3269, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123142: Tue Apr 14 16:52:40 2026 read: IOPS=10, BW=42.8KiB/s (43.8kB/s)(4976KiB/116241msec) slat (usec): min=2534, max=54165, avg=14301.92, stdev=8035.98 clat (msec): min=31, max=964, avg=462.64, stdev=113.76 lat (msec): min=38, max=994, avg=476.95, stdev=114.66 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 481], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 751], 99.50th=[ 869], 99.90th=[ 953], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 120, per=24.60%, avg=42.31, stdev=21.53, samples=229 iops : min= 1, max= 30, avg=10.03, stdev= 5.40, samples=229 write: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5264KiB/116241msec); 0 zone resets slat (usec): min=27, max=16850, avg=466.49, stdev=1365.79 clat (msec): min=237, max=1790, avg=960.50, stdev=264.45 lat (msec): min=239, max=1791, avg=960.98, stdev=264.40 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1586], 99.50th=[ 1670], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 95, per=25.02%, avg=44.78, stdev=18.89, samples=230 iops : min= 1, max= 23, avg=10.65, stdev= 4.76, samples=230 lat (msec) : 50=0.12%, 100=0.16%, 250=0.82%, 500=33.24%, 750=25.04% lat (msec) : 1000=18.20%, 2000=22.42% cpu : usr=0.14%, sys=0.76%, ctx=3267, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123143: Tue Apr 14 16:52:40 2026 read: IOPS=10, BW=43.0KiB/s (44.1kB/s)(4984KiB/115795msec) slat (usec): min=54, max=67025, avg=14167.74, stdev=7726.44 clat (msec): min=14, max=1322, avg=464.29, stdev=118.86 lat (msec): min=16, max=1331, avg=478.47, stdev=119.83 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 288], 10.00th=[ 330], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 489], | 70.00th=[ 518], 80.00th=[ 558], 90.00th=[ 617], 95.00th=[ 651], | 99.00th=[ 751], 99.50th=[ 860], 99.90th=[ 961], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 112, per=24.90%, avg=42.83, stdev=20.53, samples=228 iops : min= 1, max= 28, avg=10.16, stdev= 5.18, samples=228 write: IOPS=11, BW=45.4KiB/s (46.5kB/s)(5256KiB/115795msec); 0 zone resets slat (usec): min=29, max=22442, avg=492.48, stdev=1445.85 clat (msec): min=123, max=1786, avg=954.93, stdev=270.10 lat (msec): min=124, max=1787, avg=955.42, stdev=270.09 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 793], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1603], 99.50th=[ 1670], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 103, per=25.14%, avg=45.00, stdev=18.29, samples=227 iops : min= 1, max= 25, avg=10.70, stdev= 4.59, samples=227 lat (msec) : 20=0.04%, 50=0.08%, 100=0.20%, 250=0.90%, 500=31.84% lat (msec) : 750=27.38%, 1000=18.05%, 2000=21.52% cpu : usr=0.13%, sys=0.69%, ctx=3401, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123144: Tue Apr 14 16:52:40 2026 read: IOPS=10, BW=43.7KiB/s (44.8kB/s)(5060KiB/115699msec) slat (usec): min=2636, max=66378, avg=14446.67, stdev=8137.78 clat (msec): min=13, max=1328, avg=462.75, stdev=126.80 lat (msec): min=17, max=1351, avg=477.20, stdev=128.04 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 284], 10.00th=[ 326], 20.00th=[ 368], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 869], 99.50th=[ 953], 99.90th=[ 1301], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 120, per=25.48%, avg=43.82, stdev=20.06, samples=226 iops : min= 1, max= 30, avg=10.40, stdev= 5.05, samples=226 write: IOPS=11, BW=44.8KiB/s (45.8kB/s)(5180KiB/115699msec); 0 zone resets slat (usec): min=28, max=17140, avg=426.05, stdev=1182.12 clat (msec): min=249, max=2106, avg=960.90, stdev=268.28 lat (msec): min=249, max=2106, avg=961.33, stdev=268.25 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1418], | 99.00th=[ 1670], 99.50th=[ 1955], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 95, per=25.03%, avg=44.81, stdev=18.74, samples=225 iops : min= 1, max= 23, avg=10.66, stdev= 4.70, samples=225 lat (msec) : 20=0.04%, 50=0.12%, 100=0.27%, 250=0.98%, 500=32.34% lat (msec) : 750=26.09%, 1000=17.81%, 2000=22.15%, >=2000=0.20% cpu : usr=0.11%, sys=0.72%, ctx=3349, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=173KiB/s (177kB/s), 42.8KiB/s-43.7KiB/s (43.8kB/s-44.8kB/s), io=19.6MiB (20.5MB), run=115699-116241msec WRITE: bw=180KiB/s (184kB/s), 44.7KiB/s-45.4KiB/s (45.8kB/s-46.5kB/s), io=20.4MiB (21.4MB), run=115699-116241msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=123166: Tue Apr 14 16:52:50 2026 read: IOPS=1, BW=78.7MiB/s (82.6MB/s)(40.0MiB/508msec) slat (nsec): min=29892k, max=29892k, avg=29892040.00, stdev= 0.00 clat (nsec): min=456692k, max=456692k, avg=456692070.00, stdev= 0.00 lat (nsec): min=486588k, max=486588k, avg=486588008.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.75%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=78.7MiB/s (82.6MB/s), 78.7MiB/s-78.7MiB/s (82.6MB/s-82.6MB/s), io=40.0MiB (41.9MB), run=508-508msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck