== sanity test 398c: run fio to test AIO ================= 09:43:22 (1787579002) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.534888 s, 78.4 MB/s osc.lustre-OST0000-osc-ffff8c00f912b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145875: Mon Aug 24 09:46:41 2026 write: IOPS=13, BW=54.6KiB/s (55.9kB/s)(10.0MiB/187687msec); 0 zone resets slat (usec): min=23, max=39402, avg=258.03, stdev=1383.68 clat (msec): min=139, max=2493, avg=1171.92, stdev=366.32 lat (msec): min=139, max=2493, avg=1172.18, stdev=366.35 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 642], 10.00th=[ 743], 20.00th=[ 885], | 30.00th=[ 961], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1469], 90.00th=[ 1687], 95.00th=[ 1838], | 99.00th=[ 2165], 99.50th=[ 2299], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=26.66%, avg=58.12, stdev=34.88, samples=346 iops : min= 1, max= 32, avg=14.06, stdev= 8.74, samples=346 lat (msec) : 250=0.31%, 500=1.37%, 750=8.55%, 1000=24.10%, 2000=62.81% lat (msec) : >=2000=2.85% cpu : usr=0.04%, sys=0.19%, ctx=1728, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145876: Mon Aug 24 09:46:41 2026 write: IOPS=13, BW=54.6KiB/s (55.9kB/s)(10.0MiB/187487msec); 0 zone resets slat (usec): min=22, max=24554, avg=219.07, stdev=1026.90 clat (msec): min=115, max=2585, avg=1171.11, stdev=374.56 lat (msec): min=115, max=2585, avg=1171.33, stdev=374.59 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 642], 10.00th=[ 735], 20.00th=[ 877], | 30.00th=[ 961], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1334], 80.00th=[ 1502], 90.00th=[ 1670], 95.00th=[ 1838], | 99.00th=[ 2198], 99.50th=[ 2333], 99.90th=[ 2500], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 128, per=26.58%, avg=57.95, stdev=36.40, samples=347 iops : min= 1, max= 32, avg=14.01, stdev= 9.13, samples=347 lat (msec) : 250=0.62%, 500=0.86%, 750=10.12%, 1000=24.26%, 2000=61.25% lat (msec) : >=2000=2.89% cpu : usr=0.04%, sys=0.18%, ctx=1541, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145877: Mon Aug 24 09:46:41 2026 write: IOPS=13, BW=54.6KiB/s (55.9kB/s)(10.0MiB/187595msec); 0 zone resets slat (usec): min=28, max=43259, avg=246.14, stdev=1363.20 clat (msec): min=176, max=2494, avg=1171.40, stdev=368.76 lat (msec): min=177, max=2494, avg=1171.65, stdev=368.79 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 659], 10.00th=[ 760], 20.00th=[ 869], | 30.00th=[ 961], 40.00th=[ 1045], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1318], 80.00th=[ 1485], 90.00th=[ 1670], 95.00th=[ 1838], | 99.00th=[ 2165], 99.50th=[ 2232], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 255, per=25.76%, avg=56.15, stdev=30.90, samples=358 iops : min= 1, max= 63, avg=13.56, stdev= 7.74, samples=358 lat (msec) : 250=0.62%, 500=1.25%, 750=7.62%, 1000=25.59%, 2000=62.11% lat (msec) : >=2000=2.81% cpu : usr=0.04%, sys=0.18%, ctx=1646, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145878: Mon Aug 24 09:46:41 2026 write: IOPS=13, BW=54.6KiB/s (55.9kB/s)(10.0MiB/187593msec); 0 zone resets slat (usec): min=27, max=22533, avg=222.81, stdev=1040.20 clat (msec): min=422, max=2579, avg=1171.26, stdev=367.76 lat (msec): min=422, max=2579, avg=1171.49, stdev=367.78 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 651], 10.00th=[ 751], 20.00th=[ 885], | 30.00th=[ 961], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1318], 80.00th=[ 1485], 90.00th=[ 1687], 95.00th=[ 1871], | 99.00th=[ 2198], 99.50th=[ 2366], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 128, per=26.35%, avg=57.45, stdev=29.35, samples=350 iops : min= 1, max= 32, avg=13.88, stdev= 7.37, samples=350 lat (msec) : 500=1.95%, 750=8.01%, 1000=26.02%, 2000=60.86%, >=2000=3.16% cpu : usr=0.05%, sys=0.18%, ctx=1561, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=218KiB/s (223kB/s), 54.6KiB/s-54.6KiB/s (55.9kB/s-55.9kB/s), io=40.0MiB (41.9MB), run=187487-187687msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145916: Mon Aug 24 09:48:48 2026 read: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5048KiB/115856msec) slat (usec): min=34, max=66069, avg=14975.96, stdev=8654.95 clat (msec): min=9, max=1199, avg=454.72, stdev=138.80 lat (msec): min=26, max=1215, avg=469.70, stdev=140.49 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 502], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 911], 99.50th=[ 944], 99.90th=[ 1150], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 119, per=25.35%, avg=43.61, stdev=22.27, samples=227 iops : min= 1, max= 29, avg=10.44, stdev= 5.62, samples=227 write: IOPS=11, BW=44.8KiB/s (45.9kB/s)(5192KiB/115856msec); 0 zone resets slat (usec): min=28, max=18138, avg=268.25, stdev=869.32 clat (msec): min=175, max=2176, avg=970.14, stdev=287.68 lat (msec): min=179, max=2176, avg=970.42, stdev=287.65 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 558], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 802], 40.00th=[ 877], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1787], 99.50th=[ 1921], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 103, per=24.97%, avg=44.70, stdev=20.87, samples=227 iops : min= 1, max= 25, avg=10.70, stdev= 5.20, samples=227 lat (msec) : 10=0.04%, 50=0.04%, 100=0.31%, 250=1.48%, 500=33.52% lat (msec) : 750=24.34%, 1000=18.01%, 2000=22.07%, >=2000=0.20% cpu : usr=0.08%, sys=0.60%, ctx=4571, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145917: Mon Aug 24 09:48:48 2026 read: IOPS=10, BW=42.8KiB/s (43.8kB/s)(4976KiB/116263msec) slat (usec): min=2777, max=72338, avg=15040.52, stdev=8689.17 clat (msec): min=16, max=943, avg=454.30, stdev=129.47 lat (msec): min=21, max=965, avg=469.35, stdev=131.05 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 275], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 464], | 70.00th=[ 502], 80.00th=[ 558], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 860], 99.50th=[ 894], 99.90th=[ 927], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 127, per=24.99%, avg=42.98, stdev=21.76, samples=227 iops : min= 1, max= 31, avg=10.27, stdev= 5.46, samples=227 write: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5264KiB/116263msec); 0 zone resets slat (usec): min=29, max=18138, avg=298.58, stdev=1081.32 clat (msec): min=388, max=2231, avg=966.86, stdev=273.62 lat (msec): min=389, max=2231, avg=967.16, stdev=273.65 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1720], 99.50th=[ 2022], 99.90th=[ 2140], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 111, per=25.48%, avg=45.61, stdev=19.18, samples=226 iops : min= 1, max= 27, avg=10.93, stdev= 4.80, samples=226 lat (msec) : 20=0.12%, 50=0.04%, 100=0.16%, 250=0.82%, 500=33.20% lat (msec) : 750=24.53%, 1000=19.61%, 2000=21.25%, >=2000=0.27% cpu : usr=0.10%, sys=0.57%, ctx=4913, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145918: Mon Aug 24 09:48:48 2026 read: IOPS=10, BW=42.9KiB/s (43.9kB/s)(4984KiB/116260msec) slat (usec): min=3017, max=68372, avg=15287.91, stdev=9059.43 clat (msec): min=18, max=961, avg=455.22, stdev=133.79 lat (msec): min=49, max=992, avg=470.51, stdev=135.63 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 435], 60.00th=[ 464], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 634], 95.00th=[ 693], | 99.00th=[ 885], 99.50th=[ 919], 99.90th=[ 936], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 110, per=25.18%, avg=43.31, stdev=21.65, samples=226 iops : min= 1, max= 27, avg=10.36, stdev= 5.44, samples=226 write: IOPS=11, BW=45.2KiB/s (46.3kB/s)(5256KiB/116260msec); 0 zone resets slat (usec): min=36, max=15268, avg=312.41, stdev=1043.88 clat (msec): min=306, max=2114, avg=966.18, stdev=271.94 lat (msec): min=306, max=2114, avg=966.50, stdev=272.02 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 575], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1653], 99.50th=[ 1821], 99.90th=[ 2056], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 112, per=25.06%, avg=44.85, stdev=20.40, samples=229 iops : min= 1, max= 28, avg=10.75, stdev= 5.11, samples=229 lat (msec) : 20=0.04%, 50=0.04%, 100=0.12%, 250=0.78%, 500=33.12% lat (msec) : 750=24.61%, 1000=19.77%, 2000=21.45%, >=2000=0.08% cpu : usr=0.09%, sys=0.60%, ctx=4740, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145919: Mon Aug 24 09:48:48 2026 read: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5060KiB/116061msec) slat (usec): min=2662, max=63831, avg=14662.20, stdev=8539.37 clat (msec): min=32, max=960, avg=457.57, stdev=135.38 lat (msec): min=38, max=991, avg=472.24, stdev=137.34 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 468], | 70.00th=[ 506], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 894], 99.50th=[ 919], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 104, per=25.61%, avg=44.05, stdev=20.21, samples=225 iops : min= 1, max= 26, avg=10.54, stdev= 5.05, samples=225 write: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5180KiB/116061msec); 0 zone resets slat (usec): min=34, max=21496, avg=311.06, stdev=1039.11 clat (msec): min=396, max=2155, avg=969.52, stdev=276.78 lat (msec): min=396, max=2155, avg=969.83, stdev=276.76 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 802], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1351], 95.00th=[ 1452], | 99.00th=[ 1687], 99.50th=[ 1871], 99.90th=[ 2072], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 87, per=24.86%, avg=44.50, stdev=18.72, samples=227 iops : min= 1, max= 21, avg=10.65, stdev= 4.68, samples=227 lat (msec) : 50=0.08%, 100=0.27%, 250=0.82%, 500=33.75%, 750=25.04% lat (msec) : 1000=18.67%, 2000=21.25%, >=2000=0.12% cpu : usr=0.10%, sys=0.57%, ctx=4577, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=173KiB/s (177kB/s), 42.8KiB/s-43.6KiB/s (43.8kB/s-44.6kB/s), io=19.6MiB (20.5MB), run=115856-116263msec WRITE: bw=180KiB/s (184kB/s), 44.6KiB/s-45.3KiB/s (45.7kB/s-46.4kB/s), io=20.4MiB (21.4MB), run=115856-116263msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145949: Mon Aug 24 09:48:59 2026 read: IOPS=1, BW=68.5MiB/s (71.8MB/s)(40.0MiB/584msec) slat (nsec): min=48173k, max=48173k, avg=48173307.00, stdev= 0.00 clat (nsec): min=512533k, max=512533k, avg=512533165.00, stdev= 0.00 lat (nsec): min=560713k, max=560713k, avg=560712513.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.06%, ctx=11, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.5MiB/s (71.8MB/s), 68.5MiB/s-68.5MiB/s (71.8MB/s-71.8MB/s), io=40.0MiB (41.9MB), run=584-584msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout