== sanity test 398c: run fio to test AIO ================= 00:17:01 (1788841021) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.456772 s, 91.8 MB/s osc.lustre-OST0000-osc-ffff9247d1042800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144988: Tue Sep 8 00:19:50 2026 write: IOPS=16, BW=64.2KiB/s (65.7kB/s)(10.0MiB/159624msec); 0 zone resets slat (usec): min=28, max=16153, avg=195.73, stdev=775.54 clat (msec): min=223, max=2105, avg=996.98, stdev=277.59 lat (msec): min=223, max=2105, avg=997.18, stdev=277.60 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1485], | 99.00th=[ 1687], 99.50th=[ 1754], 99.90th=[ 2089], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 254, per=24.88%, avg=63.70, stdev=31.97, samples=316 iops : min= 1, max= 63, avg=15.44, stdev= 8.02, samples=316 lat (msec) : 250=0.62%, 500=3.63%, 750=10.82%, 1000=38.83%, 2000=45.98% lat (msec) : >=2000=0.12% cpu : usr=0.05%, sys=0.21%, ctx=1787, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144989: Tue Sep 8 00:19:50 2026 write: IOPS=16, BW=64.1KiB/s (65.6kB/s)(10.0MiB/159790msec); 0 zone resets slat (usec): min=28, max=16141, avg=191.51, stdev=691.28 clat (msec): min=257, max=2104, avg=997.01, stdev=272.58 lat (msec): min=257, max=2104, avg=997.21, stdev=272.58 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1485], | 99.00th=[ 1687], 99.50th=[ 1787], 99.90th=[ 1838], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 255, per=25.30%, avg=64.76, stdev=30.84, samples=311 iops : min= 1, max= 63, avg=15.70, stdev= 7.75, samples=311 lat (msec) : 500=4.22%, 750=9.80%, 1000=39.84%, 2000=46.05%, >=2000=0.08% cpu : usr=0.04%, sys=0.21%, ctx=1924, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144990: Tue Sep 8 00:19:50 2026 write: IOPS=16, BW=64.1KiB/s (65.7kB/s)(10.0MiB/159695msec); 0 zone resets slat (usec): min=27, max=14643, avg=166.64, stdev=601.21 clat (msec): min=279, max=2167, avg=996.41, stdev=276.77 lat (msec): min=279, max=2167, avg=996.57, stdev=276.78 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1485], | 99.00th=[ 1737], 99.50th=[ 1787], 99.90th=[ 2140], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 255, per=25.05%, avg=64.12, stdev=27.39, samples=314 iops : min= 1, max= 63, avg=15.54, stdev= 6.87, samples=314 lat (msec) : 500=3.87%, 750=12.54%, 1000=37.11%, 2000=46.29%, >=2000=0.20% cpu : usr=0.04%, sys=0.19%, ctx=1498, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144991: Tue Sep 8 00:19:50 2026 write: IOPS=16, BW=64.1KiB/s (65.7kB/s)(10.0MiB/159698msec); 0 zone resets slat (usec): min=28, max=13636, avg=223.38, stdev=786.01 clat (msec): min=95, max=2089, avg=996.49, stdev=276.40 lat (msec): min=95, max=2089, avg=996.71, stdev=276.37 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 617], 10.00th=[ 693], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 1670], 99.50th=[ 1770], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 207, per=24.97%, avg=63.91, stdev=27.67, samples=315 iops : min= 1, max= 51, avg=15.49, stdev= 6.93, samples=315 lat (msec) : 100=0.04%, 250=0.43%, 500=3.59%, 750=11.56%, 1000=38.32% lat (msec) : 2000=45.90%, >=2000=0.16% cpu : usr=0.05%, sys=0.21%, ctx=1963, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=256KiB/s (262kB/s), 64.1KiB/s-64.2KiB/s (65.6kB/s-65.7kB/s), io=40.0MiB (41.9MB), run=159624-159790msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145024: Tue Sep 8 00:21:58 2026 read: IOPS=10, BW=42.0KiB/s (43.0kB/s)(5048KiB/117504msec) slat (usec): min=2682, max=88033, avg=14909.92, stdev=9164.36 clat (msec): min=19, max=1186, avg=455.00, stdev=150.31 lat (msec): min=31, max=1199, avg=469.91, stdev=151.74 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 426], 60.00th=[ 464], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 634], 95.00th=[ 743], | 99.00th=[ 961], 99.50th=[ 1036], 99.90th=[ 1133], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 119, per=25.19%, avg=42.83, stdev=21.72, samples=231 iops : min= 1, max= 29, avg=10.21, stdev= 5.45, samples=231 write: IOPS=11, BW=44.2KiB/s (45.2kB/s)(5192KiB/117504msec); 0 zone resets slat (usec): min=33, max=28288, avg=288.42, stdev=1356.89 clat (msec): min=373, max=2218, avg=987.34, stdev=327.35 lat (msec): min=373, max=2218, avg=987.63, stdev=327.43 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 535], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 793], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 1011], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1452], 95.00th=[ 1653], | 99.00th=[ 1888], 99.50th=[ 2022], 99.90th=[ 2165], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 103, per=24.84%, avg=43.97, stdev=21.51, samples=231 iops : min= 1, max= 25, avg=10.50, stdev= 5.39, samples=231 lat (msec) : 20=0.04%, 50=0.20%, 100=0.31%, 250=0.90%, 500=33.28% lat (msec) : 750=25.16%, 1000=18.91%, 2000=20.94%, >=2000=0.27% cpu : usr=0.09%, sys=0.62%, ctx=4838, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145025: Tue Sep 8 00:21:58 2026 read: IOPS=10, BW=42.3KiB/s (43.3kB/s)(4976KiB/117712msec) slat (usec): min=3026, max=82911, avg=15356.76, stdev=9762.20 clat (msec): min=15, max=1185, avg=461.42, stdev=154.53 lat (msec): min=21, max=1200, avg=476.79, stdev=156.72 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 464], | 70.00th=[ 493], 80.00th=[ 542], 90.00th=[ 676], 95.00th=[ 768], | 99.00th=[ 978], 99.50th=[ 1133], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 112, per=25.21%, avg=42.85, stdev=20.72, samples=227 iops : min= 1, max= 28, avg=10.22, stdev= 5.22, samples=227 write: IOPS=11, BW=44.7KiB/s (45.8kB/s)(5264KiB/117712msec); 0 zone resets slat (usec): min=31, max=13798, avg=247.12, stdev=924.28 clat (msec): min=352, max=2160, avg=976.92, stdev=322.26 lat (msec): min=353, max=2161, avg=977.17, stdev=322.27 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 776], 40.00th=[ 860], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1099], 80.00th=[ 1234], 90.00th=[ 1435], 95.00th=[ 1603], | 99.00th=[ 1888], 99.50th=[ 1989], 99.90th=[ 2089], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 111, per=25.25%, avg=44.69, stdev=21.08, samples=231 iops : min= 1, max= 27, avg=10.68, stdev= 5.30, samples=231 lat (msec) : 20=0.04%, 50=0.08%, 100=0.20%, 250=1.02%, 500=35.31% lat (msec) : 750=22.66%, 1000=19.77%, 2000=20.70%, >=2000=0.23% cpu : usr=0.08%, sys=0.52%, ctx=4571, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145026: Tue Sep 8 00:21:58 2026 read: IOPS=10, BW=42.3KiB/s (43.3kB/s)(4984KiB/117801msec) slat (usec): min=2951, max=78923, avg=15201.43, stdev=8897.56 clat (msec): min=6, max=1183, avg=458.96, stdev=146.39 lat (msec): min=18, max=1198, avg=474.17, stdev=148.22 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 651], 95.00th=[ 743], | 99.00th=[ 902], 99.50th=[ 1045], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 120, per=25.13%, avg=42.72, stdev=20.79, samples=229 iops : min= 1, max= 30, avg=10.18, stdev= 5.24, samples=229 write: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5256KiB/117801msec); 0 zone resets slat (usec): min=33, max=31741, avg=290.74, stdev=1287.92 clat (msec): min=274, max=2184, avg=981.57, stdev=326.60 lat (msec): min=274, max=2184, avg=981.86, stdev=326.64 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 776], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 1011], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1435], 95.00th=[ 1636], | 99.00th=[ 1972], 99.50th=[ 2005], 99.90th=[ 2140], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 111, per=25.09%, avg=44.41, stdev=20.52, samples=231 iops : min= 1, max= 27, avg=10.61, stdev= 5.15, samples=231 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=0.27%, 250=0.55% lat (msec) : 500=34.92%, 750=24.06%, 1000=18.55%, 2000=21.25%, >=2000=0.23% cpu : usr=0.09%, sys=0.52%, ctx=4505, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145027: Tue Sep 8 00:21:58 2026 read: IOPS=10, BW=43.2KiB/s (44.3kB/s)(5060KiB/117041msec) slat (usec): min=54, max=85397, avg=14965.44, stdev=9487.79 clat (msec): min=23, max=1156, avg=456.71, stdev=152.46 lat (msec): min=30, max=1175, avg=471.68, stdev=154.64 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 288], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 542], 90.00th=[ 651], 95.00th=[ 768], | 99.00th=[ 969], 99.50th=[ 1070], 99.90th=[ 1133], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 111, per=25.56%, avg=43.45, stdev=19.96, samples=228 iops : min= 1, max= 27, avg=10.36, stdev= 5.03, samples=228 write: IOPS=11, BW=44.3KiB/s (45.3kB/s)(5180KiB/117041msec); 0 zone resets slat (usec): min=33, max=9877, avg=233.98, stdev=634.04 clat (msec): min=213, max=2303, avg=984.05, stdev=329.33 lat (msec): min=214, max=2303, avg=984.29, stdev=329.32 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 523], 10.00th=[ 609], 20.00th=[ 701], | 30.00th=[ 785], 40.00th=[ 860], 50.00th=[ 936], 60.00th=[ 1020], | 70.00th=[ 1116], 80.00th=[ 1250], 90.00th=[ 1418], 95.00th=[ 1653], | 99.00th=[ 1888], 99.50th=[ 2005], 99.90th=[ 2232], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 95, per=24.89%, avg=44.06, stdev=21.02, samples=230 iops : min= 1, max= 23, avg=10.52, stdev= 5.26, samples=230 lat (msec) : 50=0.20%, 100=0.20%, 250=0.82%, 500=36.41%, 750=21.80% lat (msec) : 1000=19.34%, 2000=20.94%, >=2000=0.31% cpu : usr=0.09%, sys=0.53%, ctx=4651, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=170KiB/s (174kB/s), 42.3KiB/s-43.2KiB/s (43.3kB/s-44.3kB/s), io=19.6MiB (20.5MB), run=117041-117801msec WRITE: bw=177KiB/s (182kB/s), 44.2KiB/s-44.7KiB/s (45.2kB/s-45.8kB/s), io=20.4MiB (21.4MB), run=117041-117801msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145048: Tue Sep 8 00:22:09 2026 read: IOPS=1, BW=58.3MiB/s (61.1MB/s)(40.0MiB/686msec) slat (nsec): min=87723k, max=87723k, avg=87722719.00, stdev= 0.00 clat (nsec): min=567591k, max=567591k, avg=567590907.00, stdev= 0.00 lat (nsec): min=655318k, max=655318k, avg=655318413.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.90%, ctx=39, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=58.3MiB/s (61.1MB/s), 58.3MiB/s-58.3MiB/s (61.1MB/s-61.1MB/s), io=40.0MiB (41.9MB), run=686-686msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout