== sanity test 398c: run fio to test AIO ================= 18:47:02 (1787784422) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.590799 s, 71.0 MB/s osc.lustre-OST0000-osc-ffff8a1ac29a6000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145574: Wed Aug 26 18:50:00 2026 write: IOPS=15, BW=61.7KiB/s (63.2kB/s)(10.0MiB/166001msec); 0 zone resets slat (usec): min=26, max=33068, avg=277.05, stdev=1162.03 clat (msec): min=197, max=2043, avg=1036.07, stdev=252.42 lat (msec): min=197, max=2043, avg=1036.35, stdev=252.44 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 625], 10.00th=[ 768], 20.00th=[ 860], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1028], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1670], 99.50th=[ 1905], 99.90th=[ 2005], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 128, per=25.90%, avg=63.71, stdev=28.52, samples=316 iops : min= 1, max= 32, avg=15.44, stdev= 7.14, samples=316 lat (msec) : 250=0.62%, 500=1.29%, 750=6.68%, 1000=35.31%, 2000=55.98% lat (msec) : >=2000=0.12% cpu : usr=0.05%, sys=0.22%, ctx=1679, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145575: Wed Aug 26 18:50:00 2026 write: IOPS=15, BW=61.7KiB/s (63.1kB/s)(10.0MiB/166092msec); 0 zone resets slat (usec): min=29, max=15430, avg=240.44, stdev=779.60 clat (msec): min=219, max=1962, avg=1036.67, stdev=252.49 lat (msec): min=219, max=1963, avg=1036.92, stdev=252.51 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 625], 10.00th=[ 768], 20.00th=[ 860], | 30.00th=[ 927], 40.00th=[ 978], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1754], 99.50th=[ 1871], 99.90th=[ 1921], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 128, per=24.86%, avg=61.16, stdev=28.03, samples=329 iops : min= 1, max= 32, avg=14.81, stdev= 7.06, samples=329 lat (msec) : 250=0.62%, 500=1.33%, 750=7.19%, 1000=35.74%, 2000=55.12% cpu : usr=0.04%, sys=0.24%, ctx=1646, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145576: Wed Aug 26 18:50:00 2026 write: IOPS=15, BW=61.7KiB/s (63.1kB/s)(10.0MiB/166095msec); 0 zone resets slat (usec): min=29, max=14044, avg=246.52, stdev=872.14 clat (msec): min=155, max=1973, avg=1037.34, stdev=256.53 lat (msec): min=155, max=1973, avg=1037.59, stdev=256.51 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 625], 10.00th=[ 760], 20.00th=[ 860], | 30.00th=[ 927], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1452], | 99.00th=[ 1754], 99.50th=[ 1838], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 162, per=24.93%, avg=61.33, stdev=26.21, samples=328 iops : min= 1, max= 40, avg=14.85, stdev= 6.57, samples=328 lat (msec) : 250=0.51%, 500=2.03%, 750=6.95%, 1000=35.59%, 2000=54.92% cpu : usr=0.05%, sys=0.23%, ctx=2074, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145577: Wed Aug 26 18:50:00 2026 write: IOPS=15, BW=61.6KiB/s (63.1kB/s)(10.0MiB/166172msec); 0 zone resets slat (usec): min=30, max=25252, avg=242.50, stdev=849.74 clat (msec): min=185, max=2002, avg=1037.12, stdev=249.88 lat (msec): min=185, max=2002, avg=1037.37, stdev=249.89 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 659], 10.00th=[ 760], 20.00th=[ 860], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1754], 99.50th=[ 1905], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 128, per=25.33%, avg=62.32, stdev=29.52, samples=323 iops : min= 1, max= 32, avg=15.10, stdev= 7.41, samples=323 lat (msec) : 250=0.62%, 500=1.48%, 750=7.50%, 1000=35.16%, 2000=55.20% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.23%, ctx=1840, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=246KiB/s (252kB/s), 61.6KiB/s-61.7KiB/s (63.1kB/s-63.2kB/s), io=40.0MiB (41.9MB), run=166001-166172msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145610: Wed Aug 26 18:51:59 2026 read: IOPS=11, BW=44.0KiB/s (46.0kB/s)(5048KiB/112296msec) slat (usec): min=40, max=51607, avg=14545.48, stdev=7949.41 clat (msec): min=12, max=1138, avg=431.44, stdev=108.45 lat (msec): min=20, max=1147, avg=446.00, stdev=109.31 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 510], 90.00th=[ 558], 95.00th=[ 617], | 99.00th=[ 726], 99.50th=[ 760], 99.90th=[ 1116], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 115, per=25.27%, avg=44.97, stdev=21.17, samples=220 iops : min= 1, max= 28, avg=10.74, stdev= 5.30, samples=220 write: IOPS=11, BW=46.2KiB/s (47.3kB/s)(5192KiB/112296msec); 0 zone resets slat (usec): min=32, max=19514, avg=326.71, stdev=986.16 clat (msec): min=139, max=2050, avg=949.52, stdev=237.36 lat (msec): min=139, max=2052, avg=949.85, stdev=237.38 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 558], 10.00th=[ 676], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1334], | 99.00th=[ 1452], 99.50th=[ 1620], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 112, per=24.59%, avg=45.48, stdev=18.94, samples=223 iops : min= 1, max= 28, avg=10.87, stdev= 4.73, samples=223 lat (msec) : 20=0.04%, 50=0.20%, 100=0.12%, 250=0.90%, 500=37.46% lat (msec) : 750=20.04%, 1000=20.12%, 2000=21.02%, >=2000=0.12% cpu : usr=0.10%, sys=0.61%, ctx=4249, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145611: Wed Aug 26 18:51:59 2026 read: IOPS=11, BW=44.3KiB/s (45.4kB/s)(4976KiB/112351msec) slat (usec): min=3067, max=49938, avg=14508.82, stdev=7563.31 clat (msec): min=18, max=880, avg=432.52, stdev=106.37 lat (msec): min=26, max=885, avg=447.04, stdev=107.29 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 514], 90.00th=[ 575], 95.00th=[ 625], | 99.00th=[ 693], 99.50th=[ 718], 99.90th=[ 835], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 103, per=24.68%, avg=43.92, stdev=19.31, samples=222 iops : min= 1, max= 25, avg=10.48, stdev= 4.83, samples=222 write: IOPS=11, BW=46.9KiB/s (47.0kB/s)(5264KiB/112351msec); 0 zone resets slat (usec): min=35, max=7008, avg=236.39, stdev=512.15 clat (msec): min=299, max=1764, avg=941.03, stdev=233.29 lat (msec): min=299, max=1764, avg=941.27, stdev=233.30 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 567], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 936], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1284], 95.00th=[ 1334], | 99.00th=[ 1452], 99.50th=[ 1519], 99.90th=[ 1653], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 111, per=25.07%, avg=46.38, stdev=19.57, samples=222 iops : min= 1, max= 27, avg=11.09, stdev= 4.87, samples=222 lat (msec) : 20=0.04%, 50=0.16%, 100=0.16%, 250=0.62%, 500=37.58% lat (msec) : 750=20.00%, 1000=21.76%, 2000=19.69% cpu : usr=0.10%, sys=0.62%, ctx=4091, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145612: Wed Aug 26 18:51:59 2026 read: IOPS=11, BW=44.4KiB/s (45.4kB/s)(4984KiB/112293msec) slat (usec): min=3118, max=54031, avg=14759.08, stdev=7491.69 clat (msec): min=20, max=772, avg=430.93, stdev=104.82 lat (msec): min=33, max=787, avg=445.69, stdev=105.38 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 481], 80.00th=[ 514], 90.00th=[ 575], 95.00th=[ 617], | 99.00th=[ 701], 99.50th=[ 726], 99.90th=[ 768], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 7, max= 111, per=24.93%, avg=44.38, stdev=21.61, samples=221 iops : min= 1, max= 27, avg=10.59, stdev= 5.40, samples=221 write: IOPS=11, BW=46.8KiB/s (47.9kB/s)(5256KiB/112293msec); 0 zone resets slat (usec): min=32, max=8390, avg=269.35, stdev=700.61 clat (msec): min=320, max=1826, avg=942.14, stdev=231.76 lat (msec): min=321, max=1826, avg=942.41, stdev=231.75 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 751], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1284], 95.00th=[ 1334], | 99.00th=[ 1469], 99.50th=[ 1519], 99.90th=[ 1703], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 104, per=24.83%, avg=45.94, stdev=18.97, samples=223 iops : min= 1, max= 26, avg=10.99, stdev= 4.75, samples=223 lat (msec) : 50=0.16%, 100=0.12%, 250=0.70%, 500=37.15%, 750=20.82% lat (msec) : 1000=21.48%, 2000=19.57% cpu : usr=0.10%, sys=0.61%, ctx=4342, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145613: Wed Aug 26 18:51:59 2026 read: IOPS=11, BW=45.1KiB/s (46.1kB/s)(5060KiB/112295msec) slat (usec): min=3102, max=54644, avg=14249.73, stdev=7564.78 clat (msec): min=12, max=1612, avg=435.54, stdev=119.01 lat (msec): min=25, max=1640, avg=449.80, stdev=119.79 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 288], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 514], 90.00th=[ 575], 95.00th=[ 625], | 99.00th=[ 709], 99.50th=[ 735], 99.90th=[ 1603], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 104, per=25.49%, avg=45.37, stdev=20.66, samples=219 iops : min= 1, max= 26, avg=10.84, stdev= 5.18, samples=219 write: IOPS=11, BW=46.1KiB/s (47.2kB/s)(5180KiB/112295msec); 0 zone resets slat (usec): min=27, max=18273, avg=318.73, stdev=1081.71 clat (msec): min=266, max=2618, avg=944.69, stdev=250.38 lat (msec): min=266, max=2618, avg=945.01, stdev=250.36 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 575], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1318], | 99.00th=[ 1485], 99.50th=[ 2198], 99.90th=[ 2534], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 96, per=24.86%, avg=45.99, stdev=19.95, samples=220 iops : min= 1, max= 24, avg=11.00, stdev= 5.02, samples=220 lat (msec) : 20=0.04%, 50=0.08%, 100=0.12%, 250=0.62%, 500=38.40% lat (msec) : 750=19.69%, 1000=20.86%, 2000=19.92%, >=2000=0.27% cpu : usr=0.10%, sys=0.54%, ctx=3939, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=179KiB/s (183kB/s), 44.3KiB/s-45.1KiB/s (45.4kB/s-46.1kB/s), io=19.6MiB (20.5MB), run=112293-112351msec WRITE: bw=186KiB/s (190kB/s), 46.1KiB/s-46.9KiB/s (47.2kB/s-47.0kB/s), io=20.4MiB (21.4MB), run=112293-112351msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145633: Wed Aug 26 18:52:07 2026 read: IOPS=2, BW=100MiB/s (105MB/s)(40.0MiB/400msec) slat (nsec): min=34854k, max=34854k, avg=34853674.00, stdev= 0.00 clat (nsec): min=344937k, max=344937k, avg=344937061.00, stdev= 0.00 lat (nsec): min=379795k, max=379795k, avg=379795490.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.01%, ctx=44, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=40.0MiB (41.9MB), run=400-400msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout