== sanity test 398c: run fio to test AIO ================= 05:25:53 (1788513953) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.638091 s, 65.7 MB/s osc.lustre-OST0000-osc-ffff9f2965e33000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145511: Fri Sep 4 05:28:59 2026 write: IOPS=14, BW=59.5KiB/s (60.0kB/s)(10.0MiB/171962msec); 0 zone resets slat (usec): min=28, max=22013, avg=262.67, stdev=1111.58 clat (msec): min=319, max=2329, avg=1073.38, stdev=299.40 lat (msec): min=319, max=2329, avg=1073.64, stdev=299.46 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 835], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1070], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1435], 95.00th=[ 1552], | 99.00th=[ 1955], 99.50th=[ 2056], 99.90th=[ 2299], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 128, per=25.92%, avg=61.69, stdev=33.55, samples=326 iops : min= 1, max= 32, avg=14.89, stdev= 8.39, samples=326 lat (msec) : 500=1.95%, 750=9.84%, 1000=31.17%, 2000=56.41%, >=2000=0.62% cpu : usr=0.05%, sys=0.25%, ctx=1723, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145512: Fri Sep 4 05:28:59 2026 write: IOPS=14, BW=59.5KiB/s (60.0kB/s)(10.0MiB/171959msec); 0 zone resets slat (usec): min=29, max=21898, avg=245.41, stdev=1089.00 clat (msec): min=312, max=2364, avg=1073.46, stdev=301.50 lat (msec): min=312, max=2365, avg=1073.71, stdev=301.54 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 617], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 969], 50.00th=[ 1062], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1888], 99.50th=[ 2039], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 223, per=25.44%, avg=60.55, stdev=34.23, samples=332 iops : min= 1, max= 55, avg=14.61, stdev= 8.55, samples=332 lat (msec) : 500=2.58%, 750=9.34%, 1000=31.72%, 2000=55.74%, >=2000=0.62% cpu : usr=0.05%, sys=0.23%, ctx=1515, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145513: Fri Sep 4 05:28:59 2026 write: IOPS=14, BW=59.7KiB/s (61.2kB/s)(10.0MiB/171450msec); 0 zone resets slat (usec): min=30, max=16471, avg=240.73, stdev=934.92 clat (msec): min=209, max=2365, avg=1070.99, stdev=314.97 lat (msec): min=209, max=2365, avg=1071.23, stdev=314.97 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 902], 40.00th=[ 978], 50.00th=[ 1070], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 1921], 99.50th=[ 2039], 99.90th=[ 2299], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 199, per=25.36%, avg=60.37, stdev=33.99, samples=333 iops : min= 1, max= 49, avg=14.56, stdev= 8.53, samples=333 lat (msec) : 250=0.62%, 500=2.66%, 750=10.12%, 1000=29.53%, 2000=56.48% lat (msec) : >=2000=0.59% cpu : usr=0.04%, sys=0.26%, ctx=1658, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145514: Fri Sep 4 05:28:59 2026 write: IOPS=14, BW=59.7KiB/s (61.2kB/s)(10.0MiB/171448msec); 0 zone resets slat (usec): min=29, max=18473, avg=283.80, stdev=1101.04 clat (msec): min=233, max=2365, avg=1070.56, stdev=303.46 lat (msec): min=233, max=2365, avg=1070.85, stdev=303.49 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 617], 10.00th=[ 701], 20.00th=[ 818], | 30.00th=[ 911], 40.00th=[ 986], 50.00th=[ 1070], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1569], | 99.00th=[ 1905], 99.50th=[ 2005], 99.90th=[ 2232], 99.95th=[ 2299], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 168, per=24.76%, avg=58.92, stdev=26.71, samples=341 iops : min= 1, max= 42, avg=14.20, stdev= 6.71, samples=341 lat (msec) : 250=0.04%, 500=2.58%, 750=10.62%, 1000=28.79%, 2000=57.46% lat (msec) : >=2000=0.51% cpu : usr=0.06%, sys=0.25%, ctx=1829, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=238KiB/s (244kB/s), 59.5KiB/s-59.7KiB/s (60.0kB/s-61.2kB/s), io=40.0MiB (41.9MB), run=171448-171962msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145559: Fri Sep 4 05:31:05 2026 read: IOPS=10, BW=43.7KiB/s (44.7kB/s)(5048KiB/115523msec) slat (usec): min=3088, max=59254, avg=14963.83, stdev=8373.41 clat (msec): min=22, max=1020, avg=440.42, stdev=116.46 lat (msec): min=33, max=1039, avg=455.39, stdev=117.55 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 288], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 489], 80.00th=[ 531], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 776], 99.50th=[ 793], 99.90th=[ 869], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 126, per=25.18%, avg=43.56, stdev=19.46, samples=227 iops : min= 1, max= 31, avg=10.39, stdev= 4.90, samples=227 write: IOPS=11, BW=44.9KiB/s (46.0kB/s)(5192KiB/115523msec); 0 zone resets slat (usec): min=32, max=21016, avg=301.78, stdev=1013.35 clat (msec): min=304, max=1912, avg=978.12, stdev=265.24 lat (msec): min=304, max=1913, avg=978.43, stdev=265.20 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 542], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1620], 99.50th=[ 1687], 99.90th=[ 1871], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 128, per=24.74%, avg=44.54, stdev=19.47, samples=228 iops : min= 1, max= 32, avg=10.64, stdev= 4.87, samples=228 lat (msec) : 50=0.23%, 100=0.16%, 250=0.74%, 500=36.13%, 750=21.84% lat (msec) : 1000=17.77%, 2000=23.12% cpu : usr=0.10%, sys=0.63%, ctx=4188, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145560: Fri Sep 4 05:31:05 2026 read: IOPS=10, BW=43.1KiB/s (44.1kB/s)(4976KiB/115416msec) slat (usec): min=36, max=59955, avg=14944.52, stdev=8477.52 clat (msec): min=13, max=969, avg=437.40, stdev=123.34 lat (msec): min=18, max=982, avg=452.35, stdev=124.45 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 266], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 464], | 70.00th=[ 489], 80.00th=[ 531], 90.00th=[ 592], 95.00th=[ 659], | 99.00th=[ 768], 99.50th=[ 793], 99.90th=[ 961], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 111, per=24.69%, avg=42.71, stdev=20.37, samples=228 iops : min= 1, max= 27, avg=10.18, stdev= 5.09, samples=228 write: IOPS=11, BW=45.6KiB/s (46.7kB/s)(5264KiB/115416msec); 0 zone resets slat (usec): min=32, max=20812, avg=334.50, stdev=1154.44 clat (msec): min=189, max=1947, avg=974.42, stdev=269.54 lat (msec): min=191, max=1947, avg=974.76, stdev=269.52 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 535], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1636], 99.50th=[ 1770], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 119, per=25.10%, avg=45.18, stdev=20.73, samples=228 iops : min= 1, max= 29, avg=10.80, stdev= 5.18, samples=228 lat (msec) : 20=0.04%, 50=0.12%, 100=0.59%, 250=1.25%, 500=35.74% lat (msec) : 750=20.39%, 1000=18.48%, 2000=23.40% cpu : usr=0.11%, sys=0.62%, ctx=4331, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145561: Fri Sep 4 05:31:05 2026 read: IOPS=10, BW=43.1KiB/s (44.1kB/s)(4984KiB/115634msec) slat (usec): min=2713, max=57951, avg=14822.75, stdev=8046.88 clat (msec): min=34, max=804, avg=446.24, stdev=113.87 lat (msec): min=49, max=821, avg=461.07, stdev=115.21 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 751], 99.50th=[ 793], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 7, max= 111, per=24.59%, avg=42.54, stdev=21.35, samples=230 iops : min= 1, max= 27, avg=10.14, stdev= 5.33, samples=230 write: IOPS=11, BW=45.5KiB/s (46.5kB/s)(5256KiB/115634msec); 0 zone resets slat (usec): min=34, max=28707, avg=391.56, stdev=1658.95 clat (msec): min=217, max=1889, avg=967.41, stdev=263.15 lat (msec): min=217, max=1889, avg=967.81, stdev=263.14 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 535], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1620], 99.50th=[ 1770], 99.90th=[ 1871], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 112, per=24.89%, avg=44.79, stdev=18.74, samples=229 iops : min= 1, max= 28, avg=10.71, stdev= 4.69, samples=229 lat (msec) : 50=0.04%, 100=0.20%, 250=0.94%, 500=35.35%, 750=21.99% lat (msec) : 1000=19.34%, 2000=22.15% cpu : usr=0.11%, sys=0.65%, ctx=4566, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145562: Fri Sep 4 05:31:05 2026 read: IOPS=10, BW=43.8KiB/s (44.9kB/s)(5060KiB/115411msec) slat (usec): min=3177, max=58223, avg=14726.76, stdev=8308.55 clat (msec): min=9, max=1020, avg=441.16, stdev=120.34 lat (msec): min=17, max=1037, avg=455.90, stdev=121.23 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 485], 80.00th=[ 523], 90.00th=[ 592], 95.00th=[ 659], | 99.00th=[ 785], 99.50th=[ 835], 99.90th=[ 986], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 112, per=25.52%, avg=44.15, stdev=20.12, samples=225 iops : min= 1, max= 28, avg=10.54, stdev= 5.04, samples=225 write: IOPS=11, BW=44.9KiB/s (45.0kB/s)(5180KiB/115411msec); 0 zone resets slat (usec): min=34, max=11463, avg=270.67, stdev=718.53 clat (msec): min=162, max=1859, avg=977.00, stdev=265.19 lat (msec): min=162, max=1859, avg=977.28, stdev=265.20 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 542], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1670], 99.50th=[ 1754], 99.90th=[ 1838], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 119, per=24.54%, avg=44.17, stdev=20.26, samples=229 iops : min= 1, max= 29, avg=10.55, stdev= 5.07, samples=229 lat (msec) : 10=0.04%, 50=0.20%, 100=0.12%, 250=0.98%, 500=36.88% lat (msec) : 750=20.90%, 1000=18.87%, 2000=22.03% cpu : usr=0.11%, sys=0.58%, ctx=4375, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=174KiB/s (178kB/s), 43.1KiB/s-43.8KiB/s (44.1kB/s-44.9kB/s), io=19.6MiB (20.5MB), run=115411-115634msec WRITE: bw=181KiB/s (185kB/s), 44.9KiB/s-45.6KiB/s (45.0kB/s-46.7kB/s), io=20.4MiB (21.4MB), run=115411-115634msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145583: Fri Sep 4 05:31:13 2026 read: IOPS=2, BW=109MiB/s (115MB/s)(40.0MiB/366msec) slat (nsec): min=36445k, max=36445k, avg=36444564.00, stdev= 0.00 clat (nsec): min=315560k, max=315560k, avg=315559575.00, stdev= 0.00 lat (nsec): min=352009k, max=352009k, avg=352008789.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.84%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s), io=40.0MiB (41.9MB), run=366-366msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout