== sanity test 398c: run fio to test AIO ================= 01:31:32 (1788586292) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.689932 s, 60.8 MB/s osc.lustre-OST0000-osc-ffff8b7b51387000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145043: Sat Sep 5 01:34:35 2026 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(10.0MiB/171514msec); 0 zone resets slat (usec): min=26, max=17008, avg=219.66, stdev=913.47 clat (msec): min=84, max=2371, avg=1070.17, stdev=290.39 lat (msec): min=85, max=2371, avg=1070.39, stdev=290.36 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 659], 10.00th=[ 743], 20.00th=[ 835], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1028], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1871], 99.50th=[ 1955], 99.90th=[ 2198], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 141, per=25.04%, avg=59.60, stdev=26.81, samples=338 iops : min= 1, max= 35, avg=14.45, stdev= 6.74, samples=338 lat (msec) : 100=0.04%, 250=0.04%, 500=0.66%, 750=9.80%, 1000=36.48% lat (msec) : 2000=52.73%, >=2000=0.23% cpu : usr=0.06%, sys=0.22%, ctx=2093, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145044: Sat Sep 5 01:34:35 2026 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(10.0MiB/171639msec); 0 zone resets slat (usec): min=21, max=17087, avg=226.74, stdev=1008.08 clat (msec): min=351, max=2281, avg=1070.88, stdev=290.12 lat (msec): min=351, max=2282, avg=1071.11, stdev=290.17 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 667], 10.00th=[ 735], 20.00th=[ 835], | 30.00th=[ 911], 40.00th=[ 961], 50.00th=[ 1020], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1905], 99.50th=[ 1972], 99.90th=[ 2198], 99.95th=[ 2265], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 127, per=25.08%, avg=59.69, stdev=27.69, samples=337 iops : min= 1, max= 31, avg=14.47, stdev= 6.94, samples=337 lat (msec) : 500=0.59%, 750=10.43%, 1000=36.99%, 2000=51.56%, >=2000=0.43% cpu : usr=0.04%, sys=0.22%, ctx=1648, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145045: Sat Sep 5 01:34:35 2026 write: IOPS=14, BW=59.8KiB/s (61.2kB/s)(10.0MiB/171377msec); 0 zone resets slat (usec): min=20, max=18236, avg=222.13, stdev=914.93 clat (msec): min=316, max=2131, avg=1070.49, stdev=296.61 lat (msec): min=316, max=2131, avg=1070.72, stdev=296.62 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 651], 10.00th=[ 743], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1653], | 99.00th=[ 1905], 99.50th=[ 1955], 99.90th=[ 2056], 99.95th=[ 2123], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 128, per=26.29%, avg=62.57, stdev=29.24, samples=322 iops : min= 1, max= 32, avg=15.18, stdev= 7.35, samples=322 lat (msec) : 500=1.25%, 750=9.45%, 1000=34.80%, 2000=54.26%, >=2000=0.23% cpu : usr=0.05%, sys=0.21%, ctx=1832, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145046: Sat Sep 5 01:34:35 2026 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(10.0MiB/171572msec); 0 zone resets slat (usec): min=23, max=22226, avg=215.12, stdev=1127.35 clat (msec): min=334, max=2474, avg=1070.58, stdev=291.90 lat (msec): min=334, max=2475, avg=1070.80, stdev=291.93 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 684], 10.00th=[ 751], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1435], 95.00th=[ 1603], | 99.00th=[ 1938], 99.50th=[ 2089], 99.90th=[ 2333], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 128, per=25.81%, avg=61.44, stdev=28.62, samples=328 iops : min= 1, max= 32, avg=14.91, stdev= 7.18, samples=328 lat (msec) : 500=0.20%, 750=9.69%, 1000=36.91%, 2000=52.42%, >=2000=0.78% cpu : usr=0.05%, sys=0.21%, ctx=1632, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=239KiB/s (244kB/s), 59.7KiB/s-59.8KiB/s (61.1kB/s-61.2kB/s), io=40.0MiB (41.9MB), run=171377-171639msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145081: Sat Sep 5 01:36:45 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(5048KiB/120858msec) slat (usec): min=34, max=58085, avg=15147.01, stdev=8584.87 clat (msec): min=16, max=1122, avg=468.47, stdev=149.93 lat (msec): min=23, max=1147, avg=483.63, stdev=151.99 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 253], 10.00th=[ 309], 20.00th=[ 359], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 531], 80.00th=[ 584], 90.00th=[ 651], 95.00th=[ 743], | 99.00th=[ 902], 99.50th=[ 978], 99.90th=[ 1083], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 103, per=25.55%, avg=41.90, stdev=19.56, samples=236 iops : min= 1, max= 25, avg= 9.97, stdev= 4.87, samples=236 write: IOPS=10, BW=42.0KiB/s (43.0kB/s)(5192KiB/120858msec); 0 zone resets slat (usec): min=32, max=17341, avg=226.92, stdev=763.38 clat (msec): min=160, max=2151, avg=1018.34, stdev=305.18 lat (msec): min=160, max=2151, avg=1018.56, stdev=305.17 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 776], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1469], 95.00th=[ 1620], | 99.00th=[ 1905], 99.50th=[ 1972], 99.90th=[ 2123], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 104, per=25.01%, avg=42.76, stdev=20.80, samples=237 iops : min= 1, max= 26, avg=10.20, stdev= 5.25, samples=237 lat (msec) : 20=0.08%, 50=0.12%, 100=0.39%, 250=2.19%, 500=30.20% lat (msec) : 750=21.99%, 1000=21.76%, 2000=23.12%, >=2000=0.16% cpu : usr=0.11%, sys=0.59%, ctx=4512, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145082: Sat Sep 5 01:36:45 2026 read: IOPS=10, BW=40.0KiB/s (41.9kB/s)(4976KiB/121468msec) slat (usec): min=2921, max=61662, avg=15221.31, stdev=8993.03 clat (msec): min=42, max=1151, avg=467.60, stdev=142.01 lat (msec): min=54, max=1171, avg=482.83, stdev=144.79 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 642], 95.00th=[ 718], | 99.00th=[ 919], 99.50th=[ 986], 99.90th=[ 1133], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 112, per=24.90%, avg=40.84, stdev=20.29, samples=238 iops : min= 1, max= 28, avg= 9.71, stdev= 5.10, samples=238 write: IOPS=10, BW=43.3KiB/s (44.4kB/s)(5264KiB/121468msec); 0 zone resets slat (usec): min=31, max=19454, avg=277.20, stdev=1235.78 clat (msec): min=264, max=2076, avg=1016.99, stdev=296.49 lat (msec): min=264, max=2076, avg=1017.27, stdev=296.53 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 785], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1804], 99.50th=[ 1905], 99.90th=[ 2022], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 104, per=25.22%, avg=43.13, stdev=19.94, samples=239 iops : min= 1, max= 26, avg=10.29, stdev= 5.03, samples=239 lat (msec) : 50=0.04%, 100=0.12%, 250=1.29%, 500=31.17%, 750=22.34% lat (msec) : 1000=21.64%, 2000=23.24%, >=2000=0.16% cpu : usr=0.09%, sys=0.57%, ctx=4606, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145083: Sat Sep 5 01:36:45 2026 read: IOPS=10, BW=40.0KiB/s (41.0kB/s)(4984KiB/121652msec) slat (usec): min=2370, max=62441, avg=15090.95, stdev=8817.32 clat (msec): min=12, max=1198, avg=462.53, stdev=135.29 lat (msec): min=21, max=1233, avg=477.63, stdev=137.37 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 268], 10.00th=[ 313], 20.00th=[ 359], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 477], | 70.00th=[ 518], 80.00th=[ 567], 90.00th=[ 634], 95.00th=[ 693], | 99.00th=[ 835], 99.50th=[ 885], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 127, per=25.17%, avg=41.27, stdev=21.34, samples=237 iops : min= 1, max= 31, avg= 9.82, stdev= 5.36, samples=237 write: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5256KiB/121652msec); 0 zone resets slat (usec): min=26, max=21440, avg=280.15, stdev=1204.05 clat (msec): min=190, max=2061, avg=1025.24, stdev=308.06 lat (msec): min=190, max=2061, avg=1025.52, stdev=308.09 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 844], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1636], | 99.00th=[ 1888], 99.50th=[ 1938], 99.90th=[ 2039], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 103, per=24.98%, avg=42.71, stdev=18.31, samples=240 iops : min= 1, max= 25, avg=10.19, stdev= 4.63, samples=240 lat (msec) : 20=0.04%, 50=0.12%, 100=0.23%, 250=1.33%, 500=32.03% lat (msec) : 750=21.37%, 1000=20.98%, 2000=23.79%, >=2000=0.12% cpu : usr=0.08%, sys=0.54%, ctx=4431, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145084: Sat Sep 5 01:36:45 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(5060KiB/121476msec) slat (usec): min=3191, max=57311, avg=15221.31, stdev=8654.83 clat (msec): min=30, max=1164, avg=466.10, stdev=139.19 lat (msec): min=40, max=1174, avg=481.33, stdev=141.09 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 259], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 527], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 709], | 99.00th=[ 860], 99.50th=[ 944], 99.90th=[ 1133], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 119, per=25.57%, avg=41.93, stdev=20.26, samples=236 iops : min= 1, max= 29, avg= 9.99, stdev= 5.11, samples=236 write: IOPS=10, BW=42.6KiB/s (43.7kB/s)(5180KiB/121476msec); 0 zone resets slat (usec): min=29, max=22212, avg=274.52, stdev=1162.07 clat (msec): min=276, max=2187, avg=1027.07, stdev=309.04 lat (msec): min=276, max=2187, avg=1027.35, stdev=309.10 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 600], 10.00th=[ 693], 20.00th=[ 785], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1620], | 99.00th=[ 1905], 99.50th=[ 1955], 99.90th=[ 2123], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 88, per=24.71%, avg=42.26, stdev=18.90, samples=239 iops : min= 1, max= 22, avg=10.07, stdev= 4.73, samples=239 lat (msec) : 50=0.04%, 100=0.23%, 250=1.68%, 500=30.66%, 750=23.12% lat (msec) : 1000=21.56%, 2000=22.62%, >=2000=0.08% cpu : usr=0.09%, sys=0.65%, ctx=4720, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=165KiB/s (169kB/s), 40.0KiB/s-41.8KiB/s (41.9kB/s-42.8kB/s), io=19.6MiB (20.5MB), run=120858-121652msec WRITE: bw=172KiB/s (176kB/s), 42.6KiB/s-43.3KiB/s (43.7kB/s-44.4kB/s), io=20.4MiB (21.4MB), run=120858-121652msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145105: Sat Sep 5 01:36:55 2026 read: IOPS=1, BW=72.3MiB/s (75.8MB/s)(40.0MiB/553msec) slat (nsec): min=35477k, max=35477k, avg=35476785.00, stdev= 0.00 clat (nsec): min=506820k, max=506820k, avg=506819886.00, stdev= 0.00 lat (nsec): min=542301k, max=542301k, avg=542301261.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.99%, ctx=31, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.3MiB/s (75.8MB/s), 72.3MiB/s-72.3MiB/s (75.8MB/s-75.8MB/s), io=40.0MiB (41.9MB), run=553-553msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout