== sanity test 398c: run fio to test AIO ================= 16:32:36 (1787776356) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.57493 s, 73.0 MB/s osc.lustre-OST0000-osc-ffff9a6e79bb4800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145083: Wed Aug 26 16:35:39 2026 write: IOPS=14, BW=59.8KiB/s (61.3kB/s)(10.0MiB/171148msec); 0 zone resets slat (usec): min=24, max=29958, avg=286.13, stdev=1133.62 clat (msec): min=131, max=1904, avg=1068.17, stdev=254.79 lat (msec): min=132, max=1905, avg=1068.46, stdev=254.77 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 609], 10.00th=[ 760], 20.00th=[ 877], | 30.00th=[ 953], 40.00th=[ 1011], 50.00th=[ 1070], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1385], 95.00th=[ 1469], | 99.00th=[ 1653], 99.50th=[ 1720], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 200, per=25.20%, avg=60.22, stdev=27.54, samples=334 iops : min= 1, max= 50, avg=14.55, stdev= 6.90, samples=334 lat (msec) : 250=0.08%, 500=2.54%, 750=7.11%, 1000=29.06%, 2000=61.21% cpu : usr=0.05%, sys=0.24%, ctx=2003, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145084: Wed Aug 26 16:35:39 2026 write: IOPS=14, BW=59.9KiB/s (61.3kB/s)(10.0MiB/171049msec); 0 zone resets slat (usec): min=25, max=18594, avg=242.13, stdev=846.22 clat (msec): min=184, max=1903, avg=1067.69, stdev=256.18 lat (msec): min=184, max=1903, avg=1067.94, stdev=256.19 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 617], 10.00th=[ 785], 20.00th=[ 877], | 30.00th=[ 944], 40.00th=[ 1003], 50.00th=[ 1062], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1385], 95.00th=[ 1469], | 99.00th=[ 1703], 99.50th=[ 1720], 99.90th=[ 1838], 99.95th=[ 1888], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 128, per=25.73%, avg=61.49, stdev=30.58, samples=327 iops : min= 1, max= 32, avg=14.87, stdev= 7.67, samples=327 lat (msec) : 250=0.62%, 500=2.03%, 750=6.48%, 1000=30.43%, 2000=60.43% cpu : usr=0.04%, sys=0.24%, ctx=1723, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145085: Wed Aug 26 16:35:39 2026 write: IOPS=14, BW=59.9KiB/s (61.4kB/s)(10.0MiB/170823msec); 0 zone resets slat (usec): min=30, max=17105, avg=279.86, stdev=1010.25 clat (msec): min=151, max=1836, avg=1066.88, stdev=260.19 lat (msec): min=151, max=1836, avg=1067.17, stdev=260.23 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 625], 10.00th=[ 743], 20.00th=[ 877], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1284], 90.00th=[ 1385], 95.00th=[ 1469], | 99.00th=[ 1636], 99.50th=[ 1703], 99.90th=[ 1787], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 183, per=25.04%, avg=59.84, stdev=27.60, samples=336 iops : min= 1, max= 45, avg=14.46, stdev= 6.94, samples=336 lat (msec) : 250=0.62%, 500=2.66%, 750=7.23%, 1000=27.73%, 2000=61.76% cpu : usr=0.05%, sys=0.23%, ctx=1823, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145086: Wed Aug 26 16:35:39 2026 write: IOPS=14, BW=59.8KiB/s (61.3kB/s)(10.0MiB/171149msec); 0 zone resets slat (usec): min=26, max=16628, avg=232.64, stdev=845.79 clat (msec): min=349, max=1823, avg=1068.26, stdev=251.55 lat (msec): min=349, max=1823, avg=1068.50, stdev=251.54 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 634], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1385], 95.00th=[ 1469], | 99.00th=[ 1620], 99.50th=[ 1687], 99.90th=[ 1804], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 216, per=25.28%, avg=60.43, stdev=30.03, samples=333 iops : min= 1, max= 54, avg=14.60, stdev= 7.51, samples=333 lat (msec) : 500=3.28%, 750=6.76%, 1000=28.16%, 2000=61.80% cpu : usr=0.04%, sys=0.22%, ctx=1728, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=239KiB/s (245kB/s), 59.8KiB/s-59.9KiB/s (61.3kB/s-61.4kB/s), io=40.0MiB (41.9MB), run=170823-171149msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145121: Wed Aug 26 16:37:48 2026 read: IOPS=10, BW=42.1KiB/s (43.2kB/s)(5048KiB/119786msec) slat (usec): min=3029, max=64241, avg=14980.55, stdev=8074.21 clat (msec): min=18, max=1149, avg=460.82, stdev=123.56 lat (msec): min=29, max=1177, avg=475.81, stdev=124.29 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 510], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 676], | 99.00th=[ 852], 99.50th=[ 927], 99.90th=[ 1133], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 111, per=25.06%, avg=41.86, stdev=20.95, samples=236 iops : min= 1, max= 27, avg= 9.95, stdev= 5.29, samples=236 write: IOPS=10, BW=43.3KiB/s (44.4kB/s)(5192KiB/119786msec); 0 zone resets slat (usec): min=27, max=12837, avg=313.53, stdev=932.31 clat (msec): min=363, max=2185, avg=1010.04, stdev=290.92 lat (msec): min=363, max=2187, avg=1010.36, stdev=290.93 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 600], 10.00th=[ 651], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 911], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1401], 95.00th=[ 1536], | 99.00th=[ 1888], 99.50th=[ 2005], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 95, per=24.74%, avg=43.05, stdev=18.50, samples=236 iops : min= 1, max= 23, avg=10.25, stdev= 4.64, samples=236 lat (msec) : 20=0.04%, 50=0.23%, 100=0.12%, 250=0.78%, 500=32.46% lat (msec) : 750=24.10%, 1000=17.89%, 2000=24.14%, >=2000=0.23% cpu : usr=0.09%, sys=0.63%, ctx=4645, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145122: Wed Aug 26 16:37:48 2026 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(4976KiB/119620msec) slat (usec): min=33, max=84289, avg=15051.00, stdev=8361.56 clat (msec): min=7, max=1072, avg=459.46, stdev=123.40 lat (msec): min=11, max=1091, avg=474.52, stdev=124.77 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 296], 10.00th=[ 326], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 667], | 99.00th=[ 835], 99.50th=[ 877], 99.90th=[ 944], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7, max= 136, per=24.67%, avg=41.20, stdev=21.24, samples=236 iops : min= 1, max= 34, avg= 9.79, stdev= 5.37, samples=236 write: IOPS=11, BW=44.0KiB/s (45.1kB/s)(5264KiB/119620msec); 0 zone resets slat (usec): min=33, max=21543, avg=370.51, stdev=1244.56 clat (msec): min=169, max=2211, avg=1004.62, stdev=296.83 lat (msec): min=169, max=2211, avg=1004.99, stdev=296.80 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 575], 10.00th=[ 651], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1921], 99.50th=[ 2039], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8, max= 104, per=25.09%, avg=43.65, stdev=18.55, samples=236 iops : min= 2, max= 26, avg=10.41, stdev= 4.67, samples=236 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.20%, 250=0.70% lat (msec) : 500=32.50%, 750=23.01%, 1000=19.53%, 2000=23.52%, >=2000=0.35% cpu : usr=0.10%, sys=0.68%, ctx=4913, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145123: Wed Aug 26 16:37:48 2026 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(4984KiB/119784msec) slat (msec): min=2, max=566, avg=15.60, stdev=17.67 clat (msec): min=21, max=1161, avg=461.35, stdev=126.55 lat (msec): min=30, max=1199, avg=476.96, stdev=127.43 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 477], | 70.00th=[ 502], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 869], 99.50th=[ 969], 99.90th=[ 1133], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 104, per=24.91%, avg=41.60, stdev=20.50, samples=235 iops : min= 1, max= 26, avg= 9.89, stdev= 5.15, samples=235 write: IOPS=10, BW=43.9KiB/s (44.9kB/s)(5256KiB/119784msec); 0 zone resets slat (usec): min=31, max=15748, avg=302.99, stdev=911.02 clat (msec): min=327, max=2209, avg=1002.69, stdev=300.90 lat (msec): min=327, max=2212, avg=1002.99, stdev=300.97 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1418], 95.00th=[ 1552], | 99.00th=[ 1821], 99.50th=[ 1972], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 111, per=24.88%, avg=43.30, stdev=19.73, samples=237 iops : min= 1, max= 27, avg=10.32, stdev= 4.97, samples=237 lat (msec) : 50=0.23%, 100=0.16%, 250=0.90%, 500=33.36%, 750=22.38% lat (msec) : 1000=19.49%, 2000=23.28%, >=2000=0.20% cpu : usr=0.08%, sys=0.57%, ctx=4514, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145124: Wed Aug 26 16:37:48 2026 read: IOPS=10, BW=42.4KiB/s (43.5kB/s)(5060KiB/119202msec) slat (usec): min=2982, max=64707, avg=14918.28, stdev=7982.47 clat (msec): min=27, max=1081, avg=464.31, stdev=123.84 lat (msec): min=41, max=1101, avg=479.24, stdev=125.01 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 667], | 99.00th=[ 844], 99.50th=[ 919], 99.90th=[ 969], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 128, per=25.15%, avg=42.00, stdev=20.65, samples=236 iops : min= 1, max= 32, avg= 9.99, stdev= 5.21, samples=236 write: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5180KiB/119202msec); 0 zone resets slat (usec): min=30, max=10370, avg=285.68, stdev=760.27 clat (msec): min=379, max=2155, avg=1001.58, stdev=290.64 lat (msec): min=380, max=2155, avg=1001.87, stdev=290.61 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 911], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1821], 99.50th=[ 1972], 99.90th=[ 2123], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 119, per=24.66%, avg=42.91, stdev=20.53, samples=236 iops : min= 1, max= 29, avg=10.22, stdev= 5.15, samples=236 lat (msec) : 50=0.08%, 100=0.16%, 250=0.66%, 500=32.73%, 750=23.55% lat (msec) : 1000=18.83%, 2000=23.75%, >=2000=0.23% cpu : usr=0.10%, sys=0.62%, ctx=4731, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=168KiB/s (172kB/s), 41.6KiB/s-42.4KiB/s (42.6kB/s-43.5kB/s), io=19.6MiB (20.5MB), run=119202-119786msec WRITE: bw=174KiB/s (179kB/s), 43.3KiB/s-44.0KiB/s (44.4kB/s-45.1kB/s), io=20.4MiB (21.4MB), run=119202-119786msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145146: Wed Aug 26 16:37:58 2026 read: IOPS=1, BW=68.6MiB/s (71.9MB/s)(40.0MiB/583msec) slat (nsec): min=45506k, max=45506k, avg=45506122.00, stdev= 0.00 clat (nsec): min=512993k, max=512993k, avg=512993233.00, stdev= 0.00 lat (nsec): min=558506k, max=558506k, avg=558506025.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.20%, ctx=44, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.6MiB/s (71.9MB/s), 68.6MiB/s-68.6MiB/s (71.9MB/s-71.9MB/s), io=40.0MiB (41.9MB), run=583-583msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout