== sanity test 398c: run fio to test AIO ================= 05:39:13 (1786181953) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.599845 s, 69.9 MB/s osc.lustre-OST0000-osc-ffff98fa7086e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145716: Sat Aug 8 05:42:10 2026 write: IOPS=15, BW=61.7KiB/s (63.1kB/s)(10.0MiB/166090msec); 0 zone resets slat (usec): min=36, max=23070, avg=239.93, stdev=788.80 clat (msec): min=278, max=2142, avg=1036.57, stdev=256.46 lat (msec): min=278, max=2142, avg=1036.81, stdev=256.47 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 667], 10.00th=[ 743], 20.00th=[ 835], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1020], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1905], 99.50th=[ 1972], 99.90th=[ 2123], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 128, per=25.58%, avg=62.93, stdev=30.41, samples=320 iops : min= 1, max= 32, avg=15.26, stdev= 7.65, samples=320 lat (msec) : 500=0.82%, 750=10.04%, 1000=36.21%, 2000=52.66%, >=2000=0.27% cpu : usr=0.04%, sys=0.27%, ctx=2007, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145717: Sat Aug 8 05:42:10 2026 write: IOPS=15, BW=61.6KiB/s (63.0kB/s)(10.0MiB/166340msec); 0 zone resets slat (usec): min=35, max=16989, avg=262.28, stdev=842.86 clat (msec): min=360, max=1976, avg=1037.68, stdev=252.01 lat (msec): min=360, max=1976, avg=1037.94, stdev=252.02 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 659], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1838], 99.50th=[ 1905], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 128, per=25.04%, avg=61.60, stdev=24.99, samples=327 iops : min= 1, max= 32, avg=14.92, stdev= 6.29, samples=327 lat (msec) : 500=0.20%, 750=10.43%, 1000=37.62%, 2000=51.76% cpu : usr=0.04%, sys=0.25%, ctx=2339, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145718: Sat Aug 8 05:42:10 2026 write: IOPS=15, BW=61.7KiB/s (63.2kB/s)(10.0MiB/166011msec); 0 zone resets slat (usec): min=37, max=20838, avg=225.42, stdev=646.90 clat (msec): min=214, max=2085, avg=1036.94, stdev=259.32 lat (msec): min=214, max=2085, avg=1037.17, stdev=259.34 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 667], 10.00th=[ 751], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1502], | 99.00th=[ 1888], 99.50th=[ 1938], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=25.99%, avg=63.93, stdev=33.35, samples=315 iops : min= 1, max= 32, avg=15.51, stdev= 8.34, samples=315 lat (msec) : 250=0.04%, 500=0.78%, 750=9.38%, 1000=38.75%, 2000=50.86% lat (msec) : >=2000=0.20% cpu : usr=0.04%, sys=0.24%, ctx=2091, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145719: Sat Aug 8 05:42:10 2026 write: IOPS=15, BW=61.6KiB/s (63.1kB/s)(10.0MiB/166202msec); 0 zone resets slat (usec): min=36, max=24548, avg=247.88, stdev=917.81 clat (msec): min=130, max=1986, avg=1036.82, stdev=255.14 lat (msec): min=131, max=1986, avg=1037.07, stdev=255.15 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 659], 10.00th=[ 743], 20.00th=[ 835], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1888], 99.50th=[ 1921], 99.90th=[ 1955], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 128, per=25.71%, avg=63.25, stdev=27.12, samples=318 iops : min= 1, max= 32, avg=15.35, stdev= 6.78, samples=318 lat (msec) : 250=0.04%, 500=0.23%, 750=9.80%, 1000=39.26%, 2000=50.66% cpu : usr=0.04%, sys=0.24%, ctx=1682, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=246KiB/s (252kB/s), 61.6KiB/s-61.7KiB/s (63.0kB/s-63.2kB/s), io=40.0MiB (41.9MB), run=166011-166340msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145753: Sat Aug 8 05:44:21 2026 read: IOPS=10, BW=42.3KiB/s (43.4kB/s)(5048KiB/119228msec) slat (usec): min=59, max=72117, avg=15597.62, stdev=9874.91 clat (msec): min=14, max=1911, avg=462.29, stdev=187.24 lat (msec): min=26, max=1936, avg=477.90, stdev=190.40 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 224], 10.00th=[ 257], 20.00th=[ 309], | 30.00th=[ 359], 40.00th=[ 401], 50.00th=[ 439], 60.00th=[ 481], | 70.00th=[ 523], 80.00th=[ 592], 90.00th=[ 693], 95.00th=[ 802], | 99.00th=[ 1011], 99.50th=[ 1167], 99.90th=[ 1838], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 119, per=25.64%, avg=42.81, stdev=22.38, samples=231 iops : min= 1, max= 29, avg=10.21, stdev= 5.61, samples=231 write: IOPS=10, BW=43.5KiB/s (44.6kB/s)(5192KiB/119228msec); 0 zone resets slat (usec): min=38, max=12527, avg=295.10, stdev=764.96 clat (msec): min=203, max=2542, avg=1003.28, stdev=366.59 lat (msec): min=203, max=2543, avg=1003.58, stdev=366.62 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 477], 10.00th=[ 550], 20.00th=[ 667], | 30.00th=[ 793], 40.00th=[ 877], 50.00th=[ 961], 60.00th=[ 1083], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1502], 95.00th=[ 1670], | 99.00th=[ 1955], 99.50th=[ 2089], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 112, per=25.23%, avg=43.90, stdev=22.34, samples=231 iops : min= 1, max= 28, avg=10.48, stdev= 5.63, samples=231 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=4.38%, 500=30.47% lat (msec) : 750=24.34%, 1000=16.68%, 2000=23.55%, >=2000=0.31% cpu : usr=0.12%, sys=0.68%, ctx=4668, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145754: Sat Aug 8 05:44:21 2026 read: IOPS=10, BW=41.5KiB/s (42.5kB/s)(4976KiB/119931msec) slat (msec): min=2, max=100, avg=15.61, stdev= 9.94 clat (msec): min=28, max=1209, avg=460.32, stdev=167.32 lat (msec): min=35, max=1240, avg=475.95, stdev=170.25 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 228], 10.00th=[ 262], 20.00th=[ 317], | 30.00th=[ 359], 40.00th=[ 401], 50.00th=[ 435], 60.00th=[ 485], | 70.00th=[ 531], 80.00th=[ 592], 90.00th=[ 684], 95.00th=[ 776], | 99.00th=[ 919], 99.50th=[ 969], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 111, per=24.94%, avg=41.64, stdev=22.34, samples=233 iops : min= 1, max= 27, avg= 9.91, stdev= 5.63, samples=233 write: IOPS=10, BW=43.9KiB/s (44.9kB/s)(5264KiB/119931msec); 0 zone resets slat (usec): min=46, max=31568, avg=327.73, stdev=1141.03 clat (msec): min=271, max=2205, avg=1005.76, stdev=361.58 lat (msec): min=272, max=2206, avg=1006.09, stdev=361.66 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 481], 10.00th=[ 558], 20.00th=[ 684], | 30.00th=[ 776], 40.00th=[ 860], 50.00th=[ 944], 60.00th=[ 1083], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1485], 95.00th=[ 1670], | 99.00th=[ 1938], 99.50th=[ 2056], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 128, per=24.90%, avg=43.32, stdev=22.55, samples=238 iops : min= 1, max= 32, avg=10.33, stdev= 5.70, samples=238 lat (msec) : 50=0.20%, 100=0.04%, 250=3.67%, 500=30.31%, 750=24.96% lat (msec) : 1000=17.30%, 2000=23.12%, >=2000=0.39% cpu : usr=0.10%, sys=0.66%, ctx=4649, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145755: Sat Aug 8 05:44:21 2026 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(4984KiB/119803msec) slat (usec): min=2451, max=79874, avg=15444.34, stdev=9628.08 clat (msec): min=25, max=1186, avg=463.68, stdev=173.49 lat (msec): min=41, max=1218, avg=479.13, stdev=176.64 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 228], 10.00th=[ 262], 20.00th=[ 317], | 30.00th=[ 363], 40.00th=[ 405], 50.00th=[ 443], 60.00th=[ 481], | 70.00th=[ 523], 80.00th=[ 609], 90.00th=[ 701], 95.00th=[ 785], | 99.00th=[ 953], 99.50th=[ 1020], 99.90th=[ 1150], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 111, per=25.35%, avg=42.33, stdev=22.51, samples=231 iops : min= 1, max= 27, avg=10.08, stdev= 5.67, samples=231 write: IOPS=10, BW=43.9KiB/s (44.9kB/s)(5256KiB/119803msec); 0 zone resets slat (usec): min=37, max=19357, avg=359.59, stdev=1059.74 clat (msec): min=287, max=2244, avg=1001.91, stdev=354.77 lat (msec): min=287, max=2245, avg=1002.27, stdev=354.82 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 472], 10.00th=[ 575], 20.00th=[ 676], | 30.00th=[ 793], 40.00th=[ 869], 50.00th=[ 953], 60.00th=[ 1070], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1687], | 99.00th=[ 1905], 99.50th=[ 2005], 99.90th=[ 2165], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 120, per=25.08%, avg=43.65, stdev=22.17, samples=235 iops : min= 1, max= 30, avg=10.42, stdev= 5.59, samples=235 lat (msec) : 50=0.12%, 100=0.20%, 250=3.52%, 500=30.59%, 750=24.22% lat (msec) : 1000=17.89%, 2000=23.20%, >=2000=0.27% cpu : usr=0.08%, sys=0.65%, ctx=4680, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145756: Sat Aug 8 05:44:21 2026 read: IOPS=10, BW=42.4KiB/s (43.4kB/s)(5060KiB/119289msec) slat (usec): min=2601, max=73085, avg=15333.34, stdev=9193.63 clat (msec): min=24, max=1465, avg=462.58, stdev=176.19 lat (msec): min=37, max=1481, avg=477.92, stdev=178.93 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 226], 10.00th=[ 262], 20.00th=[ 313], | 30.00th=[ 363], 40.00th=[ 405], 50.00th=[ 443], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 592], 90.00th=[ 693], 95.00th=[ 785], | 99.00th=[ 961], 99.50th=[ 1083], 99.90th=[ 1334], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 120, per=26.19%, avg=43.74, stdev=22.63, samples=227 iops : min= 1, max= 30, avg=10.44, stdev= 5.70, samples=227 write: IOPS=10, BW=43.4KiB/s (44.5kB/s)(5180KiB/119289msec); 0 zone resets slat (usec): min=44, max=40189, avg=348.85, stdev=1338.49 clat (msec): min=266, max=2376, avg=1004.21, stdev=366.38 lat (msec): min=266, max=2376, avg=1004.56, stdev=366.39 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 481], 10.00th=[ 558], 20.00th=[ 676], | 30.00th=[ 776], 40.00th=[ 860], 50.00th=[ 953], 60.00th=[ 1070], | 70.00th=[ 1183], 80.00th=[ 1318], 90.00th=[ 1502], 95.00th=[ 1670], | 99.00th=[ 1955], 99.50th=[ 2299], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 96, per=24.82%, avg=43.19, stdev=22.15, samples=234 iops : min= 1, max= 24, avg=10.30, stdev= 5.60, samples=234 lat (msec) : 50=0.20%, 100=0.12%, 250=3.91%, 500=30.90%, 750=24.26% lat (msec) : 1000=17.34%, 2000=22.85%, >=2000=0.43% cpu : usr=0.09%, sys=0.62%, ctx=4665, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=167KiB/s (171kB/s), 41.5KiB/s-42.4KiB/s (42.5kB/s-43.4kB/s), io=19.6MiB (20.5MB), run=119228-119931msec WRITE: bw=174KiB/s (178kB/s), 43.4KiB/s-43.9KiB/s (44.5kB/s-44.9kB/s), io=20.4MiB (21.4MB), run=119228-119931msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145777: Sat Aug 8 05:44:35 2026 read: IOPS=1, BW=60.3MiB/s (63.3MB/s)(40.0MiB/663msec) slat (nsec): min=33668k, max=33668k, avg=33668161.00, stdev= 0.00 clat (nsec): min=605452k, max=605452k, avg=605452246.00, stdev= 0.00 lat (nsec): min=639128k, max=639128k, avg=639127807.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 609], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.11%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=60.3MiB/s (63.3MB/s), 60.3MiB/s-60.3MiB/s (63.3MB/s-63.3MB/s), io=40.0MiB (41.9MB), run=663-663msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout