== sanity test 398c: run fio to test AIO ================= 11:51:39 (1783957899) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.396979 s, 106 MB/s osc.lustre-OST0000-osc-ffff9545466e5000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146138: Mon Jul 13 11:54:28 2026 write: IOPS=16, BW=64.1KiB/s (65.7kB/s)(10.0MiB/159650msec); 0 zone resets slat (usec): min=38, max=23992, avg=375.14, stdev=1074.04 clat (msec): min=111, max=2502, avg=996.35, stdev=297.10 lat (msec): min=112, max=2502, avg=996.73, stdev=297.07 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 592], 10.00th=[ 701], 20.00th=[ 785], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1569], | 99.00th=[ 1955], 99.50th=[ 2165], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 142, per=25.04%, avg=64.09, stdev=29.35, samples=314 iops : min= 1, max= 35, avg=15.54, stdev= 7.38, samples=314 lat (msec) : 250=0.27%, 500=2.54%, 750=12.97%, 1000=42.34%, 2000=41.05% lat (msec) : >=2000=0.82% cpu : usr=0.06%, sys=0.35%, ctx=2224, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146139: Mon Jul 13 11:54:28 2026 write: IOPS=16, BW=64.2KiB/s (65.8kB/s)(10.0MiB/159468msec); 0 zone resets slat (usec): min=36, max=36700, avg=343.22, stdev=1145.28 clat (msec): min=197, max=2364, avg=995.82, stdev=303.03 lat (msec): min=197, max=2367, avg=996.17, stdev=303.09 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 592], 10.00th=[ 701], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1368], 95.00th=[ 1586], | 99.00th=[ 2056], 99.50th=[ 2140], 99.90th=[ 2299], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 252, per=25.60%, avg=65.54, stdev=31.57, samples=307 iops : min= 1, max= 63, avg=15.90, stdev= 7.93, samples=307 lat (msec) : 250=0.62%, 500=2.81%, 750=12.46%, 1000=41.41%, 2000=41.64% lat (msec) : >=2000=1.05% cpu : usr=0.05%, sys=0.33%, ctx=1910, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146140: Mon Jul 13 11:54:28 2026 write: IOPS=16, BW=64.1KiB/s (65.7kB/s)(10.0MiB/159647msec); 0 zone resets slat (usec): min=37, max=21521, avg=359.44, stdev=1047.08 clat (msec): min=336, max=2380, avg=996.32, stdev=303.54 lat (msec): min=336, max=2380, avg=996.69, stdev=303.52 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 584], 10.00th=[ 684], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1351], 95.00th=[ 1603], | 99.00th=[ 2056], 99.50th=[ 2198], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 128, per=25.28%, avg=64.73, stdev=28.71, samples=311 iops : min= 1, max= 32, avg=15.69, stdev= 7.18, samples=311 lat (msec) : 500=1.95%, 750=15.55%, 1000=40.00%, 2000=41.37%, >=2000=1.13% cpu : usr=0.06%, sys=0.32%, ctx=2216, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146141: Mon Jul 13 11:54:28 2026 write: IOPS=16, BW=64.1KiB/s (65.6kB/s)(10.0MiB/159811msec); 0 zone resets slat (usec): min=36, max=15331, avg=347.77, stdev=939.73 clat (msec): min=335, max=2499, avg=997.24, stdev=296.65 lat (msec): min=335, max=2499, avg=997.59, stdev=296.68 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1552], | 99.00th=[ 2039], 99.50th=[ 2123], 99.90th=[ 2265], 99.95th=[ 2366], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 247, per=25.05%, avg=64.12, stdev=32.84, samples=314 iops : min= 1, max= 61, avg=15.54, stdev= 8.23, samples=314 lat (msec) : 500=2.62%, 750=12.77%, 1000=42.58%, 2000=40.94%, >=2000=1.09% cpu : usr=0.05%, sys=0.32%, ctx=1950, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=256KiB/s (262kB/s), 64.1KiB/s-64.2KiB/s (65.6kB/s-65.8kB/s), io=40.0MiB (41.9MB), run=159468-159811msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146174: Mon Jul 13 11:56:42 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(5048KiB/120672msec) slat (usec): min=3133, max=65949, avg=16459.80, stdev=9674.28 clat (msec): min=30, max=1873, avg=479.90, stdev=180.93 lat (msec): min=43, max=1887, avg=496.37, stdev=183.38 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 257], 10.00th=[ 292], 20.00th=[ 338], | 30.00th=[ 384], 40.00th=[ 418], 50.00th=[ 456], 60.00th=[ 498], | 70.00th=[ 542], 80.00th=[ 600], 90.00th=[ 684], 95.00th=[ 785], | 99.00th=[ 1062], 99.50th=[ 1116], 99.90th=[ 1737], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 119, per=25.48%, avg=42.29, stdev=21.21, samples=234 iops : min= 1, max= 29, avg=10.10, stdev= 5.35, samples=234 write: IOPS=10, BW=43.0KiB/s (44.1kB/s)(5192KiB/120672msec); 0 zone resets slat (usec): min=41, max=10812, avg=330.48, stdev=821.26 clat (msec): min=312, max=3212, avg=998.43, stdev=320.09 lat (msec): min=312, max=3212, avg=998.77, stdev=320.07 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1435], 95.00th=[ 1569], | 99.00th=[ 1938], 99.50th=[ 2165], 99.90th=[ 3037], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min= 7, max= 95, per=25.20%, avg=43.60, stdev=20.08, samples=233 iops : min= 1, max= 23, avg=10.42, stdev= 5.02, samples=233 lat (msec) : 50=0.12%, 100=0.16%, 250=1.88%, 500=28.95%, 750=25.62% lat (msec) : 1000=20.35%, 2000=22.58%, >=2000=0.35% cpu : usr=0.10%, sys=0.64%, ctx=4315, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146175: Mon Jul 13 11:56:42 2026 read: IOPS=10, BW=41.2KiB/s (42.2kB/s)(4976KiB/120673msec) slat (usec): min=3042, max=83935, avg=16332.01, stdev=10002.15 clat (msec): min=33, max=2306, avg=475.65, stdev=183.83 lat (msec): min=40, max=2341, avg=491.99, stdev=186.32 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 255], 10.00th=[ 300], 20.00th=[ 338], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 451], 60.00th=[ 489], | 70.00th=[ 531], 80.00th=[ 584], 90.00th=[ 659], 95.00th=[ 793], | 99.00th=[ 1053], 99.50th=[ 1502], 99.90th=[ 2039], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 5, max= 112, per=25.19%, avg=41.82, stdev=21.25, samples=233 iops : min= 1, max= 28, avg= 9.98, stdev= 5.33, samples=233 write: IOPS=10, BW=43.6KiB/s (44.7kB/s)(5264KiB/120673msec); 0 zone resets slat (usec): min=41, max=17205, avg=397.78, stdev=1088.47 clat (msec): min=279, max=2830, avg=997.23, stdev=319.62 lat (msec): min=279, max=2830, avg=997.63, stdev=319.57 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1401], 95.00th=[ 1586], | 99.00th=[ 1888], 99.50th=[ 2072], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 104, per=25.25%, avg=43.69, stdev=19.53, samples=236 iops : min= 1, max= 26, avg=10.45, stdev= 4.93, samples=236 lat (msec) : 50=0.20%, 100=0.20%, 250=1.64%, 500=29.84%, 750=25.27% lat (msec) : 1000=19.18%, 2000=23.28%, >=2000=0.39% cpu : usr=0.11%, sys=0.68%, ctx=4480, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146176: Mon Jul 13 11:56:42 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(4984KiB/120672msec) slat (usec): min=63, max=96994, avg=16473.72, stdev=9775.14 clat (msec): min=14, max=1907, avg=475.91, stdev=167.04 lat (msec): min=19, max=1936, avg=492.40, stdev=169.41 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 257], 10.00th=[ 292], 20.00th=[ 342], | 30.00th=[ 384], 40.00th=[ 422], 50.00th=[ 460], 60.00th=[ 498], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 743], | 99.00th=[ 986], 99.50th=[ 1083], 99.90th=[ 1569], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 127, per=25.39%, avg=42.15, stdev=23.00, samples=232 iops : min= 1, max= 31, avg=10.06, stdev= 5.80, samples=232 write: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5256KiB/120672msec); 0 zone resets slat (usec): min=45, max=17497, avg=409.68, stdev=1180.79 clat (msec): min=334, max=3186, avg=1000.32, stdev=330.56 lat (msec): min=336, max=3186, avg=1000.73, stdev=330.51 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1418], 95.00th=[ 1569], | 99.00th=[ 2056], 99.50th=[ 2299], 99.90th=[ 2970], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 7, max= 135, per=25.28%, avg=43.73, stdev=21.50, samples=234 iops : min= 1, max= 33, avg=10.46, stdev= 5.40, samples=234 lat (msec) : 20=0.04%, 50=0.12%, 100=0.39%, 250=1.68%, 500=28.52% lat (msec) : 750=26.37%, 1000=19.61%, 2000=22.66%, >=2000=0.62% cpu : usr=0.10%, sys=0.64%, ctx=4341, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146177: Mon Jul 13 11:56:42 2026 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5060KiB/120671msec) slat (usec): min=3293, max=80376, avg=16522.19, stdev=9482.69 clat (msec): min=5, max=1420, avg=472.53, stdev=159.76 lat (msec): min=19, max=1435, avg=489.06, stdev=161.83 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 347], | 30.00th=[ 384], 40.00th=[ 418], 50.00th=[ 456], 60.00th=[ 493], | 70.00th=[ 535], 80.00th=[ 600], 90.00th=[ 651], 95.00th=[ 751], | 99.00th=[ 986], 99.50th=[ 1083], 99.90th=[ 1385], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 112, per=25.57%, avg=42.45, stdev=22.13, samples=234 iops : min= 1, max= 28, avg=10.14, stdev= 5.58, samples=234 write: IOPS=10, BW=42.9KiB/s (43.0kB/s)(5180KiB/120671msec); 0 zone resets slat (usec): min=45, max=12236, avg=389.86, stdev=952.78 clat (msec): min=289, max=2830, avg=1007.81, stdev=331.46 lat (msec): min=289, max=2830, avg=1008.20, stdev=331.42 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 2056], 99.50th=[ 2265], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 103, per=24.99%, avg=43.24, stdev=21.02, samples=234 iops : min= 1, max= 25, avg=10.33, stdev= 5.30, samples=234 lat (msec) : 10=0.04%, 20=0.04%, 50=0.16%, 100=0.04%, 250=1.68% lat (msec) : 500=29.96%, 750=25.78%, 1000=18.95%, 2000=22.81%, >=2000=0.55% cpu : usr=0.10%, sys=0.67%, ctx=4600, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=166KiB/s (170kB/s), 41.2KiB/s-41.9KiB/s (42.2kB/s-42.9kB/s), io=19.6MiB (20.5MB), run=120671-120673msec WRITE: bw=173KiB/s (177kB/s), 42.9KiB/s-43.6KiB/s (43.0kB/s-44.7kB/s), io=20.4MiB (21.4MB), run=120671-120673msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146199: Mon Jul 13 11:56:55 2026 read: IOPS=1, BW=67.0MiB/s (70.3MB/s)(40.0MiB/597msec) slat (nsec): min=42395k, max=42395k, avg=42394588.00, stdev= 0.00 clat (nsec): min=530355k, max=530355k, avg=530355425.00, stdev= 0.00 lat (nsec): min=572754k, max=572754k, avg=572754263.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 531], 10.00th=[ 531], 20.00th=[ 531], | 30.00th=[ 531], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 531], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.01%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.0MiB/s (70.3MB/s), 67.0MiB/s-67.0MiB/s (70.3MB/s-70.3MB/s), io=40.0MiB (41.9MB), run=597-597msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout