== sanity test 398c: run fio to test AIO ================= 02:11:20 (1789020680) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.575227 s, 72.9 MB/s osc.lustre-OST0000-osc-ffff950044ab5800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147526: Thu Sep 10 02:14:18 2026 write: IOPS=15, BW=61.3KiB/s (62.8kB/s)(10.0MiB/167076msec); 0 zone resets slat (usec): min=29, max=29511, avg=267.77, stdev=1061.28 clat (msec): min=86, max=2048, avg=1042.90, stdev=310.88 lat (msec): min=88, max=2048, avg=1043.17, stdev=310.88 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 592], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1821], 99.50th=[ 1871], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 199, per=24.99%, avg=60.96, stdev=26.87, samples=330 iops : min= 1, max= 49, avg=14.74, stdev= 6.76, samples=330 lat (msec) : 100=0.04%, 250=1.25%, 500=2.77%, 750=10.39%, 1000=32.30% lat (msec) : 2000=53.05%, >=2000=0.20% cpu : usr=0.06%, sys=0.24%, ctx=1883, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147527: Thu Sep 10 02:14:18 2026 write: IOPS=15, BW=61.3KiB/s (62.7kB/s)(10.0MiB/167151msec); 0 zone resets slat (usec): min=30, max=22029, avg=269.50, stdev=924.63 clat (msec): min=97, max=1960, avg=1043.80, stdev=312.84 lat (msec): min=97, max=1960, avg=1044.07, stdev=312.81 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 600], 10.00th=[ 693], 20.00th=[ 802], | 30.00th=[ 885], 40.00th=[ 961], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1485], 95.00th=[ 1603], | 99.00th=[ 1804], 99.50th=[ 1838], 99.90th=[ 1905], 99.95th=[ 1921], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 199, per=24.90%, avg=60.76, stdev=30.25, samples=331 iops : min= 1, max= 49, avg=14.69, stdev= 7.59, samples=331 lat (msec) : 100=0.04%, 250=0.82%, 500=3.12%, 750=10.66%, 1000=32.50% lat (msec) : 2000=52.85% cpu : usr=0.06%, sys=0.24%, ctx=1960, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147528: Thu Sep 10 02:14:18 2026 write: IOPS=15, BW=61.3KiB/s (62.7kB/s)(10.0MiB/167154msec); 0 zone resets slat (usec): min=28, max=31649, avg=279.08, stdev=1332.13 clat (msec): min=187, max=1954, avg=1043.40, stdev=312.20 lat (msec): min=187, max=1954, avg=1043.68, stdev=312.22 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1620], | 99.00th=[ 1838], 99.50th=[ 1888], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 255, per=25.44%, avg=62.08, stdev=34.64, samples=324 iops : min= 1, max= 63, avg=15.02, stdev= 8.68, samples=324 lat (msec) : 250=1.25%, 500=2.77%, 750=8.98%, 1000=35.16%, 2000=51.84% cpu : usr=0.06%, sys=0.25%, ctx=1848, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147529: Thu Sep 10 02:14:18 2026 write: IOPS=15, BW=61.2KiB/s (62.7kB/s)(10.0MiB/167286msec); 0 zone resets slat (usec): min=26, max=20676, avg=238.77, stdev=926.73 clat (msec): min=205, max=2051, avg=1044.66, stdev=305.28 lat (msec): min=206, max=2051, avg=1044.90, stdev=305.26 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 617], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1821], 99.50th=[ 1854], 99.90th=[ 1938], 99.95th=[ 1955], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 254, per=25.14%, avg=61.34, stdev=32.52, samples=328 iops : min= 1, max= 63, avg=14.84, stdev= 8.17, samples=328 lat (msec) : 250=0.62%, 500=3.28%, 750=10.74%, 1000=30.98%, 2000=54.34% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.23%, ctx=1800, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=245KiB/s (251kB/s), 61.2KiB/s-61.3KiB/s (62.7kB/s-62.8kB/s), io=40.0MiB (41.9MB), run=167076-167286msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147562: Thu Sep 10 02:16:23 2026 read: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5048KiB/116330msec) slat (usec): min=43, max=76638, avg=15420.93, stdev=9226.09 clat (msec): min=7, max=1115, avg=446.65, stdev=136.79 lat (msec): min=11, max=1130, avg=462.08, stdev=139.07 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 460], | 70.00th=[ 498], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 693], | 99.00th=[ 869], 99.50th=[ 953], 99.90th=[ 1036], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 103, per=25.46%, avg=43.29, stdev=21.66, samples=228 iops : min= 1, max= 25, avg=10.33, stdev= 5.40, samples=228 write: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5192KiB/116330msec); 0 zone resets slat (usec): min=34, max=16594, avg=279.75, stdev=840.78 clat (msec): min=154, max=2375, avg=983.35, stdev=338.86 lat (msec): min=154, max=2376, avg=983.63, stdev=338.86 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 518], 10.00th=[ 600], 20.00th=[ 693], | 30.00th=[ 776], 40.00th=[ 852], 50.00th=[ 927], 60.00th=[ 1036], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1921], 99.50th=[ 2039], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 112, per=25.07%, avg=44.38, stdev=20.56, samples=228 iops : min= 1, max= 28, avg=10.61, stdev= 5.20, samples=228 lat (msec) : 10=0.04%, 50=0.16%, 100=0.23%, 250=0.98%, 500=35.47% lat (msec) : 750=24.38%, 1000=16.60%, 2000=21.80%, >=2000=0.35% cpu : usr=0.11%, sys=0.66%, ctx=4795, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147563: Thu Sep 10 02:16:23 2026 read: IOPS=10, BW=42.4KiB/s (43.4kB/s)(4976KiB/117483msec) slat (usec): min=3158, max=65515, avg=15051.83, stdev=8913.33 clat (msec): min=35, max=1462, avg=455.92, stdev=147.84 lat (msec): min=50, max=1475, avg=470.98, stdev=149.42 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 284], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 460], | 70.00th=[ 506], 80.00th=[ 558], 90.00th=[ 642], 95.00th=[ 718], | 99.00th=[ 911], 99.50th=[ 986], 99.90th=[ 1267], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 112, per=24.71%, avg=42.01, stdev=21.37, samples=231 iops : min= 1, max= 28, avg=10.01, stdev= 5.38, samples=231 write: IOPS=11, BW=44.8KiB/s (45.9kB/s)(5264KiB/117483msec); 0 zone resets slat (usec): min=32, max=21847, avg=325.06, stdev=1194.56 clat (msec): min=339, max=2326, avg=979.34, stdev=324.28 lat (msec): min=339, max=2327, avg=979.66, stdev=324.29 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 550], 10.00th=[ 609], 20.00th=[ 701], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 927], 60.00th=[ 1011], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1435], 95.00th=[ 1586], | 99.00th=[ 1888], 99.50th=[ 2039], 99.90th=[ 2265], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 144, per=25.36%, avg=44.89, stdev=22.14, samples=230 iops : min= 1, max= 36, avg=10.74, stdev= 5.53, samples=230 lat (msec) : 50=0.08%, 100=0.16%, 250=0.43%, 500=34.02%, 750=25.20% lat (msec) : 1000=18.55%, 2000=21.21%, >=2000=0.35% cpu : usr=0.12%, sys=0.69%, ctx=4691, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147564: Thu Sep 10 02:16:23 2026 read: IOPS=10, BW=42.4KiB/s (43.4kB/s)(4984KiB/117477msec) slat (usec): min=2755, max=88319, avg=15661.86, stdev=9769.83 clat (msec): min=35, max=1260, avg=448.78, stdev=134.84 lat (msec): min=47, max=1290, avg=464.45, stdev=136.87 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 502], 80.00th=[ 542], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 844], 99.50th=[ 894], 99.90th=[ 1250], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 7, max= 119, per=25.31%, avg=43.04, stdev=22.44, samples=227 iops : min= 1, max= 29, avg=10.27, stdev= 5.65, samples=227 write: IOPS=11, BW=44.7KiB/s (45.8kB/s)(5256KiB/117477msec); 0 zone resets slat (usec): min=38, max=17676, avg=279.28, stdev=932.50 clat (msec): min=314, max=2361, avg=986.11, stdev=330.03 lat (msec): min=314, max=2361, avg=986.39, stdev=330.03 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 535], 10.00th=[ 609], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 1011], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1921], 99.50th=[ 2005], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 120, per=25.20%, avg=44.61, stdev=21.05, samples=230 iops : min= 1, max= 30, avg=10.67, stdev= 5.26, samples=230 lat (msec) : 50=0.04%, 100=0.16%, 250=0.82%, 500=34.38%, 750=24.57% lat (msec) : 1000=18.83%, 2000=20.86%, >=2000=0.35% cpu : usr=0.12%, sys=0.63%, ctx=5004, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147565: Thu Sep 10 02:16:23 2026 read: IOPS=10, BW=43.5KiB/s (44.6kB/s)(5060KiB/116258msec) slat (usec): min=3029, max=79560, avg=15276.54, stdev=9260.35 clat (msec): min=8, max=1019, avg=445.48, stdev=135.06 lat (msec): min=14, max=1031, avg=460.77, stdev=137.19 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 493], 80.00th=[ 542], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 852], 99.50th=[ 877], 99.90th=[ 1011], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 111, per=25.71%, avg=43.70, stdev=22.18, samples=227 iops : min= 1, max= 27, avg=10.44, stdev= 5.57, samples=227 write: IOPS=11, BW=44.6KiB/s (45.6kB/s)(5180KiB/116258msec); 0 zone resets slat (usec): min=32, max=20518, avg=316.85, stdev=1074.73 clat (msec): min=342, max=2377, avg=983.02, stdev=329.19 lat (msec): min=342, max=2377, avg=983.34, stdev=329.18 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 535], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 927], 60.00th=[ 1011], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1435], 95.00th=[ 1620], | 99.00th=[ 1921], 99.50th=[ 2005], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 104, per=24.92%, avg=44.11, stdev=20.10, samples=229 iops : min= 1, max= 26, avg=10.54, stdev= 5.04, samples=229 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.39%, 250=0.86% lat (msec) : 500=35.16%, 750=23.95%, 1000=18.32%, 2000=20.86%, >=2000=0.27% cpu : usr=0.12%, sys=0.65%, ctx=4918, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=171KiB/s (175kB/s), 42.4KiB/s-43.5KiB/s (43.4kB/s-44.6kB/s), io=19.6MiB (20.5MB), run=116258-117483msec WRITE: bw=178KiB/s (182kB/s), 44.6KiB/s-44.8KiB/s (45.6kB/s-45.9kB/s), io=20.4MiB (21.4MB), run=116258-117483msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147588: Thu Sep 10 02:16:34 2026 read: IOPS=1, BW=62.3MiB/s (65.3MB/s)(40.0MiB/642msec) slat (nsec): min=70793k, max=70793k, avg=70792970.00, stdev= 0.00 clat (nsec): min=558318k, max=558318k, avg=558317891.00, stdev= 0.00 lat (nsec): min=629116k, max=629116k, avg=629115580.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.03%, ctx=14, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.3MiB/s (65.3MB/s), 62.3MiB/s-62.3MiB/s (65.3MB/s-65.3MB/s), io=40.0MiB (41.9MB), run=642-642msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout