== sanity test 398c: run fio to test AIO ================= 15:42:39 (1775763759) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.597002 s, 70.3 MB/s osc.lustre-OST0000-osc-ffff9fa7d1485000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148025: Thu Apr 9 15:45:37 2026 write: IOPS=15, BW=61.3KiB/s (62.8kB/s)(10.0MiB/167091msec); 0 zone resets slat (usec): min=51, max=24523, avg=418.89, stdev=1129.67 clat (msec): min=155, max=2084, avg=1043.37, stdev=266.75 lat (msec): min=155, max=2084, avg=1043.79, stdev=266.75 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 659], 10.00th=[ 743], 20.00th=[ 835], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1485], | 99.00th=[ 1653], 99.50th=[ 1838], 99.90th=[ 2022], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 241, per=24.78%, avg=60.45, stdev=28.45, samples=332 iops : min= 1, max= 60, avg=14.53, stdev= 7.11, samples=332 lat (msec) : 250=0.78%, 500=1.29%, 750=8.09%, 1000=34.49%, 2000=55.16% lat (msec) : >=2000=0.20% cpu : usr=0.06%, sys=0.35%, ctx=2247, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148026: Thu Apr 9 15:45:37 2026 write: IOPS=15, BW=61.1KiB/s (62.6kB/s)(10.0MiB/167459msec); 0 zone resets slat (usec): min=38, max=16279, avg=424.07, stdev=1063.18 clat (msec): min=256, max=2151, avg=1045.02, stdev=260.98 lat (msec): min=256, max=2152, avg=1045.45, stdev=261.03 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 667], 10.00th=[ 768], 20.00th=[ 835], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1485], | 99.00th=[ 1636], 99.50th=[ 1787], 99.90th=[ 2005], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 128, per=25.33%, avg=61.80, stdev=27.43, samples=325 iops : min= 1, max= 32, avg=14.87, stdev= 6.85, samples=325 lat (msec) : 500=1.95%, 750=7.07%, 1000=36.88%, 2000=53.98%, >=2000=0.12% cpu : usr=0.06%, sys=0.35%, ctx=2116, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148027: Thu Apr 9 15:45:37 2026 write: IOPS=15, BW=61.1KiB/s (62.5kB/s)(10.0MiB/167671msec); 0 zone resets slat (usec): min=51, max=22967, avg=446.34, stdev=1178.96 clat (msec): min=237, max=2093, avg=1046.42, stdev=260.19 lat (msec): min=237, max=2093, avg=1046.87, stdev=260.21 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 676], 10.00th=[ 760], 20.00th=[ 835], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 1670], 99.50th=[ 1754], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 127, per=24.70%, avg=60.26, stdev=25.87, samples=333 iops : min= 1, max= 31, avg=14.49, stdev= 6.46, samples=333 lat (msec) : 250=0.62%, 500=0.98%, 750=7.77%, 1000=36.09%, 2000=54.38% lat (msec) : >=2000=0.16% cpu : usr=0.05%, sys=0.36%, ctx=2444, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148028: Thu Apr 9 15:45:37 2026 write: IOPS=15, BW=61.1KiB/s (62.6kB/s)(10.0MiB/167548msec); 0 zone resets slat (usec): min=50, max=22162, avg=487.57, stdev=1335.14 clat (msec): min=125, max=2092, avg=1045.80, stdev=262.23 lat (msec): min=126, max=2093, avg=1046.29, stdev=262.20 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 676], 10.00th=[ 751], 20.00th=[ 844], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 1670], 99.50th=[ 1720], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 183, per=24.78%, avg=60.47, stdev=27.50, samples=332 iops : min= 1, max= 45, avg=14.53, stdev= 6.87, samples=332 lat (msec) : 250=0.59%, 500=1.52%, 750=7.85%, 1000=35.23%, 2000=54.65% lat (msec) : >=2000=0.16% cpu : usr=0.05%, sys=0.37%, ctx=2452, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=244KiB/s (250kB/s), 61.1KiB/s-61.3KiB/s (62.5kB/s-62.8kB/s), io=40.0MiB (41.9MB), run=167091-167671msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148063: Thu Apr 9 15:47:44 2026 read: IOPS=10, BW=43.3KiB/s (44.3kB/s)(5048KiB/116646msec) slat (usec): min=3122, max=65505, avg=15724.07, stdev=7982.25 clat (msec): min=48, max=918, avg=454.02, stdev=114.53 lat (msec): min=71, max=939, avg=469.76, stdev=115.55 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 300], 10.00th=[ 326], 20.00th=[ 363], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 502], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 768], 99.50th=[ 835], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 7, max= 111, per=25.33%, avg=43.31, stdev=21.30, samples=228 iops : min= 1, max= 27, avg=10.27, stdev= 5.37, samples=228 write: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5192KiB/116646msec); 0 zone resets slat (usec): min=57, max=12849, avg=470.70, stdev=937.72 clat (msec): min=405, max=1838, avg=977.68, stdev=250.03 lat (msec): min=405, max=1840, avg=978.16, stdev=250.00 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 625], 10.00th=[ 676], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1636], 99.50th=[ 1687], 99.90th=[ 1804], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 104, per=24.62%, avg=44.07, stdev=20.35, samples=230 iops : min= 1, max= 26, avg=10.46, stdev= 5.08, samples=230 lat (msec) : 50=0.04%, 100=0.16%, 250=0.66%, 500=33.95%, 750=23.91% lat (msec) : 1000=19.88%, 2000=21.41% cpu : usr=0.10%, sys=0.75%, ctx=4664, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148064: Thu Apr 9 15:47:44 2026 read: IOPS=10, BW=42.6KiB/s (43.7kB/s)(4976KiB/116697msec) slat (usec): min=3253, max=64608, avg=15711.02, stdev=7842.47 clat (msec): min=13, max=873, avg=449.31, stdev=110.58 lat (msec): min=26, max=917, avg=465.03, stdev=112.11 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 300], 10.00th=[ 326], 20.00th=[ 363], | 30.00th=[ 388], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 751], 99.50th=[ 818], 99.90th=[ 852], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 112, per=24.78%, avg=42.38, stdev=20.80, samples=229 iops : min= 1, max= 28, avg=10.04, stdev= 5.23, samples=229 write: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5264KiB/116697msec); 0 zone resets slat (usec): min=58, max=17671, avg=514.87, stdev=1237.21 clat (msec): min=233, max=1888, avg=976.60, stdev=261.33 lat (msec): min=234, max=1888, avg=977.12, stdev=261.33 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 584], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1653], 99.50th=[ 1737], 99.90th=[ 1804], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 96, per=25.03%, avg=44.80, stdev=18.78, samples=230 iops : min= 1, max= 24, avg=10.64, stdev= 4.73, samples=230 lat (msec) : 20=0.04%, 50=0.12%, 100=0.12%, 250=0.31%, 500=34.34% lat (msec) : 750=23.83%, 1000=18.28%, 2000=22.97% cpu : usr=0.11%, sys=0.78%, ctx=4943, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148065: Thu Apr 9 15:47:44 2026 read: IOPS=10, BW=42.7KiB/s (43.8kB/s)(4984KiB/116645msec) slat (usec): min=3954, max=60458, avg=15929.49, stdev=7951.37 clat (msec): min=44, max=911, avg=450.77, stdev=115.04 lat (msec): min=56, max=929, avg=466.71, stdev=116.20 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 296], 10.00th=[ 321], 20.00th=[ 359], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 435], 60.00th=[ 464], | 70.00th=[ 502], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 785], 99.50th=[ 827], 99.90th=[ 902], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 7, max= 95, per=25.08%, avg=42.89, stdev=18.94, samples=228 iops : min= 1, max= 23, avg=10.17, stdev= 4.78, samples=228 write: IOPS=11, BW=45.1KiB/s (46.1kB/s)(5256KiB/116645msec); 0 zone resets slat (usec): min=51, max=15983, avg=533.98, stdev=1142.04 clat (msec): min=455, max=1801, avg=974.50, stdev=252.19 lat (msec): min=455, max=1801, avg=975.04, stdev=252.20 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 600], 10.00th=[ 659], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1670], 99.50th=[ 1720], 99.90th=[ 1754], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 111, per=24.75%, avg=44.29, stdev=18.69, samples=231 iops : min= 1, max= 27, avg=10.52, stdev= 4.72, samples=231 lat (msec) : 50=0.04%, 100=0.23%, 250=0.51%, 500=33.63%, 750=24.18% lat (msec) : 1000=19.22%, 2000=22.19% cpu : usr=0.12%, sys=0.79%, ctx=5007, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148066: Thu Apr 9 15:47:44 2026 read: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5060KiB/116091msec) slat (usec): min=79, max=68029, avg=15544.42, stdev=8003.23 clat (msec): min=19, max=1056, avg=451.20, stdev=121.69 lat (msec): min=33, max=1068, avg=466.77, stdev=123.22 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 292], 10.00th=[ 321], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 506], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 810], 99.50th=[ 852], 99.90th=[ 1036], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 111, per=25.24%, avg=43.16, stdev=19.16, samples=229 iops : min= 1, max= 27, avg=10.23, stdev= 4.81, samples=229 write: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5180KiB/116091msec); 0 zone resets slat (usec): min=44, max=18048, avg=570.78, stdev=1328.25 clat (msec): min=175, max=1771, avg=976.85, stdev=253.44 lat (msec): min=175, max=1771, avg=977.43, stdev=253.39 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 609], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1334], 95.00th=[ 1401], | 99.00th=[ 1670], 99.50th=[ 1720], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 96, per=24.55%, avg=43.94, stdev=19.49, samples=230 iops : min= 1, max= 24, avg=10.43, stdev= 4.91, samples=230 lat (msec) : 20=0.04%, 50=0.20%, 100=0.20%, 250=0.82%, 500=33.63% lat (msec) : 750=23.32%, 1000=19.14%, 2000=22.66% cpu : usr=0.10%, sys=0.81%, ctx=5078, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172KiB/s (176kB/s), 42.6KiB/s-43.6KiB/s (43.7kB/s-44.6kB/s), io=19.6MiB (20.5MB), run=116091-116697msec WRITE: bw=179KiB/s (183kB/s), 44.5KiB/s-45.1KiB/s (45.6kB/s-46.2kB/s), io=20.4MiB (21.4MB), run=116091-116697msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148095: Thu Apr 9 15:47:53 2026 read: IOPS=2, BW=81.6MiB/s (85.6MB/s)(40.0MiB/490msec) slat (nsec): min=23939k, max=23939k, avg=23938836.00, stdev= 0.00 clat (nsec): min=449529k, max=449529k, avg=449529004.00, stdev= 0.00 lat (nsec): min=473472k, max=473472k, avg=473472402.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.25%, ctx=4, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=81.6MiB/s (85.6MB/s), 81.6MiB/s-81.6MiB/s (85.6MB/s-85.6MB/s), io=40.0MiB (41.9MB), run=490-490msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout