== sanity test 398c: run fio to test AIO ================= 11:00:23 (1743519623) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.425503 s, 98.6 MB/s osc.lustre-OST0000-osc-ffff8f52fb484000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=531157: Tue Apr 1 11:00:37 2025 write: IOPS=361, BW=1446KiB/s (1480kB/s)(10.0MiB/7084msec); 0 zone resets slat (usec): min=139, max=3827, avg=598.91, stdev=305.20 clat (msec): min=7, max=109, avg=43.59, stdev=13.08 lat (msec): min=8, max=110, avg=44.19, stdev=13.21 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 36], | 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 40], 60.00th=[ 41], | 70.00th=[ 42], 80.00th=[ 45], 90.00th=[ 67], 95.00th=[ 79], | 99.00th=[ 87], 99.50th=[ 91], 99.90th=[ 105], 99.95th=[ 109], | 99.99th=[ 110] bw ( KiB/s): min= 822, max= 1720, per=25.23%, avg=1447.21, stdev=327.85, samples=14 iops : min= 205, max= 430, avg=361.71, stdev=82.10, samples=14 lat (msec) : 10=0.04%, 20=0.12%, 50=84.57%, 100=15.16%, 250=0.12% cpu : usr=0.44%, sys=21.23%, ctx=1628, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=531158: Tue Apr 1 11:00:37 2025 write: IOPS=358, BW=1434KiB/s (1469kB/s)(10.0MiB/7139msec); 0 zone resets slat (usec): min=141, max=3673, avg=599.76, stdev=321.76 clat (usec): min=26876, max=97518, avg=43743.82, stdev=12845.16 lat (usec): min=27352, max=99134, avg=44344.74, stdev=12984.50 clat percentiles (usec): | 1.00th=[31589], 5.00th=[33424], 10.00th=[34866], 20.00th=[36439], | 30.00th=[37487], 40.00th=[38536], 50.00th=[39584], 60.00th=[40109], | 70.00th=[41681], 80.00th=[44827], 90.00th=[65799], 95.00th=[78119], | 99.00th=[86508], 99.50th=[88605], 99.90th=[96994], 99.95th=[96994], | 99.99th=[98042] bw ( KiB/s): min= 822, max= 1720, per=25.05%, avg=1436.71, stdev=328.11, samples=14 iops : min= 205, max= 430, avg=359.07, stdev=82.18, samples=14 lat (msec) : 50=83.98%, 100=16.02% cpu : usr=0.42%, sys=21.13%, ctx=1608, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=531159: Tue Apr 1 11:00:37 2025 write: IOPS=358, BW=1434KiB/s (1469kB/s)(10.0MiB/7140msec); 0 zone resets slat (usec): min=158, max=3645, avg=594.36, stdev=291.42 clat (usec): min=23282, max=97316, avg=43766.78, stdev=12892.66 lat (usec): min=23794, max=99806, avg=44362.21, stdev=13026.79 clat percentiles (usec): | 1.00th=[31327], 5.00th=[33424], 10.00th=[34866], 20.00th=[36439], | 30.00th=[37487], 40.00th=[38536], 50.00th=[39584], 60.00th=[40633], | 70.00th=[41681], 80.00th=[44303], 90.00th=[66847], 95.00th=[78119], | 99.00th=[86508], 99.50th=[88605], 99.90th=[91751], 99.95th=[93848], | 99.99th=[96994] bw ( KiB/s): min= 806, max= 1728, per=25.06%, avg=1437.29, stdev=329.02, samples=14 iops : min= 201, max= 432, avg=359.21, stdev=82.41, samples=14 lat (msec) : 50=83.91%, 100=16.09% cpu : usr=0.43%, sys=20.89%, ctx=1456, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=531160: Tue Apr 1 11:00:37 2025 write: IOPS=359, BW=1437KiB/s (1471kB/s)(10.0MiB/7128msec); 0 zone resets slat (usec): min=159, max=6058, avg=591.58, stdev=323.63 clat (usec): min=5651, max=97570, avg=43738.22, stdev=13071.11 lat (usec): min=6143, max=98433, avg=44330.88, stdev=13204.57 clat percentiles (usec): | 1.00th=[31065], 5.00th=[33424], 10.00th=[34866], 20.00th=[36439], | 30.00th=[37487], 40.00th=[38536], 50.00th=[39584], 60.00th=[40109], | 70.00th=[41681], 80.00th=[44303], 90.00th=[66847], 95.00th=[78119], | 99.00th=[87557], 99.50th=[90702], 99.90th=[93848], 99.95th=[96994], | 99.99th=[98042] bw ( KiB/s): min= 798, max= 1744, per=25.09%, avg=1438.93, stdev=335.09, samples=14 iops : min= 199, max= 436, avg=359.64, stdev=83.91, samples=14 lat (msec) : 10=0.08%, 20=0.04%, 50=83.98%, 100=15.90% cpu : usr=0.43%, sys=20.84%, ctx=1500, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=5737KiB/s (5874kB/s), 1434KiB/s-1446KiB/s (1469kB/s-1480kB/s), io=40.0MiB (41.9MB), run=7084-7140msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=531171: Tue Apr 1 11:00:52 2025 read: IOPS=122, BW=489KiB/s (501kB/s)(5048KiB/10327msec) slat (usec): min=351, max=24816, avg=6470.29, stdev=2019.11 clat (msec): min=5, max=110, avg=62.05, stdev=13.60 lat (msec): min=5, max=122, avg=68.52, stdev=14.12 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 66], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 81], 95.00th=[ 86], | 99.00th=[ 95], 99.50th=[ 100], 99.90th=[ 105], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 367, max= 666, per=25.12%, avg=487.55, stdev=65.93, samples=20 iops : min= 91, max= 166, avg=121.50, stdev=16.56, samples=20 write: IOPS=125, BW=503KiB/s (515kB/s)(5192KiB/10327msec); 0 zone resets slat (usec): min=149, max=5138, avg=635.91, stdev=284.41 clat (msec): min=12, max=103, avg=59.83, stdev=11.89 lat (msec): min=12, max=104, avg=60.47, stdev=11.93 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 75], 95.00th=[ 81], | 99.00th=[ 91], 99.50th=[ 95], 99.90th=[ 102], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 400, max= 682, per=24.86%, avg=502.25, stdev=78.31, samples=20 iops : min= 100, max= 170, avg=125.20, stdev=19.58, samples=20 lat (msec) : 10=0.12%, 20=0.27%, 50=18.09%, 100=81.29%, 250=0.23% cpu : usr=0.26%, sys=21.61%, ctx=1700, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=531172: Tue Apr 1 11:00:52 2025 read: IOPS=120, BW=483KiB/s (494kB/s)(4976KiB/10310msec) slat (usec): min=656, max=14701, avg=6503.59, stdev=1848.82 clat (msec): min=27, max=106, avg=61.40, stdev=13.11 lat (msec): min=31, max=112, avg=67.91, stdev=13.54 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 72], 90.00th=[ 79], 95.00th=[ 85], | 99.00th=[ 94], 99.50th=[ 96], 99.90th=[ 104], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 408, max= 576, per=24.84%, avg=482.05, stdev=47.38, samples=20 iops : min= 102, max= 144, avg=120.15, stdev=11.93, samples=20 write: IOPS=127, BW=511KiB/s (523kB/s)(5264KiB/10310msec); 0 zone resets slat (usec): min=121, max=3939, avg=634.67, stdev=301.81 clat (msec): min=18, max=117, avg=59.88, stdev=11.94 lat (msec): min=18, max=118, avg=60.52, stdev=11.98 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 82], | 99.00th=[ 91], 99.50th=[ 97], 99.90th=[ 112], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 392, max= 772, per=25.20%, avg=509.10, stdev=88.38, samples=20 iops : min= 98, max= 193, avg=126.95, stdev=22.25, samples=20 lat (msec) : 20=0.08%, 50=20.74%, 100=78.91%, 250=0.27% cpu : usr=0.23%, sys=21.77%, ctx=1731, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=531173: Tue Apr 1 11:00:52 2025 read: IOPS=120, BW=482KiB/s (494kB/s)(4984KiB/10338msec) slat (usec): min=890, max=17960, avg=6529.87, stdev=1993.37 clat (msec): min=7, max=109, avg=61.80, stdev=14.13 lat (msec): min=10, max=116, avg=68.33, stdev=14.62 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 81], 95.00th=[ 85], | 99.00th=[ 99], 99.50th=[ 101], 99.90th=[ 108], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 383, max= 685, per=24.74%, avg=480.15, stdev=68.89, samples=20 iops : min= 95, max= 171, avg=119.65, stdev=17.32, samples=20 write: IOPS=127, BW=508KiB/s (521kB/s)(5256KiB/10338msec); 0 zone resets slat (usec): min=148, max=2766, avg=623.92, stdev=256.87 clat (msec): min=7, max=104, avg=59.67, stdev=12.26 lat (msec): min=7, max=105, avg=60.30, stdev=12.30 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 75], 95.00th=[ 81], | 99.00th=[ 93], 99.50th=[ 97], 99.90th=[ 104], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 392, max= 620, per=24.75%, avg=499.90, stdev=72.94, samples=20 iops : min= 98, max= 155, avg=124.65, stdev=18.31, samples=20 lat (msec) : 10=0.23%, 20=0.39%, 50=20.00%, 100=79.06%, 250=0.31% cpu : usr=0.24%, sys=21.62%, ctx=1714, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=531174: Tue Apr 1 11:00:52 2025 read: IOPS=122, BW=490KiB/s (502kB/s)(5060KiB/10328msec) slat (usec): min=563, max=15833, avg=6450.14, stdev=1861.42 clat (msec): min=10, max=105, avg=61.22, stdev=12.66 lat (msec): min=16, max=112, avg=67.67, stdev=13.06 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 79], 95.00th=[ 83], | 99.00th=[ 93], 99.50th=[ 96], 99.90th=[ 102], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 400, max= 645, per=25.16%, avg=488.40, stdev=59.24, samples=20 iops : min= 100, max= 161, avg=121.75, stdev=14.80, samples=20 write: IOPS=125, BW=502KiB/s (514kB/s)(5180KiB/10328msec); 0 zone resets slat (usec): min=101, max=5575, avg=646.94, stdev=361.89 clat (msec): min=10, max=119, avg=60.19, stdev=12.25 lat (msec): min=10, max=119, avg=60.84, stdev=12.29 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 64], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 81], | 99.00th=[ 89], 99.50th=[ 92], 99.90th=[ 116], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 367, max= 629, per=24.73%, avg=499.55, stdev=67.20, samples=20 iops : min= 91, max= 157, avg=124.60, stdev=16.89, samples=20 lat (msec) : 20=0.12%, 50=19.77%, 100=79.84%, 250=0.27% cpu : usr=0.28%, sys=21.62%, ctx=1715, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1941KiB/s (1988kB/s), 482KiB/s-490KiB/s (494kB/s-502kB/s), io=19.6MiB (20.5MB), run=10310-10338msec WRITE: bw=2021KiB/s (2069kB/s), 502KiB/s-511KiB/s (514kB/s-523kB/s), io=20.4MiB (21.4MB), run=10310-10338msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=531179: Tue Apr 1 11:00:58 2025 read: IOPS=4, BW=182MiB/s (191MB/s)(40.0MiB/220msec) slat (nsec): min=64222k, max=64222k, avg=64221667.00, stdev= 0.00 clat (nsec): min=148619k, max=148619k, avg=148619187.00, stdev= 0.00 lat (nsec): min=212844k, max=212844k, avg=212843619.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=28.77%, ctx=3, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=182MiB/s (191MB/s), 182MiB/s-182MiB/s (191MB/s-191MB/s), io=40.0MiB (41.9MB), run=220-220msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout