== sanity test 398c: run fio to test AIO ================= 03:20:47 (1734682847) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.130197 s, 322 MB/s osc.lustre-OST0000-osc-ffff8800b6cff800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=28101: Fri Dec 20 03:22:01 2024 write: IOPS=35, BW=143KiB/s (146kB/s)(10.0MiB/71706msec) slat (usec): min=13, max=1790, avg=113.70, stdev=169.43 clat (msec): min=55, max=870, avg=447.29, stdev=116.47 lat (msec): min=55, max=870, avg=447.41, stdev=116.48 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 288], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 510], 80.00th=[ 558], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 760], 99.50th=[ 785], 99.90th=[ 835], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 64, max= 248, per=24.90%, avg=142.18, stdev=38.75, samples=143 iops : min= 16, max= 62, avg=35.40, stdev= 9.67, samples=143 lat (msec) : 100=0.27%, 250=0.98%, 500=67.77%, 750=29.77%, 1000=1.21% cpu : usr=0.17%, sys=0.56%, ctx=1712, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28102: Fri Dec 20 03:22:01 2024 write: IOPS=35, BW=143KiB/s (147kB/s)(10.0MiB/71530msec) slat (usec): min=10, max=3168, avg=106.13, stdev=177.35 clat (msec): min=55, max=850, avg=446.47, stdev=118.28 lat (msec): min=55, max=850, avg=446.57, stdev=118.29 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 288], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 659], | 99.00th=[ 760], 99.50th=[ 785], 99.90th=[ 835], 99.95th=[ 844], | 99.99th=[ 852] bw ( KiB/s): min= 48, max= 256, per=24.89%, avg=142.13, stdev=39.38, samples=143 iops : min= 12, max= 64, avg=35.35, stdev= 9.90, samples=143 lat (msec) : 100=0.62%, 250=0.59%, 500=68.05%, 750=29.53%, 1000=1.21% cpu : usr=0.09%, sys=0.59%, ctx=1553, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28103: Fri Dec 20 03:22:01 2024 write: IOPS=35, BW=143KiB/s (146kB/s)(10.0MiB/71659msec) slat (usec): min=12, max=2052, avg=117.29, stdev=171.53 clat (msec): min=56, max=880, avg=446.99, stdev=116.56 lat (msec): min=56, max=880, avg=447.11, stdev=116.57 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 288], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 760], 99.50th=[ 793], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 877] bw ( KiB/s): min= 63, max= 248, per=24.89%, avg=142.11, stdev=38.70, samples=143 iops : min= 15, max= 62, avg=35.33, stdev= 9.65, samples=143 lat (msec) : 100=0.27%, 250=0.94%, 500=67.81%, 750=29.73%, 1000=1.25% cpu : usr=0.11%, sys=0.64%, ctx=1671, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28104: Fri Dec 20 03:22:01 2024 write: IOPS=35, BW=143KiB/s (146kB/s)(10.0MiB/71729msec) slat (usec): min=11, max=2029, avg=114.61, stdev=188.71 clat (msec): min=56, max=881, avg=447.46, stdev=115.84 lat (msec): min=56, max=881, avg=447.58, stdev=115.85 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 292], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 506], 80.00th=[ 558], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 760], 99.50th=[ 785], 99.90th=[ 844], 99.95th=[ 852], | 99.99th=[ 885] bw ( KiB/s): min= 48, max= 256, per=24.91%, avg=142.22, stdev=39.51, samples=143 iops : min= 12, max= 64, avg=35.43, stdev= 9.91, samples=143 lat (msec) : 100=0.12%, 250=0.86%, 500=68.48%, 750=29.18%, 1000=1.37% cpu : usr=0.11%, sys=0.59%, ctx=1607, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=571KiB/s (585kB/s), 143KiB/s-143KiB/s (146kB/s-147kB/s), io=40.0MiB (41.9MB), run=71530-71729msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28171: Fri Dec 20 03:22:37 2024 read: IOPS=35, BW=142KiB/s (145kB/s)(5048KiB/35569msec) slat (usec): min=989, max=12765, avg=2644.70, stdev=1267.32 clat (msec): min=4, max=501, avg=144.77, stdev=48.06 lat (msec): min=6, max=505, avg=147.42, stdev=48.62 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 88], 10.00th=[ 96], 20.00th=[ 106], | 30.00th=[ 117], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 178], 90.00th=[ 207], 95.00th=[ 232], | 99.00th=[ 292], 99.50th=[ 321], 99.90th=[ 405], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 40, max= 240, per=25.08%, avg=141.23, stdev=42.19, samples=71 iops : min= 10, max= 60, avg=35.17, stdev=10.56, samples=71 write: IOPS=36, BW=146KiB/s (149kB/s)(5192KiB/35569msec) slat (usec): min=15, max=1044, avg=96.43, stdev=102.62 clat (msec): min=117, max=658, avg=294.10, stdev=91.40 lat (msec): min=117, max=658, avg=294.20, stdev=91.40 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 167], 10.00th=[ 186], 20.00th=[ 218], | 30.00th=[ 243], 40.00th=[ 262], 50.00th=[ 279], 60.00th=[ 305], | 70.00th=[ 330], 80.00th=[ 368], 90.00th=[ 414], 95.00th=[ 460], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 651], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 55, max= 312, per=24.77%, avg=145.17, stdev=45.52, samples=71 iops : min= 13, max= 78, avg=36.15, stdev=11.43, samples=71 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=7.27%, 250=58.09% lat (msec) : 500=32.85%, 750=1.60% cpu : usr=0.10%, sys=1.66%, ctx=3288, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28172: Fri Dec 20 03:22:37 2024 read: IOPS=34, BW=140KiB/s (143kB/s)(4976KiB/35593msec) slat (usec): min=990, max=11779, avg=2610.37, stdev=1286.36 clat (msec): min=4, max=309, avg=144.01, stdev=43.55 lat (msec): min=6, max=313, avg=146.63, stdev=44.13 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 88], 10.00th=[ 96], 20.00th=[ 110], | 30.00th=[ 121], 40.00th=[ 131], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 176], 90.00th=[ 203], 95.00th=[ 228], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 55, max= 248, per=24.76%, avg=139.38, stdev=47.35, samples=71 iops : min= 13, max= 62, avg=34.68, stdev=11.85, samples=71 write: IOPS=36, BW=148KiB/s (151kB/s)(5264KiB/35593msec) slat (usec): min=15, max=1514, avg=101.19, stdev=121.73 clat (msec): min=124, max=625, avg=293.13, stdev=89.72 lat (msec): min=125, max=625, avg=293.23, stdev=89.74 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 163], 10.00th=[ 188], 20.00th=[ 218], | 30.00th=[ 239], 40.00th=[ 259], 50.00th=[ 275], 60.00th=[ 305], | 70.00th=[ 334], 80.00th=[ 368], 90.00th=[ 418], 95.00th=[ 460], | 99.00th=[ 550], 99.50th=[ 584], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 72, max= 264, per=25.07%, avg=146.93, stdev=42.98, samples=71 iops : min= 18, max= 66, avg=36.56, stdev=10.75, samples=71 lat (msec) : 10=0.08%, 20=0.08%, 50=0.23%, 100=6.09%, 250=58.98% lat (msec) : 500=33.36%, 750=1.17% cpu : usr=0.38%, sys=1.42%, ctx=3058, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28173: Fri Dec 20 03:22:37 2024 read: IOPS=35, BW=140KiB/s (143kB/s)(4984KiB/35593msec) slat (usec): min=22, max=11477, avg=2662.63, stdev=1313.78 clat (msec): min=15, max=425, avg=144.88, stdev=45.29 lat (msec): min=17, max=428, avg=147.55, stdev=45.85 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 86], 10.00th=[ 96], 20.00th=[ 109], | 30.00th=[ 120], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 178], 90.00th=[ 207], 95.00th=[ 230], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 422], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 56, max= 256, per=24.84%, avg=139.87, stdev=45.39, samples=71 iops : min= 14, max= 64, avg=34.83, stdev=11.32, samples=71 write: IOPS=36, BW=148KiB/s (151kB/s)(5256KiB/35593msec) slat (usec): min=13, max=2195, avg=110.33, stdev=149.23 clat (msec): min=60, max=657, avg=292.96, stdev=90.50 lat (msec): min=60, max=657, avg=293.07, stdev=90.51 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 163], 10.00th=[ 186], 20.00th=[ 218], | 30.00th=[ 243], 40.00th=[ 262], 50.00th=[ 279], 60.00th=[ 305], | 70.00th=[ 334], 80.00th=[ 363], 90.00th=[ 418], 95.00th=[ 460], | 99.00th=[ 558], 99.50th=[ 584], 99.90th=[ 617], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 55, max= 263, per=25.00%, avg=146.52, stdev=43.56, samples=71 iops : min= 13, max= 65, avg=36.49, stdev=10.95, samples=71 lat (msec) : 20=0.04%, 50=0.08%, 100=6.37%, 250=58.48%, 500=33.59% lat (msec) : 750=1.45% cpu : usr=0.49%, sys=1.39%, ctx=3271, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28174: Fri Dec 20 03:22:37 2024 read: IOPS=35, BW=142KiB/s (146kB/s)(5060KiB/35593msec) slat (usec): min=906, max=13790, avg=2647.19, stdev=1311.91 clat (msec): min=3, max=465, avg=144.37, stdev=44.62 lat (msec): min=5, max=475, avg=147.02, stdev=45.25 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 110], | 30.00th=[ 120], 40.00th=[ 129], 50.00th=[ 136], 60.00th=[ 146], | 70.00th=[ 159], 80.00th=[ 176], 90.00th=[ 201], 95.00th=[ 224], | 99.00th=[ 279], 99.50th=[ 309], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 24, max= 263, per=25.17%, avg=141.69, stdev=47.43, samples=71 iops : min= 6, max= 65, avg=35.30, stdev=11.84, samples=71 write: IOPS=36, BW=146KiB/s (149kB/s)(5180KiB/35593msec) slat (usec): min=12, max=1755, avg=101.89, stdev=139.40 clat (msec): min=123, max=676, avg=295.10, stdev=90.63 lat (msec): min=123, max=676, avg=295.21, stdev=90.64 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 167], 10.00th=[ 186], 20.00th=[ 215], | 30.00th=[ 243], 40.00th=[ 262], 50.00th=[ 284], 60.00th=[ 309], | 70.00th=[ 338], 80.00th=[ 363], 90.00th=[ 414], 95.00th=[ 460], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 659], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 71, max= 256, per=24.70%, avg=144.73, stdev=38.55, samples=71 iops : min= 17, max= 64, avg=36.06, stdev= 9.67, samples=71 lat (msec) : 4=0.04%, 50=0.08%, 100=5.43%, 250=59.49%, 500=33.40% lat (msec) : 750=1.56% cpu : usr=0.44%, sys=1.35%, ctx=3124, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=564KiB/s (577kB/s), 140KiB/s-142KiB/s (143kB/s-146kB/s), io=19.6MiB (20.5MB), run=35569-35593msec WRITE: bw=587KiB/s (601kB/s), 146KiB/s-148KiB/s (149kB/s-151kB/s), io=20.4MiB (21.4MB), run=35569-35593msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=28206: Fri Dec 20 03:22:39 2024 read: IOPS=5, BW=215MiB/s (226MB/s)(40.0MiB/186msec) slat (nsec): min=8634.1k, max=8634.1k, avg=8634115.00, stdev= 0.00 clat (nsec): min=168137k, max=168137k, avg=168136605.00, stdev= 0.00 lat (nsec): min=176794k, max=176794k, avg=176794495.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.78%, ctx=38, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=215MiB/s (226MB/s), 215MiB/s-215MiB/s (226MB/s-226MB/s), io=40.0MiB (41.9MB), run=186-186msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout