== sanity test 398c: run fio to test AIO ================= 05:27:43 (1727774863) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.099127 s, 423 MB/s osc.lustre-OST0000-osc-ffff8800b6e51800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=19453: Tue Oct 1 05:28:11 2024 write: IOPS=93, BW=376KiB/s (385kB/s)(10.0MiB/27244msec) slat (usec): min=13, max=3007, avg=87.36, stdev=85.81 clat (msec): min=14, max=303, avg=170.10, stdev=33.14 lat (msec): min=14, max=303, avg=170.19, stdev=33.14 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 116], 10.00th=[ 132], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 188], 80.00th=[ 199], 90.00th=[ 211], 95.00th=[ 222], | 99.00th=[ 241], 99.50th=[ 255], 99.90th=[ 296], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 288, max= 664, per=25.03%, avg=375.48, stdev=57.80, samples=54 iops : min= 72, max= 166, avg=93.81, stdev=14.47, samples=54 lat (msec) : 20=0.08%, 50=0.20%, 100=2.93%, 250=96.17%, 500=0.62% cpu : usr=0.21%, sys=0.86%, ctx=2535, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19454: Tue Oct 1 05:28:11 2024 write: IOPS=93, BW=375KiB/s (384kB/s)(10.0MiB/27289msec) slat (usec): min=14, max=1771, avg=87.70, stdev=69.45 clat (msec): min=80, max=271, avg=170.32, stdev=30.98 lat (msec): min=80, max=271, avg=170.41, stdev=30.98 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 120], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 197], 90.00th=[ 211], 95.00th=[ 222], | 99.00th=[ 241], 99.50th=[ 249], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 271] bw ( KiB/s): min= 288, max= 544, per=24.95%, avg=374.30, stdev=51.56, samples=54 iops : min= 72, max= 136, avg=93.52, stdev=12.91, samples=54 lat (msec) : 100=2.27%, 250=97.27%, 500=0.47% cpu : usr=0.24%, sys=0.80%, ctx=2592, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19455: Tue Oct 1 05:28:11 2024 write: IOPS=93, BW=375KiB/s (384kB/s)(10.0MiB/27289msec) slat (usec): min=13, max=4794, avg=88.78, stdev=120.67 clat (msec): min=68, max=271, avg=170.30, stdev=30.92 lat (msec): min=69, max=274, avg=170.40, stdev=30.93 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 163], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 241], 99.50th=[ 249], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 280, max= 528, per=24.93%, avg=374.00, stdev=51.51, samples=54 iops : min= 70, max= 132, avg=93.44, stdev=12.91, samples=54 lat (msec) : 100=2.07%, 250=97.50%, 500=0.43% cpu : usr=0.28%, sys=0.78%, ctx=2605, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19456: Tue Oct 1 05:28:11 2024 write: IOPS=93, BW=375KiB/s (384kB/s)(10.0MiB/27277msec) slat (usec): min=15, max=1735, avg=88.93, stdev=68.21 clat (msec): min=12, max=276, avg=170.27, stdev=32.43 lat (msec): min=13, max=276, avg=170.36, stdev=32.43 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 117], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 197], 90.00th=[ 211], 95.00th=[ 222], | 99.00th=[ 245], 99.50th=[ 253], 99.90th=[ 264], 99.95th=[ 271], | 99.99th=[ 275] bw ( KiB/s): min= 296, max= 624, per=24.98%, avg=374.67, stdev=55.64, samples=54 iops : min= 74, max= 156, avg=93.56, stdev=13.96, samples=54 lat (msec) : 20=0.08%, 50=0.04%, 100=2.66%, 250=96.52%, 500=0.70% cpu : usr=0.21%, sys=0.84%, ctx=2682, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1501KiB/s (1537kB/s), 375KiB/s-376KiB/s (384kB/s-385kB/s), io=40.0MiB (41.9MB), run=27244-27289msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19488: Tue Oct 1 05:28:29 2024 read: IOPS=75, BW=304KiB/s (311kB/s)(5048KiB/16618msec) slat (usec): min=758, max=3738, avg=1284.30, stdev=329.79 clat (usec): min=1410, max=141490, avg=70514.10, stdev=12969.51 lat (msec): min=2, max=142, avg=71.80, stdev=12.98 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 91], | 99.00th=[ 99], 99.50th=[ 108], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 216, max= 376, per=25.22%, avg=303.67, stdev=42.85, samples=33 iops : min= 54, max= 94, avg=75.85, stdev=10.73, samples=33 write: IOPS=78, BW=312KiB/s (320kB/s)(5192KiB/16618msec) slat (usec): min=15, max=310, avg=64.40, stdev=35.71 clat (msec): min=39, max=230, avg=134.64, stdev=31.57 lat (msec): min=39, max=231, avg=134.71, stdev=31.57 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 87], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 116], 40.00th=[ 127], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 211], 99.50th=[ 215], 99.90th=[ 230], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 247, max= 400, per=24.83%, avg=311.42, stdev=32.71, samples=33 iops : min= 61, max= 100, avg=77.79, stdev= 8.21, samples=33 lat (msec) : 2=0.08%, 20=0.27%, 50=1.56%, 100=54.38%, 250=43.71% cpu : usr=0.24%, sys=1.67%, ctx=2991, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19489: Tue Oct 1 05:28:29 2024 read: IOPS=74, BW=299KiB/s (306kB/s)(4976KiB/16641msec) slat (usec): min=733, max=3036, avg=1266.03, stdev=292.97 clat (msec): min=11, max=130, avg=70.65, stdev=12.01 lat (msec): min=11, max=131, avg=71.92, stdev=12.02 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 86], 95.00th=[ 90], | 99.00th=[ 101], 99.50th=[ 108], 99.90th=[ 130], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 192, max= 400, per=24.74%, avg=297.85, stdev=54.87, samples=33 iops : min= 48, max= 100, avg=74.39, stdev=13.77, samples=33 write: IOPS=79, BW=316KiB/s (324kB/s)(5264KiB/16641msec) slat (usec): min=14, max=366, avg=65.07, stdev=36.88 clat (msec): min=44, max=230, avg=134.04, stdev=32.09 lat (msec): min=45, max=230, avg=134.11, stdev=32.09 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 83], 10.00th=[ 92], 20.00th=[ 105], | 30.00th=[ 116], 40.00th=[ 128], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 163], 90.00th=[ 174], 95.00th=[ 184], | 99.00th=[ 211], 99.50th=[ 213], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 248, max= 376, per=25.16%, avg=315.55, stdev=31.77, samples=33 iops : min= 62, max= 94, avg=78.82, stdev= 7.96, samples=33 lat (msec) : 20=0.12%, 50=1.56%, 100=54.69%, 250=43.63% cpu : usr=0.22%, sys=1.68%, ctx=2993, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19490: Tue Oct 1 05:28:29 2024 read: IOPS=74, BW=299KiB/s (306kB/s)(4984KiB/16659msec) slat (usec): min=729, max=3081, avg=1269.13, stdev=305.05 clat (usec): min=1841, max=108208, avg=70495.25, stdev=12860.74 lat (msec): min=3, max=109, avg=71.77, stdev=12.87 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 91], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 108], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 215, max= 472, per=24.84%, avg=299.06, stdev=59.74, samples=33 iops : min= 53, max= 118, avg=74.70, stdev=14.97, samples=33 write: IOPS=78, BW=316KiB/s (323kB/s)(5256KiB/16659msec) slat (usec): min=16, max=270, avg=65.79, stdev=36.08 clat (msec): min=40, max=232, avg=134.41, stdev=31.61 lat (msec): min=40, max=232, avg=134.48, stdev=31.61 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 84], 10.00th=[ 93], 20.00th=[ 107], | 30.00th=[ 118], 40.00th=[ 128], 50.00th=[ 134], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 161], 90.00th=[ 174], 95.00th=[ 186], | 99.00th=[ 213], 99.50th=[ 222], 99.90th=[ 228], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 256, max= 392, per=24.93%, avg=312.64, stdev=37.34, samples=33 iops : min= 64, max= 98, avg=78.09, stdev= 9.35, samples=33 lat (msec) : 2=0.04%, 20=0.39%, 50=1.76%, 100=53.52%, 250=44.30% cpu : usr=0.27%, sys=1.63%, ctx=2919, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19491: Tue Oct 1 05:28:29 2024 read: IOPS=76, BW=305KiB/s (312kB/s)(5060KiB/16598msec) slat (usec): min=67, max=3239, avg=1269.33, stdev=317.79 clat (usec): min=1282, max=198863, avg=70742.14, stdev=14531.75 lat (msec): min=2, max=201, avg=72.01, stdev=14.56 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 78], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 90], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 165], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 183, max= 432, per=25.36%, avg=305.36, stdev=57.38, samples=33 iops : min= 45, max= 108, avg=76.27, stdev=14.39, samples=33 write: IOPS=78, BW=312KiB/s (320kB/s)(5180KiB/16598msec) slat (usec): min=14, max=293, avg=65.50, stdev=35.95 clat (msec): min=16, max=265, avg=134.47, stdev=33.71 lat (msec): min=16, max=265, avg=134.54, stdev=33.71 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 83], 10.00th=[ 92], 20.00th=[ 106], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 220], 99.50th=[ 230], 99.90th=[ 253], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 248, max= 376, per=24.83%, avg=311.42, stdev=31.57, samples=33 iops : min= 62, max= 94, avg=77.79, stdev= 7.91, samples=33 lat (msec) : 2=0.04%, 10=0.23%, 20=0.59%, 50=1.84%, 100=54.10% lat (msec) : 250=42.97%, 500=0.23% cpu : usr=0.32%, sys=1.61%, ctx=2948, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1205KiB/s (1234kB/s), 299KiB/s-305KiB/s (306kB/s-312kB/s), io=19.6MiB (20.5MB), run=16598-16659msec WRITE: bw=1254KiB/s (1284kB/s), 312KiB/s-316KiB/s (320kB/s-324kB/s), io=20.4MiB (21.4MB), run=16598-16659msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=19508: Tue Oct 1 05:28:30 2024 read: IOPS=12, BW=519MiB/s (545MB/s)(40.0MiB/77msec) slat (nsec): min=8306.7k, max=8306.7k, avg=8306701.00, stdev= 0.00 clat (nsec): min=64392k, max=64392k, avg=64391687.00, stdev= 0.00 lat (nsec): min=72721k, max=72721k, avg=72720991.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[64226], 5.00th=[64226], 10.00th=[64226], 20.00th=[64226], | 30.00th=[64226], 40.00th=[64226], 50.00th=[64226], 60.00th=[64226], | 70.00th=[64226], 80.00th=[64226], 90.00th=[64226], 95.00th=[64226], | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=10.53%, ctx=16, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=40.0MiB (41.9MB), run=77-77msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout