== sanity test 398c: run fio to test AIO ================= 12:28:43 (1729096123) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0881097 s, 476 MB/s osc.lustre-OST0000-osc-ffff8801301d0000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=28629: Wed Oct 16 12:29:13 2024 write: IOPS=88, BW=354KiB/s (362kB/s)(10.0MiB/28955msec) slat (usec): min=12, max=634, avg=68.72, stdev=45.25 clat (msec): min=19, max=341, avg=180.79, stdev=42.67 lat (msec): min=20, max=341, avg=180.86, stdev=42.68 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 121], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 257], | 99.00th=[ 309], 99.50th=[ 321], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 342] bw ( KiB/s): min= 216, max= 656, per=25.01%, avg=352.35, stdev=70.69, samples=57 iops : min= 54, max= 164, avg=88.04, stdev=17.70, samples=57 lat (msec) : 20=0.04%, 50=0.66%, 100=1.64%, 250=91.72%, 500=5.94% cpu : usr=0.11%, sys=0.69%, ctx=2481, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28630: Wed Oct 16 12:29:13 2024 write: IOPS=88, BW=353KiB/s (361kB/s)(10.0MiB/29049msec) slat (usec): min=12, max=2300, avg=67.15, stdev=61.17 clat (msec): min=20, max=337, avg=181.26, stdev=41.66 lat (msec): min=20, max=337, avg=181.33, stdev=41.67 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 122], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 216, max= 552, per=24.91%, avg=350.97, stdev=62.71, samples=58 iops : min= 54, max= 138, avg=87.69, stdev=15.65, samples=58 lat (msec) : 50=0.12%, 100=1.99%, 250=92.03%, 500=5.86% cpu : usr=0.18%, sys=0.62%, ctx=2500, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28631: Wed Oct 16 12:29:13 2024 write: IOPS=88, BW=353KiB/s (362kB/s)(10.0MiB/28995msec) slat (usec): min=13, max=1250, avg=65.40, stdev=49.45 clat (msec): min=30, max=342, avg=180.99, stdev=42.93 lat (msec): min=30, max=342, avg=181.05, stdev=42.93 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 121], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 257], | 99.00th=[ 309], 99.50th=[ 321], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 342] bw ( KiB/s): min= 184, max= 640, per=24.95%, avg=351.53, stdev=72.74, samples=57 iops : min= 46, max= 160, avg=87.84, stdev=18.18, samples=57 lat (msec) : 50=0.62%, 100=1.33%, 250=91.37%, 500=6.68% cpu : usr=0.16%, sys=0.62%, ctx=2215, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28632: Wed Oct 16 12:29:13 2024 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29050msec) slat (usec): min=13, max=621, avg=67.50, stdev=41.93 clat (msec): min=30, max=342, avg=181.26, stdev=41.46 lat (msec): min=30, max=342, avg=181.33, stdev=41.46 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 123], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 192, max= 536, per=24.91%, avg=350.98, stdev=64.06, samples=58 iops : min= 48, max= 134, avg=87.71, stdev=16.02, samples=58 lat (msec) : 50=0.12%, 100=1.52%, 250=92.23%, 500=6.13% cpu : usr=0.13%, sys=0.67%, ctx=2537, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1410KiB/s (1444kB/s), 352KiB/s-354KiB/s (361kB/s-362kB/s), io=40.0MiB (41.9MB), run=28955-29050msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28665: Wed Oct 16 12:29:31 2024 read: IOPS=75, BW=302KiB/s (309kB/s)(5048KiB/16708msec) slat (usec): min=663, max=4764, avg=1199.98, stdev=291.16 clat (msec): min=2, max=123, avg=69.71, stdev=12.69 lat (msec): min=3, max=124, avg=70.91, stdev=12.73 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 105], 99.50th=[ 114], 99.90th=[ 123], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 216, max= 408, per=25.25%, avg=302.48, stdev=53.21, samples=33 iops : min= 54, max= 102, avg=75.58, stdev=13.31, samples=33 write: IOPS=77, BW=311KiB/s (318kB/s)(5192KiB/16708msec) slat (usec): min=13, max=293, avg=53.86, stdev=25.41 clat (msec): min=59, max=252, avg=136.51, stdev=28.49 lat (msec): min=59, max=252, avg=136.56, stdev=28.49 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 94], 10.00th=[ 102], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 128], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 176], 95.00th=[ 184], | 99.00th=[ 199], 99.50th=[ 205], 99.90th=[ 247], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 216, max= 424, per=24.90%, avg=310.48, stdev=38.67, samples=33 iops : min= 54, max= 106, avg=77.58, stdev= 9.68, samples=33 lat (msec) : 4=0.04%, 20=0.20%, 50=1.84%, 100=51.29%, 250=46.60% lat (msec) : 500=0.04% cpu : usr=0.19%, sys=1.38%, ctx=3541, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28666: Wed Oct 16 12:29:31 2024 read: IOPS=74, BW=298KiB/s (305kB/s)(4976KiB/16722msec) slat (usec): min=16, max=5171, avg=1218.47, stdev=334.84 clat (usec): min=1436, max=136738, avg=69665.07, stdev=13459.04 lat (msec): min=2, max=138, avg=70.89, stdev=13.50 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 91], | 99.00th=[ 112], 99.50th=[ 115], 99.90th=[ 132], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 200, max= 416, per=24.78%, avg=296.91, stdev=52.38, samples=33 iops : min= 50, max= 104, avg=74.18, stdev=13.07, samples=33 write: IOPS=78, BW=315KiB/s (322kB/s)(5264KiB/16722msec) slat (usec): min=13, max=226, avg=53.39, stdev=24.52 clat (msec): min=28, max=213, avg=136.09, stdev=28.77 lat (msec): min=28, max=213, avg=136.14, stdev=28.77 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 92], 10.00th=[ 101], 20.00th=[ 110], | 30.00th=[ 120], 40.00th=[ 129], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 174], 95.00th=[ 184], | 99.00th=[ 201], 99.50th=[ 205], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 248, max= 384, per=25.25%, avg=314.85, stdev=35.24, samples=33 iops : min= 62, max= 96, avg=78.67, stdev= 8.84, samples=33 lat (msec) : 2=0.08%, 4=0.04%, 10=0.04%, 20=0.20%, 50=1.56% lat (msec) : 100=50.59%, 250=47.50% cpu : usr=0.12%, sys=1.47%, ctx=3382, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28667: Wed Oct 16 12:29:31 2024 read: IOPS=74, BW=298KiB/s (305kB/s)(4984KiB/16741msec) slat (usec): min=683, max=3138, avg=1208.44, stdev=280.60 clat (usec): min=1457, max=119888, avg=69103.81, stdev=12768.42 lat (msec): min=2, max=120, avg=70.31, stdev=12.79 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 60], | 30.00th=[ 63], 40.00th=[ 65], 50.00th=[ 69], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 84], 95.00th=[ 89], | 99.00th=[ 106], 99.50th=[ 114], 99.90th=[ 120], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 208, max= 464, per=24.93%, avg=298.64, stdev=58.95, samples=33 iops : min= 52, max= 116, avg=74.64, stdev=14.71, samples=33 write: IOPS=78, BW=314KiB/s (321kB/s)(5256KiB/16741msec) slat (usec): min=13, max=288, avg=52.90, stdev=23.86 clat (msec): min=66, max=211, avg=136.76, stdev=28.45 lat (msec): min=66, max=211, avg=136.82, stdev=28.45 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 111], | 30.00th=[ 121], 40.00th=[ 128], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 163], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 199], 99.50th=[ 207], 99.90th=[ 209], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 240, max= 384, per=24.98%, avg=311.48, stdev=32.39, samples=33 iops : min= 60, max= 96, avg=77.85, stdev= 8.12, samples=33 lat (msec) : 2=0.04%, 20=0.23%, 50=2.19%, 100=50.78%, 250=46.76% cpu : usr=0.17%, sys=1.38%, ctx=3513, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28668: Wed Oct 16 12:29:31 2024 read: IOPS=75, BW=304KiB/s (311kB/s)(5060KiB/16672msec) slat (usec): min=723, max=4215, avg=1213.22, stdev=308.60 clat (usec): min=1402, max=154670, avg=69653.75, stdev=12921.71 lat (msec): min=2, max=155, avg=70.87, stdev=12.94 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 75], 80.00th=[ 79], 90.00th=[ 84], 95.00th=[ 90], | 99.00th=[ 111], 99.50th=[ 121], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 216, max= 456, per=25.43%, avg=304.70, stdev=52.43, samples=33 iops : min= 54, max= 114, avg=76.15, stdev=13.11, samples=33 write: IOPS=77, BW=311KiB/s (318kB/s)(5180KiB/16672msec) slat (usec): min=14, max=232, avg=53.19, stdev=26.08 clat (msec): min=61, max=297, avg=136.24, stdev=30.03 lat (msec): min=61, max=297, avg=136.29, stdev=30.03 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 92], 10.00th=[ 102], 20.00th=[ 111], | 30.00th=[ 118], 40.00th=[ 126], 50.00th=[ 134], 60.00th=[ 142], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 178], 95.00th=[ 186], | 99.00th=[ 205], 99.50th=[ 247], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 216, max= 384, per=24.86%, avg=310.03, stdev=34.22, samples=33 iops : min= 54, max= 96, avg=77.48, stdev= 8.54, samples=33 lat (msec) : 2=0.04%, 20=0.12%, 50=1.41%, 100=51.88%, 250=46.33% lat (msec) : 500=0.23% cpu : usr=0.22%, sys=1.35%, ctx=3512, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1199KiB/s (1228kB/s), 298KiB/s-304KiB/s (305kB/s-311kB/s), io=19.6MiB (20.5MB), run=16672-16741msec WRITE: bw=1248KiB/s (1278kB/s), 311KiB/s-315KiB/s (318kB/s-322kB/s), io=20.4MiB (21.4MB), run=16672-16741msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=28685: Wed Oct 16 12:29:32 2024 read: IOPS=11, BW=471MiB/s (493MB/s)(40.0MiB/85msec) slat (nsec): min=6210.3k, max=6210.3k, avg=6210271.00, stdev= 0.00 clat (nsec): min=72352k, max=72352k, avg=72351711.00, stdev= 0.00 lat (nsec): min=78573k, max=78573k, avg=78572532.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[71828], 5.00th=[71828], 10.00th=[71828], 20.00th=[71828], | 30.00th=[71828], 40.00th=[71828], 50.00th=[71828], 60.00th=[71828], | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.14%, ctx=16, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=471MiB/s (493MB/s), 471MiB/s-471MiB/s (493MB/s-493MB/s), io=40.0MiB (41.9MB), run=85-85msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout