== sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 12:06:59 (1788365219)
On MGS 192.168.202.134, active = nodemap.active=1
On MGS 192.168.202.134, TBF = 
ost.OSS.ost_io.nrs_policies=tbf nodemap
ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.145354 s, 115 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627041 s, 16.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152976 s, 6.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134071 s, 7.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19075 s, 5.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200028 s, 5.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176549 s, 5.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196195 s, 5.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193649 s, 5.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212377 s, 4.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236735 s, 4.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21732 s, 4.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213696 s, 4.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229491 s, 4.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215704 s, 4.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234298 s, 4.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230836 s, 4.5 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffffa04b3fb0d000.lock_unused_count=1
ldlm.namespaces.lustre-OST0001-osc-ffffa04b3fb0d000.lock_unused_count=4
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491815 s, 21.3 MB/s1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out

1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756696 s, 13.9 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534738 s, 19.6 MB/s1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521791 s, 20.1 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510635 s, 20.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605706 s, 17.3 MB/s

1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550959 s, 19.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596487 s, 17.6 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619594 s, 16.9 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595058 s, 17.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060503 s, 17.3 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629705 s, 16.7 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601662 s, 17.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594336 s, 17.6 MB/s

1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758422 s, 13.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622221 s, 16.9 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg234-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg234-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 0
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 4.89408 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 4.96931 s, 21.1 MB/s
Read runtime is 6 s, speed is 16.666666 IOPS
cleanup_tbf /mnt/lustre/d77kj.sanityn
Waiting for MDT destroys to complete
On MGS 192.168.202.134, TBF.id = 
On MGS 192.168.202.134, active = nodemap.active=0
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.162091 s, 104 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037677 s, 27.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820017 s, 12.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896746 s, 11.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121978 s, 8.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129448 s, 8.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120433 s, 8.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13393 s, 7.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115913 s, 9.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114016 s, 9.2 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129392 s, 8.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110245 s, 9.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128566 s, 8.2 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115217 s, 9.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134568 s, 7.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133557 s, 7.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125453 s, 8.4 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119491 s, 87.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128779 s, 81.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021919 s, 47.8 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150885 s, 69.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127407 s, 82.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0034102 s, 307 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152043 s, 69.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024855 s, 42.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273793 s, 38.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201758 s, 52.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237495 s, 44.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00301611 s, 348 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187399 s, 56.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355827 s, 29.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189499 s, 55.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00491442 s, 213 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffffa04b3fb0d000.lock_unused_count=1
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg234-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg234-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.777745 s, 135 MB/s
Write runtime is 2 s, speed is 50.000000 IOPS
write rate (50.000000) should exceed limit (20 * 1)
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.548276 s, 191 MB/s
Read runtime is 2 s, speed is 50.000000 IOPS
read rate (50.000000) should exceed limit (20 * 1)
cleanup_tbf /mnt/lustre/d77kj.sanityn
Waiting for MDT destroys to complete
ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw
ost.OSS.ost_io.nrs_policies=fifo
ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode
oleg234-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument
oleg234-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument
oleg234-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument
oleg234-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument
pdsh@oleg234-client: oleg234-server: ssh exited with exit code 22
On MGS 192.168.202.134, active = nodemap.active=1
On MGS 192.168.202.134, TBF = 
ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.15616 s, 107 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207095 s, 50.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044671 s, 23.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567323 s, 18.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614688 s, 17.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635652 s, 16.5 MB/s
ldlm.namespaces.lustre-OST0001-osc-ffffa04b3fb0d000.lock_unused_count=2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517965 s, 20.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892851 s, 11.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872616 s, 12.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113064 s, 9.3 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1315 s, 8.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125428 s, 8.4 MB/s

1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164915 s, 6.4 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161926 s, 6.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156793 s, 6.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172782 s, 6.1 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out
ldlm.namespaces.lustre-OST0000-osc-ffffa04b3fb0d000.lock_unused_count=5
ldlm.namespaces.lustre-OST0001-osc-ffffa04b3fb0d000.lock_unused_count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00410828 s, 255 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156084 s, 67.2 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164716 s, 6.4 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108237 s, 96.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00521537 s, 201 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467651 s, 224 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00521479 s, 201 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00757222 s, 138 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448908 s, 234 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430293 s, 244 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00471128 s, 223 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241148 s, 43.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170681 s, 61.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015755 s, 66.6 MB/s1+0 records in
1+0 records out

1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411333 s, 25.5 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119944 s, 87.4 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119865 s, 87.5 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg234-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg234-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 0
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 4.89323 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 9.73348 s, 10.8 MB/s
Read runtime is 10 s, speed is 10.000000 IOPS
cleanup_tbf /mnt/lustre/d77kj.sanityn
Waiting for MDT destroys to complete
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_policies=fifo
On MGS 192.168.202.134, TBF.id = 
On MGS 192.168.202.134, active = nodemap.active=0