== sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 01:38:59 (1788673139)
On MGS 192.168.206.158, active = nodemap.active=1
On MGS 192.168.206.158, TBF = 
ost.OSS.ost_io.nrs_policies=tbf nodemap
ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.515136 s, 32.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142852 s, 7.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170205 s, 6.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157328 s, 6.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148132 s, 7.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191687 s, 5.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155326 s, 6.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218055 s, 4.8 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17633 s, 5.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253388 s, 4.1 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194301 s, 5.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235675 s, 4.4 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242958 s, 4.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221861 s, 4.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21845 s, 4.8 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242481 s, 4.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35223 s, 3.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00819167 s, 128 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642009 s, 16.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318135 s, 33.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486008 s, 21.6 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131522 s, 8.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280579 s, 37.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11444 s, 9.2 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8d205af0d800.lock_unused_count=2
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588274 s, 17.8 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682144 s, 15.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516462 s, 20.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588522 s, 178 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447769 s, 23.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434536 s, 24.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397976 s, 26.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300957 s, 34.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175637 s, 59.7 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg658-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg658-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 0
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 5.35311 s, 19.6 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 5.13684 s, 20.4 MB/s
Read runtime is 7 s, speed is 14.285714 IOPS
cleanup_tbf /mnt/lustre/d77kj.sanityn
Waiting for MDT destroys to complete
On MGS 192.168.206.158, TBF.id = 
On MGS 192.168.206.158, active = nodemap.active=0
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.319625 s, 52.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685304 s, 15.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683663 s, 15.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766044 s, 13.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765784 s, 13.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699494 s, 15.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119142 s, 8.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119503 s, 8.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812839 s, 12.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826745 s, 12.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168645 s, 6.2 MB/s
ldlm.namespaces.lustre-OST0001-osc-ffff8d205af0d800.lock_unused_count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17957 s, 5.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160101 s, 6.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178407 s, 5.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183456 s, 5.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228057 s, 4.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214571 s, 4.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542045 s, 19.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487527 s, 21.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545185 s, 19.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272133 s, 38.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312146 s, 33.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575102 s, 18.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351349 s, 29.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397779 s, 26.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644567 s, 16.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461905 s, 22.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514628 s, 20.4 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617288 s, 17.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615568 s, 17.0 MB/s

1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649222 s, 16.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671674 s, 15.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783393 s, 13.4 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8d205af0d800.lock_unused_count=5
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg658-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg658-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 3.23098 s, 32.5 MB/s
Write runtime is 4 s, speed is 25.000000 IOPS
write rate (25.000000) should exceed limit (20 * 1)
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 2.83791 s, 36.9 MB/s
Read runtime is 4 s, speed is 25.000000 IOPS
read rate (25.000000) should exceed limit (20 * 1)
cleanup_tbf /mnt/lustre/d77kj.sanityn
Waiting for MDT destroys to complete
ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw
ost.OSS.ost_io.nrs_policies=fifo
ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode
oleg658-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument
oleg658-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument
oleg658-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument
oleg658-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument
pdsh@oleg658-client: oleg658-server: ssh exited with exit code 22
On MGS 192.168.206.158, active = nodemap.active=1
On MGS 192.168.206.158, TBF = 
ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.411966 s, 40.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881122 s, 11.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581469 s, 18.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128807 s, 8.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124299 s, 8.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142652 s, 7.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165731 s, 6.3 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149191 s, 7.0 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212935 s, 4.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192448 s, 5.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21234 s, 4.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2555 s, 4.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202587 s, 5.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187493 s, 5.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250279 s, 4.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236099 s, 4.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205175 s, 5.1 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8d205af0d800.lock_unused_count=1
ldlm.namespaces.lustre-OST0001-osc-ffff8d205af0d800.lock_unused_count=6
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00202594 s, 518 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533859 s, 19.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379198 s, 277 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00810244 s, 129 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653162 s, 16.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582234 s, 18.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817598 s, 12.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103445 s, 10.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738987 s, 14.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122665 s, 8.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139424 s, 7.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504247 s, 20.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714804 s, 14.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419723 s, 25.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361558 s, 29.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783302 s, 13.4 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg658-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg658-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 0
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 5.28214 s, 19.9 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 9.95077 s, 10.5 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77kj.sanityn
Waiting for MDT destroys to complete
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_policies=fifo
On MGS 192.168.206.158, TBF.id = 
On MGS 192.168.206.158, active = nodemap.active=0