== sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 18:37:43 (1790980663)
On MGS 192.168.204.108, active = nodemap.active=1
On MGS 192.168.204.108, TBF = 
ost.OSS.ost_io.nrs_policies=tbf nodemap
ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.687564 s, 24.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119397 s, 8.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146658 s, 7.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172065 s, 6.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19448 s, 5.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201017 s, 5.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192832 s, 5.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156178 s, 6.7 MB/s
ldlm.namespaces.lustre-OST0001-osc-ffff989583fd2800.lock_unused_count=5
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26228 s, 4.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258174 s, 4.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264131 s, 4.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295039 s, 3.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384178 s, 2.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338094 s, 3.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287494 s, 3.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346913 s, 3.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431535 s, 2.4 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff989583fd2800.lock_unused_count=7
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450119 s, 2.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448987 s, 2.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45124 s, 2.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453134 s, 2.3 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460868 s, 2.3 MB/s1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487498 s, 2.2 MB/s

1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495152 s, 2.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486895 s, 2.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483051 s, 2.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.502338 s, 2.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488978 s, 2.1 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.542397 s, 1.9 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.506171 s, 2.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488448 s, 2.1 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488048 s, 2.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.515303 s, 2.0 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg408-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg408-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 0
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 9.20079 s, 11.4 MB/s
Write runtime is 10 s, speed is 10.000000 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 5.37694 s, 19.5 MB/s
Read runtime is 7 s, speed is 14.285714 IOPS
cleanup_tbf /mnt/lustre/d77kj.sanityn
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
On MGS 192.168.204.108, TBF.id = 
On MGS 192.168.204.108, active = nodemap.active=0
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.876923 s, 19.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173322 s, 6.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211761 s, 5.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128822 s, 8.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180247 s, 5.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291102 s, 3.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240725 s, 4.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198125 s, 5.3 MB/s1+0 records in
1+0 records out

1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218254 s, 4.8 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269318 s, 3.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250166 s, 4.2 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff989583fd2800.lock_unused_count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424426 s, 2.5 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414148 s, 2.5 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321472 s, 3.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36671 s, 2.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475781 s, 2.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443172 s, 2.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874819 s, 12.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796581 s, 13.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854097 s, 12.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17362 s, 6.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948219 s, 11.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793029 s, 13.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117605 s, 8.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820999 s, 12.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118951 s, 8.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186604 s, 5.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127432 s, 8.2 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132987 s, 7.9 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152225 s, 6.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117898 s, 8.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154635 s, 6.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152307 s, 6.9 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg408-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg408-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 10.003 s, 10.5 MB/s
Write runtime is 11 s, speed is 9.090909 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 3.83156 s, 27.4 MB/s
Read runtime is 5 s, speed is 20.000000 IOPS
cleanup_tbf /mnt/lustre/d77kj.sanityn
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw
ost.OSS.ost_io.nrs_policies=fifo
ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode
oleg408-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument
oleg408-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument
oleg408-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument
oleg408-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 22
On MGS 192.168.204.108, active = nodemap.active=1
On MGS 192.168.204.108, TBF = 
ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.559872 s, 30.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098321 s, 10.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485038 s, 21.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833981 s, 12.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250263 s, 4.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344213 s, 3.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333435 s, 3.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360045 s, 2.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367064 s, 2.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336156 s, 3.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308655 s, 3.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297201 s, 3.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401683 s, 2.6 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32548 s, 3.2 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328575 s, 3.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440646 s, 2.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394379 s, 2.7 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff989583fd2800.lock_unused_count=1
ldlm.namespaces.lustre-OST0001-osc-ffff989583fd2800.lock_unused_count=2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00479078 s, 219 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00822198 s, 128 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118378 s, 88.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00563101 s, 186 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203411 s, 51.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169423 s, 6.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118945 s, 8.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113785 s, 9.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134081 s, 7.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099031 s, 10.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119609 s, 8.8 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684751 s, 15.3 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146268 s, 7.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113348 s, 9.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332033 s, 31.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988805 s, 10.6 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg408-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg408-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 0
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 10.0324 s, 10.5 MB/s
Write runtime is 11 s, speed is 9.090909 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 10.0526 s, 10.4 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77kj.sanityn
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_policies=fifo
On MGS 192.168.204.108, TBF.id = 
On MGS 192.168.204.108, active = nodemap.active=0