== sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 20:58:53 (1787792333)
On MGS 192.168.201.140, active = nodemap.active=1
On MGS 192.168.201.140, TBF = 
ost.OSS.ost_io.nrs_policies=tbf nodemap
ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.499755 s, 33.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302955 s, 34.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028943 s, 36.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074182 s, 14.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748274 s, 14.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640183 s, 16.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653454 s, 16.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138866 s, 7.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134194 s, 7.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106874 s, 9.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144086 s, 7.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186203 s, 5.6 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200228 s, 5.2 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180523 s, 5.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183315 s, 5.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208881 s, 5.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212045 s, 4.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541017 s, 19.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316674 s, 33.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00807904 s, 130 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681783 s, 15.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245251 s, 42.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00621372 s, 169 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693613 s, 15.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197378 s, 53.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107134 s, 97.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612388 s, 17.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635493 s, 16.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602806 s, 174 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126004 s, 83.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010299 s, 102 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144383 s, 72.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291721 s, 35.9 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg140-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg140-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 0
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 7.60298 s, 13.8 MB/s
Write runtime is 9 s, speed is 11.111111 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 5.37981 s, 19.5 MB/s
Read runtime is 6 s, speed is 16.666666 IOPS
cleanup_tbf /mnt/lustre/d77kj.sanityn
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
On MGS 192.168.201.140, TBF.id = 
On MGS 192.168.201.140, active = nodemap.active=0
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.476363 s, 35.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167924 s, 62.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635485 s, 16.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569569 s, 18.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759894 s, 13.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10252 s, 10.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150482 s, 7.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140171 s, 7.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270429 s, 3.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26863 s, 3.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19674 s, 5.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233092 s, 4.5 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255694 s, 4.1 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234938 s, 4.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288968 s, 3.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234023 s, 4.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374901 s, 2.8 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff9b224a20b000.lock_unused_count=4
ldlm.namespaces.lustre-OST0001-osc-ffff9b224a20b000.lock_unused_count=2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251 s, 41.8 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246546 s, 42.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254677 s, 41.2 MB/s1+0 records in
1+0 records out

1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227388 s, 46.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066661 s, 15.7 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291461 s, 36.0 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350202 s, 29.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366137 s, 28.6 MB/s

1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044345 s, 23.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377083 s, 27.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442193 s, 23.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486019 s, 21.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505996 s, 20.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464588 s, 22.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497764 s, 21.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063838 s, 16.4 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg140-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg140-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 6.62405 s, 15.8 MB/s
Write runtime is 8 s, speed is 12.500000 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 2.45621 s, 42.7 MB/s
Read runtime is 3 s, speed is 33.333333 IOPS
read rate (33.333333) should exceed limit (20 * 1)
cleanup_tbf /mnt/lustre/d77kj.sanityn
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw
ost.OSS.ost_io.nrs_policies=fifo
ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode
oleg140-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument
oleg140-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument
oleg140-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument
oleg140-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument
pdsh@oleg140-client: oleg140-server: ssh exited with exit code 22
On MGS 192.168.201.140, active = nodemap.active=1
On MGS 192.168.201.140, TBF = 
ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.56977 s, 29.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552199 s, 19.0 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556928 s, 18.8 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473628 s, 22.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655094 s, 16.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711346 s, 14.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11763 s, 8.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150453 s, 7.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108867 s, 9.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157953 s, 6.6 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206774 s, 5.1 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205775 s, 5.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886734 s, 11.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115159 s, 9.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151889 s, 6.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17093 s, 6.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253719 s, 4.1 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830207 s, 12.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286844 s, 36.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568511 s, 18.4 MB/s
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590337 s, 17.8 MB/s

1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619964 s, 16.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373153 s, 28.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480288 s, 21.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574688 s, 18.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057938 s, 18.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600084 s, 17.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065716 s, 16.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072235 s, 14.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737759 s, 14.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794228 s, 13.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821326 s, 12.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990159 s, 10.6 MB/s
removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg140-client.virtnet'
removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg140-client.virtnet'
removed directory '/mnt/lustre/d77kj.sanityn'
not_under_control: 0
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 6.06947 s, 17.3 MB/s
Write runtime is 7 s, speed is 14.285714 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 10.038 s, 10.4 MB/s
Read runtime is 12 s, speed is 8.333333 IOPS
cleanup_tbf /mnt/lustre/d77kj.sanityn
sleep 5 for ZFS MDS
sleep 5 for ZFS OST
Waiting for MDT destroys to complete
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_policies=fifo
On MGS 192.168.201.140, TBF.id = 
On MGS 192.168.201.140, active = nodemap.active=0