== sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 02:16:26 (1733814986)
ost.OSS.ost_io.nrs_policies=tbf
ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.172535 s, 97.2 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0137377 s, 76.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.013837 s, 75.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0159798 s, 65.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0359715 s, 29.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0307626 s, 34.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0378286 s, 27.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0436385 s, 24.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0434755 s, 24.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0310496 s, 33.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0383921 s, 27.3 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0407909 s, 25.7 MB/s
, 0.0483545 s, 21.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0497117 s, 21.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0378703 s, 27.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0389795 s, 26.9 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800aa019000.lock_unused_count=7
ldlm.namespaces.lustre-OST0001-osc-ffff8800aa019000.lock_unused_count=6
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0578269 s, 18.1 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0133063 s, 78.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0155013 s, 67.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0183494 s, 57.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0117033 s, 89.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0131042 s, 80.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0129801 s, 80.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0143953 s, 72.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.015184 s, 69.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0110629 s, 94.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0114611 s, 91.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0115592 s, 90.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0120809 s, 86.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0130731 s, 80.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0139033 s, 75.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0146806 s, 71.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.014929 s, 70.2 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800aa019000.lock_unused_count=2
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg352-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.90571 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73488 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.52@tcp}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.52@tcp}&opcode={ost_read} rate=10
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.167521 s, 100 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0215723 s, 48.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0147524 s, 71.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0347569 s, 30.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0179807 s, 58.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0343505 s, 30.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0337226 s, 31.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0393238 s, 26.7 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0480432 s, 21.8 MB/s
, 0.0454419 s, 23.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0350576 s, 29.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0448333 s, 23.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0701198 s, 15.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0918192 s, 11.4 MB/s
, 0.0824762 s, 12.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0736025 s, 14.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0853507 s, 12.3 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800aa019000.lock_unused_count=5
ldlm.namespaces.lustre-OST0001-osc-ffff8800aa019000.lock_unused_count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0151501 s, 69.2 MB/s
, 0.013982 s, 75.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0131214 s, 79.9 MB/s
, 0.0141498 s, 74.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0188698 s, 55.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0156528 s, 67.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0150456 s, 69.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0169363 s, 61.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0172436 s, 60.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0172225 s, 60.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0168378 s, 62.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0169994 s, 61.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0175924 s, 59.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0175957 s, 59.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.017163 s, 61.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.016945 s, 61.9 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800aa019000.lock_unused_count=1
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg352-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.91082 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73554 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.52@tcp}&jobid={dd.500} rate=20
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.104838 s, 160 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00756235 s, 139 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0132484 s, 79.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0190713 s, 55.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0134662 s, 77.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0289066 s, 36.3 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0352758 s, 29.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0367941 s, 28.5 MB/s
1048576 bytes (1.0 MB) copied, 0.0269202 s, 39.0 MB/s
, 0.025076 s, 41.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0250099 s, 41.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0378356 s, 27.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0347986 s, 30.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0387078 s, 27.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0400454 s, 26.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0387188 s, 27.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0417855 s, 25.1 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800aa019000.lock_unused_count=4
ldlm.namespaces.lustre-OST0001-osc-ffff8800aa019000.lock_unused_count=1
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00866645 s, 121 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00693187 s, 151 MB/s
, 0.00752582 s, 139 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.00764381 s, 137 MB/s
1048576 bytes (1.0 MB) copied, 0.00844097 s, 124 MB/s
, 0.00762061 s, 138 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00832345 s, 126 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00456337 s, 230 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0096447 s, 109 MB/s
, 0.0111041 s, 94.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.00136046 s, 771 MB/s
1048576 bytes (1.0 MB) copied, 0.00314347 s, 334 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0129777 s, 80.8 MB/s
, 0.0015315 s, 685 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.005338 s, 196 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00941164 s, 111 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800aa019000.lock_unused_count=1
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg352-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.91125 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.88254 s, 21.5 MB/s
Read runtime is 6 s, speed is 16.666666 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext
ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.156468 s, 107 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0127413 s, 82.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0158934 s, 66.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0385219 s, 27.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0409189 s, 25.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0451248 s, 23.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0431081 s, 24.3 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0441307 s, 23.8 MB/s
, 0.0374324 s, 28.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0397446 s, 26.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0424661 s, 24.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0556092 s, 18.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0586829 s, 17.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0489248 s, 21.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0583024 s, 18.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0505175 s, 20.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0557672 s, 18.8 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0137113 s, 76.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0161118 s, 65.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0176425 s, 59.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0158885 s, 66.0 MB/s
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0143023 s, 73.3 MB/s
, 0.0169142 s, 62.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.014624 s, 71.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0137069 s, 76.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0156322 s, 67.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0163379 s, 64.2 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00248783 s, 421 MB/s
, 0.0166205 s, 63.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0146143 s, 71.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0153726 s, 68.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0181505 s, 57.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0199879 s, 52.5 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg352-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 10, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.75881 s, 10.7 MB/s
Write runtime is 11 s, speed is 9.090909 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73914 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.90253 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.74236 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_a
ost.OSS.ost_io.nrs_tbf_rule=stop ext_b
ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.176595 s, 95.0 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0228213 s, 45.9 MB/s
, 0.0268161 s, 39.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0333086 s, 31.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0522988 s, 20.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0419351 s, 25.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.059549 s, 17.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0710488 s, 14.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0577253 s, 18.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0586766 s, 17.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0661593 s, 15.8 MB/s
1048576 bytes (1.0 MB) copied, 0.0578695 s, 18.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0552959 s, 19.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0625413 s, 16.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0794548 s, 13.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0674544 s, 15.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0756589 s, 13.9 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0142519 s, 73.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0176217 s, 59.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0140313 s, 74.7 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0164558 s, 63.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0148361 s, 70.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0172537 s, 60.8 MB/s
, 0.0192448 s, 54.5 MB/s
, 0.0175298 s, 59.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.015756 s, 66.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0157979 s, 66.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0161872 s, 64.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0158768 s, 66.0 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0169574 s, 61.8 MB/s
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0167271 s, 62.7 MB/s
1048576 bytes (1.0 MB) copied, 0.0166807 s, 62.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.019799 s, 53.0 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg352-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 5, read rate: 5
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 19.4647 s, 5.4 MB/s
Write runtime is 20 s, speed is 5.000000 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 19.4445 s, 5.4 MB/s
Read runtime is 21 s, speed is 4.761904 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug
ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.159619 s, 105 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0155073 s, 67.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0187324 s, 56.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0406398 s, 25.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0439341 s, 23.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0418327 s, 25.1 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0507327 s, 20.7 MB/s
1048576 bytes (1.0 MB) copied, 0.0492523 s, 21.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0550684 s, 19.0 MB/s
1048576 bytes (1.0 MB) copied, 0.0506191 s, 20.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0457113 s, 22.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0480134 s, 21.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0636824 s, 16.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0595047 s, 17.6 MB/s
, 0.0629835 s, 16.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0597743 s, 17.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0614494 s, 17.1 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800aa019000.lock_unused_count=3
ldlm.namespaces.lustre-OST0001-osc-ffff8800aa019000.lock_unused_count=2
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00985541 s, 106 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0170847 s, 61.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0126619 s, 82.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00919017 s, 114 MB/s
1048576 bytes (1.0 MB) copied, 0.0148435 s, 70.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0173441 s, 60.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0163793 s, 64.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0159733 s, 65.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0120542 s, 87.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.012162 s, 86.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0135367 s, 77.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0145997 s, 71.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0121808 s, 86.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0134584 s, 77.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0139932 s, 74.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0162756 s, 64.4 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800aa019000.lock_unused_count=2
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg352-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.90962 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.74277 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw
ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur
ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.188258 s, 89.1 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0185538 s, 56.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0241718 s, 43.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0456316 s, 23.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0269996 s, 38.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.044108 s, 23.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0406106 s, 25.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0508709 s, 20.6 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0664703 s, 15.8 MB/s
, 0.0591937 s, 17.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0559756 s, 18.7 MB/s
1048576 bytes (1.0 MB) copied, 0.0682365 s, 15.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0530458 s, 19.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0632577 s, 16.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0714684 s, 14.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0521879 s, 20.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0530224 s, 19.8 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0105287 s, 99.6 MB/s
1048576 bytes (1.0 MB) copied, 0.0106206 s, 98.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.012719 s, 82.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0117932 s, 88.9 MB/s
1048576 bytes (1.0 MB) copied, 0.0209272 s, 50.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0181719 s, 57.7 MB/s
1048576 bytes (1.0 MB) copied, 0.0128959 s, 81.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.014902 s, 70.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0155508 s, 67.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0161081 s, 65.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0165666 s, 63.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0171322 s, 61.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0180395 s, 58.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0179164 s, 58.5 MB/s
1048576 bytes (1.0 MB) copied, 0.0166336 s, 63.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0194815 s, 53.8 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg352-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg352-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 10, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.76262 s, 10.7 MB/s
Write runtime is 11 s, speed is 9.090909 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.74263 s, 10.8 MB/s
Read runtime is 10 s, speed is 10.000000 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73648 s, 10.8 MB/s
Write runtime is 11 s, speed is 9.090909 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.72057 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_a
ost.OSS.ost_io.nrs_tbf_rule=stop ext_b
ost.OSS.ost_io.nrs_policies=fifo