== sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 07:38:27 (1739536707)
ost.OSS.ost_io.nrs_policies=tbf
ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.102608 s, 164 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00821249 s, 128 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0154191 s, 68.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0314521 s, 33.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0295908 s, 35.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0279439 s, 37.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0256616 s, 40.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0378656 s, 27.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0354351 s, 29.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0326771 s, 32.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0395078 s, 26.5 MB/s
, 0.028544 s, 36.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0379039 s, 27.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0355002 s, 29.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0349552 s, 30.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0549536 s, 19.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0630756 s, 16.6 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e1c800.lock_unused_count=2
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00924278 s, 113 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00938714 s, 112 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00837551 s, 125 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00874866 s, 120 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0112385 s, 93.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.00931291 s, 113 MB/s
1048576 bytes (1.0 MB) copied, 0.0103778 s, 101 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00846791 s, 124 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0014402 s, 728 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00983147 s, 107 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0133369 s, 78.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0100988 s, 104 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00171684 s, 611 MB/s
1048576 bytes (1.0 MB) copied, 0.0117888 s, 88.9 MB/s
1048576 bytes (1.0 MB) copied, 0.00651529 s, 161 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0120917 s, 86.7 MB/s
ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e1c800.lock_unused_count=3
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.89915 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73945 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg415-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.15@tcp}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.15@tcp}&opcode={ost_read} rate=10
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.112879 s, 149 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.009447 s, 111 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0224394 s, 46.7 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0207955 s, 50.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0232564 s, 45.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0339956 s, 30.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0336859 s, 31.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0181437 s, 57.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0343878 s, 30.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.026255 s, 39.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0293733 s, 35.7 MB/s
1048576 bytes (1.0 MB) copied, 0.0396635 s, 26.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.023466 s, 44.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0361085 s, 29.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0180703 s, 58.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0346991 s, 30.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.034156 s, 30.7 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00545868 s, 192 MB/s
, 0.00630406 s, 166 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00942852 s, 111 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00612257 s, 171 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00959915 s, 109 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00761818 s, 138 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00751508 s, 140 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00783369 s, 134 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00840965 s, 125 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00859168 s, 122 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0103919 s, 101 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00987273 s, 106 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0105454 s, 99.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00994546 s, 105 MB/s
, 0.00982508 s, 107 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0101009 s, 104 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.88832 s, 21.5 MB/s
Write runtime is 5 s, speed is 20.000000 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.72738 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg415-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.15@tcp}&jobid={dd.500} rate=20
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.110884 s, 151 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0101732 s, 103 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00883966 s, 119 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0131596 s, 79.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0180871 s, 58.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0302119 s, 34.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0343104 s, 30.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0318845 s, 32.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0273051 s, 38.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0392843 s, 26.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0224321 s, 46.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0271867 s, 38.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0328503 s, 31.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0293343 s, 35.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0300549 s, 34.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0280601 s, 37.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0230056 s, 45.6 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e1c800.lock_unused_count=1
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00856483 s, 122 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00573295 s, 183 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00514294 s, 204 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0077869 s, 135 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00135656 s, 773 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00993949 s, 105 MB/s
, 0.0054529 s, 192 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00797342 s, 132 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00732391 s, 143 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00789281 s, 133 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00822757 s, 127 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.00148647 s, 705 MB/s
1048576 bytes (1.0 MB) copied, 0.00117408 s, 893 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00275698 s, 380 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00461023 s, 227 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00198168 s, 529 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.89085 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.88017 s, 21.5 MB/s
Read runtime is 6 s, speed is 16.666666 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg415-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg415-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext
ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.132917 s, 126 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00944337 s, 111 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0123618 s, 84.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.025546 s, 41.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0266289 s, 39.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0189891 s, 55.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0307694 s, 34.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0339963 s, 30.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0288859 s, 36.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0375402 s, 27.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0369299 s, 28.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0422953 s, 24.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0423017 s, 24.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0421696 s, 24.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0342657 s, 30.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0884696 s, 11.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.088357 s, 11.9 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e1c800.lock_unused_count=2
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0266776 s, 39.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0283857 s, 36.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0276633 s, 37.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0253037 s, 41.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0277341 s, 37.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0286607 s, 36.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0273421 s, 38.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0284179 s, 36.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0282766 s, 37.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0278581 s, 37.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0288945 s, 36.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0288371 s, 36.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0303375 s, 34.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0301564 s, 34.8 MB/s
ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e1c800.lock_unused_count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0302674 s, 34.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0311884 s, 33.6 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 10, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.742 s, 10.8 MB/s
Write runtime is 11 s, speed is 9.090909 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.72719 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg415-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg415-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.88403 s, 21.5 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.72554 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg415-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg415-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_a
ost.OSS.ost_io.nrs_tbf_rule=stop ext_b
ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.0990774 s, 169 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00806786 s, 130 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0155786 s, 67.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.015233 s, 68.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0211758 s, 49.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0188397 s, 55.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0144912 s, 72.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0207714 s, 50.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0127653 s, 82.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0240611 s, 43.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0339016 s, 30.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0379301 s, 27.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0245614 s, 42.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0392524 s, 26.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.038532 s, 27.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0305797 s, 34.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0506478 s, 20.7 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0060169 s, 174 MB/s
, 0.00509067 s, 206 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00925862 s, 113 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00537169 s, 195 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00638537 s, 164 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00759184 s, 138 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00909571 s, 115 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0103149 s, 102 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00139781 s, 750 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0083608 s, 125 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00851591 s, 123 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0096343 s, 109 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0101837 s, 103 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0114978 s, 91.2 MB/s
, 0.0104364 s, 100 MB/s
1048576 bytes (1.0 MB) copied, 0.0116743 s, 89.8 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 5, read rate: 5
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 19.6356 s, 5.3 MB/s
Write runtime is 20 s, speed is 5.000000 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 19.4258 s, 5.4 MB/s
Read runtime is 21 s, speed is 4.761904 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg415-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg415-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug
ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.109132 s, 154 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00686067 s, 153 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0115615 s, 90.7 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0133641 s, 78.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0242602 s, 43.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0206702 s, 50.7 MB/s
1048576 bytes (1.0 MB) copied, 0.0301214 s, 34.8 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0327709 s, 32.0 MB/s
1048576 bytes (1.0 MB) copied, 0.0428807 s, 24.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0208514 s, 50.3 MB/s
1048576 bytes (1.0 MB) copied, 0.0117477 s, 89.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0322889 s, 32.5 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0390791 s, 26.8 MB/s
1048576 bytes (1.0 MB) copied, 0.0238819 s, 43.9 MB/s
, 0.0170285 s, 61.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.027749 s, 37.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0226474 s, 46.3 MB/s
ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e1c800.lock_unused_count=1
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00886653 s, 118 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00791916 s, 132 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00809592 s, 130 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00893488 s, 117 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0118285 s, 88.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0097715 s, 107 MB/s
, 0.00974123 s, 108 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0110457 s, 94.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00997617 s, 105 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0100718 s, 104 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0115158 s, 91.1 MB/s
, 0.0111448 s, 94.1 MB/s
1048576 bytes (1.0 MB) copied, 0.011972 s, 87.6 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0116874 s, 89.7 MB/s
, 0.010115 s, 104 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0138335 s, 75.8 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.88804 s, 21.5 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.72636 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg415-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw
ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur
ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.11006 s, 152 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_w]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00809451 s, 130 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0084789 s, 124 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0226034 s, 46.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0244398 s, 42.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0230797 s, 45.4 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0340592 s, 30.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.034161 s, 30.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0283653 s, 37.0 MB/s
1048576 bytes (1.0 MB) copied, 0.031834 s, 32.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0351009 s, 29.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0344079 s, 30.5 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0401777 s, 26.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0381908 s, 27.5 MB/s
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0391416 s, 26.8 MB/s
1048576 bytes (1.0 MB) copied, 0.0371973 s, 28.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0339172 s, 30.9 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [declare] [-a] [pids_r]
execvp fails running declare (13): Permission denied
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.00789972 s, 133 MB/s
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00724264 s, 145 MB/s
, 0.00917372 s, 114 MB/s
, 0.00710109 s, 148 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0065332 s, 160 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00917282 s, 114 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00823336 s, 127 MB/s
1048576 bytes (1.0 MB) copied, 0.00976243 s, 107 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00901264 s, 116 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00883436 s, 119 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00920135 s, 114 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00869633 s, 121 MB/s
, 0.0100129 s, 105 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00862758 s, 122 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00911506 s, 115 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0117928 s, 88.9 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 10, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73856 s, 10.8 MB/s
Write runtime is 10 s, speed is 10.000000 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73213 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg415-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg415-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73746 s, 10.8 MB/s
Write runtime is 11 s, speed is 9.090909 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.72851 s, 10.8 MB/s
Read runtime is 10 s, speed is 10.000000 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg415-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg415-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_a
ost.OSS.ost_io.nrs_tbf_rule=stop ext_b
ost.OSS.ost_io.nrs_policies=fifo