== sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 06:14:48 (1730805288)
ost.OSS.ost_io.nrs_policies=tbf
ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.136958 s, 122 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_w]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00787298 s, 133 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0220294 s, 47.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.028015 s, 37.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0138126 s, 75.9 MB/s
1048576 bytes (1.0 MB) copied, 0.032192 s, 32.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0333553 s, 31.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0302698 s, 34.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0339766 s, 30.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0319278 s, 32.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0421069 s, 24.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0391367 s, 26.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0321637 s, 32.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0406536 s, 25.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0594843 s, 17.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0625109 s, 16.8 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff88012aaaa000.lock_unused_count=1
ldlm.namespaces.lustre-OST0001-osc-ffff88012aaaa000.lock_unused_count=4
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0501917 s, 20.9 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_r]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0111047 s, 94.4 MB/s
1048576 bytes (1.0 MB) copied, 0.00939192 s, 112 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00930253 s, 113 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0101391 s, 103 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0111862 s, 93.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0117489 s, 89.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0105725 s, 99.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0120164 s, 87.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0126534 s, 82.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.01171 s, 89.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0134454 s, 78.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0132588 s, 79.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.013413 s, 78.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.013944 s, 75.2 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff88012aaaa000.lock_unused_count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0132801 s, 79.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0172607 s, 60.7 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg210-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.90139 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.74386 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg210-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg210-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.10@tcp}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.10@tcp}&opcode={ost_read} rate=10
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.172669 s, 97.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0178137 s, 58.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0260942 s, 40.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0443589 s, 23.6 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0479151 s, 21.9 MB/s
1048576 bytes (1.0 MB) copied, 0.052387 s, 20.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0343087 s, 30.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0418076 s, 25.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0398714 s, 26.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0316992 s, 33.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.058399 s, 18.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0500348 s, 21.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0532788 s, 19.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0388898 s, 27.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.044372 s, 23.6 MB/s
, 0.0479941 s, 21.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0463776 s, 22.6 MB/s
ldlm.namespaces.lustre-OST0001-osc-ffff88012aaaa000.lock_unused_count=1
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0160931 s, 65.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0191895 s, 54.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0141775 s, 74.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0144703 s, 72.5 MB/s
1048576 bytes (1.0 MB) copied, 0.0172886 s, 60.7 MB/s
1048576 bytes (1.0 MB) copied, 0.0177884 s, 58.9 MB/s
, 0.0191101 s, 54.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.017762 s, 59.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0176096 s, 59.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.015235 s, 68.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.018898 s, 55.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0163737 s, 64.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0184425 s, 56.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0172937 s, 60.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.016562 s, 63.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0197049 s, 53.2 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg210-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.91307 s, 21.3 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.74946 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg210-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
oleg210-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_w
ost.OSS.ost_io.nrs_tbf_rule=stop ext_r
ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.10@tcp}&jobid={dd.500} rate=20
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.132091 s, 127 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_w]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00986177 s, 106 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0129295 s, 81.1 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0374855 s, 28.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0349555 s, 30.0 MB/s
1048576 bytes (1.0 MB) copied, 0.0397791 s, 26.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0340734 s, 30.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0411154 s, 25.5 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0454542 s, 23.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0423256 s, 24.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0369035 s, 28.4 MB/s
1048576 bytes (1.0 MB) copied, 0.0383557 s, 27.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0391939 s, 26.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0496325 s, 21.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0413103 s, 25.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0440468 s, 23.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0475502 s, 22.1 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_r]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00887321 s, 118 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0101404 s, 103 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0126114 s, 83.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0114145 s, 91.9 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0110511 s, 94.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0138025 s, 76.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0124105 s, 84.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0128854 s, 81.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0132888 s, 78.9 MB/s
1048576 bytes (1.0 MB) copied, 0.0153675 s, 68.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.014107 s, 74.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0155759 s, 67.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0142688 s, 73.5 MB/s
, 0.0160297 s, 65.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0144847 s, 72.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0178003 s, 58.9 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg210-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 20
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.90708 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.88269 s, 21.5 MB/s
Read runtime is 6 s, speed is 16.666666 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg210-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg210-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext
ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.126503 s, 133 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_w]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0100972 s, 104 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0133002 s, 78.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0259872 s, 40.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0178454 s, 58.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0230353 s, 45.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0261786 s, 40.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0289118 s, 36.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0202629 s, 51.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0353447 s, 29.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0352752 s, 29.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0305648 s, 34.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.03658 s, 28.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0396062 s, 26.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0367011 s, 28.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0378501 s, 27.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0376059 s, 27.9 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_r]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0125712 s, 83.4 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0167551 s, 62.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0112968 s, 92.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0151802 s, 69.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0148186 s, 70.8 MB/s
1048576 bytes (1.0 MB) copied, 0.0186481 s, 56.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0167083 s, 62.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0155565 s, 67.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00224075 s, 468 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00551215 s, 190 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0022116 s, 474 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0185057 s, 56.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0186181 s, 56.3 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0183628 s, 57.1 MB/s
, 0.0175914 s, 59.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0230358 s, 45.5 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg210-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 10, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.76155 s, 10.7 MB/s
Write runtime is 11 s, speed is 9.090909 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.74658 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg210-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg210-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.88822 s, 21.5 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73504 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg210-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
oleg210-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_a
ost.OSS.ost_io.nrs_tbf_rule=stop ext_b
ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.12062 s, 139 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_w]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00964446 s, 109 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00952973 s, 110 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0262754 s, 39.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0319176 s, 32.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0342478 s, 30.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0293892 s, 35.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0278869 s, 37.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0322657 s, 32.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0368292 s, 28.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0358385 s, 29.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0299181 s, 35.0 MB/s
1048576 bytes (1.0 MB) copied, 0.0421293 s, 24.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0385796 s, 27.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0321485 s, 32.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0345507 s, 30.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0354937 s, 29.5 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_r]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00828518 s, 127 MB/s
, 0.00848066 s, 124 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0103401 s, 101 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00980147 s, 107 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00979914 s, 107 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0101887 s, 103 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00851035 s, 123 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00888657 s, 118 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0120033 s, 87.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0111109 s, 94.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0108377 s, 96.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00986311 s, 106 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0120518 s, 87.0 MB/s
1048576 bytes (1.0 MB) copied, 0.0104067 s, 101 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0114024 s, 92.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0123799 s, 84.7 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg210-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 5, read rate: 5
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 19.6407 s, 5.3 MB/s
Write runtime is 21 s, speed is 4.761904 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 19.4466 s, 5.4 MB/s
Read runtime is 20 s, speed is 5.000000 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg210-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg210-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug
ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.134932 s, 124 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_w]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00968866 s, 108 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0162443 s, 64.6 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0200189 s, 52.4 MB/s
, 0.0223364 s, 46.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0355894 s, 29.5 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.029472 s, 35.6 MB/s
1048576 bytes (1.0 MB) copied, 0.0248619 s, 42.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0285265 s, 36.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0442986 s, 23.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0439945 s, 23.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0385424 s, 27.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0344522 s, 30.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0314822 s, 33.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0335873 s, 31.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0388303 s, 27.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0527588 s, 19.9 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff88012aaaa000.lock_unused_count=1
ldlm.namespaces.lustre-OST0001-osc-ffff88012aaaa000.lock_unused_count=2
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_r]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00907951 s, 115 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0103584 s, 101 MB/s
1048576 bytes (1.0 MB) copied, 0.00857105 s, 122 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00982166 s, 107 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0116601 s, 89.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00487189 s, 215 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0095448 s, 110 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0103708 s, 101 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00819112 s, 128 MB/s
1048576 bytes (1.0 MB) copied, 0.0107449 s, 97.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0118202 s, 88.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00360793 s, 291 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0101506 s, 103 MB/s
1048576 bytes (1.0 MB) copied, 0.00542212 s, 193 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00152633 s, 687 MB/s
ldlm.namespaces.lustre-OST0000-osc-ffff88012aaaa000.lock_unused_count=2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00690815 s, 152 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg210-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 4.90008 s, 21.4 MB/s
Write runtime is 6 s, speed is 16.666666 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73692 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg210-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg210-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw
ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur
ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20
ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet] [bs=1M] [count=16]
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 0.153779 s, 109 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_w]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0173425 s, 60.5 MB/s
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0220407 s, 47.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0250703 s, 41.8 MB/s
, 0.0267117 s, 39.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0291733 s, 35.9 MB/s
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0295587 s, 35.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0276903 s, 37.9 MB/s
1048576 bytes (1.0 MB) copied, 0.0152164 s, 68.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0197244 s, 53.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0295117 s, 35.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0201829 s, 52.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0142938 s, 73.4 MB/s
1048576 bytes (1.0 MB) copied, 0.0278607 s, 37.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0204777 s, 51.2 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0425535 s, 24.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0541671 s, 19.4 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [declare] [-a] [pids_r]
execvp fails running declare (2): No such file or directory
1+0 records in
1+0 records out
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0161276 s, 65.0 MB/s
1048576 bytes (1.0 MB) copied, 0.00385045 s, 272 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0132064 s, 79.4 MB/s
1048576 bytes (1.0 MB) copied, 0.0175582 s, 59.7 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0153359 s, 68.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0168782 s, 62.1 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00716319 s, 146 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0167054 s, 62.8 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0159773 s, 65.6 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0160094 s, 65.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0175158 s, 59.9 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0189153 s, 55.4 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0185577 s, 56.5 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied1+0 records in
1+0 records out
, 0.0189651 s, 55.3 MB/s
1048576 bytes (1.0 MB) copied, 0.0176694 s, 59.3 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0206754 s, 50.7 MB/s
removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg210-client.virtnet'
removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg210-client.virtnet'
removed directory: '/mnt/lustre/d77k.sanityn'
cpu_npartitions on ost1 is 1
Limited write rate: 10, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.75758 s, 10.7 MB/s
Write runtime is 10 s, speed is 10.000000 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.73654 s, 10.8 MB/s
Read runtime is 10 s, speed is 10.000000 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg210-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
oleg210-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
cpu_npartitions on ost1 is 1
Limited write rate: 20, read rate: 10
Verify the write rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.74371 s, 10.8 MB/s
Write runtime is 10 s, speed is 10.000000 IOPS
Verify the read rate is under TBF control
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct]
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.74547 s, 10.8 MB/s
Read runtime is 11 s, speed is 9.090909 IOPS
cleanup_tbf /mnt/lustre/d77k.sanityn
Waiting for MDT destroys to complete
oleg210-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported
oleg210-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
ost.OSS.ost_io.nrs_tbf_rule=stop ext_a
ost.OSS.ost_io.nrs_tbf_rule=stop ext_b
ost.OSS.ost_io.nrs_policies=fifo