== sanity-hsm test 25b: Restore lost file (HS_LOST flag) after release ========================================================== 18:41:08 (1715208068) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d25b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d25b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d25b.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d25b.sanity-hsm/f25b.sanity-hsm': No such file or directory sanity-hsm test_25b: @@@@@@ FAIL: cannot create /mnt/lustre2/d25b.sanity-hsm/f25b.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2172:test_25b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2184:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_25b.debug_log.oleg123-server.1715208069.log.4y9pPT" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_25b.dmesg.oleg123-server.1715208069.log.WdOuWn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d25b.sanity-hsm/f25b.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Update not seen after 200s: want 'log' got '' sanity-hsm test_25b: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2175:test_25b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2184:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_25b.*.1715208272.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_25b.debug_log.oleg123-server.1715208272.log.8TYnOI" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_25b.dmesg.oleg123-server.1715208272.log.jHxbiN" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]