== sanity-hsm test 29c: Archive/delete/remove by FID, using a file list. ========================================================== 19:08:59 (1715209739) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d29c.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d29c.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d29c.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-1': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0186924 s, 0.0 kB/s cannot create file '/mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-1' sanity-hsm test_29c: @@@@@@ FAIL: cannot create /mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-1 with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2471:test_29c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2492:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.debug_log.oleg123-server.1715209740.log.g3sSN4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.dmesg.oleg123-server.1715209740.log.dUORMb" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0632677 s, 16.6 MB/s dd: error writing '/mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-3': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0188228 s, 0.0 kB/s cannot create file '/mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-3' sanity-hsm test_29c: @@@@@@ FAIL: cannot create /mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-3 with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2473:test_29c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2492:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.debug_log.oleg123-server.1715209742.log.rumsm3" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.dmesg.oleg123-server.1715209742.log.WEHKif" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d29c.sanity-hsm/f29c.sanity-hsm-[1-3]: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_29c: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2476:test_29c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2492:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_29c.*.1715209945.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.debug_log.oleg123-server.1715209945.log.xrhFaP" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.dmesg.oleg123-server.1715209945.log.ozJ7oB" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]