-----============= acceptance-small: sanity-hsm ============----- Wed May 8 18:02:18 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: === sanity-hsm: start setup 18:02:22 (1715205742) === Starting client oleg123-client.virtnet: -o user_xattr,flock oleg123-server@tcp:/lustre /mnt/lustre2 Started clients oleg123-client.virtnet: 192.168.201.123@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg123-client.virtnet: executing check_config_client /mnt/lustre oleg123-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg123-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a9618000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b6238800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9618000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6238800.idle_timeout=debug disable quota as required oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-hsm: finish setup 18:02:29 (1715205749) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7462] nobody:x:99: Killing existing copytools on oleg123-client.virtnet pkill: pidfile not valid Try `pkill --help' for more information. Set HSM on and start Set sanity-hsm HSM policy debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 1A: lfs hsm flags root/non-root access ========================================================== 18:02:52 (1715205772) running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_clear] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--exists] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] lfs hsm_set: change hsm flags for '/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm' failed: Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] PASS 1A (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 1a: mmap & cat a HSM released file ==== 18:02:55 (1715205775) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0349129 s, 30.0 MB/s Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Changed after 1s: from 'WAITING' to 'STARTED' Waiting 180s for 'SUCCEED' Waiting 170s for 'SUCCEED' Waiting 160s for 'SUCCEED' Waiting 150s for 'SUCCEED' Waiting 140s for 'SUCCEED' Waiting 130s for 'SUCCEED' Waiting 110s for 'SUCCEED' Waiting 100s for 'SUCCEED' Changed after 101s: from 'STARTED' to 'FAILED' Waiting 90s for 'SUCCEED' Changed after 116s: from 'FAILED' to '' Waiting 80s for 'SUCCEED' Waiting 70s for 'SUCCEED' Waiting 60s for 'SUCCEED' Waiting 50s for 'SUCCEED' Waiting 20s for 'SUCCEED' Waiting 10s for 'SUCCEED' Waiting 0s for 'SUCCEED' Update not seen after 200s: want 'SUCCEED' got '' sanity-hsm test_1a: @@@@@@ FAIL: request on 0x200000403:0x2:0x0 is not SUCCEED on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:577:test_1a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:586:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_1a.*.1715205978.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_1a.debug_log.oleg123-server.1715205978.log.VMHt1r" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_1a.dmesg.oleg123-server.1715205978.log.gN2i6z" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 1a (204s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 1b: Archive, Release and Restore composite file ========================================================== 18:06:20 (1715205980) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019873 s, 52.8 MB/s Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm Waiting 200s for 'SUCCEED' Changed after 1s: from 'WAITING' to 'STARTED' Waiting 170s for 'SUCCEED' Waiting 160s for 'SUCCEED' Waiting 150s for 'SUCCEED' Waiting 140s for 'SUCCEED' Waiting 130s for 'SUCCEED' Waiting 120s for 'SUCCEED' Waiting 100s for 'SUCCEED' Updated after 102s: want 'SUCCEED' got 'SUCCEED' release /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm Cannot stat /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm: Input/output error sanity-hsm test_1b: @@@@@@ FAIL: could not release file Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:603:test_1bde_base() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:620:test_1b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:622:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_1b.*.1715206083.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_1b.debug_log.oleg123-server.1715206083.log.UdIH2a" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_1b.dmesg.oleg123-server.1715206083.log.gFP7QK" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 1b (104s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 1c: Check setting archive-id in lfs hsm_set ========================================================== 18:08:06 (1715206086) running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 32 running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is still 32 running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 33 running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 16 PASS 1c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 1d: Archive, Release and Restore DoM file ========================================================== 18:08:09 (1715206089) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191409 s, 54.8 MB/s Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm Waiting 200s for 'SUCCEED' Changed after 1s: from 'WAITING' to 'STARTED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Waiting 170s for 'SUCCEED' Waiting 160s for 'SUCCEED' Waiting 150s for 'SUCCEED' Waiting 140s for 'SUCCEED' Waiting 130s for 'SUCCEED' Waiting 120s for 'SUCCEED' Waiting 100s for 'SUCCEED' Updated after 102s: want 'SUCCEED' got 'SUCCEED' release /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm Cannot stat /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm: Input/output error sanity-hsm test_1d: @@@@@@ FAIL: could not release file Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:603:test_1bde_base() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:693:test_1d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:695:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_1d.*.1715206192.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_1d.debug_log.oleg123-server.1715206192.log.SHlRwh" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_1d.dmesg.oleg123-server.1715206192.log.AhOsvr" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 1d (103s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 1e: Archive, Release and Restore SEL file ========================================================== 18:09:54 (1715206194) lfs mkdir: cannot resolve path '/mnt/lustre/d1e.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d1e.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d1e.sanity-hsm': Cannot send after transport endpoint shutdown lfs setstripe: cannot create composite file '/mnt/lustre/d1e.sanity-hsm': Cannot send after transport endpoint shutdown sanity-hsm test_1e: @@@@@@ FAIL: failed to set default stripe Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:703:test_1e() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:734:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_1e.*.1715206195.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_1e.debug_log.oleg123-server.1715206195.log.mKePgn" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_1e.dmesg.oleg123-server.1715206195.log.HabgXE" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 1e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 1f: DoM file release after restore ==== 18:09:57 (1715206197) lfs mkdir: cannot resolve path '/mnt/lustre/d1f.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d1f.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d1f.sanity-hsm': Cannot send after transport endpoint shutdown lfs setstripe: cannot create composite file '/mnt/lustre/d1f.sanity-hsm': Cannot send after transport endpoint shutdown sanity-hsm test_1f: @@@@@@ FAIL: failed to set default stripe Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:743:test_1f() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:759:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_1f.*.1715206198.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_1f.debug_log.oleg123-server.1715206198.log.Tm9k9C" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_1f.dmesg.oleg123-server.1715206198.log.FcVbo3" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 1f (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 2: Check file dirtyness when doing setattr ========================================================== 18:10:00 (1715206200) lfs mkdir: cannot resolve path '/mnt/lustre/d2.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d2.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d2.sanity-hsm': Cannot send after transport endpoint shutdown 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00128127 s, 0.0 kB/s 0x240000403:0x2:0x0 lfs hsm_state: get HSM state for '/mnt/lustre/d2.sanity-hsm/f2.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_2: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d2.sanity-hsm/f2.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11439:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:767:test_2() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:788:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_2.debug_log.oleg123-server.1715206201.log.RVx5Qd" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_2.dmesg.oleg123-server.1715206201.log.WtOTYM" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] sanity-hsm test_2: @@@@@@ FAIL: hsm flags on /mnt/lustre/d2.sanity-hsm/f2.sanity-hsm are Dumping lctl log to /tmp/testlogs//sanity-hsm.test_2.*.1715206201.log != 0x00000000 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11440:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:767:test_2() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:788:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_2.*.1715206203.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_2.debug_log.oleg123-server.1715206203.log.lw8Hfo" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_2.dmesg.oleg123-server.1715206203.log.JyBde0" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 2 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 3: Check file dirtyness when opening for write ========================================================== 18:10:05 (1715206205) lfs mkdir: cannot resolve path '/mnt/lustre/d3.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d3.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d3.sanity-hsm': Cannot send after transport endpoint shutdown cp: failed to access '/mnt/lustre/d3.sanity-hsm/f3.sanity-hsm': Cannot send after transport endpoint shutdown lfs hsm_state: get HSM state for '/mnt/lustre/d3.sanity-hsm/f3.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_3: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d3.sanity-hsm/f3.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11439:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:796:test_3() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:843:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_3.debug_log.oleg123-server.1715206206.log.nXKW3b" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_3.dmesg.oleg123-server.1715206206.log.RUi9CX" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] sanity-hsm test_3: @@@@@@ FAIL: hsm flags on /mnt/lustre/d3.sanity-hsm/f3.sanity-hsm are Dumping lctl log to /tmp/testlogs//sanity-hsm.test_3.*.1715206206.log != 0x00000000 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11440:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:796:test_3() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:843:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_3.*.1715206207.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_3.debug_log.oleg123-server.1715206207.log.Wpwjfg" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_3.dmesg.oleg123-server.1715206207.log.yCDSw4" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 3 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 4: Useless cancel must not be registered ========================================================== 18:10:09 (1715206209) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0328003 s, 32.0 MB/s Cannot stat /mnt/lustre/d4.sanity-hsm/f4.sanity-hsm: Cannot send after transport endpoint shutdown PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 8: Test default archive number ======== 18:10:13 (1715206213) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_8/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d8.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d8.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d8.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d8.sanity-hsm/f8.sanity-hsm': No such file or directory sanity-hsm test_8: @@@@@@ FAIL: cannot create /mnt/lustre2/d8.sanity-hsm/f8.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:861:test_8() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:867:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_8.debug_log.oleg123-server.1715206214.log.J5VSm2" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_8.dmesg.oleg123-server.1715206214.log.cLUYt7" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d8.sanity-hsm/f8.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_8: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:863:test_8() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:867:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_8.*.1715206416.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_8.debug_log.oleg123-server.1715206416.log.7yCGEg" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_8.dmesg.oleg123-server.1715206416.log.vCjX0U" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 8 (204s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 9A: Use of explicit archive number, with dedicated copytool ========================================================== 18:13:39 (1715206419) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_9A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 3 "/mnt/lustre2"' oleg123-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg123-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=1bc50a3e-f07c-4b55-83c3-5f3c3d406528 archive_id=3 requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=1bc50a3e-f07c-4b55-83c3-5f3c3d406528 archive_id=3 requests=[current:0 ok:0 errors:0] on mds2 lfs mkdir: cannot resolve path '/mnt/lustre/d9A.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d9A.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d9A.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d9A.sanity-hsm/f9A.sanity-hsm': No such file or directory sanity-hsm test_9A: @@@@@@ FAIL: cannot create /mnt/lustre2/d9A.sanity-hsm/f9A.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:882:test_9A() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:888:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_9A.debug_log.oleg123-server.1715206426.log.PXaRJs" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_9A.dmesg.oleg123-server.1715206426.log.VpquIw" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d9A.sanity-hsm/f9A.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_9A: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:884:test_9A() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:888:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_9A.*.1715206629.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_9A.debug_log.oleg123-server.1715206629.log.9Dqhh1" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_9A.dmesg.oleg123-server.1715206629.log.qp8xWE" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 9A (211s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 9a: Multiple remote agents ============ 18:17:12 (1715206632) SKIP: sanity-hsm test_9a Need 3 or more clients, have 1 SKIP 9a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 10a: Archive a file =================== 18:17:14 (1715206634) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d10a.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d10a.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d10a.sanity-hsm': Cannot send after transport endpoint shutdown mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d10a.sanity-hsm/f10a.sanity-hsm': No such file or directory sanity-hsm test_10a: @@@@@@ FAIL: cannot create /mnt/lustre2/d10a.sanity-hsm/f10a.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:923:test_10a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:943:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_10a.debug_log.oleg123-server.1715206636.log.zY4aZ1" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_10a.dmesg.oleg123-server.1715206636.log.ZICw6W" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d10a.sanity-hsm/f10a.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_10a: @@@@@@ FAIL: hsm_archive failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:925:test_10a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:943:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_10a.*.1715206637.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_10a.debug_log.oleg123-server.1715206637.log.v7XDPw" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_10a.dmesg.oleg123-server.1715206637.log.YFRtTv" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 10a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 10b: Archive of non dirty file must work without doing request ========================================================== 18:17:20 (1715206640) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d10b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d10b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d10b.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d10b.sanity-hsm/f10b.sanity-hsm': No such file or directory sanity-hsm test_10b: @@@@@@ FAIL: cannot create /mnt/lustre2/d10b.sanity-hsm/f10b.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:951:test_10b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:960:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_10b.debug_log.oleg123-server.1715206641.log.X8k3NG" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_10b.dmesg.oleg123-server.1715206641.log.ULlScQ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d10b.sanity-hsm/f10b.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_10b: @@@@@@ FAIL: archive request failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:952:test_10b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:960:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_10b.*.1715206642.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_10b.debug_log.oleg123-server.1715206642.log.1sEifS" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_10b.dmesg.oleg123-server.1715206642.log.RJZZC4" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 10b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 10c: Check forbidden archive ========== 18:17:25 (1715206645) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d10c.sanity-hsm/f10c.sanity-hsm': No such file or directory sanity-hsm test_10c: @@@@@@ FAIL: cannot create /mnt/lustre2/d10c.sanity-hsm/f10c.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:968:test_10c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:973:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_10c.debug_log.oleg123-server.1715206646.log.N3sPLu" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_10c.dmesg.oleg123-server.1715206646.log.QvAR9P" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] lfs hsm_set: change hsm flags for '/mnt/lustre/d10c.sanity-hsm/f10c.sanity-hsm' failed: Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d10c.sanity-hsm/f10c.sanity-hsm: Cannot send after transport endpoint shutdown FAIL 10c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 10d: Archive a file on the default archive id ========================================================== 18:17:29 (1715206649) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d10d.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d10d.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d10d.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d10d.sanity-hsm/f10d.sanity-hsm': No such file or directory sanity-hsm test_10d: @@@@@@ FAIL: cannot create /mnt/lustre2/d10d.sanity-hsm/f10d.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:981:test_10d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:990:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_10d.debug_log.oleg123-server.1715206650.log.qLSd2K" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_10d.dmesg.oleg123-server.1715206650.log.LYw9jh" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d10d.sanity-hsm/f10d.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_10d: @@@@@@ FAIL: cannot archive /mnt/lustre/d10d.sanity-hsm/f10d.sanity-hsm Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:982:test_10d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:990:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_10d.*.1715206651.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_10d.debug_log.oleg123-server.1715206651.log.0FfgnO" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_10d.dmesg.oleg123-server.1715206651.log.srbYqn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 10d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 11a: Import a file ==================== 18:17:34 (1715206654) lfs mkdir: cannot resolve path '/mnt/lustre/d11a.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d11a.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d11a.sanity-hsm': Cannot send after transport endpoint shutdown mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown sanity-hsm test_11a: @@@@@@ FAIL: cannot create directory '/mnt/lustre/d11a.sanity-hsm' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11273:__lhsmtool_import() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11380:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:997:test_11a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1016:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_11a.*.1715206654.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_11a.debug_log.oleg123-server.1715206654.log.JAOKSp" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_11a.dmesg.oleg123-server.1715206654.log.H5ZeO7" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 11a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 11b: Import a deleted file using its FID ========================================================== 18:17:37 (1715206657) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_11b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d11b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d11b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d11b.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d11b.sanity-hsm/f11b.sanity-hsm': No such file or directory sanity-hsm test_11b: @@@@@@ FAIL: cannot create /mnt/lustre2/d11b.sanity-hsm/f11b.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1024:test_11b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1038:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_11b.debug_log.oleg123-server.1715206658.log.eZFBQp" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_11b.dmesg.oleg123-server.1715206658.log.P8VeVh" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_11b: @@@@@@ FAIL: hsm_archive failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1026:test_11b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1038:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_11b.*.1715206659.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_11b.debug_log.oleg123-server.1715206659.log.v0wRjw" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_11b.dmesg.oleg123-server.1715206659.log.4hAx4q" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 11b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 11c: Import a file to a directory with a pool ========================================================== 18:17:42 (1715206662) Creating new pool oleg123-server: Pool lustre.test_11c created Adding targets to pool oleg123-server: OST lustre-OST0001_UUID added to pool lustre.test_11c mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown lfs setstripe: cannot resolve path '/mnt/lustre/d11c.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs setstripe: '/mnt/lustre/d11c.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown sanity-hsm test_11c: @@@@@@ FAIL: cannot create directory '/mnt/lustre/d11c.sanity-hsm' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11273:__lhsmtool_import() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11380:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1048:test_11c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1050:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_11c.*.1715206667.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_11c.debug_log.oleg123-server.1715206667.log.XdSG0U" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_11c.dmesg.oleg123-server.1715206667.log.o23rWa" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Destroy the created pools: test_11c lustre.test_11c oleg123-server: OST lustre-OST0001_UUID removed from pool lustre.test_11c oleg123-server: Pool lustre.test_11c destroyed FAIL 11c (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12a: Restore an imported file explicitly ========================================================== 18:17:54 (1715206674) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12a.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12a.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12a.sanity-hsm': Cannot send after transport endpoint shutdown mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown sanity-hsm test_12a: @@@@@@ FAIL: cannot create directory '/mnt/lustre/d12a.sanity-hsm' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11273:__lhsmtool_import() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11380:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1060:test_12a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1076:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12a.*.1715206675.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12a.debug_log.oleg123-server.1715206675.log.lKDNyo" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12a.dmesg.oleg123-server.1715206675.log.ejCWCZ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12b: Restore an imported file implicitly ========================================================== 18:17:58 (1715206678) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12b.sanity-hsm': Cannot send after transport endpoint shutdown mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown sanity-hsm test_12b: @@@@@@ FAIL: cannot create directory '/mnt/lustre/d12b.sanity-hsm' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11273:__lhsmtool_import() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11380:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1086:test_12b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1099:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12b.*.1715206679.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12b.debug_log.oleg123-server.1715206679.log.QiY0I7" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12b.dmesg.oleg123-server.1715206679.log.OsiVkS" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12c: Restore a file with stripe of 2 == 18:18:02 (1715206682) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12c.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12c.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12c.sanity-hsm': Cannot send after transport endpoint shutdown lfs setstripe: unable to open '/mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm': Cannot send after transport endpoint shutdown (108) mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm': Cannot send after transport endpoint shutdown dd: failed to open '/mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm': Cannot send after transport endpoint shutdown cannot create file '/mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm' md5sum: /mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm: Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm: Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_12c: @@@@@@ FAIL: release /mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1116:test_12c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1122:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12c.*.1715206683.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12c.debug_log.oleg123-server.1715206683.log.ZGmKf6" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12c.dmesg.oleg123-server.1715206683.log.W1ZUU0" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12d: Restore of a non archived, non released file must work ========================================================== 18:18:06 (1715206686) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12d.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12d.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12d.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12d.sanity-hsm/f12d.sanity-hsm': No such file or directory sanity-hsm test_12d: @@@@@@ FAIL: cannot create /mnt/lustre2/d12d.sanity-hsm/f12d.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1131:test_12d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1145:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12d.debug_log.oleg123-server.1715206686.log.eGJrVM" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12d.dmesg.oleg123-server.1715206686.log.jQhSDQ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12d.sanity-hsm/f12d.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_12d: @@@@@@ FAIL: restore of non archived file failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1132:test_12d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1145:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12d.*.1715206688.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12d.debug_log.oleg123-server.1715206688.log.WjpVXV" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12d.dmesg.oleg123-server.1715206688.log.MBrjz2" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12e: Check forbidden restore ========== 18:18:10 (1715206690) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12e.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12e.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12e.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12e.sanity-hsm/f12e.sanity-hsm': No such file or directory sanity-hsm test_12e: @@@@@@ FAIL: cannot create /mnt/lustre2/d12e.sanity-hsm/f12e.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1154:test_12e() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1166:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12e.debug_log.oleg123-server.1715206691.log.pFjOcs" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12e.dmesg.oleg123-server.1715206691.log.pqwAwH" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12e.sanity-hsm/f12e.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_12e: @@@@@@ FAIL: archive request failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1155:test_12e() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1166:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12e.*.1715206692.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12e.debug_log.oleg123-server.1715206692.log.LixVss" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12e.dmesg.oleg123-server.1715206692.log.S5U2jK" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12e (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12f: Restore a released file explicitly ========================================================== 18:18:14 (1715206694) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12f.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12f.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12f.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12f.sanity-hsm/f12f.sanity-hsm': No such file or directory sanity-hsm test_12f: @@@@@@ FAIL: cannot create /mnt/lustre2/d12f.sanity-hsm/f12f.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1174:test_12f() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1189:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12f.debug_log.oleg123-server.1715206695.log.u55Rpj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12f.dmesg.oleg123-server.1715206695.log.lY2H4K" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12f.sanity-hsm/f12f.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Update not seen after 200s: want 'log' got '' sanity-hsm test_12f: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1177:test_12f() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1189:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12f.*.1715206897.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12f.debug_log.oleg123-server.1715206897.log.VZyHgU" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12f.dmesg.oleg123-server.1715206897.log.FArcpT" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12f (204s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12g: Restore a released file implicitly ========================================================== 18:21:41 (1715206901) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12g.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12g.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12g.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12g.sanity-hsm/f12g.sanity-hsm': No such file or directory sanity-hsm test_12g: @@@@@@ FAIL: cannot create /mnt/lustre2/d12g.sanity-hsm/f12g.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1197:test_12g() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1211:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12g.debug_log.oleg123-server.1715206902.log.D5oxCP" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12g.dmesg.oleg123-server.1715206902.log.tSjR10" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12g.sanity-hsm/f12g.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_12g: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1200:test_12g() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1211:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12g.*.1715207105.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12g.debug_log.oleg123-server.1715207105.log.OMiEKi" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12g.dmesg.oleg123-server.1715207105.log.HsO1T3" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12g (205s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12h: Restore a released file implicitly from a second node ========================================================== 18:25:08 (1715207108) SKIP: sanity-hsm test_12h Need 2 or more clients, have 1 SKIP 12h (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12m: Archive/release/implicit restore ========================================================== 18:25:12 (1715207112) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12m/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12m.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12m.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12m.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12m.sanity-hsm/f12m.sanity-hsm': No such file or directory sanity-hsm test_12m: @@@@@@ FAIL: cannot create /mnt/lustre2/d12m.sanity-hsm/f12m.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1243:test_12m() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1253:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12m.debug_log.oleg123-server.1715207113.log.NE1N8t" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12m.dmesg.oleg123-server.1715207113.log.GsoNQz" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12m.sanity-hsm/f12m.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_12m: @@@@@@ FAIL: archive of /mnt/lustre/d12m.sanity-hsm/f12m.sanity-hsm failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1244:test_12m() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1253:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12m.*.1715207114.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12m.debug_log.oleg123-server.1715207114.log.vs7xsb" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12m.dmesg.oleg123-server.1715207114.log.H8WO5k" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12m (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12n: Import/implicit restore/release == 18:25:18 (1715207118) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12n/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown sanity-hsm test_12n: @@@@@@ FAIL: cannot create directory '/mnt/lustre/d12n.sanity-hsm' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11273:__lhsmtool_import() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11380:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1263:test_12n() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1270:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12n.*.1715207119.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12n.debug_log.oleg123-server.1715207119.log.BRZO5U" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12n.dmesg.oleg123-server.1715207119.log.HdseNg" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12n (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ========================================================== 18:25:22 (1715207122) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12o/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12o.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12o.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12o.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12o.sanity-hsm/f12o.sanity-hsm': No such file or directory sanity-hsm test_12o: @@@@@@ FAIL: cannot create /mnt/lustre2/d12o.sanity-hsm/f12o.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1278:test_12o() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1319:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12o.debug_log.oleg123-server.1715207124.log.U42cCJ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12o.dmesg.oleg123-server.1715207124.log.QJJilh" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12o.sanity-hsm/f12o.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Update not seen after 200s: want 'log' got '' sanity-hsm test_12o: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1281:test_12o() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1319:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12o.*.1715207326.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12o.debug_log.oleg123-server.1715207326.log.dJRFp8" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12o.dmesg.oleg123-server.1715207326.log.ZGOHxf" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12o (205s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12p: implicit restore of a file on copytool mount point ========================================================== 18:28:50 (1715207330) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12p/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12p.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12p.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12p.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12p.sanity-hsm/f12p.sanity-hsm': No such file or directory sanity-hsm test_12p: @@@@@@ FAIL: cannot create /mnt/lustre2/d12p.sanity-hsm/f12p.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1327:test_12p() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1337:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12p.debug_log.oleg123-server.1715207331.log.g8ljSO" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12p.dmesg.oleg123-server.1715207331.log.EDXUW7" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12p.sanity-hsm/f12p.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_12p: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1330:test_12p() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1337:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12p.*.1715207534.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12p.debug_log.oleg123-server.1715207534.log.afK0OD" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12p.dmesg.oleg123-server.1715207534.log.xvoCex" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12p (205s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12q: file attributes are refreshed after restore ========================================================== 18:32:18 (1715207538) Starting client: oleg123-client.virtnet: -o user_xattr,flock oleg123-server@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12q/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' lfs mkdir: cannot resolve path '/mnt/lustre/d12q.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12q.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12q.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d12q.sanity-hsm/f12q.sanity-hsm': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0110251 s, 0.0 kB/s cannot create file '/mnt/lustre2/d12q.sanity-hsm/f12q.sanity-hsm' sanity-hsm test_12q: @@@@@@ FAIL: cannot create /mnt/lustre2/d12q.sanity-hsm/f12q.sanity-hsm with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1354:test_12q() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1398:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12q.debug_log.oleg123-server.1715207539.log.z7CLaq" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12q.dmesg.oleg123-server.1715207539.log.omrrGv" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d12q.sanity-hsm/f12q.sanity-hsm': Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d12q.sanity-hsm/f12q.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_12q: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1358:test_12q() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1398:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12q.*.1715207742.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12q.debug_log.oleg123-server.1715207742.log.RH7NF0" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12q.dmesg.oleg123-server.1715207742.log.h57P0F" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Stopping client oleg123-client.virtnet /mnt/lustre3 (opts:) FAIL 12q (205s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12r: lseek restores released file ===== 18:35:45 (1715207745) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12r/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12r.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12r.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12r.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12r.sanity-hsm/f12r.sanity-hsm': No such file or directory sanity-hsm test_12r: @@@@@@ FAIL: cannot create /mnt/lustre2/d12r.sanity-hsm/f12r.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1406:test_12r() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1418:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12r.debug_log.oleg123-server.1715207747.log.pvrL23" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12r.dmesg.oleg123-server.1715207747.log.BYimOV" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12r.sanity-hsm/f12r.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_12r: @@@@@@ FAIL: archive of /mnt/lustre/d12r.sanity-hsm/f12r.sanity-hsm failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1408:test_12r() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1418:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12r.*.1715207748.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12r.debug_log.oleg123-server.1715207748.log.xal8NL" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12r.dmesg.oleg123-server.1715207748.log.5KYbWH" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12r (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12s: race between restore requests ==== 18:35:52 (1715207752) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12s/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12s.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12s.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12s.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12s.sanity-hsm/f12s.sanity-hsm': No such file or directory sanity-hsm test_12s: @@@@@@ FAIL: cannot create /mnt/lustre2/d12s.sanity-hsm/f12s.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1432:test_12s() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1449:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12s.debug_log.oleg123-server.1715207753.log.89Rt0I" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12s.dmesg.oleg123-server.1715207753.log.UEStLR" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12s.sanity-hsm/f12s.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_12s: @@@@@@ FAIL: archive of /mnt/lustre/d12s.sanity-hsm/f12s.sanity-hsm failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1434:test_12s() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1449:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12s.*.1715207755.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12s.debug_log.oleg123-server.1715207755.log.VasF3u" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12s.dmesg.oleg123-server.1715207755.log.Oh0QmI" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12s (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12t: Multiple parallel reads for a HSM imported file ========================================================== 18:35:58 (1715207758) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12t/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12t.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12t.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12t.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d12t.sanity-hsm/f12t.sanity-hsm': No such file or directory sanity-hsm test_12t: @@@@@@ FAIL: cannot create /mnt/lustre2/d12t.sanity-hsm/f12t.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1457:test_12t() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1479:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_12t.debug_log.oleg123-server.1715207760.log.ne5Drw" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12t.dmesg.oleg123-server.1715207760.log.yv5O6V" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_12t: @@@@@@ FAIL: failed to HSM archive /mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1461:test_12t() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1479:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12t.*.1715207761.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12t.debug_log.oleg123-server.1715207761.log.l2Wp6i" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12t.dmesg.oleg123-server.1715207761.log.kX1ljN" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12t (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 12u: Multiple reads on multiple HSM imported files in parallel ========================================================== 18:36:05 (1715207765) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12u/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d12u.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d12u.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d12u.sanity-hsm': Cannot send after transport endpoint shutdown sanity-hsm test_12u: @@@@@@ FAIL: failed to mkdir /mnt/lustre/d12u.sanity-hsm Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1487:test_12u() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1521:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_12u.*.1715207766.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_12u.debug_log.oleg123-server.1715207766.log.xyxob9" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_12u.dmesg.oleg123-server.1715207766.log.1Mc1AP" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 12u (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 13: Recursively import and restore a directory ========================================================== 18:36:09 (1715207769) mkdir: cannot create directory '/mnt/lustre': File exists sanity-hsm test_13: @@@@@@ FAIL: cannot create directory '/mnt/lustre' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11273:__lhsmtool_import() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11380:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1538:test_13() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1553:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_13.*.1715207773.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_13.debug_log.oleg123-server.1715207773.log.1m18f1" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_13.dmesg.oleg123-server.1715207773.log.meQRi1" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 13 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 14: Rebind archived file to a new fid ========================================================== 18:36:17 (1715207777) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_14/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d14.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d14.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d14.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d14.sanity-hsm/f14.sanity-hsm': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0193579 s, 0.0 kB/s cannot create file '/mnt/lustre2/d14.sanity-hsm/f14.sanity-hsm' sanity-hsm test_14: @@@@@@ FAIL: cannot create /mnt/lustre2/d14.sanity-hsm/f14.sanity-hsm with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1563:test_14() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1583:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_14.debug_log.oleg123-server.1715207778.log.9Z9Erw" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_14.dmesg.oleg123-server.1715207778.log.SWMg3H" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] md5sum: /mnt/lustre/d14.sanity-hsm/f14.sanity-hsm: Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d14.sanity-hsm/f14.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_14: @@@@@@ FAIL: could not archive file Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1565:test_14() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1583:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_14.*.1715207780.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_14.debug_log.oleg123-server.1715207780.log.z9dk1o" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_14.dmesg.oleg123-server.1715207780.log.0O6X1E" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 14 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 15: Rebind a list of files ============ 18:36:23 (1715207783) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_15/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d15.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d15.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d15.sanity-hsm': Cannot send after transport endpoint shutdown 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0660966 s, 15.9 MB/s md5sum: /mnt/lustre/d15.sanity-hsm/f15.sanity-hsm.1: Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d15.sanity-hsm/f15.sanity-hsm.1: Cannot send after transport endpoint shutdown sanity-hsm test_15: @@@@@@ FAIL: could not archive file Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1601:test_15() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1631:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_15.*.1715207784.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_15.debug_log.oleg123-server.1715207784.log.Nmuh1w" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_15.dmesg.oleg123-server.1715207784.log.1gzbXZ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 15 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 16: Test CT bandwith control option === 18:36:28 (1715207788) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_16/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.0214256 s, 979 MB/s lfs mkdir: cannot resolve path '/mnt/lustre/d16.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d16.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d16.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d16.sanity-hsm/f16.sanity-hsm': No such file or directory sanity-hsm test_16: @@@@@@ FAIL: cannot create /mnt/lustre2/d16.sanity-hsm/f16.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1645:test_16() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1657:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_16.debug_log.oleg123-server.1715207789.log.wQ3EOm" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_16.dmesg.oleg123-server.1715207789.log.BKP8yZ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d16.sanity-hsm/f16.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Update not seen after 200s: want 'log' got '' sanity-hsm test_16: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1649:test_16() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1657:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_16.*.1715207990.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_16.debug_log.oleg123-server.1715207990.log.myNmdc" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_16.dmesg.oleg123-server.1715207990.log.xhjFMm" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 16 (203s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 20: Release is not permitted ========== 18:39:53 (1715207993) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00211925 s, 0.0 kB/s 0x200000403:0x8:0x0 Cannot stat /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm: Cannot send after transport endpoint shutdown lfs hsm_set: change hsm flags for '/mnt/lustre/d20.sanity-hsm/f20.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_20: @@@@@@ FAIL: could not add flag Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1667:test_20() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1684:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_20.*.1715207994.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_20.debug_log.oleg123-server.1715207994.log.rTiIji" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_20.dmesg.oleg123-server.1715207994.log.dGeahD" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 20 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 21: Simple release tests ============== 18:39:58 (1715207998) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_21/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d21.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d21.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d21.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d21.sanity-hsm/test_release': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0183908 s, 0.0 kB/s cannot create file '/mnt/lustre2/d21.sanity-hsm/test_release' sanity-hsm test_21: @@@@@@ FAIL: cannot create /mnt/lustre2/d21.sanity-hsm/test_release with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1694:test_21() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1751:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_21.debug_log.oleg123-server.1715207999.log.kVHWaZ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_21.dmesg.oleg123-server.1715207999.log.GSWdYv" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] lfs hsm_state: get HSM state for '/mnt/lustre/d21.sanity-hsm/test_release' failed: Cannot send after transport endpoint shutdown sanity-hsm test_21: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d21.sanity-hsm/test_release failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11439:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1695:test_21() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1751:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_21.debug_log.oleg123-server.1715208001.log.4ibnIK" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_21.dmesg.oleg123-server.1715208001.log.XHulPl" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] sanity-hsm test_21: @@@@@@ FAIL: hsm flags on /mnt/lustre/d21.sanity-hsm/test_release are Dumping lctl log to /tmp/testlogs//sanity-hsm.test_21.*.1715208001.log != 0x00000000 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11440:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1695:test_21() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1751:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_21.*.1715208002.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_21.debug_log.oleg123-server.1715208002.log.sYhYgo" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_21.dmesg.oleg123-server.1715208002.log.9UATw3" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 21 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 22: Could not swap a release file ===== 18:40:05 (1715208005) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_22/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d22.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d22.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d22.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d22.sanity-hsm/test_release': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00986072 s, 0.0 kB/s cannot create file '/mnt/lustre2/d22.sanity-hsm/test_release' sanity-hsm test_22: @@@@@@ FAIL: cannot create /mnt/lustre2/d22.sanity-hsm/test_release with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1762:test_22() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1777:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_22.debug_log.oleg123-server.1715208006.log.Ubpyee" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_22.dmesg.oleg123-server.1715208006.log.p3PIj3" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] lfs hsm_state: get HSM state for '/mnt/lustre/d22.sanity-hsm/test_release' failed: Cannot send after transport endpoint shutdown sanity-hsm test_22: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d22.sanity-hsm/test_release failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11439:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1763:test_22() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1777:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_22.debug_log.oleg123-server.1715208007.log.RNtFFJ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_22.dmesg.oleg123-server.1715208007.log.iFOaoC" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] sanity-hsm test_22: @@@@@@ FAIL: hsm flags on /mnt/lustre/d22.sanity-hsm/test_release are Dumping lctl log to /tmp/testlogs//sanity-hsm.test_22.*.1715208007.log != 0x00000000 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11440:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1763:test_22() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1777:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_22.*.1715208009.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_22.debug_log.oleg123-server.1715208009.log.vT1LFa" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_22.dmesg.oleg123-server.1715208009.log.BDPxR6" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 22 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 23: Release does not change a/mtime (utime) ========================================================== 18:40:12 (1715208012) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_23/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d23.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d23.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d23.sanity-hsm': Cannot send after transport endpoint shutdown 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321944 s, 32.6 MB/s lfs hsm_state: get HSM state for '/mnt/lustre/d23.sanity-hsm/test_mtime' failed: Cannot send after transport endpoint shutdown sanity-hsm test_23: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d23.sanity-hsm/test_mtime failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11439:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1788:test_23() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1805:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_23.debug_log.oleg123-server.1715208013.log.pIlmGf" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_23.dmesg.oleg123-server.1715208013.log.LYgndm" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] sanity-hsm test_23: @@@@@@ FAIL: hsm flags on /mnt/lustre/d23.sanity-hsm/test_mtime are Dumping lctl log to /tmp/testlogs//sanity-hsm.test_23.*.1715208013.log != 0x00000000 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11440:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1788:test_23() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1805:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_23.*.1715208014.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_23.debug_log.oleg123-server.1715208014.log.0g5QZQ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_23.dmesg.oleg123-server.1715208014.log.HOKWA1" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 23 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 24a: Archive, release, and restore does not change a/mtime (i/o) ========================================================== 18:40:17 (1715208017) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d24a.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d24a.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d24a.sanity-hsm': Cannot send after transport endpoint shutdown 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0478223 s, 21.9 MB/s lfs hsm_state: get HSM state for '/mnt/lustre/d24a.sanity-hsm/f24a.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_24a: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d24a.sanity-hsm/f24a.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11439:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1824:test_24a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1911:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_24a.debug_log.oleg123-server.1715208019.log.k0HhtI" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24a.dmesg.oleg123-server.1715208019.log.MXP6o5" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] sanity-hsm test_24a: @@@@@@ FAIL: hsm flags on /mnt/lustre/d24a.sanity-hsm/f24a.sanity-hsm are Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24a.*.1715208019.log != 0x00000000 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11440:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1824:test_24a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1911:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24a.*.1715208020.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_24a.debug_log.oleg123-server.1715208020.log.o2wJzs" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24a.dmesg.oleg123-server.1715208020.log.NiQ3ET" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 24a (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 24b: root can archive, release, and restore user files ========================================================== 18:40:24 (1715208024) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d24b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d24b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d24b.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d24b.sanity-hsm/f24b.sanity-hsm': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0198276 s, 0.0 kB/s cannot create file '/mnt/lustre2/d24b.sanity-hsm/f24b.sanity-hsm' sanity-hsm test_24b: @@@@@@ FAIL: cannot create /mnt/lustre2/d24b.sanity-hsm/f24b.sanity-hsm with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1925:test_24b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1957:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_24b.debug_log.oleg123-server.1715208025.log.DstrVf" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24b.dmesg.oleg123-server.1715208025.log.cDAzkT" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] md5sum: /mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm: Cannot send after transport endpoint shutdown chown: cannot access '/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm': Cannot send after transport endpoint shutdown sanity-hsm test_24b: @@@@@@ FAIL: cannot chown '/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm' to '500' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1929:test_24b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1957:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24b.*.1715208027.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_24b.debug_log.oleg123-server.1715208027.log.FXq2c0" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24b.dmesg.oleg123-server.1715208027.log.8LVJ1H" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 24b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 24c: check that user,group,other request masks work ========================================================== 18:40:30 (1715208030) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d24c.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d24c.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d24c.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d24c.sanity-hsm/f24c.sanity-hsm': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0198763 s, 0.0 kB/s cannot create file '/mnt/lustre2/d24c.sanity-hsm/f24c.sanity-hsm' sanity-hsm test_24c: @@@@@@ FAIL: cannot create /mnt/lustre2/d24c.sanity-hsm/f24c.sanity-hsm with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:1987:test_24c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2022:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24c.*.1715208033.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_24c.debug_log.oleg123-server.1715208033.log.NyMcPW" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24c.dmesg.oleg123-server.1715208033.log.xPo9qT" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 24c (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 24d: check that read-only mounts are respected ========================================================== 18:40:38 (1715208038) lfs mkdir: cannot resolve path '/mnt/lustre/d24d.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d24d.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d24d.sanity-hsm': Cannot send after transport endpoint shutdown 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0661604 s, 15.8 MB/s 0x240000403:0xe:0x0 lfs getstripe: llapi_semantic_traverse: Failed to open '/mnt/lustre/d24d.sanity-hsm/f24d.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs: getstripe for '/mnt/lustre/d24d.sanity-hsm/f24d.sanity-hsm' failed: Cannot send after transport endpoint shutdown Starting client: oleg123-client.virtnet: -o user_xattr,flock oleg123-server@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Cannot send HSM request (use of /mnt/lustre2/d24d.sanity-hsm/f24d.sanity-hsm): Read-only file system lfs hsm_state: get HSM state for '/mnt/lustre/d24d.sanity-hsm/f24d.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_24d: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d24d.sanity-hsm/f24d.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11439:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2054:test_24d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2071:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_24d.debug_log.oleg123-server.1715208040.log.u45lyI" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24d.dmesg.oleg123-server.1715208040.log.3AbGKY" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] sanity-hsm test_24d: @@@@@@ FAIL: hsm flags on /mnt/lustre/d24d.sanity-hsm/f24d.sanity-hsm are Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24d.*.1715208040.log != 0x00000000 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11440:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2054:test_24d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2071:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24d.*.1715208042.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_24d.debug_log.oleg123-server.1715208042.log.RZnhBz" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24d.dmesg.oleg123-server.1715208042.log.o1t2eU" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Stopping client oleg123-client.virtnet /mnt/lustre3 (opts:) FAIL 24d (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 24e: tar succeeds on HSM released files ========================================================== 18:40:45 (1715208045) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d24e.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d24e.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d24e.sanity-hsm': Cannot send after transport endpoint shutdown 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0543689 s, 19.3 MB/s Cannot stat /mnt/lustre/d24e.sanity-hsm/f24e.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_24e: @@@@@@ FAIL: cannot archive /mnt/lustre/d24e.sanity-hsm/f24e.sanity-hsm Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2081:test_24e() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2090:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24e.*.1715208047.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_24e.debug_log.oleg123-server.1715208047.log.NAGEFJ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24e.dmesg.oleg123-server.1715208047.log.NUt9nh" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 24e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 24f: root can archive, release, and restore tar files ========================================================== 18:40:50 (1715208050) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d24f.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d24f.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d24f.sanity-hsm': Cannot send after transport endpoint shutdown mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d24f.sanity-hsm/f24f.sanity-hsm': No such file or directory sanity-hsm test_24f: @@@@@@ FAIL: cannot create /mnt/lustre2/d24f.sanity-hsm/f24f.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2099:test_24f() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2115:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_24f.debug_log.oleg123-server.1715208051.log.kTBJHu" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24f.dmesg.oleg123-server.1715208051.log.i66xre" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] md5sum: /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm: Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_24f: @@@@@@ FAIL: hsm_archive failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2103:test_24f() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2115:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24f.*.1715208053.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_24f.debug_log.oleg123-server.1715208053.log.ZsluNb" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24f.dmesg.oleg123-server.1715208053.log.KAImSZ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 24f (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 24g: write by non-owner still sets dirty ========================================================== 18:40:56 (1715208056) RUNAS = 'runas -u 500 -g 500' Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d24g.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d24g.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d24g.sanity-hsm': Cannot send after transport endpoint shutdown chmod: cannot access '/mnt/lustre/d24g.sanity-hsm': Cannot send after transport endpoint shutdown tee: /mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm: Cannot send after transport endpoint shutdown Please listen carefully as our options have changed. lfs path2fid: cannot get fid for '/mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm': Cannot send after transport endpoint shutdown chmod: cannot access '/mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm': Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm: Cannot send after transport endpoint shutdown lfs hsm_state: get HSM state for '/mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_24g: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11439:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2137:test_24g() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2142:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_24g.debug_log.oleg123-server.1715208058.log.8PGi1c" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24g.dmesg.oleg123-server.1715208058.log.hrbnMd" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] sanity-hsm test_24g: @@@@@@ FAIL: hsm flags on /mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm are Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24g.*.1715208058.log != 0x00000009 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11440:check_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2137:test_24g() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2142:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24g.*.1715208059.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_24g.debug_log.oleg123-server.1715208059.log.qm98H4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_24g.dmesg.oleg123-server.1715208059.log.svV6S9" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 24g (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 25a: Restore lost file (HS_LOST flag) from import ========================================================== 18:41:03 (1715208063) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown sanity-hsm test_25a: @@@@@@ FAIL: cannot create directory '/mnt/lustre/d25a.sanity-hsm' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11273:__lhsmtool_import() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11380:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2153:test_25a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2162:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_25a.*.1715208064.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_25a.debug_log.oleg123-server.1715208064.log.mXHEz5" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_25a.dmesg.oleg123-server.1715208064.log.Ppvaqn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 25a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 25b: Restore lost file (HS_LOST flag) after release ========================================================== 18:41:08 (1715208068) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d25b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d25b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d25b.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d25b.sanity-hsm/f25b.sanity-hsm': No such file or directory sanity-hsm test_25b: @@@@@@ FAIL: cannot create /mnt/lustre2/d25b.sanity-hsm/f25b.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2172:test_25b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2184:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_25b.debug_log.oleg123-server.1715208069.log.4y9pPT" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_25b.dmesg.oleg123-server.1715208069.log.WdOuWn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d25b.sanity-hsm/f25b.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Update not seen after 200s: want 'log' got '' sanity-hsm test_25b: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2175:test_25b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2184:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_25b.*.1715208272.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_25b.debug_log.oleg123-server.1715208272.log.8TYnOI" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_25b.dmesg.oleg123-server.1715208272.log.jHxbiN" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 25b (205s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 26A: Remove the archive of a valid file ========================================================== 18:44:35 (1715208275) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d26A.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d26A.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d26A.sanity-hsm': Cannot send after transport endpoint shutdown 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0025653 s, 0.0 kB/s Cannot stat /mnt/lustre/d26A.sanity-hsm/f26A.sanity-hsm: Cannot send after transport endpoint shutdown Waiting 200s for 'SUCCEED' Waiting 180s for 'SUCCEED' Waiting 160s for 'SUCCEED' Waiting 150s for 'SUCCEED' Waiting 140s for 'SUCCEED' Waiting 120s for 'SUCCEED' Waiting 100s for 'SUCCEED' Waiting 90s for 'SUCCEED' Waiting 80s for 'SUCCEED' Waiting 70s for 'SUCCEED' Waiting 50s for 'SUCCEED' Waiting 30s for 'SUCCEED' Waiting 20s for 'SUCCEED' Waiting 10s for 'SUCCEED' Waiting 0s for 'SUCCEED' Update not seen after 200s: want 'SUCCEED' got '' sanity-hsm test_26A: @@@@@@ FAIL: request on 0x240000403:0x10:0x0 is not SUCCEED on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2196:test_26A() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2203:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_26A.*.1715208478.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_26A.debug_log.oleg123-server.1715208478.log.THVG4S" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_26A.dmesg.oleg123-server.1715208478.log.r3ARgG" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 26A (205s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 26a: Remove Archive On Last Unlink (RAoLU) policy ========================================================== 18:48:02 (1715208482) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d26a.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d26a.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d26a.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d26a.sanity-hsm/f26a.sanity-hsm': No such file or directory sanity-hsm test_26a: @@@@@@ FAIL: cannot create /mnt/lustre2/d26a.sanity-hsm/f26a.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2214:test_26a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2256:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_26a.debug_log.oleg123-server.1715208483.log.3XGReU" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_26a.dmesg.oleg123-server.1715208483.log.K3gQZT" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d26a.sanity-hsm/f26a.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_26a: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2217:test_26a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2256:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_26a.*.1715208686.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_26a.debug_log.oleg123-server.1715208686.log.qMTjhS" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_26a.dmesg.oleg123-server.1715208686.log.FFYvGq" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 26a (205s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 26b: RAoLU policy when CDT off ======== 18:51:29 (1715208689) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d26b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d26b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d26b.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d26b.sanity-hsm/f26b.sanity-hsm': No such file or directory sanity-hsm test_26b: @@@@@@ FAIL: cannot create /mnt/lustre2/d26b.sanity-hsm/f26b.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2264:test_26b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2288:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_26b.debug_log.oleg123-server.1715208691.log.EoKETQ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_26b.dmesg.oleg123-server.1715208691.log.bAVCmB" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d26b.sanity-hsm/f26b.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_26b: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2267:test_26b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2288:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_26b.*.1715208894.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_26b.debug_log.oleg123-server.1715208894.log.WAHscB" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_26b.dmesg.oleg123-server.1715208894.log.zm0diW" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 26b (206s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 26c: RAoLU effective when file closed ========================================================== 18:54:57 (1715208897) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d26c.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d26c.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d26c.sanity-hsm': Cannot send after transport endpoint shutdown cp: cannot create regular file '/mnt/lustre2/d26c.sanity-hsm/f26c.sanity-hsm': No such file or directory sanity-hsm test_26c: @@@@@@ FAIL: cannot create /mnt/lustre2/d26c.sanity-hsm/f26c.sanity-hsm with cp, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2296:test_26c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2335:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_26c.debug_log.oleg123-server.1715208898.log.9CgTot" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_26c.dmesg.oleg123-server.1715208898.log.xoVYr0" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d26c.sanity-hsm/f26c.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_26c: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2299:test_26c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2335:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_26c.*.1715209101.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_26c.debug_log.oleg123-server.1715209101.log.H0GxWP" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_26c.dmesg.oleg123-server.1715209101.log.zrLCtW" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 26c (205s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 26d: RAoLU when Client eviction ======= 18:58:24 (1715209104) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d26d.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d26d.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d26d.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d26d.sanity-hsm/f26d.sanity-hsm': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0136427 s, 0.0 kB/s cannot create file '/mnt/lustre2/d26d.sanity-hsm/f26d.sanity-hsm' sanity-hsm test_26d: @@@@@@ FAIL: cannot create /mnt/lustre2/d26d.sanity-hsm/f26d.sanity-hsm with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2343:test_26d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2373:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_26d.debug_log.oleg123-server.1715209106.log.ppEVOt" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_26d.dmesg.oleg123-server.1715209106.log.wBGqNL" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d26d.sanity-hsm/f26d.sanity-hsm: Cannot send after transport endpoint shutdown sanity-hsm test_26d: @@@@@@ FAIL: could not archive file Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2345:test_26d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2373:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_26d.*.1715209107.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_26d.debug_log.oleg123-server.1715209107.log.ZUg1qc" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_26d.dmesg.oleg123-server.1715209107.log.AO8RIy" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 26d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 27a: Remove the archive of an imported file (Operation not permitted) ========================================================== 18:58:31 (1715209111) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB) copied, 0.360329 s, 113 MB/s mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown sanity-hsm test_27a: @@@@@@ FAIL: cannot create directory '/mnt/lustre/d27a.sanity-hsm' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11273:__lhsmtool_import() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11380:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2381:test_27a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2388:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_27a.*.1715209112.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_27a.debug_log.oleg123-server.1715209112.log.humlys" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_27a.dmesg.oleg123-server.1715209112.log.hNvBa2" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 27a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 27b: Remove the archive of a relased file (Operation not permitted) ========================================================== 18:58:36 (1715209116) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d27b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d27b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d27b.sanity-hsm': Cannot send after transport endpoint shutdown 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00221629 s, 0.0 kB/s Cannot stat /mnt/lustre/d27b.sanity-hsm/f27b.sanity-hsm: Cannot send after transport endpoint shutdown Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Waiting 160s for 'SUCCEED' Waiting 150s for 'SUCCEED' Waiting 130s for 'SUCCEED' Waiting 120s for 'SUCCEED' Waiting 110s for 'SUCCEED' Waiting 100s for 'SUCCEED' Waiting 90s for 'SUCCEED' Waiting 80s for 'SUCCEED' Waiting 60s for 'SUCCEED' Waiting 50s for 'SUCCEED' Waiting 40s for 'SUCCEED' Waiting 30s for 'SUCCEED' Waiting 20s for 'SUCCEED' Waiting 0s for 'SUCCEED' Update not seen after 200s: want 'SUCCEED' got '' sanity-hsm test_27b: @@@@@@ FAIL: request on 0x240000403:0x12:0x0 is not SUCCEED on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2399:test_27b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2406:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_27b.*.1715209318.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_27b.debug_log.oleg123-server.1715209318.log.0GfXOC" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_27b.dmesg.oleg123-server.1715209318.log.6cf2XS" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 27b (203s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 28: Concurrent archive/file remove ==== 19:02:01 (1715209321) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_28/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d28.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d28.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d28.sanity-hsm': Cannot send after transport endpoint shutdown 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00169866 s, 0.0 kB/s Cannot stat /mnt/lustre/d28.sanity-hsm/f28.sanity-hsm: Cannot send after transport endpoint shutdown Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Waiting 170s for 'SUCCEED' Waiting 160s for 'SUCCEED' Waiting 150s for 'SUCCEED' Waiting 140s for 'SUCCEED' Waiting 130s for 'SUCCEED' Waiting 120s for 'SUCCEED' Waiting 110s for 'SUCCEED' Waiting 100s for 'SUCCEED' Waiting 90s for 'SUCCEED' Waiting 80s for 'SUCCEED' Waiting 70s for 'SUCCEED' Waiting 50s for 'SUCCEED' Waiting 30s for 'SUCCEED' Waiting 10s for 'SUCCEED' Update not seen after 200s: want 'SUCCEED' got '' sanity-hsm test_28: @@@@@@ FAIL: request on 0x200000403:0x14:0x0 is not SUCCEED on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2417:test_28() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2428:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_28.*.1715209523.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_28.debug_log.oleg123-server.1715209523.log.Qx0AOM" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_28.dmesg.oleg123-server.1715209523.log.AnC97I" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 28 (203s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 29a: Tests --mntpath and --archive options ========================================================== 19:05:27 (1715209527) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 7 "/mnt/lustre"' sanity-hsm test_29a: @@@@@@ FAIL: Failed to start copytool agt1 on 'oleg123-client.virtnet' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11399:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2434:test_29a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2446:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_29a.*.1715209528.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_29a.debug_log.oleg123-server.1715209528.log.aq4BZu" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_29a.dmesg.oleg123-server.1715209528.log.DrL8bD" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 29a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 29b: Archive/delete/remove by FID from the archive. ========================================================== 19:05:31 (1715209531) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d29b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d29b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d29b.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d29b.sanity-hsm/f29b.sanity-hsm': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0181003 s, 0.0 kB/s cannot create file '/mnt/lustre2/d29b.sanity-hsm/f29b.sanity-hsm' sanity-hsm test_29b: @@@@@@ FAIL: cannot create /mnt/lustre2/d29b.sanity-hsm/f29b.sanity-hsm with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2454:test_29b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2464:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_29b.debug_log.oleg123-server.1715209533.log.1inYZI" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_29b.dmesg.oleg123-server.1715209533.log.8EXnf4" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d29b.sanity-hsm/f29b.sanity-hsm: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 160s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 70s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 20s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_29b: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2457:test_29b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2464:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_29b.*.1715209736.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_29b.debug_log.oleg123-server.1715209736.log.XW6oC6" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_29b.dmesg.oleg123-server.1715209736.log.MK9ni1" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 29b (206s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 29c: Archive/delete/remove by FID, using a file list. ========================================================== 19:08:59 (1715209739) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d29c.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d29c.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d29c.sanity-hsm': Cannot send after transport endpoint shutdown dd: error writing '/mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-1': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0186924 s, 0.0 kB/s cannot create file '/mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-1' sanity-hsm test_29c: @@@@@@ FAIL: cannot create /mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-1 with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2471:test_29c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2492:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.debug_log.oleg123-server.1715209740.log.g3sSN4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.dmesg.oleg123-server.1715209740.log.dUORMb" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0632677 s, 16.6 MB/s dd: error writing '/mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-3': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0188228 s, 0.0 kB/s cannot create file '/mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-3' sanity-hsm test_29c: @@@@@@ FAIL: cannot create /mnt/lustre2/d29c.sanity-hsm/f29c.sanity-hsm-3 with dd, status=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:243:create_small_file() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2473:test_29c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2492:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.debug_log.oleg123-server.1715209742.log.rumsm3" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.dmesg.oleg123-server.1715209742.log.WEHKif" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d29c.sanity-hsm/f29c.sanity-hsm-[1-3]: Cannot send after transport endpoint shutdown oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 200s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 190s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 180s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 170s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 150s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 140s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 130s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 120s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 110s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 100s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 90s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 80s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 60s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 50s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 40s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 30s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 10s for 'log' oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory oleg123-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory Waiting 0s for 'log' Update not seen after 200s: want 'log' got '' sanity-hsm test_29c: @@@@@@ FAIL: request on Dumping is not log on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2476:test_29c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2492:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_29c.*.1715209945.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.debug_log.oleg123-server.1715209945.log.xrhFaP" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_29c.dmesg.oleg123-server.1715209945.log.ozJ7oB" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 29c (208s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ========================================================== 19:12:29 (1715209949) SKIP: sanity-hsm test_29d Need 3 or more clients, have 1 SKIP 29d (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 30a: Restore at exec (import case) ==== 19:12:31 (1715209951) SKIP: sanity-hsm test_30a Need 2 or more clients, have 1 SKIP 30a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 30b: Restore at exec (release case) === 19:12:34 (1715209954) SKIP: sanity-hsm test_30b Need 2 or more clients, have 1 SKIP 30b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 30c: Update during exec of released file must fail ========================================================== 19:12:37 (1715209957) SKIP: sanity-hsm test_30c Need 2 or more clients, have 1 SKIP 30c (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 31a: Import a large file and check size during restore ========================================================== 19:12:39 (1715209959) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d31a.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d31a.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d31a.sanity-hsm': Cannot send after transport endpoint shutdown 39+0 records in 39+0 records out 40894464 bytes (41 MB) copied, 0.322458 s, 127 MB/s mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown sanity-hsm test_31a: @@@@@@ FAIL: cannot create directory '/mnt/lustre/d31a.sanity-hsm' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11273:__lhsmtool_import() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11380:copytool() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2696:test_31a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2705:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_31a.*.1715209961.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_31a.debug_log.oleg123-server.1715209961.log.yOGKyq" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31a.dmesg.oleg123-server.1715209961.log.REUmUP" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 31a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ========================================================== 19:12:43 (1715209963) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d31b.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d31b.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d31b.sanity-hsm': Cannot send after transport endpoint shutdown mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown dd: failed to open '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown cannot create file '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' Cannot stat /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm: Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm: Cannot send after transport endpoint shutdown stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2662:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715209965.log.nfYJ2U" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715209965.log.d5dswv" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm: Cannot send after transport endpoint shutdown stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715209976.log.wbVTsD" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715209976.log.7SG1GH" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715209988.log.hItxnK" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715209988.log.DIkibj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715209999.log.S1pNa9" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715209999.log.amJBTc" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715210011.log.fenStj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715210011.log.mjtUNR" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715210023.log.psWUan" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715210023.log.etUz2p" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715210035.log.GiV9qE" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715210035.log.FlXuWb" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715210046.log.I909dO" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715210046.log.faPLoQ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715210058.log.pbQMo2" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715210058.log.f5Zcpz" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715210070.log.4ZzRRh" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715210070.log.uXdipj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715210081.log.agORXN" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715210081.log.k2UtVk" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31b.sanity-hsm/f31b.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2720:test_31b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2725:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.debug_log.oleg123-server.1715210093.log.tTjzrM" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31b.dmesg.oleg123-server.1715210093.log.8MtyAN" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] restore is too long FAIL 31b (133s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 31c: Restore a large aligned file and check size during restore ========================================================== 19:14:58 (1715210098) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d31c.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d31c.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d31c.sanity-hsm': Cannot send after transport endpoint shutdown mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown dd: failed to open '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown cannot create file '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' Cannot stat /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm: Cannot send after transport endpoint shutdown Cannot stat /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm: Cannot send after transport endpoint shutdown stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2662:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210100.log.67X1iW" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210100.log.hUMNzf" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] Cannot stat /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm: Cannot send after transport endpoint shutdown stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210112.log.KLMEm3" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210112.log.ROgOcR" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210124.log.QqygMl" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210124.log.ymuwdE" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210135.log.WBXMzy" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210135.log.wmt3Kl" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210147.log.JbiWTF" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210147.log.KdMXEX" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210159.log.qDRGrR" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210159.log.5YaGGD" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210170.log.nlZEu3" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210170.log.hWhWtk" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210182.log.Pi2Nzc" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210182.log.qtHJfY" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210194.log.nP9Vax" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210194.log.vDiH6P" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210207.log.5i4nuR" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210207.log.RIZrnF" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210218.log.o1AX31" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210218.log.350hJk" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown size seen is right: == lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main() rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210230.log.aIdaoe" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210230.log.LU2HO1" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] restore is too long FAIL 31c (135s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 33: Kill a restore waiting process ==== 19:17:15 (1715210235) lfs mkdir: cannot resolve path '/mnt/lustre/d33.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d33.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d33.sanity-hsm': Cannot send after transport endpoint shutdown 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00223724 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_33/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Cannot stat /mnt/lustre/d33.sanity-hsm/f33.sanity-hsm: Cannot send after transport endpoint shutdown Waiting 200s for 'SUCCEED' Waiting 180s for 'SUCCEED' Waiting 160s for 'SUCCEED' Waiting 140s for 'SUCCEED' Waiting 120s for 'SUCCEED' Waiting 100s for 'SUCCEED' Waiting 80s for 'SUCCEED' Waiting 70s for 'SUCCEED' Waiting 60s for 'SUCCEED' Waiting 50s for 'SUCCEED' Waiting 30s for 'SUCCEED' Waiting 10s for 'SUCCEED' Waiting 0s for 'SUCCEED' Update not seen after 200s: want 'SUCCEED' got '' sanity-hsm test_33: @@@@@@ FAIL: request on 0x240000403:0x16:0x0 is not SUCCEED on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2755:test_33() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2774:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_33.*.1715210438.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_33.debug_log.oleg123-server.1715210438.log.GJHeXv" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_33.dmesg.oleg123-server.1715210438.log.vwZTk7" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 33 (204s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 34: Remove file during restore ======== 19:20:42 (1715210442) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_34/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d34.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d34.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d34.sanity-hsm': Cannot send after transport endpoint shutdown 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.002028 s, 0.0 kB/s Cannot stat /mnt/lustre/d34.sanity-hsm/f34.sanity-hsm: Cannot send after transport endpoint shutdown Waiting 200s for 'SUCCEED' Waiting 180s for 'SUCCEED' Waiting 170s for 'SUCCEED' Waiting 160s for 'SUCCEED' Waiting 150s for 'SUCCEED' Waiting 130s for 'SUCCEED' Waiting 120s for 'SUCCEED' Waiting 100s for 'SUCCEED' Waiting 90s for 'SUCCEED' Waiting 80s for 'SUCCEED' Waiting 70s for 'SUCCEED' Waiting 60s for 'SUCCEED' Waiting 50s for 'SUCCEED' Waiting 30s for 'SUCCEED' Waiting 20s for 'SUCCEED' Waiting 0s for 'SUCCEED' Update not seen after 200s: want 'SUCCEED' got '' sanity-hsm test_34: @@@@@@ FAIL: request on 0x200000403:0x1a:0x0 is not SUCCEED on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2785:test_34() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2809:main() Dumping lctl log to /tmp/testlogs//sanity-hsm.test_34.*.1715210644.log rsync: chown "/tmp/testlogs/.sanity-hsm.test_34.debug_log.oleg123-server.1715210644.log.lMI5UF" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-hsm.test_34.dmesg.oleg123-server.1715210644.log.rqgFKX" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 34 (203s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-hsm test 35: Overwrite file during restore ===== 19:24:07 (1715210647) Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_35/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' lfs mkdir: cannot resolve path '/mnt/lustre/d35.sanity-hsm': Cannot send after transport endpoint shutdown (108) lfs mkdir: '/mnt/lustre/d35.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108) lfs setdirstripe: cannot create dir '/mnt/lustre/d35.sanity-hsm': Cannot send after transport endpoint shutdown 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0024506 s, 0.0 kB/s Cannot stat /mnt/lustre/d35.sanity-hsm/f35.sanity-hsm: Cannot send after transport endpoint shutdown Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED'