== sanity-hsm test 9A: Use of explicit archive number, with dedicated copytool ========================================================== 18:13:35 (1715206415)
Starting copytool 'agt1' on 'oleg238-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_9A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 3  "/mnt/lustre2"'
oleg238-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool
oleg238-client.virtnet: executing get_client_uuid /mnt/lustre2
found agent uuid=cbae74da-7571-45ad-b971-0e0378bbd4e4 archive_id=3 requests=[current:0 ok:0 errors:0] on mds1
lfs mkdir: cannot resolve path '/mnt/lustre/d9A.sanity-hsm': Cannot send after transport endpoint shutdown (108)
lfs mkdir: '/mnt/lustre/d9A.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108)
lfs setdirstripe: cannot create dir '/mnt/lustre/d9A.sanity-hsm': Cannot send after transport endpoint shutdown
cp: cannot create regular file '/mnt/lustre2/d9A.sanity-hsm/f9A.sanity-hsm': No such file or directory
 sanity-hsm test_9A: @@@@@@ FAIL: cannot create /mnt/lustre2/d9A.sanity-hsm/f9A.sanity-hsm with cp, status=1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:197:file_creation_failure()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:366:copy_file()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:882:test_9A()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:888:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_9A.debug_log.oleg238-server.1715206419.log.TI8oLW" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_9A.dmesg.oleg238-server.1715206419.log.tf7C5C" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
Cannot stat /mnt/lustre/d9A.sanity-hsm/f9A.sanity-hsm: Cannot send after transport endpoint shutdown
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 200s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 180s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 170s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 160s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 150s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 140s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 130s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 110s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 100s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 90s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 80s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 70s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 60s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 50s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 40s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 30s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 20s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Waiting 10s for 'log'
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
oleg238-server: error: get_param: param_path 'mdt/lustre-MDT000to/hsm/actions': No such file or directory
Update not seen after 200s: want 'log' got ''
 sanity-hsm test_9A: @@@@@@ FAIL: request on Dumping is not log on mds1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11524:wait_request_state()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:884:test_9A()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:888:main()
Dumping lctl log to /tmp/testlogs//sanity-hsm.test_9A.*.1715206621.log
rsync: chown "/tmp/testlogs/.sanity-hsm.test_9A.debug_log.oleg238-server.1715206621.log.hOo1TA" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_9A.dmesg.oleg238-server.1715206621.log.cTagXO" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]