== sanity-hsm test 407: Check for double RESTORE records in llog ========================================================== 17:16:26 (1718399786)
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.0014253 s, 0.0 kB/s
Starting copytool 'agt1' on 'oleg215-client.virtnet' with cmdline 'lhsmtool_posix --hsm-root=/tmp/arc1/sanity-hsm.test_407/ --archive-format=v2 --daemon --pid-file=/var/run/lhsmtool_posix.pid  "/mnt/lustre2"'
Waiting 200s for 'SUCCEED'
fail_val=5
fail_loc=0x164
Copytool is suspended on oleg215-client.virtnet
mdt.lustre-MDT0000.hsm.actions=
lrh=[type=10680000 len=136 idx=5/1] fid=[0x200002b11:0x2:0x0] dfid=[0x200002b11:0x2:0x0] compound/cookie=0x0/0x666cb305 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=5/2] fid=[0x200002b12:0x2:0x0] dfid=[0x200002b12:0x2:0x0] compound/cookie=0x0/0x666cb306 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
reboot facets: mds1
Failover mds1 to oleg215-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
fail_loc=0
mdt.lustre-MDT0000.hsm.actions=
lrh=[type=10680000 len=136 idx=5/1] fid=[0x200002b11:0x2:0x0] dfid=[0x200002b11:0x2:0x0] compound/cookie=0x0/0x666cb305 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=5/2] fid=[0x200002b12:0x2:0x0] dfid=[0x200002b12:0x2:0x0] compound/cookie=0x0/0x666cb306 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=5/3] fid=[0x200002b12:0x2:0x0] dfid=[0x200002b12:0x2:0x0] compound/cookie=0x0/0x666cb307 action=RESTORE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
Copytool is continued on oleg215-client.virtnet
Waiting 100s for ''
mdt.lustre-MDT0000.hsm.actions=
lrh=[type=10680000 len=136 idx=5/1] fid=[0x200002b11:0x2:0x0] dfid=[0x200002b11:0x2:0x0] compound/cookie=0x0/0x666cb305 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=5/2] fid=[0x200002b12:0x2:0x0] dfid=[0x200002b12:0x2:0x0] compound/cookie=0x0/0x666cb306 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=5/3] fid=[0x200002b12:0x2:0x0] dfid=[0x200002b12:0x2:0x0] compound/cookie=0x0/0x666cb307 action=RESTORE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
Waiting 90s for ''
Waiting 80s for ''
Waiting 70s for ''
Waiting 60s for ''
Waiting 50s for ''
Waiting 40s for ''
Waiting 10s for ''
Waiting 0s for ''
Update not seen after 100s: want '' got 'lrh=[type=10680000 len=136 idx=5/3] fid=[0x200002b12:0x2:0x0] dfid=[0x200002b12:0x2:0x0] compound/cookie=0x0/0x666cb307 action=RESTORE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]'
 sanity-hsm test_407: @@@@@@ FAIL: requests did not complete 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6427:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:415:wait_all_done()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:5324:test_407()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6744:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6791:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6632:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:5326:main()
Dumping lctl log to /tmp/testlogs//sanity-hsm.test_407.*.1718399920.log
rsync: chown "/tmp/testlogs/.sanity-hsm.test_407.debug_log.oleg215-server.1718399920.log.29EcGR" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_407.dmesg.oleg215-server.1718399920.log.Tljfrt" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]