== sanity-hsm test 407: Check for double RESTORE records in llog ========================================================== 02:40:05 (1733643605)
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00122441 s, 0.0 kB/s
Starting copytool 'agt1' on 'oleg156-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_407/ --daemon --pid-file=/var/run/lhsmtool_posix.pid  "/mnt/lustre2"'
fail_val=5
fail_loc=0x164
Copytool is suspended on oleg156-client.virtnet
mdt.lustre-MDT0000.hsm.actions=
lrh=[type=10680000 len=136 idx=6/1] fid=[0x200000403:0x231:0x0] dfid=[0x200000403:0x231:0x0] compound/cookie=0x0/0x67554cf5 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/2] fid=[0x200000403:0x235:0x0] dfid=[0x200000403:0x235:0x0] compound/cookie=0x0/0x67554cf6 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x67554cf7 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x67554cf8 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x67554cf9 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x67554cfa action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x67554cfb action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x67554cfc action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x67554cfd action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x67554cfe action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
Failing mds1 on oleg156-server
Stopping /mnt/lustre-mds1 (opts:) on oleg156-server
02:40:12 (1733643612) shut down
facet: mds1 facet_host: oleg156-server facet_failover_host: oleg156-server
Failover mds1 to oleg156-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
mdt.lustre-MDT0000.hsm.actions=
lrh=[type=10680000 len=136 idx=6/1] fid=[0x200000403:0x231:0x0] dfid=[0x200000403:0x231:0x0] compound/cookie=0x0/0x67554cf5 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/2] fid=[0x200000403:0x235:0x0] dfid=[0x200000403:0x235:0x0] compound/cookie=0x0/0x67554cf6 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x67554cf7 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x67554cf8 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x67554cf9 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x67554cfa action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x67554cfb action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x67554cfc action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x67554cfd action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x67554cfe action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x67554cff action=RESTORE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[]
oleg156-server: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1
Started lustre-MDT0000
02:40:43 (1733643643) targets are mounted
02:40:43 (1733643643) facet_failover done
oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
fail_loc=0
mdt.lustre-MDT0000.hsm.actions=
lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x67554cf8 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x67554cf9 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x67554cfa action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x67554cfb action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x67554cfc action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x67554cfd action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x67554cfe action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]
lrh=[type=10680000 len=136 idx=6/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x67554cff action=RESTORE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
Copytool is continued on oleg156-client.virtnet
d41d8cd98f00b204e9800998ecf8427e  /mnt/lustre/d407.sanity-hsm/f407.sanity-hsm
d41d8cd98f00b204e9800998ecf8427e  /mnt/lustre2/d407.sanity-hsm/f407.sanity-hsm
pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1