== conf-sanity test 27b: Reacquire MGS lock after failover ========================================================== 21:54:24 (1749606864)
start mds service on oleg408-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
start ost1 service on oleg408-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg408-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
mount lustre  on /mnt/lustre.....
Starting client: oleg408-client.virtnet:  -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre
mount.lustre: mount 192.168.204.108@tcp:/lustre at /mnt/lustre failed: Input/output error
Is the MGS running?
Failing mds1 on oleg408-server
21:57:14 (1749607034) shut down
facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server
Failover mds1 to oleg408-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg408-server: mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: File exists
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
 conf-sanity test_27b: @@@@@@ FAIL: Restart of mds1 failed! 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:2139:mount_facets()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:4327:facet_failover()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1120:test_27b()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1130:main()
Dumping lctl log to /tmp/testlogs//conf-sanity.test_27b.*.1749607057.log
21:57:38 (1749607058) targets are mounted
21:57:38 (1749607058) facet_failover done
rsync: chown "/tmp/testlogs/.conf-sanity.test_27b.debug_log.oleg408-server.1749607057.log.smcWqq" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.conf-sanity.test_27b.dmesg.oleg408-server.1749607057.log.2qNEvU" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
oleg408-server: error: get_param: param_path 'mdt//identity_acquire_expire': No such file or directory
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
oleg408-server: error: get_param: param_path 'mdt//identity_acquire_expire': No such file or directory
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2
Setting .mdt.identity_acquire_expire from  to 5
oleg408-server: error: conf_param: Unknown error 524
pdsh@oleg408-client: oleg408-server: ssh exited with exit code 12
 conf-sanity test_27b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lctl conf_param .mdt.identity_acquire_expire failed 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7343:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6143:set_persistent_param()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6160:set_persistent_param_and_check()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1121:test_27b()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7720:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7783:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7582:run_test()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:1130:main()
Dumping lctl log to /tmp/testlogs//conf-sanity.test_27b.*.1749607060.log
rsync: chown "/tmp/testlogs/.conf-sanity.test_27b.debug_log.oleg408-server.1749607060.log.ctkpfE" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.conf-sanity.test_27b.dmesg.oleg408-server.1749607060.log.yM0Jud" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]