== recovery-small test 155: failover after client remount ========================================================== 17:11:34 (1717017094)
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
Starting client: oleg328-client.virtnet:  -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre
Replay barrier on lustre-MDT0000
Failing mds1 on oleg328-server
Stopping /mnt/lustre-mds1 (opts:) on oleg328-server
17:11:42 (1717017102) shut down
Failover mds1 to oleg328-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
17:11:56 (1717017116) targets are mounted
17:11:56 (1717017116) facet_failover done
 recovery-small test_155: @@@@@@ FAIL: total 2263
drwxr-xr-x 3 root root   11776 May 29 16:47 d107.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:48 d108.recovery-small
drwxrwxrwx 2 root root   11776 May 29 15:51 d10c.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:57 d135.recovery-small
drwxr-xr-x 4 root root   11776 May 29 16:58 d137.recovery-small
drwxr-xr-x 2 root root 1420800 May 29 17:07 d144c.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:52 d14.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:54 d18a.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:54 d18b.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:54 d18c.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:58 d20a.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:58 d20b.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:06 d24a.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:06 d24b.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:08 d27.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:13 d50.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:17 d51.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:33 d52.recovery-small
drwxr-xr-x 2 root root  469504 May 29 16:43 d60.recovery-small
-rw-r--r-- 1 root root      12 May 29 15:51 f10d.recovery-small
-rwxrwxrwx 1 root root       0 May 29 16:51 f113.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115a.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:49 f115b.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115c.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115d.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115e.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115f.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115g.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:50 f120.recovery-small
-rw-r--r-- 1 root root     512 May 29 16:56 f131.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:56 f132.recovery-small
-rw-r--r-- 1 root root       0 May 29 17:07 f147.recovery-small
-rw-r--r-- 1 root root       0 May 29 17:11 f155.recovery-small
-rw-r--r-- 1 root root  102400 May 29 15:52 f16.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:58 f19c.recovery-small
-rw-r--r-- 1 root root       0 May 29 15:44 f1.recovery-small
-rw-rw-rw- 1 root root       0 May 29 15:44 f1.recovery-small.2
-rw-r--r-- 1 root root       0 May 29 16:06 f23.recovery-small
-rw-r--r-- 1 root root   10240 May 29 15:46 f4.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:33 f53a.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:34 f53b.recovery-small
-rwxr-xr-x 1 root root       0 May 29 16:34 f53c.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:34 f54.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:34 f54.recovery-small.1
-rw-r--r-- 1 root root     512 May 29 16:41 f59.recovery-small
-rw-r--r-- 1 root root       1 May 29 16:44 f65.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:44 f66.recovery-small
-rw-r--r-- 1 root root       0 May 29 15:48 f8.recovery-small != total 2263
drwxr-xr-x 3 root root   11776 May 29 16:47 d107.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:48 d108.recovery-small
drwxrwxrwx 2 root root   11776 May 29 15:51 d10c.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:57 d135.recovery-small
drwxr-xr-x 4 root root   11776 May 29 16:58 d137.recovery-small
drwxr-xr-x 2 root root 1420800 May 29 17:07 d144c.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:52 d14.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:54 d18a.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:54 d18b.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:54 d18c.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:58 d20a.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:58 d20b.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:06 d24a.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:06 d24b.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:08 d27.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:13 d50.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:17 d51.recovery-small
drwxr-xr-x 2 root root   11776 May 29 16:33 d52.recovery-small
drwxr-xr-x 2 root root  469504 May 29 16:43 d60.recovery-small
-rw-r--r-- 1 root root      12 May 29 15:51 f10d.recovery-small
-rwxrwxrwx 1 root root       0 May 29 16:51 f113.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115a.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:49 f115b.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115c.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115d.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115e.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115f.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:49 f115g.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:50 f120.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:56 f131.recovery-small
-rw-r--r-- 1 root root    4096 May 29 16:56 f132.recovery-small
-rw-r--r-- 1 root root       0 May 29 17:07 f147.recovery-small
-rw-r--r-- 1 root root       0 May 29 17:11 f155.recovery-small
-rw-r--r-- 1 root root  102400 May 29 15:52 f16.recovery-small
drwxr-xr-x 2 root root   11776 May 29 15:58 f19c.recovery-small
-rw-r--r-- 1 root root       0 May 29 15:44 f1.recovery-small
-rw-rw-rw- 1 root root       0 May 29 15:44 f1.recovery-small.2
-rw-r--r-- 1 root root       0 May 29 16:06 f23.recovery-small
-rw-r--r-- 1 root root   10240 May 29 15:46 f4.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:33 f53a.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:34 f53b.recovery-small
-rwxr-xr-x 1 root root       0 May 29 16:34 f53c.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:34 f54.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:34 f54.recovery-small.1
-rw-r--r-- 1 root root     512 May 29 16:41 f59.recovery-small
-rw-r--r-- 1 root root       1 May 29 16:44 f65.recovery-small
-rw-r--r-- 1 root root       0 May 29 16:44 f66.recovery-small
-rw-r--r-- 1 root root       0 May 29 15:48 f8.recovery-small 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7068:error()
  = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3568:test_155()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7413:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7476:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7299:run_test()
  = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3570:main()
Dumping lctl log to /tmp/testlogs//recovery-small.test_155.*.1717017118.log
rsync: chown "/tmp/testlogs/.recovery-small.test_155.debug_log.oleg328-server.1717017118.log.0tGnhO" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.recovery-small.test_155.dmesg.oleg328-server.1717017118.log.P1Trgo" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]