== recovery-small test 155: failover after client remount ========================================================== 18:03:09 (1746136989) Stopping client oleg215-client.virtnet /mnt/lustre (opts:) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre Replay barrier on lustre-MDT0000 Failing mds1 on oleg215-server Stopping /mnt/lustre-mds1 (opts:) on oleg215-server 18:03:16 (1746136996) shut down facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server Failover mds1 to oleg215-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 18:03:29 (1746137009) targets are mounted 18:03:29 (1746137009) facet_failover done recovery-small test_155: @@@@@@ FAIL: total 2259 drwxr-xr-x 3 root root 11776 May 1 17:36 d107.recovery-small drwxr-xr-x 2 root root 11776 May 1 17:36 d108.recovery-small drwxrwxrwx 2 root root 11776 May 1 16:37 d10c.recovery-small drwxr-xr-x 2 root root 11776 May 1 17:46 d135.recovery-small drwxr-xr-x 4 root root 11776 May 1 17:47 d137.recovery-small drwxr-xr-x 2 root root 1416704 May 1 17:58 d144c.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:38 d14.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:40 d18a.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:40 d18b.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:41 d18c.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:44 d20a.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:44 d20b.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:52 d24a.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:52 d24b.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:54 d27.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:59 d50.recovery-small drwxr-xr-x 2 root root 11776 May 1 17:03 d51.recovery-small drwxr-xr-x 2 root root 11776 May 1 17:19 d52.recovery-small drwxr-xr-x 2 root root 469504 May 1 17:30 d60.recovery-small -rw-r--r-- 1 root root 12 May 1 16:37 f10d.recovery-small -rwxrwxrwx 1 root root 0 May 1 17:40 f113.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115a.recovery-small -rw-r--r-- 1 root root 0 May 1 17:38 f115b.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115c.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115d.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115e.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115f.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115g.recovery-small -rw-r--r-- 1 root root 0 May 1 17:39 f120.recovery-small -rw-r--r-- 1 root root 512 May 1 17:45 f131.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:45 f132.recovery-small -rw-r--r-- 1 root root 0 May 1 17:59 f147.recovery-small -rw-r--r-- 1 root root 0 May 1 18:03 f155.recovery-small -rw-r--r-- 1 root root 102400 May 1 16:39 f16.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:44 f19c.recovery-small -rw-r--r-- 1 root root 0 May 1 16:30 f1.recovery-small -rw-rw-rw- 1 root root 0 May 1 16:31 f1.recovery-small.2 -rw-r--r-- 1 root root 0 May 1 16:52 f23.recovery-small -rw-r--r-- 1 root root 10240 May 1 16:32 f4.recovery-small -rw-r--r-- 1 root root 0 May 1 17:19 f53a.recovery-small -rw-r--r-- 1 root root 0 May 1 17:20 f53b.recovery-small -rwxr-xr-x 1 root root 0 May 1 17:20 f53c.recovery-small -rw-r--r-- 1 root root 0 May 1 17:20 f54.recovery-small -rw-r--r-- 1 root root 0 May 1 17:20 f54.recovery-small.1 -rw-r--r-- 1 root root 512 May 1 17:27 f59.recovery-small -rw-r--r-- 1 root root 1 May 1 17:31 f65.recovery-small -rw-r--r-- 1 root root 0 May 1 17:31 f66.recovery-small -rw-r--r-- 1 root root 0 May 1 16:34 f8.recovery-small != total 2259 drwxr-xr-x 3 root root 11776 May 1 17:36 d107.recovery-small drwxr-xr-x 2 root root 11776 May 1 17:36 d108.recovery-small drwxrwxrwx 2 root root 11776 May 1 16:37 d10c.recovery-small drwxr-xr-x 2 root root 11776 May 1 17:46 d135.recovery-small drwxr-xr-x 4 root root 11776 May 1 17:47 d137.recovery-small drwxr-xr-x 2 root root 1416704 May 1 17:58 d144c.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:38 d14.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:40 d18a.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:40 d18b.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:41 d18c.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:44 d20a.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:44 d20b.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:52 d24a.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:52 d24b.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:54 d27.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:59 d50.recovery-small drwxr-xr-x 2 root root 11776 May 1 17:03 d51.recovery-small drwxr-xr-x 2 root root 11776 May 1 17:19 d52.recovery-small drwxr-xr-x 2 root root 469504 May 1 17:30 d60.recovery-small -rw-r--r-- 1 root root 12 May 1 16:37 f10d.recovery-small -rwxrwxrwx 1 root root 0 May 1 17:40 f113.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115a.recovery-small -rw-r--r-- 1 root root 0 May 1 17:38 f115b.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115c.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115d.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115e.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115f.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:38 f115g.recovery-small -rw-r--r-- 1 root root 0 May 1 17:39 f120.recovery-small -rw-r--r-- 1 root root 0 May 1 17:45 f131.recovery-small -rw-r--r-- 1 root root 4096 May 1 17:45 f132.recovery-small -rw-r--r-- 1 root root 0 May 1 17:59 f147.recovery-small -rw-r--r-- 1 root root 0 May 1 18:03 f155.recovery-small -rw-r--r-- 1 root root 102400 May 1 16:39 f16.recovery-small drwxr-xr-x 2 root root 11776 May 1 16:44 f19c.recovery-small -rw-r--r-- 1 root root 0 May 1 16:30 f1.recovery-small -rw-rw-rw- 1 root root 0 May 1 16:31 f1.recovery-small.2 -rw-r--r-- 1 root root 0 May 1 16:52 f23.recovery-small -rw-r--r-- 1 root root 10240 May 1 16:32 f4.recovery-small -rw-r--r-- 1 root root 0 May 1 17:19 f53a.recovery-small -rw-r--r-- 1 root root 0 May 1 17:20 f53b.recovery-small -rwxr-xr-x 1 root root 0 May 1 17:20 f53c.recovery-small -rw-r--r-- 1 root root 0 May 1 17:20 f54.recovery-small -rw-r--r-- 1 root root 0 May 1 17:20 f54.recovery-small.1 -rw-r--r-- 1 root root 512 May 1 17:27 f59.recovery-small -rw-r--r-- 1 root root 1 May 1 17:31 f65.recovery-small -rw-r--r-- 1 root root 0 May 1 17:31 f66.recovery-small -rw-r--r-- 1 root root 0 May 1 16:34 f8.recovery-small Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3620:test_155() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3622:main() Dumping lctl log to /tmp/testlogs//recovery-small.test_155.*.1746137010.log rsync: chown "/tmp/testlogs/.recovery-small.test_155.debug_log.oleg215-server.1746137010.log.j0T1oc" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.recovery-small.test_155.dmesg.oleg215-server.1746137010.log.Ej8KmZ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]