== replay-single test 70c: tar 1mdts recovery ============ 01:25:03 (1782365103) Starting client oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre Started clients oleg224-client.virtnet: 192.168.202.124@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar 129699 tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5376 2202880 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3335168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3376128 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 23552 6711296 1% /mnt/lustre tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets test_70c fail mds1 1 times Failing mds1 on oleg224-server Stopping /mnt/lustre-mds1 (opts:) on oleg224-server 01:27:18 (1782365238) shut down facet: mds1 facet_host: oleg224-server facet_failover_host: oleg224-server Failover mds1 to oleg224-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 01:27:33 (1782365253) targets are mounted 01:27:33 (1782365253) facet_failover done tar: etc/rdma/modules: Cannot mkdir: Input/output error tar: etc/rdma/modules/roce.conf: Cannot open: File exists tar: etc/rdma/modules/rdma.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/rdma/modules/opa.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/rdma/modules/iwarp.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/rdma/modules/infiniband.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/rdma/modules: Cannot utime: Cannot send after transport endpoint shutdown tar: etc/rdma/modules: Cannot change ownership to uid 0, gid 0: Cannot send after transport endpoint shutdown tar: etc/rdma: Cannot utime: Cannot send after transport endpoint shutdown tar: etc/rdma: Cannot change ownership to uid 0, gid 0: Cannot send after transport endpoint shutdown tar: etc/rc.d: Cannot mkdir: Input/output error tar: etc/rc.d/rc.local: Cannot open: Cannot send after transport endpoint shutdown tar: etc/rc.d/rc6.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/rc.d/rc5.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/rc.d/rc4.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/rc.d/rc3.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/rc.d/rc2.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/rc.d/rc1.d: Cannot mkdir: Cannot send after transport endpoint shutdown pdsh@oleg224-client: oleg224-client: ssh exited with exit code 5 tar: etc/rc.d/rc0.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/rc.d/init.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/rc.d/init.d/README: Cannot open: Cannot send after transport endpoint shutdown tar: etc/rc.d/init.d/functions: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d: Cannot mkdir: Input/output error tar: etc/profile.d/which2.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/sh.local: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/gawk.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/gawk.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/debuginfod.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/cvs.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/cvs.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/csh.local: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/colorzgrep.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/colorls.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/colorls.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/colorgrep.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/colorgrep.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/colorxzgrep.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/which2.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/colorzgrep.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/colorxzgrep.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/debuginfod.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/lang.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/lang.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/less.sh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/profile.d/less.csh: Cannot open: Cannot send after transport endpoint shutdown tar: etc/prelink.conf.d: Cannot mkdir: Input/output error tar: etc/postfix: Cannot mkdir: Input/output error tar: etc/postfix/postfix-files: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/virtual: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/transport: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/relocated: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/master.cf.proto: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/master.cf: Cannot hard link to 'etc/postfix/master.cf.proto': Cannot send after transport endpoint shutdown tar: etc/postfix/main.cf.proto: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/main.cf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/header_checks: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/generic: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/dynamicmaps.cf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/canonical: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/access: Cannot open: Cannot send after transport endpoint shutdown tar: etc/postfix/postfix-files.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/postfix/dynamicmaps.cf.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/popt.d: Cannot mkdir: Input/output error tar: etc/polkit-1: Cannot mkdir: Input/output error tar: etc/polkit-1/rules.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/polkit-1/rules.d/50-default.rules: Cannot open: Cannot send after transport endpoint shutdown tar: etc/polkit-1/rules.d/49-polkit-pkla-compat.rules: Cannot open: Cannot send after transport endpoint shutdown tar: etc/polkit-1/localauthority: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/polkit-1/localauthority/90-mandatory.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/polkit-1/localauthority/50-local.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/polkit-1/localauthority/30-site.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/polkit-1/localauthority/20-org.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/polkit-1/localauthority/10-vendor.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/polkit-1/localauthority.conf.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pm: Cannot mkdir: Input/output error tar: etc/pm/sleep.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pm/power.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pm/config.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki: Cannot mkdir: Input/output error tar: etc/pki/tls: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/tls/cert.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/tls/openssl.cnf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/tls/ct_log_list.cnf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/tls/private: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/tls/private/postfix.key: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/tls/misc: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/tls/certs: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/tls/certs/ca-bundle.trust.crt: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/tls/certs/ca-bundle.crt: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/tls/certs/postfix.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/swid: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/swid/CA: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/rsyslog: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/rpm-gpg: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/rpm-gpg/RPM-GPG-KEY-rockytesting: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-8-Testing: Cannot create symlink to 'RPM-GPG-KEY-rockytesting': Cannot send after transport endpoint shutdown tar: etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-8: Cannot create symlink to 'RPM-GPG-KEY-rockyofficial': Cannot send after transport endpoint shutdown tar: etc/pki/product: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/product/491.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/product/479.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/product-default: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/nssdb: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/nssdb/secmod.db: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/nssdb/pkcs11.txt: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/nssdb/key4.db: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/nssdb/key3.db: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/nssdb/cert9.db: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/nssdb/cert8.db: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/java: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/java/cacerts: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/entitlement: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/entitlement/3434645747755415203.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/entitlement/3434645747755415203-key.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/consumer: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/consumer/cert.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/consumer/key.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/ca-legacy.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/source: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/source/README: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/source/blacklist: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/source/anchors: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/source/ca-bundle.legacy.crt: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/pem: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/pem/README: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/openssl: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/openssl/README: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/java: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/java/README: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/java/cacerts: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/edk2: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/edk2/README: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/edk2/cacerts.bin: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/extracted/README: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pki/ca-trust/README: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pkcs11: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pkcs11/modules: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pam.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/pam.d/subscription-manager: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/systemd-user: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/crond: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/polkit-1: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/vlock: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/sudo-i: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/sudo: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/su-l: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/su: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/sshd: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/smtp.postfix: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/runuser-l: Cannot open: Cannot send after transport endpoint shutdown tar: etc/pam.d/runuser: Cannot open: Cannot send after transport endpoint shutdown oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec tar: Exiting with failure status due to previous errors replay-single test_70c: @@@@@@ FAIL: tar failed in loop Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7447:error() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:2158:test_70c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7823:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7886:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7686:run_test() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:2173:main() Dumping lctl log to /tmp/testlogs//replay-single.test_70c.*.1782365448.log replay-single test_70c: @@@@@@ FAIL: 1: tar failed rsync: chown "/tmp/testlogs/.replay-single.test_70c.debug_log.oleg224-server.1782365448.log.Wlcj63" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-single.test_70c.dmesg.oleg224-server.1782365448.log.supd0Q" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7447:error() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:2168:test_70c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7823:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7886:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7686:run_test() = /home/green/git/lustre-release/lustre/tests/replay-single.sh:2173:main() Dumping lctl log to /tmp/testlogs//replay-single.test_70c.*.1782365450.log rsync: chown "/tmp/testlogs/.replay-single.test_70c.debug_log.oleg224-server.1782365450.log.bFxTlL" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-single.test_70c.dmesg.oleg224-server.1782365450.log.UrEBQg" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]