== replay-single test 70c: tar 2mdts recovery ============ 12:22:50 (1740676970) Starting client oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre Started clients oleg432-client.virtnet: 192.168.204.132@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar 27610 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70c.replay-single tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 7232 1280456 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4840 1282848 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 4232 3576152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13804 3559088 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 18036 7135240 1% /mnt/lustre test_70c fail mds1 1 times Failing mds1 on oleg432-server Stopping /mnt/lustre-mds1 (opts:) on oleg432-server 12:25:03 (1740677103) shut down facet: mds1 facet_host: oleg432-server facet_failover_host: oleg432-server Failover mds1 to oleg432-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 12:25:17 (1740677117) targets are mounted 12:25:17 (1740677117) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 7444 1280244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5000 1282688 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9872 3568864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5400 3571640 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 15272 7140504 1% /mnt/lustre test_70c fail mds2 2 times Failing mds2 on oleg432-server Stopping /mnt/lustre-mds2 (opts:) on oleg432-server 12:27:41 (1740677261) shut down facet: mds2 facet_host: oleg432-server facet_failover_host: oleg432-server Failover mds2 to oleg432-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0001 12:27:54 (1740677274) targets are mounted 12:27:54 (1740677274) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec