-----============= acceptance-small: replay-dual ============----- Thu Aug 27 09:23:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg109-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 09:24:16 (1787837056) === Starting client oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Mount client oleg109-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Started clients oleg109-client.virtnet: 192.168.201.109@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg109-client.virtnet: executing check_config_client /mnt/lustre oleg109-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg109-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff949d10125000.idle_timeout=debug osc.lustre-OST0000-osc-ffff949d11d33800.idle_timeout=debug osc.lustre-OST0001-osc-ffff949d10125000.idle_timeout=debug osc.lustre-OST0001-osc-ffff949d11d33800.idle_timeout=debug disable quota as required oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all osd-ldiskfs.track_declares_assert=1 === replay-dual: finish setup 09:24:56 (1787837096) === == replay-dual test 0a: expired recovery with lost client ========================================================== 09:24:58 (1787837098) Check file is LU482_FAILED=/tmp/replay-dual.lu482.A47lHG UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1708 1269596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 50 open/close in 1.75 seconds: 28.58 ops/second fail_loc=0x80000514 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:25:12 (1787837112) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:25:48 (1787837148) targets are mounted 09:25:48 (1787837148) facet_failover done Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1787837261 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (170s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 09:27:48 (1787837268) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:28:01 (1787837281) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:28:32 (1787837312) targets are mounted 09:28:32 (1787837312) facet_failover done Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 PASS 0b (220s) == replay-dual test 1: |X| simple create ================= 09:31:28 (1787837488) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg109-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:31:41 (1787837501) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:32:14 (1787837534) targets are mounted 09:32:14 (1787837534) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (63s) == replay-dual test 2: |X| mkdir adir ==================== 09:32:31 (1787837551) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:32:44 (1787837564) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:33:19 (1787837599) targets are mounted 09:33:19 (1787837599) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (63s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 09:33:34 (1787837614) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:33:47 (1787837627) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:34:23 (1787837663) targets are mounted 09:34:23 (1787837663) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (66s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 09:34:40 (1787837680) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1269552 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:34:56 (1787837696) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:35:31 (1787837731) targets are mounted 09:35:31 (1787837731) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (68s) == replay-dual test 5: open, unlink |X| close ============ 09:35:48 (1787837748) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8223 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:36:05 (1787837765) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:36:39 (1787837799) targets are mounted 09:36:39 (1787837799) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (70s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 09:36:58 (1787837818) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8223 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8223 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:37:11 (1787837831) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:37:44 (1787837864) targets are mounted 09:37:44 (1787837864) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (70s) == replay-dual test 8: replay of resent request ========== 09:38:08 (1787837888) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:38:42 (1787837922) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:39:18 (1787837958) targets are mounted 09:39:18 (1787837958) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (88s) == replay-dual test 9: resending a replayed create ======= 09:39:36 (1787837976) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:39:53 (1787837993) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:40:26 (1787838026) targets are mounted 09:40:26 (1787838026) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (78s) == replay-dual test 10: resending a replayed unlink ====== 09:40:54 (1787838054) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:41:10 (1787838070) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:41:45 (1787838105) targets are mounted 09:41:45 (1787838105) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (77s) == replay-dual test 11: both clients timeout during replay ========================================================== 09:42:11 (1787838131) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:42:26 (1787838146) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:43:00 (1787838180) targets are mounted 09:43:00 (1787838180) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 11 (66s) == replay-dual test 12: open resend timeout ============== 09:43:18 (1787838198) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8223 fail_loc=0x80000302 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:43:36 (1787838216) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:44:10 (1787838250) targets are mounted 09:44:10 (1787838250) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (74s) == replay-dual test 13: close resend timeout ============= 09:44:32 (1787838272) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8223 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:44:49 (1787838289) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:45:24 (1787838324) targets are mounted 09:45:24 (1787838324) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (74s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 09:45:48 (1787838348) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 1.07 seconds: 23.40 ops/second total: 1 open/close in 0.02 seconds: 42.58 ops/second Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:46:05 (1787838365) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:46:42 (1787838402) targets are mounted 09:46:42 (1787838402) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1787838476 ; total 0 ; last 0) total: 25 unlinks in 2 seconds: 12.500000 unlinks/second Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 PASS 15a (136s) == replay-dual test 15c: remove multiple OST orphans ===== 09:48:04 (1787838484) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:50:12 (1787838612) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:50:48 (1787838648) targets are mounted 09:50:48 (1787838648) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 PASS 15c (243s) == replay-dual test 16: fail MDS during recovery (3571) == 09:52:07 (1787838727) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1748 1269556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 0.98 seconds: 25.64 ops/second total: 1 open/close in 0.02 seconds: 55.02 ops/second Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:52:23 (1787838743) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:52:57 (1787838777) targets are mounted 09:52:57 (1787838777) facet_failover done Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 09:53:21 (1787838801) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 09:53:56 (1787838836) targets are mounted 09:53:56 (1787838836) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1787838912 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 PASS 16 (190s) == replay-dual test 17: fail OST during recovery (3571) == 09:55:18 (1787838918) total: 25 open/close in 0.82 seconds: 30.59 ops/second total: 1 open/close in 0.09 seconds: 11.19 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1780 1269524 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing ost1 on oleg109-server Stopping /mnt/lustre-ost1 (opts:) on oleg109-server 09:55:34 (1787838934) shut down facet: ost1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover ost1 to oleg109-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 09:56:06 (1787838966) targets are mounted 09:56:06 (1787838966) facet_failover done Failing ost1 on oleg109-server Stopping /mnt/lustre-ost1 (opts:) on oleg109-server 09:56:32 (1787838992) shut down facet: ost1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover ost1 to oleg109-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 09:56:58 (1787839018) targets are mounted 09:56:58 (1787839018) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1787839098 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 PASS 17 (188s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 09:58:26 (1787839106) debug=+dlmtrace fail_loc=0x8000030b using seed 2412678167 running for 500 iterations total: 500 stats in 1 seconds: 500.000000 stats/second ldlm.namespaces.MGC192.168.201.109@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff949d07ddc000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff949d11d34000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff949d07ddc000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff949d11d34000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff949d07ddc000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff949d11d34000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff949d07ddc000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff949d11d34000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.201.109@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff949d07ddc000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff949d11d34000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff949d07ddc000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff949d11d34000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff949d07ddc000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff949d11d34000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff949d07ddc000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff949d11d34000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (61s) == replay-dual test 19: resend of open request =========== 09:59:27 (1787839167) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1787839266.67 total 86.86 last 0.00) total: 1 open/close in 86.86 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 10:01:12 (1787839272) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 10:01:47 (1787839307) targets are mounted 10:01:47 (1787839307) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (158s) == replay-dual test 20: recovery time is not increasing == 10:02:05 (1787839325) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 10:02:20 (1787839340) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 10:02:54 (1787839374) targets are mounted 10:02:54 (1787839374) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1632 1269672 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 10:05:33 (1787839533) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 10:06:09 (1787839569) targets are mounted 10:06:09 (1787839569) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 PASS 20 (395s) == replay-dual test 21a: commit on sharing =============== 10:08:40 (1787839720) mdt.lustre-MDT0000.commit_on_sharing=1 mdt.lustre-MDT0001.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 10:08:57 (1787839737) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 10:09:33 (1787839773) targets are mounted 10:09:33 (1787839773) facet_failover done Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 mdt.lustre-MDT0001.commit_on_sharing=0 PASS 21a (192s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 10:11:54 (1787839914) fail_loc=0x119 Failing mds2 on oleg109-server Stopping /mnt/lustre-mds2 (opts:) on oleg109-server 10:12:03 (1787839923) shut down facet: mds2 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds2 to oleg109-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0001 10:12:26 (1787839946) targets are mounted 10:12:26 (1787839946) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1636 1269668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.09 seconds: 21.65 ops/second total: 2 open/close in 0.04 seconds: 50.50 ops/second Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 10:12:56 (1787839976) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 10:13:30 (1787840010) targets are mounted 10:13:30 (1787840010) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22a (114s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 10:13:48 (1787840028) fail_loc=0x119 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server Failing mds2 on oleg109-server Stopping /mnt/lustre-mds2 (opts:) on oleg109-server 10:14:02 (1787840042) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server facet: mds2 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Failover mds2 to oleg109-server mount facets: mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 10:14:35 (1787840075) targets are mounted 10:14:35 (1787840075) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1636 1269668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.08 seconds: 24.14 ops/second total: 2 open/close in 0.07 seconds: 30.35 ops/second Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 10:15:04 (1787840104) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 10:15:39 (1787840139) targets are mounted 10:15:39 (1787840139) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22b (130s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 10:15:58 (1787840158) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 10:16:07 (1787840167) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 10:16:42 (1787840202) targets are mounted 10:16:42 (1787840202) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1648 1269656 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.16 seconds: 12.41 ops/second total: 2 open/close in 0.06 seconds: 33.35 ops/second Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 10:17:10 (1787840230) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 10:17:44 (1787840264) targets are mounted 10:17:44 (1787840264) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22c (126s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 10:18:04 (1787840284) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server Failing mds2 on oleg109-server Stopping /mnt/lustre-mds2 (opts:) on oleg109-server 10:18:30 (1787840310) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server facet: mds2 facet_host: oleg109-server facet_failover_host: oleg109-server