-----============= acceptance-small: replay-dual ============----- Thu Aug 27 09:36:31 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg210-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 09:36:55 (1787837815) === Starting client oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Mount client oleg210-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Started clients oleg210-client.virtnet: 192.168.202.110@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg210-client.virtnet: executing check_config_client /mnt/lustre oleg210-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg210-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff981bc4f02800.idle_timeout=debug osc.lustre-OST0000-osc-ffff981bd264e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff981bc4f02800.idle_timeout=debug osc.lustre-OST0001-osc-ffff981bd264e800.idle_timeout=debug disable quota as required oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 09:37:36 (1787837856) === == replay-dual test 0a: expired recovery with lost client ========================================================== 09:37:38 (1787837858) Check file is LU482_FAILED=/tmp/replay-dual.lu482.6uvcTe UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 50 open/close in 2.00 seconds: 25.05 ops/second fail_loc=0x80000514 Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:37:51 (1787837871) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:38:23 (1787837903) targets are mounted 09:38:23 (1787837903) facet_failover done Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1787838025 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (175s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 09:40:33 (1787838033) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:40:45 (1787838045) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:41:19 (1787838079) targets are mounted 09:41:19 (1787838079) facet_failover done Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 PASS 0b (222s) == replay-dual test 1: |X| simple create ================= 09:44:15 (1787838255) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:44:28 (1787838268) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:45:05 (1787838305) targets are mounted 09:45:05 (1787838305) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (68s) == replay-dual test 2: |X| mkdir adir ==================== 09:45:24 (1787838324) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:45:35 (1787838335) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:46:01 (1787838361) targets are mounted 09:46:01 (1787838361) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (56s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 09:46:19 (1787838379) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:46:30 (1787838390) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:47:03 (1787838423) targets are mounted 09:47:03 (1787838423) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (65s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 09:47:24 (1787838444) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:47:34 (1787838454) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:47:59 (1787838479) targets are mounted 09:47:59 (1787838479) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (57s) == replay-dual test 5: open, unlink |X| close ============ 09:48:21 (1787838501) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7620 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:48:32 (1787838512) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:49:05 (1787838545) targets are mounted 09:49:05 (1787838545) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (61s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 09:49:23 (1787838563) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:49:34 (1787838574) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:50:08 (1787838608) targets are mounted 09:50:08 (1787838608) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (62s) == replay-dual test 8: replay of resent request ========== 09:50:24 (1787838624) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:50:55 (1787838655) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:51:30 (1787838690) targets are mounted 09:51:30 (1787838690) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (86s) == replay-dual test 9: resending a replayed create ======= 09:51:50 (1787838710) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:52:02 (1787838722) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:52:38 (1787838758) targets are mounted 09:52:38 (1787838758) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (74s) == replay-dual test 10: resending a replayed unlink ====== 09:53:04 (1787838784) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:53:16 (1787838796) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:53:50 (1787838830) targets are mounted 09:53:50 (1787838830) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (73s) == replay-dual test 11: both clients timeout during replay ========================================================== 09:54:17 (1787838857) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:54:33 (1787838873) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:55:08 (1787838908) targets are mounted 09:55:08 (1787838908) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 12 sec fail_loc=0 PASS 11 (79s) == replay-dual test 12: open resend timeout ============== 09:55:36 (1787838936) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 fail_loc=0x80000302 Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:55:49 (1787838949) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:56:14 (1787838974) targets are mounted 09:56:14 (1787838974) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (53s) == replay-dual test 13: close resend timeout ============= 09:56:29 (1787838989) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:56:41 (1787839001) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:57:16 (1787839036) targets are mounted 09:57:16 (1787839036) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (62s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 09:57:33 (1787839053) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 0.79 seconds: 31.74 ops/second total: 1 open/close in 0.03 seconds: 32.29 ops/second Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:57:44 (1787839064) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:58:17 (1787839097) targets are mounted 09:58:17 (1787839097) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1787839173 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 PASS 15a (128s) == replay-dual test 15c: remove multiple OST orphans ===== 09:59:40 (1787839180) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 10:01:50 (1787839310) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 10:02:24 (1787839344) targets are mounted 10:02:24 (1787839344) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 PASS 15c (256s) == replay-dual test 16: fail MDS during recovery (3571) == 10:03:57 (1787839437) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 1.27 seconds: 19.69 ops/second total: 1 open/close in 0.05 seconds: 22.21 ops/second Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 10:04:10 (1787839450) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 10:04:36 (1787839476) targets are mounted 10:04:36 (1787839476) facet_failover done Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 10:05:02 (1787839502) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 10:05:35 (1787839535) targets are mounted 10:05:35 (1787839535) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1787839621 ; total 0 ; last 0) total: 25 unlinks in 2 seconds: 12.500000 unlinks/second Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 PASS 16 (191s) == replay-dual test 17: fail OST during recovery (3571) == 10:07:09 (1787839629) total: 25 open/close in 0.99 seconds: 25.30 ops/second total: 1 open/close in 0.06 seconds: 15.74 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing ost1 on oleg210-server Stopping /mnt/lustre-ost1 (opts:) on oleg210-server 10:07:21 (1787839641) shut down facet: ost1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover ost1 to oleg210-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 10:07:50 (1787839670) targets are mounted 10:07:50 (1787839670) facet_failover done Failing ost1 on oleg210-server Stopping /mnt/lustre-ost1 (opts:) on oleg210-server 10:08:16 (1787839696) shut down facet: ost1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover ost1 to oleg210-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 10:08:46 (1787839726) targets are mounted 10:08:46 (1787839726) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1787839798 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 PASS 17 (176s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 10:10:05 (1787839805) debug=+dlmtrace using seed 2417600712 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.202.110@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff981bd22a1000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff981bd86d2000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff981bd22a1000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff981bd86d2000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff981bd22a1000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff981bd86d2000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.202.110@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff981bd22a1000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff981bd86d2000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff981bd22a1000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff981bd86d2000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff981bd22a1000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff981bd86d2000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (61s) == replay-dual test 19: resend of open request =========== 10:11:07 (1787839867) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1787839962.57 total 86.71 last 0.00) total: 1 open/close in 86.71 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 10:12:49 (1787839969) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 10:13:24 (1787840004) targets are mounted 10:13:24 (1787840004) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (156s) == replay-dual test 20: recovery time is not increasing == 10:13:42 (1787840022) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 10:13:54 (1787840034) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 10:14:29 (1787840069) targets are mounted 10:14:29 (1787840069) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 10:17:05 (1787840225) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 10:17:37 (1787840257) targets are mounted 10:17:37 (1787840257) facet_failover done