-----============= acceptance-small: replay-dual ============----- Tue Sep 8 07:09:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg259-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 07:10:21 (1788865821) === Starting client oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 Mount client oleg259-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 Started clients oleg259-client.virtnet: 192.168.202.159@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg259-client.virtnet: executing check_config_client /mnt/lustre oleg259-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg259-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8af64529a800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8af650ccd800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8af64529a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8af650ccd800.idle_timeout=debug disable quota as required oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 07:11:10 (1788865870) === == replay-dual test 0a: expired recovery with lost client ========================================================== 07:11:12 (1788865872) Check file is LU482_FAILED=/tmp/replay-dual.lu482.aefocj UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 50 open/close in 2.11 seconds: 23.72 ops/second fail_loc=0x80000514 Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:11:27 (1788865887) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:12:05 (1788865925) targets are mounted 07:12:05 (1788865925) facet_failover done Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1788866042 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (179s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 07:14:11 (1788866051) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:14:22 (1788866062) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:14:58 (1788866098) targets are mounted 07:14:58 (1788866098) facet_failover done Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 PASS 0b (127s) == replay-dual test 1: |X| simple create ================= 07:16:19 (1788866179) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:16:30 (1788866190) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:17:06 (1788866226) targets are mounted 07:17:06 (1788866226) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (66s) == replay-dual test 2: |X| mkdir adir ==================== 07:17:25 (1788866245) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:17:37 (1788866257) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:18:13 (1788866293) targets are mounted 07:18:13 (1788866293) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (68s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 07:18:33 (1788866313) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:18:45 (1788866325) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:19:19 (1788866359) targets are mounted 07:19:19 (1788866359) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (69s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 07:19:42 (1788866382) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:19:53 (1788866393) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:20:28 (1788866428) targets are mounted 07:20:28 (1788866428) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (66s) == replay-dual test 5: open, unlink |X| close ============ 07:20:48 (1788866448) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7643 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:21:00 (1788866460) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:21:36 (1788866496) targets are mounted 07:21:36 (1788866496) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (69s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 07:21:57 (1788866517) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7643 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7643 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:22:09 (1788866529) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:22:43 (1788866563) targets are mounted 07:22:43 (1788866563) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (63s) == replay-dual test 8: replay of resent request ========== 07:23:00 (1788866580) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:23:31 (1788866611) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:24:06 (1788866646) targets are mounted 07:24:06 (1788866646) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (87s) == replay-dual test 9: resending a replayed create ======= 07:24:27 (1788866667) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:24:41 (1788866681) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:25:08 (1788866708) targets are mounted 07:25:08 (1788866708) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (73s) == replay-dual test 10: resending a replayed unlink ====== 07:25:41 (1788866741) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:25:54 (1788866754) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:26:18 (1788866778) targets are mounted 07:26:18 (1788866778) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (74s) == replay-dual test 11: both clients timeout during replay ========================================================== 07:26:54 (1788866814) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:27:06 (1788866826) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:27:43 (1788866863) targets are mounted 07:27:43 (1788866863) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 12 sec fail_loc=0 PASS 11 (79s) == replay-dual test 12: open resend timeout ============== 07:28:13 (1788866893) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7643 fail_loc=0x80000302 Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:28:31 (1788866911) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:29:05 (1788866945) targets are mounted 07:29:05 (1788866945) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (60s) == replay-dual test 13: close resend timeout ============= 07:29:13 (1788866953) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7643 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:29:27 (1788866967) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:30:00 (1788867000) targets are mounted 07:30:00 (1788867000) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (62s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 07:30:16 (1788867016) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 0.78 seconds: 31.99 ops/second total: 1 open/close in 0.03 seconds: 32.72 ops/second Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:30:28 (1788867028) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:30:54 (1788867054) targets are mounted 07:30:54 (1788867054) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788867138 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 PASS 15a (128s) == replay-dual test 15c: remove multiple OST orphans ===== 07:32:24 (1788867144) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:34:22 (1788867262) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:35:00 (1788867300) targets are mounted 07:35:00 (1788867300) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 PASS 15c (247s) == replay-dual test 16: fail MDS during recovery (3571) == 07:36:32 (1788867392) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 0.84 seconds: 29.93 ops/second total: 1 open/close in 0.03 seconds: 29.97 ops/second Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:36:44 (1788867404) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:37:16 (1788867436) targets are mounted 07:37:16 (1788867436) facet_failover done Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:37:43 (1788867463) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:38:19 (1788867499) targets are mounted 07:38:19 (1788867499) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788867581 ; total 0 ; last 0) total: 25 unlinks in 2 seconds: 12.500000 unlinks/second Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 PASS 16 (197s) == replay-dual test 17: fail OST during recovery (3571) == 07:39:49 (1788867589) total: 25 open/close in 0.89 seconds: 28.20 ops/second total: 1 open/close in 0.04 seconds: 26.95 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing ost1 on oleg259-server Stopping /mnt/lustre-ost1 (opts:) on oleg259-server 07:40:01 (1788867601) shut down facet: ost1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover ost1 to oleg259-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 07:40:30 (1788867630) targets are mounted 07:40:30 (1788867630) facet_failover done Failing ost1 on oleg259-server Stopping /mnt/lustre-ost1 (opts:) on oleg259-server 07:40:56 (1788867656) shut down facet: ost1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover ost1 to oleg259-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 07:41:26 (1788867686) targets are mounted 07:41:26 (1788867686) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788867758 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 PASS 17 (176s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 07:42:45 (1788867765) debug=+dlmtrace fail_loc=0x8000030b using seed 1759795635 running for 500 iterations total: 500 stats in 1 seconds: 500.000000 stats/second ldlm.namespaces.MGC192.168.202.159@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8af645290800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8af6586d2000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8af645290800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8af6586d2000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8af645290800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8af6586d2000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.202.159@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8af645290800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8af6586d2000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8af645290800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8af6586d2000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8af645290800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8af6586d2000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (62s) == replay-dual test 19: resend of open request =========== 07:43:47 (1788867827) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 2048 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1788867922.83 total 86.97 last 0.00) total: 1 open/close in 86.97 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:45:29 (1788867929) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:45:56 (1788867956) targets are mounted 07:45:56 (1788867956) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (150s) == replay-dual test 20: recovery time is not increasing == 07:46:17 (1788867977) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:46:32 (1788867992) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:47:04 (1788868024) targets are mounted 07:47:04 (1788868024) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:49:40 (1788868180) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:50:15 (1788868215) targets are mounted 07:50:15 (1788868215) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 PASS 20 (389s) == replay-dual test 21a: commit on sharing =============== 07:52:46 (1788868366) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 07:53:00 (1788868380) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 07:53:37 (1788868417) targets are mounted 07:53:37 (1788868417) facet_failover done Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (191s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 07:55:58 (1788868558) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (3s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 07:56:02 (1788868562) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (3s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 07:56:05 (1788868565) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (4s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 07:56:09 (1788868569) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (3s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 07:56:13 (1788868573) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (4s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 07:56:17 (1788868577) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (4s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 07:56:22 (1788868582) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (3s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 07:56:25 (1788868585) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (3s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 07:56:29 (1788868589) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (95s) == replay-dual test 25: replay|resend ==================== 07:58:04 (1788868684) 1+0 records in 1+0 records out 512 bytes copied, 0.0102294 s, 50.1 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg259-server Stopping /mnt/lustre-ost1 (opts:) on oleg259-server 07:58:14 (1788868694) shut down facet: ost1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover ost1 to oleg259-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 07:58:40 (1788868720) targets are mounted 07:58:40 (1788868720) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7010: 43812 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (55s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 07:58:59 (1788868739) Starting client oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre Started clients oleg259-client.virtnet: 192.168.202.159@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45237 Started dbench loop with 45238 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 8192 7498752 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Sep 8 07:59:06 EDT 2026 waiting for dbench pid 45316 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 15 0.07 MB/sec warmup 1 sec latency 232.084 ms 1 69 1.53 MB/sec warmup 2 sec latency 182.745 ms 1 113 1.74 MB/sec warmup 3 sec latency 185.845 ms 1 143 1.51 MB/sec warmup 4 sec latency 160.682 ms test_26 fail mds1 1 times 1 166 1.35 MB/sec warmup 5 sec latency 152.945 ms 1 189 1.26 MB/sec warmup 6 sec latency 334.651 ms 1 215 1.19 MB/sec warmup 7 sec latency 165.913 ms Failing mds1 on oleg259-server 1 251 1.18 MB/sec warmup 8 sec latency 222.067 ms Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 1 279 1.16 MB/sec warmup 9 sec latency 161.845 ms 1 296 1.09 MB/sec warmup 10 sec latency 601.023 ms 1 296 0.99 MB/sec warmup 11 sec latency 1605.631 ms 1 296 0.91 MB/sec warmup 12 sec latency 2605.913 ms 07:59:19 (1788868759) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server 1 296 0.84 MB/sec warmup 13 sec latency 3606.404 ms 1 296 0.78 MB/sec warmup 14 sec latency 4607.502 ms 1 296 0.73 MB/sec warmup 15 sec latency 5607.749 ms 1 296 0.68 MB/sec warmup 16 sec latency 6609.040 ms 1 296 0.64 MB/sec warmup 17 sec latency 7609.311 ms