-----============= acceptance-small: replay-dual ============----- Thu Aug 6 23:43:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg134-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 23:43:58 (1786074238) === Starting client oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 Mount client oleg134-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 Started clients oleg134-client.virtnet: 192.168.201.134@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg134-client.virtnet: executing check_config_client /mnt/lustre oleg134-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg134-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e74c5e85800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8e74d1be5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e74c5e85800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e74d1be5800.idle_timeout=debug disable quota as required oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 23:44:41 (1786074281) === == replay-dual test 0a: expired recovery with lost client ========================================================== 23:44:44 (1786074284) Check file is LU482_FAILED=/tmp/replay-dual.lu482.sN2IYY UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 50 open/close in 1.71 seconds: 29.29 ops/second fail_loc=0x80000514 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:44:57 (1786074297) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:45:29 (1786074329) targets are mounted 23:45:29 (1786074329) facet_failover done Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1786074438 ; total 0 ; last 0) total: 50 unlinks in 3 seconds: 16.666666 unlinks/second PASS 0a (162s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 23:47:27 (1786074447) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:47:37 (1786074457) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:48:01 (1786074481) targets are mounted 23:48:01 (1786074481) facet_failover done Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 PASS 0b (207s) == replay-dual test 1: |X| simple create ================= 23:50:53 (1786074653) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:51:03 (1786074663) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:51:28 (1786074688) targets are mounted 23:51:28 (1786074688) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (53s) == replay-dual test 2: |X| mkdir adir ==================== 23:51:46 (1786074706) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:51:57 (1786074717) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:52:30 (1786074750) targets are mounted 23:52:30 (1786074750) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (61s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 23:52:47 (1786074767) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:52:57 (1786074777) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:53:22 (1786074802) targets are mounted 23:53:22 (1786074802) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (52s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 23:53:40 (1786074820) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:53:52 (1786074832) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:54:25 (1786074865) targets are mounted 23:54:25 (1786074865) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (62s) == replay-dual test 5: open, unlink |X| close ============ 23:54:42 (1786074882) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7606 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:54:54 (1786074894) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:55:29 (1786074929) targets are mounted 23:55:29 (1786074929) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (66s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 23:55:48 (1786074948) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7606 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7606 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:56:02 (1786074962) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:56:36 (1786074996) targets are mounted 23:56:36 (1786074996) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (67s) == replay-dual test 8: replay of resent request ========== 23:56:55 (1786075015) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:57:24 (1786075044) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:57:50 (1786075070) targets are mounted 23:57:50 (1786075070) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (71s) == replay-dual test 9: resending a replayed create ======= 23:58:06 (1786075086) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:58:18 (1786075098) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 23:58:46 (1786075126) targets are mounted 23:58:46 (1786075126) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (83s) == replay-dual test 10: resending a replayed unlink ====== 23:59:30 (1786075170) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 23:59:41 (1786075181) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:00:06 (1786075206) targets are mounted 00:00:07 (1786075207) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (81s) == replay-dual test 11: both clients timeout during replay ========================================================== 00:00:51 (1786075251) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:01:02 (1786075262) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:01:37 (1786075297) targets are mounted 00:01:37 (1786075297) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 13 sec fail_loc=0 PASS 11 (76s) == replay-dual test 12: open resend timeout ============== 00:02:07 (1786075327) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7606 fail_loc=0x80000302 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:02:20 (1786075340) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:02:53 (1786075373) targets are mounted 00:02:53 (1786075373) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (64s) == replay-dual test 13: close resend timeout ============= 00:03:11 (1786075391) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7606 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:03:23 (1786075403) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:03:47 (1786075427) targets are mounted 00:03:47 (1786075427) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (54s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 00:04:07 (1786075447) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 1.05 seconds: 23.74 ops/second total: 1 open/close in 0.04 seconds: 23.83 ops/second Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:04:22 (1786075462) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:04:56 (1786075496) targets are mounted 00:04:56 (1786075496) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1786075570 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 PASS 15a (130s) == replay-dual test 15c: remove multiple OST orphans ===== 00:06:17 (1786075577) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:08:16 (1786075696) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:08:41 (1786075721) targets are mounted 00:08:41 (1786075721) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 PASS 15c (230s) == replay-dual test 16: fail MDS during recovery (3571) == 00:10:07 (1786075807) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 0.91 seconds: 27.44 ops/second total: 1 open/close in 0.04 seconds: 23.84 ops/second Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:10:20 (1786075820) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:10:45 (1786075845) targets are mounted 00:10:45 (1786075845) facet_failover done Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:11:11 (1786075871) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:11:37 (1786075897) targets are mounted 00:11:37 (1786075897) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1786075972 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 PASS 16 (173s) == replay-dual test 17: fail OST during recovery (3571) == 00:13:00 (1786075980) total: 25 open/close in 0.92 seconds: 27.32 ops/second total: 1 open/close in 0.08 seconds: 13.11 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing ost1 on oleg134-server Stopping /mnt/lustre-ost1 (opts:) on oleg134-server 00:13:12 (1786075992) shut down facet: ost1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover ost1 to oleg134-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 00:13:39 (1786076019) targets are mounted 00:13:39 (1786076019) facet_failover done Failing ost1 on oleg134-server Stopping /mnt/lustre-ost1 (opts:) on oleg134-server 00:14:04 (1786076044) shut down facet: ost1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover ost1 to oleg134-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 00:14:32 (1786076072) targets are mounted 00:14:32 (1786076072) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1786076144 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 PASS 17 (171s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 00:15:51 (1786076151) debug=+dlmtrace fail_loc=0x8000030b using seed 1516185034 running for 500 iterations total: 500 stats in 2 seconds: 250.000000 stats/second ldlm.namespaces.MGC192.168.201.134@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8e74c402f000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8e74d0f67800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8e74c402f000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8e74d0f67800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8e74c402f000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8e74d0f67800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.201.134@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8e74c402f000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8e74d0f67800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8e74c402f000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8e74d0f67800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e74c402f000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e74d0f67800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (65s) == replay-dual test 19: resend of open request =========== 00:16:56 (1786076216) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1786076311.49 total 86.39 last 0.00) total: 1 open/close in 86.39 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:18:39 (1786076319) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:19:16 (1786076356) targets are mounted 00:19:16 (1786076356) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (162s) == replay-dual test 20: recovery time is not increasing == 00:19:39 (1786076379) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:19:53 (1786076393) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:20:28 (1786076428) targets are mounted 00:20:28 (1786076428) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:23:08 (1786076588) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:23:33 (1786076613) targets are mounted 00:23:33 (1786076613) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 PASS 20 (392s) == replay-dual test 21a: commit on sharing =============== 00:26:10 (1786076770) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:26:23 (1786076783) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:26:50 (1786076810) targets are mounted 00:26:50 (1786076810) facet_failover done Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (183s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 00:29:15 (1786076955) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (4s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 00:29:19 (1786076959) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (4s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 00:29:23 (1786076963) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (4s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 00:29:27 (1786076967) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (3s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 00:29:31 (1786076971) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (4s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 00:29:34 (1786076974) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (4s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 00:29:38 (1786076978) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (4s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 00:29:42 (1786076982) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (3s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 00:29:46 (1786076986) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (96s) == replay-dual test 25: replay|resend ==================== 00:31:21 (1786077081) 1+0 records in 1+0 records out 512 bytes copied, 0.0108841 s, 47.0 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg134-server Stopping /mnt/lustre-ost1 (opts:) on oleg134-server 00:31:31 (1786077091) shut down facet: ost1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover ost1 to oleg134-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 00:31:59 (1786077119) targets are mounted 00:31:59 (1786077119) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6963: 43620 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (56s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 00:32:18 (1786077138) Starting client oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre Started clients oleg134-client.virtnet: 192.168.201.134@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45037 Started dbench loop with 45038 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3729408 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 24576 7481344 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Aug 7 00:32:24 EDT 2026 waiting for dbench pid 45106 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 1 of 1 processes prepared for launch 0 sec releasing clients 1 14 0.00 MB/sec warmup 1 sec latency 2.869 ms 1 44 0.85 MB/sec warmup 2 sec latency 214.328 ms 1 80 1.18 MB/sec warmup 3 sec latency 204.047 ms 1 111 1.31 MB/sec warmup 4 sec latency 210.701 ms 1 139 1.20 MB/sec warmup 5 sec latency 215.757 ms 1 165 1.13 MB/sec warmup 6 sec latency 260.639 ms test_26 fail mds1 1 times 1 184 1.04 MB/sec warmup 7 sec latency 271.538 ms 1 207 1.01 MB/sec warmup 8 sec latency 240.256 ms Failing mds1 on oleg134-server 1 235 0.99 MB/sec warmup 9 sec latency 161.840 ms Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 1 270 1.02 MB/sec warmup 10 sec latency 130.190 ms 1 297 1.00 MB/sec warmup 11 sec latency 171.424 ms 1 305 0.93 MB/sec warmup 12 sec latency 838.646 ms 1 305 0.86 MB/sec warmup 13 sec latency 1843.874 ms 1 305 0.80 MB/sec warmup 14 sec latency 2844.120 ms 00:32:39 (1786077159) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server 1 305 0.75 MB/sec warmup 15 sec latency 3844.330 ms 1 305 0.70 MB/sec warmup 16 sec latency 4845.224 ms 1 305 0.66 MB/sec warmup 17 sec latency 5845.515 ms 1 305 0.62 MB/sec warmup 18 sec latency 6847.671 ms 1 305 0.59 MB/sec warmup 19 sec latency 7850.012 ms 1 305 0.00 MB/sec execute 1 sec latency 9850.326 ms 1 305 0.00 MB/sec execute 2 sec latency 10854.838 ms 1 305 0.00 MB/sec execute 3 sec latency 11855.036 ms 1 305 0.00 MB/sec execute 4 sec latency 12855.416 ms 1 305 0.00 MB/sec execute 5 sec latency 13857.344 ms Failover mds1 to oleg134-server mount facets: mds1 1 305 0.00 MB/sec execute 6 sec latency 14857.519 ms 1 305 0.00 MB/sec execute 7 sec latency 15857.748 ms 1 305 0.00 MB/sec execute 8 sec latency 16857.945 ms 1 305 0.00 MB/sec execute 9 sec latency 17858.128 ms 1 305 0.00 MB/sec execute 10 sec latency 18858.949 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 305 0.00 MB/sec execute 11 sec latency 19859.440 ms 1 305 0.00 MB/sec execute 12 sec latency 20859.682 ms 1 305 0.00 MB/sec execute 13 sec latency 21859.893 ms 1 305 0.00 MB/sec execute 14 sec latency 22861.479 ms 1 305 0.00 MB/sec execute 15 sec latency 23861.698 ms 1 305 0.00 MB/sec execute 16 sec latency 24862.439 ms 1 305 0.00 MB/sec execute 17 sec latency 25862.639 ms 1 305 0.00 MB/sec execute 18 sec latency 26863.002 ms 1 305 0.00 MB/sec execute 19 sec latency 27863.867 ms 1 305 0.00 MB/sec execute 20 sec latency 28869.709 ms 1 305 0.00 MB/sec execute 21 sec latency 29870.037 ms 1 305 0.00 MB/sec execute 22 sec latency 30870.172 ms 1 305 0.00 MB/sec execute 23 sec latency 31870.459 ms 1 305 0.00 MB/sec execute 24 sec latency 32870.650 ms 1 305 0.00 MB/sec execute 25 sec latency 33870.867 ms oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 1 305 0.00 MB/sec execute 26 sec latency 34871.422 ms pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 1 305 0.00 MB/sec execute 27 sec latency 35872.473 ms Started lustre-MDT0000 00:33:12 (1786077192) targets are mounted 00:33:12 (1786077192) facet_failover done 1 305 0.00 MB/sec execute 28 sec latency 36874.298 ms 1 305 0.00 MB/sec execute 29 sec latency 37874.976 ms 1 313 0.01 MB/sec execute 30 sec latency 38770.644 ms 1 367 0.06 MB/sec execute 31 sec latency 106.144 ms 1 429 0.12 MB/sec execute 32 sec latency 125.006 ms 1 475 0.16 MB/sec execute 33 sec latency 123.695 ms oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 501 0.18 MB/sec execute 34 sec latency 163.265 ms 1 522 0.19 MB/sec execute 35 sec latency 230.352 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 584 0.24 MB/sec execute 36 sec latency 173.363 ms 1 618 0.26 MB/sec execute 37 sec latency 145.052 ms 1 656 0.26 MB/sec execute 38 sec latency 167.804 ms 1 678 0.25 MB/sec execute 39 sec latency 299.520 ms 1 710 0.25 MB/sec execute 40 sec latency 136.809 ms 1 730 0.25 MB/sec execute 41 sec latency 348.087 ms 1 778 0.24 MB/sec execute 42 sec latency 89.138 ms 1 801 0.24 MB/sec execute 43 sec latency 191.810 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 35840 3709952 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 11264 3702784 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 47104 7412736 1% /mnt/lustre 1 825 0.23 MB/sec execute 44 sec latency 128.894 ms 1 873 0.23 MB/sec execute 45 sec latency 248.600 ms 1 902 0.23 MB/sec execute 46 sec latency 275.732 ms 1 923 0.22 MB/sec execute 47 sec latency 157.737 ms 1 964 0.22 MB/sec execute 48 sec latency 131.823 ms 1 990 0.22 MB/sec execute 49 sec latency 151.864 ms test_26 fail mds1 2 times 1 1060 0.23 MB/sec execute 50 sec latency 81.712 ms 1 1095 0.23 MB/sec execute 51 sec latency 124.205 ms Failing mds1 on oleg134-server 1 1134 0.23 MB/sec execute 52 sec latency 110.358 ms Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 1 1176 0.22 MB/sec execute 53 sec latency 115.493 ms 1 1201 0.22 MB/sec execute 54 sec latency 320.337 ms 1 1201 0.21 MB/sec execute 55 sec latency 1321.525 ms 1 1201 0.21 MB/sec execute 56 sec latency 2321.700 ms 00:33:40 (1786077220) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server 1 1201 0.21 MB/sec execute 57 sec latency 3321.976 ms 1 1201 0.20 MB/sec execute 58 sec latency 4322.141 ms 1 1201 0.20 MB/sec execute 59 sec latency 5322.956 ms 1 1201 0.20 MB/sec execute 60 sec latency 6324.002 ms 1 1201 0.19 MB/sec execute 61 sec latency 7324.130 ms 1 1201 0.19 MB/sec execute 62 sec latency 8324.375 ms 1 1201 0.19 MB/sec execute 63 sec latency 9324.644 ms 1 1201 0.18 MB/sec execute 64 sec latency 10326.802 ms 1 1201 0.18 MB/sec execute 65 sec latency 11326.989 ms 1 1201 0.18 MB/sec execute 66 sec latency 12327.190 ms Failover mds1 to oleg134-server mount facets: mds1 1 1201 0.18 MB/sec execute 67 sec latency 13329.058 ms 1 1201 0.17 MB/sec execute 68 sec latency 14331.951 ms 1 1201 0.17 MB/sec execute 69 sec latency 15332.113 ms 1 1201 0.17 MB/sec execute 70 sec latency 16332.234 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1201 0.17 MB/sec execute 71 sec latency 17334.463 ms 1 1201 0.16 MB/sec execute 72 sec latency 18334.631 ms 1 1201 0.16 MB/sec execute 73 sec latency 19335.407 ms 1 1201 0.16 MB/sec execute 74 sec latency 20335.566 ms 1 1201 0.16 MB/sec execute 75 sec latency 21335.916 ms 1 1201 0.16 MB/sec execute 76 sec latency 22340.471 ms 1 1201 0.15 MB/sec execute 77 sec latency 23342.097 ms 1 1201 0.15 MB/sec execute 78 sec latency 24342.235 ms oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 1 1216 0.15 MB/sec execute 79 sec latency 24774.554 ms 1 1249 0.15 MB/sec execute 80 sec latency 131.431 ms pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 1 1296 0.15 MB/sec execute 81 sec latency 86.133 ms Started lustre-MDT0000 00:34:06 (1786077246) targets are mounted 00:34:06 (1786077246) facet_failover done 1 1405 0.18 MB/sec execute 82 sec latency 97.499 ms 1 1449 0.18 MB/sec execute 83 sec latency 100.843 ms 1 1481 0.18 MB/sec execute 84 sec latency 174.620 ms 1 1509 0.17 MB/sec execute 85 sec latency 209.120 ms 1 1560 0.17 MB/sec execute 86 sec latency 118.958 ms 1 1597 0.17 MB/sec execute 87 sec latency 164.831 ms 1 1632 0.17 MB/sec execute 88 sec latency 200.718 ms 1 1660 0.17 MB/sec execute 89 sec latency 280.881 ms oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1692 0.17 MB/sec execute 90 sec latency 152.403 ms 1 1754 0.17 MB/sec execute 91 sec latency 144.668 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1787 0.16 MB/sec execute 92 sec latency 129.722 ms 1 1828 0.16 MB/sec execute 93 sec latency 190.390 ms 1 1860 0.16 MB/sec execute 94 sec latency 189.516 ms 1 1924 0.16 MB/sec execute 95 sec latency 108.387 ms 1 1978 0.16 MB/sec execute 96 sec latency 83.745 ms 1 2012 0.16 MB/sec execute 97 sec latency 185.820 ms 1 2076 0.16 MB/sec execute 98 sec latency 128.868 ms 1 2132 0.16 MB/sec execute 99 sec latency 128.423 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 37888 3692544 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 34816 3705856 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 72704 7398400 1% /mnt/lustre 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 324 187.817 38770.630 Close 280 13.845 50.668 Rename 11 102.143 152.365 Unlink 28 31.340 56.992 Qpathinfo 266 111.194 24774.528 Qfileinfo 87 2.269 28.180 Qfsinfo 33 4.318 32.530 Sfileinfo 34 71.958 142.158 Find 88 8.539 85.039 WriteX 335 13.396 83.658 ReadX 336 2.531 167.771 Flush 30 126.944 348.070 Throughput 0.155787 MB/sec 1 clients 1 procs max_latency=38770.644 ms stopping dbench on /mnt/lustre at Fri Aug 7 00:34:28 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished test_26 fail mds1 3 times looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Aug 7 00:34:34 EDT 2026 waiting for dbench pid 47304 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 1 of 1 processes prepared for launch 0 sec releasing clients 0 10 0.00 MB/sec warmup 1 sec latency 431.897 ms 0 10 0.00 MB/sec warmup 2 sec latency 1434.328 ms 0 10 0.00 MB/sec warmup 3 sec latency 2436.908 ms 00:34:38 (1786077278) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server 0 10 0.00 MB/sec warmup 4 sec latency 3437.143 ms 0 10 0.00 MB/sec warmup 5 sec latency 4437.644 ms 0 10 0.00 MB/sec warmup 6 sec latency 5438.320 ms 0 10 0.00 MB/sec warmup 7 sec latency 6438.655 ms 0 10 0.00 MB/sec warmup 8 sec latency 7438.849 ms 0 10 0.00 MB/sec warmup 9 sec latency 8440.653 ms 0 10 0.00 MB/sec warmup 10 sec latency 9441.570 ms 0 10 0.00 MB/sec warmup 11 sec latency 10441.854 ms 0 10 0.00 MB/sec warmup 12 sec latency 11442.922 ms 0 10 0.00 MB/sec warmup 13 sec latency 12443.184 ms Failover mds1 to oleg134-server 0 10 0.00 MB/sec warmup 14 sec latency 13447.477 ms mount facets: mds1 0 10 0.00 MB/sec warmup 15 sec latency 14448.646 ms 0 10 0.00 MB/sec warmup 16 sec latency 15448.870 ms 0 10 0.00 MB/sec warmup 17 sec latency 16449.741 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 0 10 0.00 MB/sec warmup 18 sec latency 17449.987 ms 0 10 0.00 MB/sec warmup 19 sec latency 18450.491 ms 0 10 0.00 MB/sec warmup 20 sec latency 19450.623 ms 0 10 0.00 MB/sec warmup 21 sec latency 20457.868 ms 0 10 0.00 MB/sec warmup 22 sec latency 21458.164 ms 0 10 0.00 MB/sec warmup 23 sec latency 22458.464 ms 0 10 0.00 MB/sec warmup 24 sec latency 23458.916 ms oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 0 10 0.00 MB/sec warmup 25 sec latency 24459.147 ms pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 0 10 0.00 MB/sec warmup 26 sec latency 25459.501 ms Started lustre-MDT0000 00:35:01 (1786077301) targets are mounted 00:35:01 (1786077301) facet_failover done 0 10 0.00 MB/sec warmup 27 sec latency 26460.912 ms 0 10 0.00 MB/sec warmup 28 sec latency 27461.079 ms 0 10 0.00 MB/sec warmup 29 sec latency 28461.336 ms 0 10 0.00 MB/sec warmup 30 sec latency 29461.641 ms 0 10 0.00 MB/sec warmup 31 sec latency 30461.831 ms 0 10 0.00 MB/sec warmup 32 sec latency 31462.235 ms 0 10 0.00 MB/sec warmup 33 sec latency 32462.947 ms 0 10 0.00 MB/sec warmup 34 sec latency 33466.908 ms 0 10 0.00 MB/sec warmup 35 sec latency 34467.198 ms 0 10 0.00 MB/sec warmup 36 sec latency 35467.601 ms 0 10 0.00 MB/sec warmup 37 sec latency 36467.856 ms 0 10 0.00 MB/sec warmup 38 sec latency 37468.666 ms 0 10 0.00 MB/sec warmup 39 sec latency 38469.289 ms 0 10 0.00 MB/sec warmup 40 sec latency 39470.251 ms 0 10 0.00 MB/sec warmup 41 sec latency 40470.473 ms 0 10 0.00 MB/sec warmup 42 sec latency 41472.326 ms 1 108 2.20 MB/sec execute 1 sec latency 193.349 ms 1 140 1.56 MB/sec execute 2 sec latency 143.632 ms 1 189 1.56 MB/sec execute 3 sec latency 164.075 ms 1 215 1.37 MB/sec execute 4 sec latency 165.255 ms oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 244 1.27 MB/sec execute 5 sec latency 192.271 ms 1 271 1.21 MB/sec execute 6 sec latency 174.189 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 297 1.15 MB/sec execute 7 sec latency 209.203 ms 1 324 1.11 MB/sec execute 8 sec latency 171.911 ms 1 353 1.08 MB/sec execute 9 sec latency 179.069 ms 1 397 1.12 MB/sec execute 10 sec latency 114.295 ms 1 429 1.11 MB/sec execute 11 sec latency 157.185 ms 1 466 1.12 MB/sec execute 12 sec latency 117.579 ms 1 508 1.13 MB/sec execute 13 sec latency 121.444 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3685376 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 27648 3707904 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 64512 7393280 1% /mnt/lustre 1 551 1.16 MB/sec execute 14 sec latency 167.337 ms 1 639 1.21 MB/sec execute 15 sec latency 115.943 ms 1 669 1.14 MB/sec execute 16 sec latency 216.059 ms 1 694 1.07 MB/sec execute 17 sec latency 165.154 ms 1 721 1.02 MB/sec execute 18 sec latency 193.631 ms 1 767 0.97 MB/sec execute 19 sec latency 270.857 ms 1 796 0.93 MB/sec execute 20 sec latency 193.875 ms test_26 fail mds1 4 times 1 849 0.89 MB/sec execute 21 sec latency 145.355 ms 1 877 0.85 MB/sec execute 22 sec latency 206.814 ms Failing mds1 on oleg134-server 1 905 0.82 MB/sec execute 23 sec latency 121.368 ms Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 1 926 0.79 MB/sec execute 24 sec latency 344.061 ms 1 948 0.75 MB/sec execute 25 sec latency 682.394 ms 1 948 0.73 MB/sec execute 26 sec latency 1683.475 ms 1 948 0.70 MB/sec execute 27 sec latency 2683.696 ms 00:35:45 (1786077345) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server 1 948 0.67 MB/sec execute 28 sec latency 3683.921 ms 1 948 0.65 MB/sec execute 29 sec latency 4685.757 ms 1 948 0.63 MB/sec execute 30 sec latency 5686.022 ms 1 948 0.61 MB/sec execute 31 sec latency 6686.248 ms 1 948 0.59 MB/sec execute 32 sec latency 7686.895 ms 1 948 0.57 MB/sec execute 33 sec latency 8687.778 ms 1 948 0.56 MB/sec execute 34 sec latency 9688.041 ms 1 948 0.54 MB/sec execute 35 sec latency 10688.429 ms 1 948 0.52 MB/sec execute 36 sec latency 11688.631 ms 1 948 0.51 MB/sec execute 37 sec latency 12689.260 ms Failover mds1 to oleg134-server mount facets: mds1 1 948 0.50 MB/sec execute 38 sec latency 13689.869 ms 1 948 0.48 MB/sec execute 39 sec latency 14690.139 ms 1 948 0.47 MB/sec execute 40 sec latency 15692.725 ms 1 948 0.46 MB/sec execute 41 sec latency 16695.508 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 948 0.45 MB/sec execute 42 sec latency 17695.685 ms 1 948 0.44 MB/sec execute 43 sec latency 18695.933 ms 1 948 0.43 MB/sec execute 44 sec latency 19697.728 ms 1 948 0.42 MB/sec execute 45 sec latency 20700.805 ms 1 948 0.41 MB/sec execute 46 sec latency 21701.650 ms 1 948 0.40 MB/sec execute 47 sec latency 22701.826 ms 1 948 0.39 MB/sec execute 48 sec latency 23703.105 ms 1 948 0.39 MB/sec execute 49 sec latency 24703.397 ms 1 948 0.38 MB/sec execute 50 sec latency 25707.815 ms 1 948 0.37 MB/sec execute 51 sec latency 26708.262 ms 1 948 0.36 MB/sec execute 52 sec latency 27709.147 ms 1 948 0.36 MB/sec execute 53 sec latency 28711.455 ms 1 948 0.35 MB/sec execute 54 sec latency 29711.760 ms 1 948 0.34 MB/sec execute 55 sec latency 30711.942 ms 1 948 0.34 MB/sec execute 56 sec latency 31712.472 ms oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 1 948 0.33 MB/sec execute 57 sec latency 32712.693 ms 1 948 0.33 MB/sec execute 58 sec latency 33715.864 ms pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 1 948 0.32 MB/sec execute 59 sec latency 34719.161 ms Started lustre-MDT0000 00:36:17 (1786077377) targets are mounted 00:36:17 (1786077377) facet_failover done 1 948 0.31 MB/sec execute 60 sec latency 35719.404 ms 1 948 0.31 MB/sec execute 61 sec latency 36720.510 ms 1 948 0.30 MB/sec execute 62 sec latency 37721.309 ms 1 948 0.30 MB/sec execute 63 sec latency 38721.446 ms 1 949 0.30 MB/sec execute 64 sec latency 39648.123 ms 1 972 0.29 MB/sec execute 65 sec latency 280.442 ms 1 988 0.29 MB/sec execute 66 sec latency 347.427 ms 1 1054 0.30 MB/sec execute 67 sec latency 140.280 ms 1 1100 0.29 MB/sec execute 68 sec latency 82.768 ms 1 1145 0.29 MB/sec execute 69 sec latency 96.527 ms oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1180 0.29 MB/sec execute 70 sec latency 156.301 ms 1 1207 0.28 MB/sec execute 71 sec latency 152.757 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1236 0.28 MB/sec execute 72 sec latency 158.121 ms 1 1268 0.28 MB/sec execute 73 sec latency 133.010 ms 1 1403 0.31 MB/sec execute 74 sec latency 115.269 ms 1 1447 0.31 MB/sec execute 75 sec latency 109.471 ms 1 1492 0.30 MB/sec execute 76 sec latency 105.744 ms 1 1558 0.30 MB/sec execute 77 sec latency 81.442 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1609 0.30 MB/sec execute 78 sec latency 112.830 ms 1 1653 0.29 MB/sec execute 79 sec latency 228.727 ms 1 1706 0.29 MB/sec execute 80 sec latency 93.569 ms 1 1783 0.29 MB/sec execute 81 sec latency 69.116 ms 1 1845 0.29 MB/sec execute 82 sec latency 98.935 ms 1 1886 0.28 MB/sec execute 83 sec latency 297.486 ms 1 1956 0.28 MB/sec execute 84 sec latency 228.763 ms 1 2018 0.28 MB/sec execute 85 sec latency 83.688 ms 1 2107 0.28 MB/sec execute 86 sec latency 82.295 ms 1 2158 0.27 MB/sec execute 87 sec latency 94.984 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Aug 7 00:36:45 EDT 2026 with return code 0 47304 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47304 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (304s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 00:37:23 (1786077443) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0274995 s, 149 kB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg134-server Stopping /mnt/lustre-ost1 (opts:) on oleg134-server 00:37:55 (1786077475) shut down facet: ost1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover ost1 to oleg134-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.017115 s, 239 kB/s oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 00:38:18 (1786077498) targets are mounted 00:38:18 (1786077498) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (73s) == replay-dual test 29: replay vs update with the same xid ========================================================== 00:38:37 (1786077517) SKIP: replay-dual test_29 needs >= 2 MDTs SKIP 29 (4s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 00:38:41 (1786077521) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0724859 s, 565 kB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0790071 s, 518 kB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 00:38:50 (1786077530) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 00:39:26 (1786077566) targets are mounted 00:39:26 (1786077566) facet_failover done 160+0 records in 160+0 records out 81920 bytes (82 kB, 80 KiB) copied, 49.8432 s, 1.6 kB/s oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (66s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 00:39:47 (1786077587) Failing ost1 on oleg134-server Stopping /mnt/lustre-ost1 (opts:) on oleg134-server Creating to objid 2273 on ost lustre-OST0000... 00:39:55 (1786077595) shut down facet: ost1 facet_host: oleg134-server facet_failover_host: oleg134-server total: 32 open/close in 1.08 seconds: 29.54 ops/second at_max=0 fail_loc=0x80001420 file /mnt/lustre2/d31.replay-dual/mdtdir/f31.replay-dual is ready Failover ost1 to oleg134-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 00:40:17 (1786077617) targets are mounted 00:40:17 (1786077617) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL IDLE state after 0 sec pids: 53247 53248 53253 53254 53255 53256 53257 53258 53259 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 530757 393 530364 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118252 460 117792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118186 394 117792 1% /mnt/lustre[OST:1] filesystem_summary: 235977 393 235584 1% /mnt/lustre at_max=600 PASS 31 (48s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 00:40:35 (1786077635) SKIP: replay-dual test_32 needs >= 2 MDTs SKIP 32 (3s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 00:40:38 (1786077638) SKIP: replay-dual test_33 ldiskfs only test SKIP 33 (3s) == replay-dual test complete, duration 3422 sec ========== 00:40:41 (1786077641) === replay-dual: start cleanup 00:40:43 (1786077643) === Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg134-client.virtnet /mnt/lustre2 opts: === replay-dual: finish cleanup 00:40:50 (1786077650) ===