-----============= acceptance-small: replay-dual ============----- Tue Apr 8 15:28:13 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg305-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex: No such file or directory excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 15:28:19 (1744140499) === Starting client oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg305-client.virtnet: executing check_config_client /mnt/lustre oleg305-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg305-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5859800.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012aacd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5859800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012aacd000.idle_timeout=debug disable quota as required oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 15:28:26 (1744140506) === == replay-dual test 0a: expired recovery with lost client ========================================================== 15:28:26 (1744140506) Check file is LU482_FAILED=/tmp/replay-dual.lu482.nY8H5O UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 50 open/close in 0.44 seconds: 114.29 ops/second fail_loc=0x80000514 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:28:29 (1744140509) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:28:42 (1744140522) targets are mounted 15:28:42 (1744140522) facet_failover done Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1744140604 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second PASS 0a (99s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 15:30:05 (1744140605) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:30:08 (1744140608) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:30:20 (1744140620) targets are mounted 15:30:20 (1744140620) facet_failover done Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 PASS 0b (87s) == replay-dual test 1: |X| simple create ================= 15:31:32 (1744140692) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:31:34 (1744140694) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:31:46 (1744140706) targets are mounted 15:31:46 (1744140706) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (19s) == replay-dual test 2: |X| mkdir adir ==================== 15:31:51 (1744140711) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:31:53 (1744140713) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:32:05 (1744140725) targets are mounted 15:32:05 (1744140725) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (19s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 15:32:10 (1744140730) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:32:12 (1744140732) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:32:25 (1744140745) targets are mounted 15:32:25 (1744140745) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (19s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 15:32:29 (1744140749) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:32:31 (1744140751) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:32:44 (1744140764) targets are mounted 15:32:44 (1744140764) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (20s) == replay-dual test 5: open, unlink |X| close ============ 15:32:49 (1744140769) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7273 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:32:51 (1744140771) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:33:04 (1744140784) targets are mounted 15:33:04 (1744140784) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (19s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 15:33:08 (1744140788) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7273 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7273 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:33:11 (1744140791) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:33:23 (1744140803) targets are mounted 15:33:23 (1744140803) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (20s) == replay-dual test 8: replay of resent request ========== 15:33:28 (1744140808) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:33:47 (1744140827) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:33:59 (1744140839) targets are mounted 15:33:59 (1744140839) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (36s) == replay-dual test 9: resending a replayed create ======= 15:34:04 (1744140844) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:34:07 (1744140847) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:34:19 (1744140859) targets are mounted 15:34:19 (1744140859) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (35s) == replay-dual test 10: resending a replayed unlink ====== 15:34:39 (1744140879) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:34:42 (1744140882) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:34:54 (1744140894) targets are mounted 15:34:54 (1744140894) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (36s) == replay-dual test 11: both clients timeout during replay ========================================================== 15:35:15 (1744140915) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:35:17 (1744140917) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:35:30 (1744140930) targets are mounted 15:35:30 (1744140930) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec fail_loc=0 PASS 11 (33s) == replay-dual test 12: open resend timeout ============== 15:35:48 (1744140948) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7273 fail_loc=0x80000302 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:35:51 (1744140951) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:36:03 (1744140963) targets are mounted 15:36:03 (1744140963) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (18s) == replay-dual test 13: close resend timeout ============= 15:36:06 (1744140966) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7273 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:36:08 (1744140968) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:36:21 (1744140981) targets are mounted 15:36:21 (1744140981) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (19s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 15:36:25 (1744140985) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.16 seconds: 152.01 ops/second total: 1 open/close in 0.01 seconds: 152.04 ops/second Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:36:28 (1744140988) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:36:40 (1744141000) targets are mounted 15:36:40 (1744141000) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1744141073 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 PASS 15a (89s) == replay-dual test 15c: remove multiple OST orphans ===== 15:37:54 (1744141074) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:38:18 (1744141098) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:38:31 (1744141111) targets are mounted 15:38:31 (1744141111) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 PASS 15c (111s) == replay-dual test 16: fail MDS during recovery (3571) == 15:39:45 (1744141185) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7522304 1% /mnt/lustre total: 25 open/close in 0.21 seconds: 118.31 ops/second total: 1 open/close in 0.01 seconds: 116.74 ops/second Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:39:48 (1744141188) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:40:00 (1744141200) targets are mounted 15:40:00 (1744141200) facet_failover done Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:40:21 (1744141221) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:40:34 (1744141234) targets are mounted 15:40:34 (1744141234) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1744141313 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 PASS 16 (130s) == replay-dual test 17: fail OST during recovery (3571) == 15:41:55 (1744141315) total: 25 open/close in 0.37 seconds: 67.34 ops/second total: 1 open/close in 0.02 seconds: 63.50 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing ost1 on oleg305-server Stopping /mnt/lustre-ost1 (opts:) on oleg305-server 15:41:58 (1744141318) shut down facet: ost1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover ost1 to oleg305-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 15:42:11 (1744141331) targets are mounted 15:42:11 (1744141331) facet_failover done Failing ost1 on oleg305-server Stopping /mnt/lustre-ost1 (opts:) on oleg305-server 15:42:32 (1744141352) shut down facet: ost1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover ost1 to oleg305-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 15:42:45 (1744141365) targets are mounted 15:42:45 (1744141365) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1744141443 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 PASS 17 (135s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 15:44:10 (1744141450) debug=+dlmtrace fail_loc=0x8000030b using seed 4189082454 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second ldlm.namespaces.MGC192.168.203.105@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8ce9000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012dbf4800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800a8ce9000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff88012dbf4800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800a8ce9000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff88012dbf4800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.203.105@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8ce9000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012dbf4800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800a8ce9000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff88012dbf4800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800a8ce9000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012dbf4800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (46s) == replay-dual test 19: resend of open request =========== 15:44:56 (1744141496) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1744141583.65 total 86.03 last 0.00) total: 1 open/close in 86.03 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:46:25 (1744141585) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:46:37 (1744141597) targets are mounted 15:46:37 (1744141597) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (106s) == replay-dual test 20: recovery time is not increasing == 15:46:42 (1744141602) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:46:44 (1744141604) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:46:57 (1744141617) targets are mounted 15:46:57 (1744141617) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:49:22 (1744141762) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:49:35 (1744141775) targets are mounted 15:49:35 (1744141775) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 PASS 20 (316s) == replay-dual test 21a: commit on sharing =============== 15:51:58 (1744141918) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 15:52:01 (1744141921) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:52:13 (1744141933) targets are mounted 15:52:13 (1744141933) facet_failover done Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (156s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 15:54:34 (1744142074) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (2s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 15:54:36 (1744142076) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (1s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 15:54:37 (1744142077) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (1s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 15:54:38 (1744142078) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (1s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 15:54:39 (1744142079) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (0s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 15:54:39 (1744142079) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (1s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 15:54:40 (1744142080) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (1s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 15:54:41 (1744142081) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (1s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 15:54:42 (1744142082) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (86s) == replay-dual test 25: replay|resend ==================== 15:56:09 (1744142169) 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.0023457 s, 218 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg305-server Stopping /mnt/lustre-ost1 (opts:) on oleg305-server 15:56:11 (1744142171) shut down facet: ost1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover ost1 to oleg305-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 15:56:24 (1744142184) targets are mounted 15:56:24 (1744142184) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6811: 14005 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (20s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 15:56:29 (1744142189) Starting client oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 15530 Started dbench loop with 15531 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Apr 8 15:56:31 EDT 2025 waiting for dbench pid 15562 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 255 9.66 MB/sec warmup 1 sec latency 25.163 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3631104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3704832 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7335936 1% /mnt/lustre 1 503 8.73 MB/sec warmup 2 sec latency 29.314 ms 1 723 7.10 MB/sec warmup 3 sec latency 65.596 ms 1 892 5.42 MB/sec warmup 4 sec latency 51.844 ms test_26 fail mds1 1 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 1142 4.60 MB/sec warmup 5 sec latency 57.072 ms 15:56:37 (1744142197) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 1142 3.83 MB/sec warmup 6 sec latency 1057.219 ms 1 1142 3.28 MB/sec warmup 7 sec latency 2057.343 ms 1 1142 2.87 MB/sec warmup 8 sec latency 3057.526 ms 1 1142 2.55 MB/sec warmup 9 sec latency 4057.767 ms 1 1142 2.30 MB/sec warmup 10 sec latency 5057.959 ms 1 1142 2.09 MB/sec warmup 11 sec latency 6058.192 ms 1 1142 1.92 MB/sec warmup 12 sec latency 7058.475 ms 1 1142 1.77 MB/sec warmup 13 sec latency 8058.653 ms 1 1142 1.64 MB/sec warmup 14 sec latency 9058.812 ms 1 1142 1.53 MB/sec warmup 15 sec latency 10059.013 ms Failover mds1 to oleg305-server mount facets: mds1 1 1142 1.44 MB/sec warmup 16 sec latency 11059.191 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1142 1.35 MB/sec warmup 17 sec latency 12059.329 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 1 1142 1.28 MB/sec warmup 18 sec latency 13059.445 ms Started lustre-MDT0000 15:56:49 (1744142209) targets are mounted 15:56:49 (1744142209) facet_failover done 1 1142 1.21 MB/sec warmup 19 sec latency 14059.604 ms 1 1142 0.00 MB/sec execute 1 sec latency 16060.038 ms 1 1142 0.00 MB/sec execute 2 sec latency 17060.179 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1296 0.16 MB/sec execute 3 sec latency 17506.876 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1618 0.79 MB/sec execute 4 sec latency 51.547 ms 1 1886 0.68 MB/sec execute 5 sec latency 38.981 ms 1 2187 0.63 MB/sec execute 6 sec latency 21.341 ms 1 2499 0.82 MB/sec execute 7 sec latency 39.016 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4736 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 38912 3553280 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3639296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7192576 1% /mnt/lustre 1 3028 1.32 MB/sec execute 8 sec latency 34.704 ms 1 3636 1.74 MB/sec execute 9 sec latency 26.870 ms test_26 fail mds1 2 times 1 3885 1.68 MB/sec execute 10 sec latency 15.901 ms Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 4068 1.55 MB/sec execute 11 sec latency 215.974 ms 15:57:02 (1744142222) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 4068 1.42 MB/sec execute 12 sec latency 1216.107 ms 1 4068 1.31 MB/sec execute 13 sec latency 2216.357 ms 1 4068 1.22 MB/sec execute 14 sec latency 3216.554 ms 1 4068 1.14 MB/sec execute 15 sec latency 4216.777 ms 1 4068 1.07 MB/sec execute 16 sec latency 5217.045 ms 1 4068 1.00 MB/sec execute 17 sec latency 6217.288 ms 1 4068 0.95 MB/sec execute 18 sec latency 7217.572 ms 1 4068 0.90 MB/sec execute 19 sec latency 8217.805 ms 1 4068 0.85 MB/sec execute 20 sec latency 9218.096 ms 1 4068 0.81 MB/sec execute 21 sec latency 10218.348 ms Failover mds1 to oleg305-server mount facets: mds1 1 4068 0.78 MB/sec execute 22 sec latency 11218.572 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4068 0.74 MB/sec execute 23 sec latency 12218.910 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1 4068 0.71 MB/sec execute 24 sec latency 13219.029 ms pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:57:15 (1744142235) targets are mounted 15:57:15 (1744142235) facet_failover done 1 4068 0.68 MB/sec execute 25 sec latency 14219.260 ms 1 4068 0.66 MB/sec execute 26 sec latency 15219.537 ms 1 4068 0.63 MB/sec execute 27 sec latency 16219.722 ms 1 4068 0.61 MB/sec execute 28 sec latency 17219.881 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4248 0.60 MB/sec execute 29 sec latency 17448.034 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4483 0.60 MB/sec execute 30 sec latency 31.485 ms 1 4795 0.62 MB/sec execute 31 sec latency 30.865 ms 1 5137 0.70 MB/sec execute 32 sec latency 46.935 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6400 2202112 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3465216 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 21504 3486720 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 63488 6951936 1% /mnt/lustre 1 5362 0.68 MB/sec execute 33 sec latency 37.876 ms 1 5650 0.67 MB/sec execute 34 sec latency 37.861 ms 1 5946 0.69 MB/sec execute 35 sec latency 55.074 ms test_26 fail mds1 3 times 1 6317 0.78 MB/sec execute 36 sec latency 34.506 ms Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 6573 0.80 MB/sec execute 37 sec latency 316.694 ms 15:57:28 (1744142248) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 6573 0.78 MB/sec execute 38 sec latency 1316.870 ms 1 6573 0.76 MB/sec execute 39 sec latency 2317.017 ms 1 6573 0.74 MB/sec execute 40 sec latency 3317.167 ms 1 6573 0.72 MB/sec execute 41 sec latency 4317.386 ms 1 6573 0.71 MB/sec execute 42 sec latency 5317.607 ms 1 6573 0.69 MB/sec execute 43 sec latency 6317.846 ms 1 6573 0.67 MB/sec execute 44 sec latency 7318.003 ms 1 6573 0.66 MB/sec execute 45 sec latency 8318.160 ms 1 6573 0.64 MB/sec execute 46 sec latency 9318.318 ms 1 6573 0.63 MB/sec execute 47 sec latency 10318.488 ms Failover mds1 to oleg305-server mount facets: mds1 1 6573 0.62 MB/sec execute 48 sec latency 11318.579 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 6573 0.61 MB/sec execute 49 sec latency 12318.751 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:57:41 (1744142261) targets are mounted 15:57:41 (1744142261) facet_failover done 1 6573 0.59 MB/sec execute 50 sec latency 13318.951 ms 1 6573 0.58 MB/sec execute 51 sec latency 14319.062 ms 1 6573 0.57 MB/sec execute 52 sec latency 15319.219 ms 1 6573 0.56 MB/sec execute 53 sec latency 16319.500 ms 1 6573 0.55 MB/sec execute 54 sec latency 17319.682 ms 1 6573 0.54 MB/sec execute 55 sec latency 18319.857 ms 1 6573 0.53 MB/sec execute 56 sec latency 19320.078 ms 1 6573 0.52 MB/sec execute 57 sec latency 20320.225 ms 1 6573 0.51 MB/sec execute 58 sec latency 21320.401 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 6933 0.54 MB/sec execute 59 sec latency 21681.776 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7231 0.58 MB/sec execute 60 sec latency 20.204 ms 1 7396 0.59 MB/sec execute 61 sec latency 27.395 ms 1 7545 0.58 MB/sec execute 62 sec latency 23.126 ms 1 7679 0.57 MB/sec execute 63 sec latency 34.613 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 7808 2200704 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3402752 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3493888 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 6896640 1% /mnt/lustre 1 7878 0.57 MB/sec execute 64 sec latency 92.489 ms 1 8082 0.57 MB/sec execute 65 sec latency 29.211 ms 1 8288 0.58 MB/sec execute 66 sec latency 52.421 ms test_26 fail mds1 4 times Failing mds1 on oleg305-server 1 8562 0.62 MB/sec execute 67 sec latency 29.394 ms Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 8661 0.61 MB/sec execute 68 sec latency 560.068 ms 15:57:59 (1744142279) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 8661 0.60 MB/sec execute 69 sec latency 1560.255 ms 1 8661 0.59 MB/sec execute 70 sec latency 2560.474 ms 1 8661 0.58 MB/sec execute 71 sec latency 3560.685 ms 1 8661 0.57 MB/sec execute 72 sec latency 4560.863 ms 1 8661 0.57 MB/sec execute 73 sec latency 5561.046 ms 1 8661 0.56 MB/sec execute 74 sec latency 6561.195 ms 1 8661 0.55 MB/sec execute 75 sec latency 7561.364 ms 1 8661 0.54 MB/sec execute 76 sec latency 8561.689 ms 1 8661 0.54 MB/sec execute 77 sec latency 9561.959 ms 1 8661 0.53 MB/sec execute 78 sec latency 10562.084 ms Failover mds1 to oleg305-server mount facets: mds1 1 8661 0.52 MB/sec execute 79 sec latency 11562.247 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 8661 0.52 MB/sec execute 80 sec latency 12562.379 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:58:12 (1744142292) targets are mounted 15:58:12 (1744142292) facet_failover done 1 8661 0.51 MB/sec execute 81 sec latency 13562.504 ms 1 8661 0.50 MB/sec execute 82 sec latency 14562.611 ms 1 8661 0.50 MB/sec execute 83 sec latency 15562.774 ms 1 8661 0.49 MB/sec execute 84 sec latency 16562.980 ms 1 8661 0.49 MB/sec execute 85 sec latency 17563.144 ms 1 8661 0.48 MB/sec execute 86 sec latency 18563.369 ms 1 8661 0.48 MB/sec execute 87 sec latency 19563.508 ms 1 8742 0.47 MB/sec execute 88 sec latency 20180.880 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 9033 0.47 MB/sec execute 89 sec latency 28.383 ms 1 9451 0.48 MB/sec execute 90 sec latency 34.213 ms 1 9868 0.52 MB/sec execute 91 sec latency 30.067 ms 1 10437 0.55 MB/sec execute 92 sec latency 31.267 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 9472 2199040 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 45056 3404800 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3420160 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 69632 6824960 2% /mnt/lustre 1 10797 0.58 MB/sec execute 93 sec latency 14.078 ms 1 11056 0.59 MB/sec execute 94 sec latency 14.652 ms 1 11325 0.58 MB/sec execute 95 sec latency 23.790 ms test_26 fail mds1 5 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 11528 0.58 MB/sec execute 96 sec latency 60.459 ms 15:58:28 (1744142308) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 11528 0.58 MB/sec execute 97 sec latency 1025.618 ms 1 11528 0.57 MB/sec execute 98 sec latency 2025.843 ms 1 11528 0.57 MB/sec execute 99 sec latency 3026.025 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec Failover mds1 to oleg305-server mount facets: mds1 1 cleanup 107 sec Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 108 sec oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 1 cleanup 109 sec Started lustre-MDT0000 15:58:40 (1744142320) targets are mounted 15:58:40 (1744142320) facet_failover done 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 0 cleanup 113 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1794 31.346 21681.754 Close 1308 27.947 17506.839 Rename 74 15.255 29.377 Unlink 366 4.911 13.744 Qpathinfo 1652 15.077 20180.850 Qfileinfo 286 0.441 4.542 Qfsinfo 295 0.568 3.599 Sfileinfo 138 8.963 16.659 Find 630 1.240 17.222 WriteX 873 1.925 10.834 ReadX 2836 0.127 27.318 LockX 6 1.357 1.537 UnlockX 6 1.561 1.858 Flush 123 21.901 92.476 Throughput 0.565708 MB/sec 1 clients 1 procs max_latency=21681.776 ms stopping dbench on /mnt/lustre at Tue Apr 8 15:58:44 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Apr 8 15:58:46 EDT 2025 waiting for dbench pid 20750 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 289 10.70 MB/sec warmup 1 sec latency 27.201 ms 1 642 10.53 MB/sec warmup 2 sec latency 16.211 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10752 2197760 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 49152 3478528 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 51200 3420160 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 100352 6898688 2% /mnt/lustre 1 873 7.22 MB/sec warmup 3 sec latency 60.930 ms 1 1161 5.75 MB/sec warmup 4 sec latency 44.802 ms 1 1551 5.22 MB/sec warmup 5 sec latency 34.041 ms test_26 fail mds1 6 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 1905 4.40 MB/sec warmup 6 sec latency 33.353 ms 15:58:53 (1744142333) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 2002 3.79 MB/sec warmup 7 sec latency 772.752 ms 1 2002 3.31 MB/sec warmup 8 sec latency 1773.369 ms 1 2002 2.94 MB/sec warmup 9 sec latency 2773.542 ms 1 2002 2.65 MB/sec warmup 10 sec latency 3773.711 ms 1 2002 2.41 MB/sec warmup 11 sec latency 4773.880 ms 1 2002 2.21 MB/sec warmup 12 sec latency 5774.042 ms 1 2002 2.04 MB/sec warmup 13 sec latency 6774.194 ms 1 2002 1.89 MB/sec warmup 14 sec latency 7774.347 ms 1 2002 1.77 MB/sec warmup 15 sec latency 8774.493 ms 1 2002 1.66 MB/sec warmup 16 sec latency 9774.688 ms Failover mds1 to oleg305-server mount facets: mds1 1 2002 1.56 MB/sec warmup 17 sec latency 10774.811 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2002 1.47 MB/sec warmup 18 sec latency 11774.940 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1 2002 1.39 MB/sec warmup 19 sec latency 12775.117 ms pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:59:05 (1744142345) targets are mounted 15:59:05 (1744142345) facet_failover done 1 2002 0.00 MB/sec execute 1 sec latency 14775.488 ms 1 2002 0.00 MB/sec execute 2 sec latency 15775.618 ms 1 2002 0.00 MB/sec execute 3 sec latency 16775.747 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2345 0.35 MB/sec execute 4 sec latency 16814.846 ms 1 2752 1.09 MB/sec execute 5 sec latency 57.494 ms 1 3278 1.47 MB/sec execute 6 sec latency 59.807 ms 1 3759 1.89 MB/sec execute 7 sec latency 24.948 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10880 2197632 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3564544 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 53248 3523584 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7088128 2% /mnt/lustre 1 3995 1.69 MB/sec execute 8 sec latency 17.536 ms 1 4243 1.53 MB/sec execute 9 sec latency 77.551 ms 1 4448 1.43 MB/sec execute 10 sec latency 60.545 ms test_26 fail mds1 7 times Failing mds1 on oleg305-server 1 4727 1.42 MB/sec execute 11 sec latency 23.826 ms Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 4842 1.31 MB/sec execute 12 sec latency 593.592 ms 15:59:18 (1744142358) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 4842 1.21 MB/sec execute 13 sec latency 1593.786 ms 1 4842 1.13 MB/sec execute 14 sec latency 2594.018 ms 1 4842 1.05 MB/sec execute 15 sec latency 3594.220 ms 1 4842 0.98 MB/sec execute 16 sec latency 4594.375 ms 1 4842 0.93 MB/sec execute 17 sec latency 5594.503 ms 1 4842 0.88 MB/sec execute 18 sec latency 6594.722 ms 1 4842 0.83 MB/sec execute 19 sec latency 7594.873 ms 1 4842 0.79 MB/sec execute 20 sec latency 8595.055 ms 1 4842 0.75 MB/sec execute 21 sec latency 9595.305 ms 1 4842 0.72 MB/sec execute 22 sec latency 10595.447 ms Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4842 0.69 MB/sec execute 23 sec latency 11595.555 ms 1 4842 0.66 MB/sec execute 24 sec latency 12595.702 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:59:30 (1744142370) targets are mounted 15:59:30 (1744142370) facet_failover done 1 4842 0.63 MB/sec execute 25 sec latency 13595.863 ms 1 4842 0.61 MB/sec execute 26 sec latency 14596.023 ms 1 4842 0.58 MB/sec execute 27 sec latency 15596.253 ms 1 4842 0.56 MB/sec execute 28 sec latency 16596.399 ms 1 4842 0.54 MB/sec execute 29 sec latency 17596.587 ms 1 4842 0.53 MB/sec execute 30 sec latency 18596.770 ms 1 4842 0.51 MB/sec execute 31 sec latency 19596.923 ms 1 4842 0.49 MB/sec execute 32 sec latency 20597.041 ms 1 4985 0.56 MB/sec execute 33 sec latency 21331.224 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5249 0.55 MB/sec execute 34 sec latency 57.777 ms 1 5519 0.54 MB/sec execute 35 sec latency 49.355 ms 1 5881 0.57 MB/sec execute 36 sec latency 13.627 ms 1 6180 0.59 MB/sec execute 37 sec latency 54.394 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 7680 2200704 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3707904 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 56320 3686400 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 77824 7394304 2% /mnt/lustre 1 6767 0.71 MB/sec execute 38 sec latency 52.074 ms 1 7244 0.80 MB/sec execute 39 sec latency 15.189 ms 1 7514 0.82 MB/sec execute 40 sec latency 13.805 ms test_26 fail mds1 8 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 7754 0.80 MB/sec execute 41 sec latency 19.908 ms 1 7788 0.78 MB/sec execute 42 sec latency 717.688 ms 15:59:48 (1744142388) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 7788 0.77 MB/sec execute 43 sec latency 1717.899 ms 1 7788 0.75 MB/sec execute 44 sec latency 2718.073 ms 1 7788 0.73 MB/sec execute 45 sec latency 3718.242 ms 1 7788 0.72 MB/sec execute 46 sec latency 4718.407 ms 1 7788 0.70 MB/sec execute 47 sec latency 5718.679 ms 1 7788 0.69 MB/sec execute 48 sec latency 6718.819 ms 1 7788 0.67 MB/sec execute 49 sec latency 7718.975 ms 1 7788 0.66 MB/sec execute 50 sec latency 8719.128 ms 1 7788 0.65 MB/sec execute 51 sec latency 9719.272 ms 1 7788 0.63 MB/sec execute 52 sec latency 10719.416 ms Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7788 0.62 MB/sec execute 53 sec latency 11719.536 ms 1 7788 0.61 MB/sec execute 54 sec latency 12719.713 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 16:00:00 (1744142400) targets are mounted 16:00:00 (1744142400) facet_failover done 1 7788 0.60 MB/sec execute 55 sec latency 13719.898 ms 1 7788 0.59 MB/sec execute 56 sec latency 14720.045 ms 1 7788 0.58 MB/sec execute 57 sec latency 15720.151 ms 1 7796 0.57 MB/sec execute 58 sec latency 16647.617 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8025 0.57 MB/sec execute 59 sec latency 39.389 ms 1 8337 0.58 MB/sec execute 60 sec latency 24.279 ms 1 8674 0.62 MB/sec execute 61 sec latency 65.371 ms 1 8933 0.61 MB/sec execute 62 sec latency 55.694 ms 1 9262 0.61 MB/sec execute 63 sec latency 43.659 ms 1 9546 0.62 MB/sec execute 64 sec latency 59.834 ms 1 10021 0.70 MB/sec execute 65 sec latency 45.865 ms 1 10592 0.76 MB/sec execute 66 sec latency 58.643 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Apr 8 16:00:12 EDT 2025 with return code 0 20750 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 20750 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (225s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 16:00:14 (1744142414) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00277825 s, 1.5 MB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg305-server Stopping /mnt/lustre-ost1 (opts:) on oleg305-server 16:00:34 (1744142434) shut down facet: ost1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover ost1 to oleg305-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00310865 s, 1.3 MB/s pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 16:00:47 (1744142447) targets are mounted 16:00:47 (1744142447) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (39s) == replay-dual test 29: replay vs update with the same xid ========================================================== 16:00:53 (1744142453) SKIP: replay-dual test_29 needs >= 2 MDTs SKIP 29 (1s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 16:00:54 (1744142454) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.00694757 s, 5.9 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.00943016 s, 4.3 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 16:00:57 (1744142457) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 16:01:09 (1744142469) targets are mounted 16:01:09 (1744142469) facet_failover done 160+0 records in 160+0 records out 81920 bytes (82 kB) copied, 22.1676 s, 3.7 kB/s oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (26s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 16:01:20 (1744142480) Failing ost1 on oleg305-server Stopping /mnt/lustre-ost1 (opts:) on oleg305-server 16:01:22 (1744142482) shut down facet: ost1 facet_host: oleg305-server facet_failover_host: oleg305-server Creating to objid 3201 on ost lustre-OST0000... total: 32 open/close in 0.15 seconds: 210.36 ops/second at_max=0 fail_loc=0x80001420 file /mnt/lustre2/d31.replay-dual/mdtdir/f31.replay-dual is ready Failover ost1 to oleg305-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 16:01:34 (1744142494) targets are mounted 16:01:34 (1744142494) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE FULL state after 0 sec pids: 28524 28525 28530 28531 28532 28533 28534 28535 28536 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3584 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 496475 393 496082 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118155 395 117760 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118153 393 117760 1% /mnt/lustre[OST:1] filesystem_summary: 235913 393 235520 1% /mnt/lustre at_max=600 PASS 31 (19s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 16:01:39 (1744142499) SKIP: replay-dual test_32 needs >= 2 MDTs SKIP 32 (1s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 16:01:40 (1744142500) SKIP: replay-dual test_33 ldiskfs only test SKIP 33 (1s) == replay-dual test complete, duration 2008 sec ========== 16:01:41 (1744142501) === replay-dual: start cleanup 16:01:41 (1744142501) === Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg305-client.virtnet /mnt/lustre2 opts: === replay-dual: finish cleanup 16:01:42 (1744142502) ===