-----============= acceptance-small: replay-single ============----- Tue Feb 25 04:35:16 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg456-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-single.*ex: No such file or directory excepting tests: 59 36 === replay-single: start setup 04:35:21 (1740476121) === oleg456-client.virtnet: executing check_config_client /mnt/lustre oleg456-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg456-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012abe4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012abe4000.idle_timeout=debug disable quota as required oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === replay-single: finish setup 04:35:27 (1740476127) === Cleaning up AT ... == replay-single test 135a: Server failure in lock replay phase ========================================================== 04:35:28 (1740476128) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre ldlm.cancel_unused_locks_before_replay=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg456-server Failover ost1 to oleg456-server oleg456-server.virtnet oleg456-server: oleg456-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x32d fail_val=20 debug_mb=100 debug=+info +ha +dlmtrace Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec Stopping /mnt/lustre-ost1 (opts:) on oleg456-server Failover ost1 to oleg456-server oleg456-server.virtnet oleg456-server: oleg456-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 End of sync oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg456-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0001 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 5 debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck ldlm.cancel_unused_locks_before_replay=1 PASS 135a (71s) == replay-single test 135b: Racy client reconnections in lock replay phase ========================================================== 04:36:39 (1740476199) ldlm.cancel_unused_locks_before_replay=0 osc.lustre-OST0000-osc-ffff88012abe4000.idle_timeout=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg456-server Failover ost1 to oleg456-server oleg456-server.virtnet oleg456-server: oleg456-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x330 fail_val=2 fail_loc=0x32d fail_val=900 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec last_state: 1740476210, REPLAY_LOCKS - [ 1740476228, CONNECTING ] Import lustre-OST0000-osc-[-0-9a-f]* is reconnecting osc.lustre-OST0000-osc-ffff88012abe4000.state= current_state: REPLAY_LOCKS state_history: - [ 1740476170, DISCONN ] - [ 1740476175, CONNECTING ] - [ 1740476195, DISCONN ] - [ 1740476195, CONNECTING ] - [ 1740476195, EVICTED ] - [ 1740476195, RECOVER ] - [ 1740476195, FULL ] - [ 1740476205, DISCONN ] - [ 1740476205, CONNECTING ] - [ 1740476205, DISCONN ] - [ 1740476210, CONNECTING ] - [ 1740476210, REPLAY ] - [ 1740476210, REPLAY_LOCKS ] - [ 1740476228, CONNECTING ] - [ 1740476228, REPLAY ] - [ 1740476228, REPLAY_LOCKS ] fail_loc=0 fail_loc=0 End of sync debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck osc.lustre-OST0000-osc-ffff88012abe4000.idle_timeout=20 ldlm.cancel_unused_locks_before_replay=1 PASS 135b (32s) == replay-single test complete, duration 114 sec ========= 04:37:11 (1740476231) === replay-single: start cleanup 04:37:11 (1740476231) === === replay-single: finish cleanup 04:37:12 (1740476232) ===