-----============= acceptance-small: replay-single ============----- Tue Feb 25 04:35:24 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg123-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-single.*ex: No such file or directory excepting tests: 59 36 === replay-single: start setup 04:35:29 (1740476129) === oleg123-client.virtnet: executing check_config_client /mnt/lustre oleg123-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg123-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b588c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b588c000.idle_timeout=debug disable quota as required oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === replay-single: finish setup 04:35:41 (1740476141) === Cleaning up AT ... == replay-single test 135a: Server failure in lock replay phase ========================================================== 04:35:42 (1740476142) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2944 2205696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3765248 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 8192 7530496 1% /mnt/lustre ldlm.cancel_unused_locks_before_replay=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg123-server Failover ost1 to oleg123-server oleg123-server.virtnet oleg123-server: oleg123-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x32d fail_val=20 debug_mb=100 debug=+info +ha +dlmtrace Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec Stopping /mnt/lustre-ost1 (opts:) on oleg123-server Failover ost1 to oleg123-server oleg123-server.virtnet oleg123-server: oleg123-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 End of sync Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg123-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 5 debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck ldlm.cancel_unused_locks_before_replay=1 PASS 135a (64s) == replay-single test 135b: Racy client reconnections in lock replay phase ========================================================== 04:36:46 (1740476206) ldlm.cancel_unused_locks_before_replay=0 osc.lustre-OST0000-osc-ffff8800b588c000.idle_timeout=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg123-server Failover ost1 to oleg123-server oleg123-server.virtnet oleg123-server: oleg123-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x330 fail_val=2 fail_loc=0x32d fail_val=900 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec last_state: 1740476211, REPLAY_LOCKS - [ 1740476229, CONNECTING ] Import lustre-OST0000-osc-[-0-9a-f]* is reconnecting osc.lustre-OST0000-osc-ffff8800b588c000.state= current_state: REPLAY_LOCKS state_history: - [ 1740476179, DISCONN ] - [ 1740476184, CONNECTING ] - [ 1740476204, DISCONN ] - [ 1740476204, CONNECTING ] - [ 1740476204, EVICTED ] - [ 1740476204, RECOVER ] - [ 1740476204, FULL ] - [ 1740476209, DISCONN ] - [ 1740476209, CONNECTING ] - [ 1740476209, DISCONN ] - [ 1740476211, CONNECTING ] - [ 1740476211, REPLAY ] - [ 1740476211, REPLAY_LOCKS ] - [ 1740476229, CONNECTING ] - [ 1740476229, REPLAY ] - [ 1740476229, REPLAY_LOCKS ] fail_loc=0 fail_loc=0 End of sync debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck osc.lustre-OST0000-osc-ffff8800b588c000.idle_timeout=20 ldlm.cancel_unused_locks_before_replay=1 PASS 135b (26s) == replay-single test complete, duration 107 sec ========= 04:37:12 (1740476232) === replay-single: start cleanup 04:37:12 (1740476232) === === replay-single: finish cleanup 04:37:13 (1740476233) ===