-----============= acceptance-small: replay-single ============----- Tue Feb 25 04:35:28 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg357-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-single.*ex: No such file or directory excepting tests: 59 36 === replay-single: start setup 04:35:33 (1740476133) === oleg357-client.virtnet: executing check_config_client /mnt/lustre oleg357-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg357-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88013767c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88013767c000.idle_timeout=debug disable quota as required oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-single: finish setup 04:35:47 (1740476147) === Cleaning up AT ... == replay-single test 135a: Server failure in lock replay phase ========================================================== 04:35:48 (1740476148) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1285936 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2800 7211240 1% /mnt/lustre ldlm.cancel_unused_locks_before_replay=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg357-server Failover ost1 to oleg357-server oleg357-server.virtnet oleg357-server: oleg357-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x32d fail_val=20 debug_mb=100 debug=+info +ha +dlmtrace Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec Stopping /mnt/lustre-ost1 (opts:) on oleg357-server Failover ost1 to oleg357-server oleg357-server.virtnet oleg357-server: oleg357-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all End of sync pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0001 debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck ldlm.cancel_unused_locks_before_replay=1 PASS 135a (59s) == replay-single test 135b: Racy client reconnections in lock replay phase ========================================================== 04:36:47 (1740476207) ldlm.cancel_unused_locks_before_replay=0 osc.lustre-OST0000-osc-ffff88013767c000.idle_timeout=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg357-server Failover ost1 to oleg357-server oleg357-server.virtnet oleg357-server: oleg357-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x330 fail_val=2 fail_loc=0x32d fail_val=900 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec last_state: 1740476213, REPLAY_LOCKS - [ 1740476232, CONNECTING ] Import lustre-OST0000-osc-[-0-9a-f]* is reconnecting osc.lustre-OST0000-osc-ffff88013767c000.state= current_state: REPLAY_LOCKS state_history: - [ 1740476189, DISCONN ] - [ 1740476194, CONNECTING ] - [ 1740476194, DISCONN ] - [ 1740476199, CONNECTING ] - [ 1740476199, EVICTED ] - [ 1740476199, RECOVER ] - [ 1740476199, FULL ] - [ 1740476209, DISCONN ] - [ 1740476209, CONNECTING ] - [ 1740476209, DISCONN ] - [ 1740476213, CONNECTING ] - [ 1740476213, REPLAY ] - [ 1740476213, REPLAY_LOCKS ] - [ 1740476232, CONNECTING ] - [ 1740476232, REPLAY ] - [ 1740476232, REPLAY_LOCKS ] fail_loc=0 fail_loc=0 End of sync debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck osc.lustre-OST0000-osc-ffff88013767c000.idle_timeout=20 ldlm.cancel_unused_locks_before_replay=1 PASS 135b (27s) == replay-single test complete, duration 106 sec ========= 04:37:14 (1740476234) === replay-single: start cleanup 04:37:15 (1740476235) === === replay-single: finish cleanup 04:37:15 (1740476235) ===