-----============= acceptance-small: replay-single ============----- Tue Feb 25 04:35:25 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg250-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-single.*ex: No such file or directory excepting tests: 59 36 === replay-single: start setup 04:35:31 (1740476131) === oleg250-client.virtnet: executing check_config_client /mnt/lustre oleg250-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg250-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b59fa800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b59fa800.idle_timeout=debug disable quota as required oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-single: finish setup 04:35:46 (1740476146) === Cleaning up AT ... == replay-single test 135a: Server failure in lock replay phase ========================================================== 04:35:48 (1740476148) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1752 1285936 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3605620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2800 7211240 1% /mnt/lustre ldlm.cancel_unused_locks_before_replay=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg250-server Failover ost1 to oleg250-server oleg250-server.virtnet oleg250-server: oleg250-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x32d fail_val=20 debug_mb=100 debug=+info +ha +dlmtrace Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec Stopping /mnt/lustre-ost1 (opts:) on oleg250-server Failover ost1 to oleg250-server oleg250-server.virtnet oleg250-server: oleg250-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 End of sync Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg250-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 5 debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck ldlm.cancel_unused_locks_before_replay=1 PASS 135a (80s) == replay-single test 135b: Racy client reconnections in lock replay phase ========================================================== 04:37:09 (1740476229) ldlm.cancel_unused_locks_before_replay=0 osc.lustre-OST0000-osc-ffff8800b59fa800.idle_timeout=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg250-server Failover ost1 to oleg250-server oleg250-server.virtnet oleg250-server: oleg250-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x330 fail_val=2 fail_loc=0x32d fail_val=900 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec last_state: 1740476236, REPLAY_LOCKS - [ 1740476254, CONNECTING ] Import lustre-OST0000-osc-[-0-9a-f]* is reconnecting osc.lustre-OST0000-osc-ffff8800b59fa800.state= current_state: REPLAY_LOCKS state_history: - [ 1740476196, DISCONN ] - [ 1740476201, CONNECTING ] - [ 1740476226, DISCONN ] - [ 1740476226, CONNECTING ] - [ 1740476226, EVICTED ] - [ 1740476226, RECOVER ] - [ 1740476226, FULL ] - [ 1740476231, DISCONN ] - [ 1740476231, CONNECTING ] - [ 1740476231, DISCONN ] - [ 1740476236, CONNECTING ] - [ 1740476236, REPLAY ] - [ 1740476236, REPLAY_LOCKS ] - [ 1740476254, CONNECTING ] - [ 1740476254, REPLAY ] - [ 1740476254, REPLAY_LOCKS ] fail_loc=0 fail_loc=0 End of sync debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck osc.lustre-OST0000-osc-ffff8800b59fa800.idle_timeout=20 ldlm.cancel_unused_locks_before_replay=1 PASS 135b (30s) == replay-single test complete, duration 133 sec ========= 04:37:38 (1740476258) === replay-single: start cleanup 04:37:38 (1740476258) === === replay-single: finish cleanup 04:37:39 (1740476259) ===