-----============= acceptance-small: replay-single ============----- Tue Feb 25 04:35:36 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg102-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-single.*ex: No such file or directory excepting tests: 59 36 === replay-single: start setup 04:35:44 (1740476144) === oleg102-client.virtnet: executing check_config_client /mnt/lustre oleg102-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg102-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5b62000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5b62000.idle_timeout=debug disable quota as required oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-single: finish setup 04:35:56 (1740476156) === Cleaning up AT ... == replay-single test 135a: Server failure in lock replay phase ========================================================== 04:35:57 (1740476157) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1776 1285912 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1616 1286072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3072 7210968 1% /mnt/lustre ldlm.cancel_unused_locks_before_replay=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg102-server Failover ost1 to oleg102-server oleg102-server.virtnet oleg102-server: oleg102-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x32d fail_val=20 debug_mb=100 debug=+info +ha +dlmtrace Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec Stopping /mnt/lustre-ost1 (opts:) on oleg102-server Failover ost1 to oleg102-server oleg102-server.virtnet oleg102-server: oleg102-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 End of sync oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg102-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0001 debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck ldlm.cancel_unused_locks_before_replay=1 PASS 135a (51s) == replay-single test 135b: Racy client reconnections in lock replay phase ========================================================== 04:36:48 (1740476208) ldlm.cancel_unused_locks_before_replay=0 osc.lustre-OST0000-osc-ffff8800b5b62000.idle_timeout=0 debug_mb=100 debug=+info +ha +dlmtrace Stopping /mnt/lustre-ost1 (opts:) on oleg102-server Failover ost1 to oleg102-server oleg102-server.virtnet oleg102-server: oleg102-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x330 fail_val=2 fail_loc=0x32d fail_val=900 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec last_state: 1740476214, REPLAY_LOCKS - [ 1740476232, CONNECTING ] Import lustre-OST0000-osc-[-0-9a-f]* is reconnecting osc.lustre-OST0000-osc-ffff8800b5b62000.state= current_state: REPLAY_LOCKS state_history: - [ 1740476192, DISCONN ] - [ 1740476197, CONNECTING ] - [ 1740476197, DISCONN ] - [ 1740476202, CONNECTING ] - [ 1740476202, EVICTED ] - [ 1740476202, RECOVER ] - [ 1740476202, FULL ] - [ 1740476212, DISCONN ] - [ 1740476212, CONNECTING ] - [ 1740476212, DISCONN ] - [ 1740476214, CONNECTING ] - [ 1740476214, REPLAY ] - [ 1740476214, REPLAY_LOCKS ] - [ 1740476232, CONNECTING ] - [ 1740476232, REPLAY ] - [ 1740476232, REPLAY_LOCKS ] fail_loc=0 fail_loc=0 End of sync debug_mb=21 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck osc.lustre-OST0000-osc-ffff8800b5b62000.idle_timeout=20 ldlm.cancel_unused_locks_before_replay=1 PASS 135b (26s) == replay-single test complete, duration 97 sec ========== 04:37:14 (1740476234) === replay-single: start cleanup 04:37:15 (1740476235) === === replay-single: finish cleanup 04:37:16 (1740476236) ===