== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 05:42:40 (1787737360) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 6528 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7530496 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0350748 s, 117 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0276876 s, 148 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0248842 s, 165 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0368442 s, 111 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0195265 s, 210 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0139703 s, 293 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0224189 s, 183 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.014495 s, 283 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0146566 s, 279 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0395729 s, 104 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0208441 s, 197 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0235373 s, 174 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0239566 s, 171 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0129121 s, 317 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0124081 s, 330 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0297186 s, 138 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0128026 s, 320 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0130517 s, 314 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0118942 s, 344 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0144228 s, 284 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg119-server Stopping /mnt/lustre-ost1 (opts:) on oleg119-server 05:42:53 (1787737373) shut down facet: ost1 facet_host: oleg119-server facet_failover_host: oleg119-server Failover ost1 to oleg119-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 05:43:18 (1787737398) targets are mounted 05:43:18 (1787737398) facet_failover done oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffffa0fe49202800.state= current_state: FULL state_history: - [ 1787737374, DISCONN ] - [ 1787737379, CONNECTING ] - [ 1787737379, DISCONN ] - [ 1787737384, CONNECTING ] - [ 1787737384, DISCONN ] - [ 1787737389, CONNECTING ] - [ 1787737389, DISCONN ] - [ 1787737391, CONNECTING ] - [ 1787737391, REPLAY ] - [ 1787737391, REPLAY_LOCKS ] - [ 1787737401, CONNECTING ] - [ 1787737401, REPLAY ] - [ 1787737401, REPLAY_LOCKS ] - [ 1787737401, REPLAY_WAIT ] - [ 1787737401, RECOVER ] - [ 1787737401, FULL ]