== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 19:54:58 (1784591698) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 6912 2201344 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7530496 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0236606 s, 173 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0807089 s, 50.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0106531 s, 384 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.031212 s, 131 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0170565 s, 240 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0177366 s, 231 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.011603 s, 353 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0275152 s, 149 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0144063 s, 284 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.013171 s, 311 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0137283 s, 298 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0142104 s, 288 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0202528 s, 202 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174963 s, 234 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0124918 s, 328 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0375386 s, 109 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0159064 s, 258 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0192386 s, 213 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.025038 s, 164 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0199845 s, 205 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg212-server Stopping /mnt/lustre-ost1 (opts:) on oleg212-server 19:55:08 (1784591708) shut down facet: ost1 facet_host: oleg212-server facet_failover_host: oleg212-server Failover ost1 to oleg212-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 19:55:27 (1784591727) targets are mounted 19:55:27 (1784591727) facet_failover done oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec osc.lustre-OST0000-osc-ffff8b85d16c4000.state= current_state: FULL state_history: - [ 1784591615, EVICTED ] - [ 1784591615, RECOVER ] - [ 1784591615, FULL ] - [ 1784591709, DISCONN ] - [ 1784591709, CONNECTING ] - [ 1784591709, DISCONN ] - [ 1784591714, CONNECTING ] - [ 1784591714, DISCONN ] - [ 1784591719, CONNECTING ] - [ 1784591719, DISCONN ] - [ 1784591723, CONNECTING ] - [ 1784591723, REPLAY ] - [ 1784591723, REPLAY_LOCKS ] - [ 1784591723, REPLAY_WAIT ] - [ 1784591733, RECOVER ] - [ 1784591733, FULL ]