== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 04:28:38 (1786696118) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 6400 2201856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7529472 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0387319 s, 106 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0379094 s, 108 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.04453 s, 92.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0687283 s, 59.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.017224 s, 238 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0210357 s, 195 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0544367 s, 75.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0140515 s, 291 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0151289 s, 271 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0250343 s, 164 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.03843 s, 107 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0250288 s, 164 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0185336 s, 221 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0179411 s, 228 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0224707 s, 182 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0194078 s, 211 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.049984 s, 81.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0220252 s, 186 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.048164 s, 85.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.023699 s, 173 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg606-server Stopping /mnt/lustre-ost1 (opts:) on oleg606-server 04:28:50 (1786696130) shut down facet: ost1 facet_host: oleg606-server facet_failover_host: oleg606-server Failover ost1 to oleg606-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 04:29:14 (1786696154) targets are mounted 04:29:14 (1786696154) facet_failover done oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff947b16439000.state= current_state: FULL state_history: - [ 1786696132, DISCONN ] - [ 1786696132, CONNECTING ] - [ 1786696132, DISCONN ] - [ 1786696137, CONNECTING ] - [ 1786696137, DISCONN ] - [ 1786696142, CONNECTING ] - [ 1786696142, DISCONN ] - [ 1786696147, CONNECTING ] - [ 1786696147, REPLAY ] - [ 1786696148, REPLAY_LOCKS ] - [ 1786696158, REPLAY_WAIT ] - [ 1786696158, CONNECTING ] - [ 1786696158, REPLAY_LOCKS ] - [ 1786696158, REPLAY_WAIT ] - [ 1786696158, RECOVER ] - [ 1786696158, FULL ]