== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 10:04:43 (1787580283) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 6528 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7529472 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0235083 s, 174 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.028782 s, 142 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0248232 s, 165 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.043088 s, 95.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0256093 s, 160 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0248466 s, 165 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.021935 s, 187 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0471032 s, 87.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0283133 s, 145 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0416262 s, 98.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0156573 s, 262 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0376454 s, 109 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0351823 s, 116 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0158039 s, 259 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0261626 s, 157 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0159648 s, 257 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0214861 s, 191 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0376119 s, 109 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0121176 s, 338 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0206562 s, 198 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg645-server Stopping /mnt/lustre-ost1 (opts:) on oleg645-server 10:04:58 (1787580298) shut down facet: ost1 facet_host: oleg645-server facet_failover_host: oleg645-server Failover ost1 to oleg645-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 10:05:24 (1787580324) targets are mounted 10:05:24 (1787580324) facet_failover done oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff8a62a02df000.state= current_state: FULL state_history: - [ 1787580297, DISCONN ] - [ 1787580302, CONNECTING ] - [ 1787580302, DISCONN ] - [ 1787580307, CONNECTING ] - [ 1787580307, DISCONN ] - [ 1787580312, CONNECTING ] - [ 1787580312, DISCONN ] - [ 1787580317, CONNECTING ] - [ 1787580317, REPLAY ] - [ 1787580317, REPLAY_LOCKS ] - [ 1787580327, CONNECTING ] - [ 1787580327, REPLAY ] - [ 1787580327, REPLAY_LOCKS ] - [ 1787580327, REPLAY_WAIT ] - [ 1787580327, RECOVER ] - [ 1787580327, FULL ]