== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 14:24:49 (1788632689) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 6528 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7529472 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0338304 s, 121 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0523217 s, 78.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0971422 s, 42.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0259603 s, 158 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0206172 s, 199 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0164116 s, 250 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0331967 s, 123 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0202366 s, 202 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.014357 s, 285 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.016233 s, 252 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0185327 s, 221 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0147429 s, 278 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0142167 s, 288 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0255684 s, 160 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0537683 s, 76.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0133835 s, 306 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0438827 s, 93.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0181105 s, 226 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0217506 s, 188 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0116391 s, 352 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg353-server Stopping /mnt/lustre-ost1 (opts:) on oleg353-server 14:25:02 (1788632702) shut down facet: ost1 facet_host: oleg353-server facet_failover_host: oleg353-server Failover ost1 to oleg353-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 14:25:27 (1788632727) targets are mounted 14:25:27 (1788632727) facet_failover done oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff90aba22e3800.state= current_state: FULL state_history: - [ 1788632642, FULL ] - [ 1788632701, DISCONN ] - [ 1788632701, CONNECTING ] - [ 1788632701, DISCONN ] - [ 1788632706, CONNECTING ] - [ 1788632706, DISCONN ] - [ 1788632711, CONNECTING ] - [ 1788632711, DISCONN ] - [ 1788632716, CONNECTING ] - [ 1788632716, DISCONN ] - [ 1788632719, CONNECTING ] - [ 1788632719, REPLAY ] - [ 1788632719, REPLAY_LOCKS ] - [ 1788632719, REPLAY_WAIT ] - [ 1788632730, RECOVER ] - [ 1788632730, FULL ]