== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 23:15:17 (1787714117) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 6528 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 8192 7528448 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.03825 s, 107 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0156294 s, 262 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0216491 s, 189 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0363322 s, 113 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0467558 s, 87.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0227393 s, 180 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0203475 s, 201 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0193238 s, 212 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0245789 s, 167 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.016718 s, 245 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0154919 s, 264 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0156569 s, 262 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0960148 s, 42.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0137072 s, 299 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0141175 s, 290 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0198551 s, 206 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0183545 s, 223 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0221873 s, 185 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0161035 s, 254 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.035237 s, 116 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg251-server Stopping /mnt/lustre-ost1 (opts:) on oleg251-server 23:15:31 (1787714131) shut down facet: ost1 facet_host: oleg251-server facet_failover_host: oleg251-server Failover ost1 to oleg251-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 23:15:54 (1787714154) targets are mounted 23:15:54 (1787714154) facet_failover done oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec osc.lustre-OST0000-osc-ffff984c18327000.state= current_state: FULL state_history: - [ 1787714003, FULL ] - [ 1787714128, DISCONN ] - [ 1787714128, CONNECTING ] - [ 1787714128, DISCONN ] - [ 1787714133, CONNECTING ] - [ 1787714133, DISCONN ] - [ 1787714138, CONNECTING ] - [ 1787714138, DISCONN ] - [ 1787714143, CONNECTING ] - [ 1787714143, DISCONN ] - [ 1787714148, CONNECTING ] - [ 1787714148, REPLAY ] - [ 1787714149, REPLAY_LOCKS ] - [ 1787714149, REPLAY_WAIT ] - [ 1787714159, RECOVER ] - [ 1787714159, FULL ]