== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 11:30:50 (1788881450) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3640 1267664 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2408 1268896 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2996 3603996 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2652 3604368 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 5648 7208364 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00901076 s, 455 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00846662 s, 484 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00891754 s, 459 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00920228 s, 445 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00858195 s, 477 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00869383 s, 471 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0089598 s, 457 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00869806 s, 471 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00868003 s, 472 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00958034 s, 428 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0087405 s, 469 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00899217 s, 456 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0085262 s, 480 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00880436 s, 465 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00894527 s, 458 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00990782 s, 413 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00880302 s, 465 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.008832 s, 464 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00907353 s, 451 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0088675 s, 462 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg640-server Stopping /mnt/lustre-ost1 (opts:) on oleg640-server 11:30:57 (1788881457) shut down facet: ost1 facet_host: oleg640-server facet_failover_host: oleg640-server Failover ost1 to oleg640-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0000 11:31:16 (1788881476) targets are mounted 11:31:16 (1788881476) facet_failover done oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec osc.lustre-OST0000-osc-ffff8b8d038de800.state= current_state: FULL state_history: - [ 1788881397, IDLE ] - [ 1788881422, CONNECTING ] - [ 1788881422, FULL ] - [ 1788881459, DISCONN ] - [ 1788881459, CONNECTING ] - [ 1788881459, DISCONN ] - [ 1788881464, CONNECTING ] - [ 1788881464, DISCONN ] - [ 1788881469, CONNECTING ] - [ 1788881469, DISCONN ] - [ 1788881474, CONNECTING ] - [ 1788881474, REPLAY ] - [ 1788881474, REPLAY_LOCKS ] - [ 1788881474, REPLAY_WAIT ] - [ 1788881485, RECOVER ] - [ 1788881485, FULL ]