== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 09:49:15 (1788270555) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3696 1267608 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2428 1268876 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2168 3604824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2524 3604496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 4692 7209320 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00561097 s, 730 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00593731 s, 690 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00593694 s, 690 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0055224 s, 742 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00553231 s, 740 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00589562 s, 695 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00682017 s, 601 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00670718 s, 611 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00553001 s, 741 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00583185 s, 702 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00564734 s, 725 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00563898 s, 726 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00602648 s, 680 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00511046 s, 801 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00591737 s, 692 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00514373 s, 796 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00496079 s, 826 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00529784 s, 773 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00639894 s, 640 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00563589 s, 727 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg655-server Stopping /mnt/lustre-ost1 (opts:) on oleg655-server 09:49:22 (1788270562) shut down facet: ost1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover ost1 to oleg655-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-OST0000 09:49:38 (1788270578) targets are mounted 09:49:38 (1788270578) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec osc.lustre-OST0000-osc-ffff9cc6844be000.state= current_state: FULL state_history: - [ 1788270561, DISCONN ] - [ 1788270563, CONNECTING ] - [ 1788270563, DISCONN ] - [ 1788270568, CONNECTING ] - [ 1788270568, DISCONN ] - [ 1788270573, CONNECTING ] - [ 1788270573, DISCONN ] - [ 1788270577, CONNECTING ] - [ 1788270577, REPLAY ] - [ 1788270577, REPLAY_LOCKS ] - [ 1788270587, CONNECTING ] - [ 1788270587, REPLAY ] - [ 1788270587, REPLAY_LOCKS ] - [ 1788270587, REPLAY_WAIT ] - [ 1788270587, RECOVER ] - [ 1788270587, FULL ]