== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 09:18:15 (1784294295) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3684 1267620 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2400 1268904 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2364 3604628 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2900 3604120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 5264 7208748 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00530208 s, 773 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0071247 s, 575 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00643838 s, 636 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00667229 s, 614 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00572331 s, 716 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00524159 s, 781 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00575003 s, 712 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00705924 s, 580 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00636096 s, 644 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00630436 s, 650 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0063915 s, 641 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00647256 s, 633 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00630969 s, 649 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00650002 s, 630 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.005678 s, 721 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00593425 s, 690 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00546772 s, 749 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00511058 s, 801 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00469715 s, 872 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00525798 s, 779 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg309-server Stopping /mnt/lustre-ost1 (opts:) on oleg309-server 09:18:20 (1784294300) shut down facet: ost1 facet_host: oleg309-server facet_failover_host: oleg309-server Failover ost1 to oleg309-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 09:18:37 (1784294317) targets are mounted 09:18:37 (1784294317) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec osc.lustre-OST0000-osc-ffff9cbe20548800.state= current_state: FULL state_history: - [ 1784294301, DISCONN ] - [ 1784294303, CONNECTING ] - [ 1784294303, DISCONN ] - [ 1784294309, CONNECTING ] - [ 1784294309, DISCONN ] - [ 1784294314, CONNECTING ] - [ 1784294314, DISCONN ] - [ 1784294316, CONNECTING ] - [ 1784294316, REPLAY ] - [ 1784294316, REPLAY_LOCKS ] - [ 1784294326, CONNECTING ] - [ 1784294326, REPLAY ] - [ 1784294326, REPLAY_LOCKS ] - [ 1784294326, REPLAY_WAIT ] - [ 1784294326, RECOVER ] - [ 1784294326, FULL ]