== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 13:48:35 (1786470515) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3620 1267684 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2412 1268892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 3396 3603596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2144 3604876 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 5540 7208472 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00467771 s, 876 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00523377 s, 783 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00512998 s, 798 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00503829 s, 813 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00436513 s, 938 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00463838 s, 883 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00428217 s, 957 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00470751 s, 870 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00498262 s, 822 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00512192 s, 800 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0043409 s, 944 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00582835 s, 703 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00530012 s, 773 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00529908 s, 773 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00479451 s, 854 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00475222 s, 862 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00465108 s, 881 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0048455 s, 845 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00484618 s, 845 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0050522 s, 811 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg331-server Stopping /mnt/lustre-ost1 (opts:) on oleg331-server 13:48:42 (1786470522) shut down facet: ost1 facet_host: oleg331-server facet_failover_host: oleg331-server Failover ost1 to oleg331-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 13:48:58 (1786470538) targets are mounted 13:48:58 (1786470538) facet_failover done oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec osc.lustre-OST0000-osc-ffff8909c4a64800.state= current_state: FULL state_history: - [ 1786470521, DISCONN ] - [ 1786470523, CONNECTING ] - [ 1786470523, DISCONN ] - [ 1786470528, CONNECTING ] - [ 1786470528, DISCONN ] - [ 1786470533, CONNECTING ] - [ 1786470533, DISCONN ] - [ 1786470537, CONNECTING ] - [ 1786470537, REPLAY ] - [ 1786470537, REPLAY_LOCKS ] - [ 1786470547, CONNECTING ] - [ 1786470547, REPLAY ] - [ 1786470547, REPLAY_LOCKS ] - [ 1786470547, REPLAY_WAIT ] - [ 1786470547, RECOVER ] - [ 1786470547, FULL ]