== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 18:11:53 (1786486313) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3604 1267700 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2364 1268940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2448 3604544 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2484 3604536 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 4932 7209080 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0082549 s, 496 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00705077 s, 581 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00704478 s, 581 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0071919 s, 570 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00795298 s, 515 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0063449 s, 646 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00663977 s, 617 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00671801 s, 610 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00783716 s, 523 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00693413 s, 591 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0059422 s, 689 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00650145 s, 630 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00639122 s, 641 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0058841 s, 696 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00590861 s, 693 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00597057 s, 686 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0071261 s, 575 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00607054 s, 675 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00556552 s, 736 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00548676 s, 747 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg328-server Stopping /mnt/lustre-ost1 (opts:) on oleg328-server 18:12:00 (1786486320) shut down facet: ost1 facet_host: oleg328-server facet_failover_host: oleg328-server Failover ost1 to oleg328-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 18:12:16 (1786486336) targets are mounted 18:12:16 (1786486336) facet_failover done oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec osc.lustre-OST0000-osc-ffff986c860cb800.state= current_state: FULL state_history: - [ 1786486319, DISCONN ] - [ 1786486321, CONNECTING ] - [ 1786486321, DISCONN ] - [ 1786486326, CONNECTING ] - [ 1786486326, DISCONN ] - [ 1786486331, CONNECTING ] - [ 1786486331, DISCONN ] - [ 1786486335, CONNECTING ] - [ 1786486335, REPLAY ] - [ 1786486335, REPLAY_LOCKS ] - [ 1786486345, CONNECTING ] - [ 1786486345, REPLAY ] - [ 1786486345, REPLAY_LOCKS ] - [ 1786486345, REPLAY_WAIT ] - [ 1786486345, RECOVER ] - [ 1786486345, FULL ]