== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 09:41:20 (1786196480) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 6272 2202112 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 8192 7528448 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00516594 s, 793 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00571295 s, 717 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00462023 s, 887 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00802234 s, 511 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00509892 s, 803 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00481418 s, 851 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00518374 s, 790 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00483405 s, 847 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00553142 s, 740 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.013667 s, 300 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00507485 s, 807 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00526534 s, 778 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00606756 s, 675 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0055872 s, 733 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00755564 s, 542 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00644257 s, 636 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00782498 s, 523 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00668183 s, 613 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00555939 s, 737 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00602069 s, 680 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg322-server Stopping /mnt/lustre-ost1 (opts:) on oleg322-server 09:41:24 (1786196484) shut down facet: ost1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover ost1 to oleg322-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 09:41:38 (1786196498) targets are mounted 09:41:38 (1786196498) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec osc.lustre-OST0000-osc-ffff8ce7cb004800.state= current_state: FULL state_history: - [ 1786196464, CONNECTING ] - [ 1786196464, FULL ] - [ 1786196487, DISCONN ] - [ 1786196487, CONNECTING ] - [ 1786196487, DISCONN ] - [ 1786196492, CONNECTING ] - [ 1786196492, DISCONN ] - [ 1786196497, CONNECTING ] - [ 1786196497, REPLAY ] - [ 1786196498, REPLAY_LOCKS ] - [ 1786196508, CONNECTING ] - [ 1786196508, REPLAY ] - [ 1786196508, REPLAY_LOCKS ] - [ 1786196508, REPLAY_WAIT ] - [ 1786196508, RECOVER ] - [ 1786196508, FULL ]