== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 19:35:47 (1786836947) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 6528 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 8192 7529472 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0100785 s, 406 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00742218 s, 552 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00900244 s, 455 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0105652 s, 388 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00540594 s, 758 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00627762 s, 652 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0082311 s, 498 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0157521 s, 260 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0170105 s, 241 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00774663 s, 529 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00579695 s, 707 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0251019 s, 163 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0058601 s, 699 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00725305 s, 565 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00967736 s, 423 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0158415 s, 259 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00685686 s, 597 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0107292 s, 382 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00649207 s, 631 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00703042 s, 583 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg211-server Stopping /mnt/lustre-ost1 (opts:) on oleg211-server 19:35:52 (1786836952) shut down facet: ost1 facet_host: oleg211-server facet_failover_host: oleg211-server Failover ost1 to oleg211-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 19:36:07 (1786836967) targets are mounted 19:36:07 (1786836967) facet_failover done oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec osc.lustre-OST0000-osc-ffff8e99492a5800.state= current_state: FULL state_history: - [ 1786836953, DISCONN ] - [ 1786836953, CONNECTING ] - [ 1786836953, DISCONN ] - [ 1786836958, CONNECTING ] - [ 1786836958, DISCONN ] - [ 1786836963, CONNECTING ] - [ 1786836963, DISCONN ] - [ 1786836966, CONNECTING ] - [ 1786836966, REPLAY ] - [ 1786836966, REPLAY_LOCKS ] - [ 1786836976, CONNECTING ] - [ 1786836976, REPLAY ] - [ 1786836976, REPLAY_LOCKS ] - [ 1786836976, REPLAY_WAIT ] - [ 1786836976, RECOVER ] - [ 1786836976, FULL ]