== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 12:36:20 (1788366980) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3672 1267632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2448 1268856 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2276 3604716 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 3100 3603920 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 5376 7208636 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0108807 s, 376 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.012161 s, 337 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00626418 s, 654 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0102842 s, 398 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00940539 s, 435 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0082722 s, 495 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00617224 s, 664 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00553922 s, 739 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00482977 s, 848 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00679891 s, 602 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00716766 s, 571 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00789067 s, 519 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0129192 s, 317 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00870179 s, 471 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00685441 s, 598 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0125658 s, 326 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00923938 s, 443 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0149186 s, 275 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00695994 s, 589 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00775084 s, 528 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg241-server Stopping /mnt/lustre-ost1 (opts:) on oleg241-server 12:36:27 (1788366987) shut down facet: ost1 facet_host: oleg241-server facet_failover_host: oleg241-server Failover ost1 to oleg241-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 12:36:43 (1788367003) targets are mounted 12:36:43 (1788367003) facet_failover done oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec osc.lustre-OST0000-osc-ffff93ba90a05000.state= current_state: FULL state_history: - [ 1788366986, DISCONN ] - [ 1788366988, CONNECTING ] - [ 1788366988, DISCONN ] - [ 1788366993, CONNECTING ] - [ 1788366993, DISCONN ] - [ 1788366998, CONNECTING ] - [ 1788366998, DISCONN ] - [ 1788367002, CONNECTING ] - [ 1788367002, REPLAY ] - [ 1788367002, REPLAY_LOCKS ] - [ 1788367012, CONNECTING ] - [ 1788367012, REPLAY ] - [ 1788367012, REPLAY_LOCKS ] - [ 1788367012, REPLAY_WAIT ] - [ 1788367012, RECOVER ] - [ 1788367012, FULL ]