== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 03:00:25 (1786345225) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3700 1267604 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2384 1268920 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2316 3604676 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2904 3604116 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 5220 7208792 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00512609 s, 799 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00462526 s, 886 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00564028 s, 726 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00475338 s, 862 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00505366 s, 811 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00487406 s, 840 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00451276 s, 908 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00535366 s, 765 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00485967 s, 843 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00511679 s, 801 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00511326 s, 801 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00530406 s, 772 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00458022 s, 894 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00469526 s, 872 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00471366 s, 869 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00626564 s, 654 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00505596 s, 810 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00488439 s, 839 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00487302 s, 841 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00441913 s, 927 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg404-server Stopping /mnt/lustre-ost1 (opts:) on oleg404-server 03:00:30 (1786345230) shut down facet: ost1 facet_host: oleg404-server facet_failover_host: oleg404-server Failover ost1 to oleg404-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-OST0000 03:00:46 (1786345246) targets are mounted 03:00:46 (1786345246) facet_failover done oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec osc.lustre-OST0000-osc-ffff8a3b49a63800.state= current_state: FULL state_history: - [ 1786345231, DISCONN ] - [ 1786345233, CONNECTING ] - [ 1786345233, DISCONN ] - [ 1786345238, CONNECTING ] - [ 1786345238, DISCONN ] - [ 1786345243, CONNECTING ] - [ 1786345243, DISCONN ] - [ 1786345245, CONNECTING ] - [ 1786345245, REPLAY ] - [ 1786345245, REPLAY_LOCKS ] - [ 1786345255, CONNECTING ] - [ 1786345255, REPLAY ] - [ 1786345255, REPLAY_LOCKS ] - [ 1786345255, REPLAY_WAIT ] - [ 1786345255, RECOVER ] - [ 1786345255, FULL ]