== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 11:09:55 (1784128195) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3692 1267612 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2392 1268912 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 3332 3603660 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2124 3604896 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 5456 7208556 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0102542 s, 399 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0108203 s, 379 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0103042 s, 398 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00955425 s, 429 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0105239 s, 389 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0102994 s, 398 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0110512 s, 371 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0113869 s, 360 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0146376 s, 280 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00940562 s, 435 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0110017 s, 372 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0101816 s, 402 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0110573 s, 370 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0106057 s, 386 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0158643 s, 258 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0119187 s, 344 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0131229 s, 312 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0122779 s, 334 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0101637 s, 403 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0138575 s, 296 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg432-server Stopping /mnt/lustre-ost1 (opts:) on oleg432-server 11:10:03 (1784128203) shut down facet: ost1 facet_host: oleg432-server facet_failover_host: oleg432-server Failover ost1 to oleg432-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 11:10:22 (1784128222) targets are mounted 11:10:22 (1784128222) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec osc.lustre-OST0000-osc-ffff8dd083e45800.state= current_state: FULL state_history: - [ 1784128114, IDLE ] - [ 1784128171, CONNECTING ] - [ 1784128171, FULL ] - [ 1784128206, DISCONN ] - [ 1784128206, CONNECTING ] - [ 1784128206, DISCONN ] - [ 1784128211, CONNECTING ] - [ 1784128211, DISCONN ] - [ 1784128216, CONNECTING ] - [ 1784128216, DISCONN ] - [ 1784128219, CONNECTING ] - [ 1784128219, REPLAY ] - [ 1784128220, REPLAY_LOCKS ] - [ 1784128220, REPLAY_WAIT ] - [ 1784128230, RECOVER ] - [ 1784128230, FULL ]