== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 11:19:13 (1786547953) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3628 1267676 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2408 1268896 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2752 3604240 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2800 3604220 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 5552 7208460 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00891395 s, 460 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00957095 s, 428 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0099305 s, 412 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00948771 s, 432 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00929138 s, 441 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00900235 s, 455 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00927402 s, 442 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00915705 s, 447 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00886985 s, 462 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00886029 s, 462 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00926837 s, 442 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00883221 s, 464 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00873078 s, 469 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00932785 s, 439 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00872964 s, 469 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00941312 s, 435 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00867428 s, 472 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00867499 s, 472 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00911658 s, 449 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00907206 s, 451 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg245-server Stopping /mnt/lustre-ost1 (opts:) on oleg245-server 11:19:23 (1786547963) shut down facet: ost1 facet_host: oleg245-server facet_failover_host: oleg245-server Failover ost1 to oleg245-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 11:19:42 (1786547982) targets are mounted 11:19:42 (1786547982) facet_failover done oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec osc.lustre-OST0000-osc-ffff940a79957800.state= current_state: FULL state_history: - [ 1786547960, DISCONN ] - [ 1786547964, CONNECTING ] - [ 1786547964, DISCONN ] - [ 1786547969, CONNECTING ] - [ 1786547969, DISCONN ] - [ 1786547974, CONNECTING ] - [ 1786547974, DISCONN ] - [ 1786547979, CONNECTING ] - [ 1786547979, REPLAY ] - [ 1786547979, REPLAY_LOCKS ] - [ 1786547989, CONNECTING ] - [ 1786547989, REPLAY ] - [ 1786547989, REPLAY_LOCKS ] - [ 1786547989, REPLAY_WAIT ] - [ 1786547989, RECOVER ] - [ 1786547989, FULL ]