== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 00:55:50 (1788843350) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3532 1267772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2352 1268952 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2244 3604748 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2328 3604692 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 4572 7209440 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0145853 s, 281 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0293872 s, 139 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0206606 s, 198 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.026001 s, 158 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0209495 s, 196 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0285667 s, 143 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0305339 s, 134 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0200088 s, 205 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.020207 s, 203 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0247426 s, 166 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.015217 s, 269 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0318004 s, 129 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0264357 s, 155 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.02943 s, 139 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0140505 s, 292 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0194732 s, 210 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0309688 s, 132 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0159383 s, 257 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0173383 s, 236 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0242489 s, 169 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg614-server Stopping /mnt/lustre-ost1 (opts:) on oleg614-server 00:56:05 (1788843365) shut down facet: ost1 facet_host: oleg614-server facet_failover_host: oleg614-server Failover ost1 to oleg614-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-OST0000 00:56:31 (1788843391) targets are mounted 00:56:31 (1788843391) facet_failover done oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff9c35611d4800.state= current_state: FULL state_history: - [ 1788843295, FULL ] - [ 1788843364, DISCONN ] - [ 1788843364, CONNECTING ] - [ 1788843364, DISCONN ] - [ 1788843369, CONNECTING ] - [ 1788843369, DISCONN ] - [ 1788843374, CONNECTING ] - [ 1788843374, DISCONN ] - [ 1788843379, CONNECTING ] - [ 1788843379, DISCONN ] - [ 1788843384, CONNECTING ] - [ 1788843384, REPLAY ] - [ 1788843386, REPLAY_LOCKS ] - [ 1788843386, REPLAY_WAIT ] - [ 1788843396, RECOVER ] - [ 1788843396, FULL ]