== recovery-small test 170: Reconnect after REPLAY_LOCKS hangs (LU-18154) ========================================================== 18:11:50 (1786313510) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3524 1267780 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2344 1268960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2412 3604580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2300 3604720 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 4712 7209300 1% /mnt/lustre 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0249634 s, 164 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0179769 s, 228 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174368 s, 235 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0169432 s, 242 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174579 s, 235 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0168718 s, 243 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0157461 s, 260 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.016422 s, 249 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.016085 s, 255 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0160919 s, 255 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.016313 s, 251 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0356914 s, 115 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0229899 s, 178 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0291238 s, 141 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0379184 s, 108 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0372843 s, 110 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0346488 s, 118 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0572343 s, 71.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0411034 s, 99.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.031099 s, 132 kB/s fail_loc=0x0000537 fail_val=10 fail_loc=0x80000537 Failing ost1 on oleg448-server Stopping /mnt/lustre-ost1 (opts:) on oleg448-server 18:12:09 (1786313529) shut down facet: ost1 facet_host: oleg448-server facet_failover_host: oleg448-server Failover ost1 to oleg448-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 18:12:42 (1786313562) targets are mounted 18:12:42 (1786313562) facet_failover done oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 60 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff9e6788c1e000.state= current_state: FULL state_history: - [ 1786313536, DISCONN ] - [ 1786313541, CONNECTING ] - [ 1786313541, DISCONN ] - [ 1786313546, CONNECTING ] - [ 1786313546, DISCONN ] - [ 1786313551, CONNECTING ] - [ 1786313551, DISCONN ] - [ 1786313555, CONNECTING ] - [ 1786313555, REPLAY ] - [ 1786313555, REPLAY_LOCKS ] - [ 1786313565, CONNECTING ] - [ 1786313565, REPLAY ] - [ 1786313565, REPLAY_LOCKS ] - [ 1786313565, REPLAY_WAIT ] - [ 1786313565, RECOVER ] - [ 1786313565, FULL ]