== conf-sanity test 84: check recovery_hard_time ========= 02:11:03 (1788588663) start mds service on oleg631-server start mds service on oleg631-server Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60 lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg631-server Start mds2: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60 lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg631-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg631-server: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg631-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg631-server: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg631-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg631-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0001 oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg631-server: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg631-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg631-server: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg631-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec recovery_time=60, timeout=20, wrap_up=5 mount lustre on /mnt/lustre..... Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre mount lustre on /mnt/lustre2..... Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 2048 379520 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 1920 379648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 382976 2048 378880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 382976 2048 378880 1% /mnt/lustre[OST:1] filesystem_summary: 765952 4096 757760 1% /mnt/lustre - open/close 705 (time 1788588792.49 total 10.01 last 70.41) total: 1000 open/close in 14.63 seconds: 68.34 ops/second fail_loc=0x20000709 fail_val=5 Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 02:13:24 (1788588804) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60 lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 02:13:47 (1788588827) targets are mounted 02:13:47 (1788588827) facet_failover done oleg631-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg631-client: oleg631-client: ssh exited with exit code 5 recovery status status: COMPLETE recovery_start: 1788588824 recovery_duration: 60 completed_clients: 2/3 replayed_requests: 185 last_transno: 12884902077 VBR: DISABLED IR: DISABLED fail_loc=0 umount lustre on /mnt/lustre..... Stopping client oleg631-client.virtnet /mnt/lustre (opts:) umount lustre on /mnt/lustre2..... Stopping client oleg631-client.virtnet /mnt/lustre2 (opts:) stop ost1 service on oleg631-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg631-server stop ost2 service on oleg631-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg631-server stop mds service on oleg631-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg631-server stop mds service on oleg631-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg631-server