[ 1861.528988] Lustre: Failing over lustre-MDT0000
[ 1861.693309] Lustre: server umount lustre-MDT0000 complete
[ 1864.167384] Lustre: lustre-MDT0000: Aborting client recovery
[ 1864.168805] LustreError: 86275:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 1864.171215] Lustre: 86321:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 1864.173442] Lustre: 86321:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 1864.175335] Lustre: 86321:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f4cf112e-56b0-4499-aacf-a956cfcc6c06@<unknown>
[ 1864.178131] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 1864.188995] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 1864.222145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1544 to 0x280000400:1633)
[ 1864.222145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1571 to 0x240000400:1633)
[ 1865.590585] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1871.284606] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:46:42 (1782362802)
[ 1877.558071] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1878.151693] Lustre: Failing over lustre-MDT0000
[ 1878.399542] Lustre: server umount lustre-MDT0000 complete
[ 1892.575860] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1894.571348] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2034 to 0x280000400:2049)
[ 1894.571563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2034 to 0x240000400:2049)
[ 1896.339342] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1896.862658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1904.946062] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:47:15 (1782362835)
[ 1910.167929] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1912.931220] Lustre: Failing over lustre-MDT0000
[ 1913.234815] Lustre: server umount lustre-MDT0000 complete
[ 1927.179559] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1927.371290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2450 to 0x240000400:2465)
[ 1927.371340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2450 to 0x280000400:2465)
[ 1930.894652] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1931.459236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1939.515963] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 00:47:50 (1782362870)
[ 1940.218918] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 1940.558268] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1940.561454] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 1942.731877] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 00:47:53 (1782362873)
[ 1947.552073] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 1950.469922] Lustre: Failing over lustre-OST0000
[ 1950.524756] Lustre: server umount lustre-OST0000 complete
[ 1951.199834] LustreError: 87951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1952.137141] LustreError: 10734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1956.319580] LustreError: 83739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1961.439700] LustreError: 87951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1961.444026] LustreError: 87951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 1964.702236] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2009.892925] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 00:49:00 (1782362940)
[ 2011.213725] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2012.108099] Lustre: Failing over lustre-MDT0000
[ 2012.305375] Lustre: server umount lustre-MDT0000 complete
[ 2026.284924] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2028.268772] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 2028.268810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2913)
[ 2030.007274] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2030.577236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2044.144271] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 00:49:34 (1782362974)
[ 2044.895270] LustreError: 93915:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 2044.896031] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2045.820177] LustreError: 93891:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2045.919489] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2881)
[ 2051.039118] LustreError: 93891:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2051.041789] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2051.656726] LustreError: 93890:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2056.671108] LustreError: 93890:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2056.674780] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2057.254710] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2062.303140] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2062.305381] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2062.898132] LustreError: 93891:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2067.935121] LustreError: 93891:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2068.505681] LustreError: 95000:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2073.567167] LustreError: 95000:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2073.570064] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2073.573207] Lustre: Skipped 1 previous similar message
[ 2079.824690] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2079.828418] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message
[ 2085.343157] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2085.345688] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message
[ 2090.975186] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2090.978048] Lustre: Skipped 2 previous similar messages
[ 2097.201950] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2097.203892] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages
[ 2102.239218] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2102.241564] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages
[ 2104.965504] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 00:50:35 (1782363035)
[ 2105.610484] LustreError: 93891:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2114.950767] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2120.070630] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2125.190654] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2130.310747] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2130.313092] Lustre: Skipped 1 previous similar message
[ 2135.430583] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2145.655152] LustreError: 93891:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2145.658828] Lustre: 93891:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e40a4caa00 x1868940988105472/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363056 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2145.672153] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2145.674577] Lustre: Skipped 3 previous similar messages
[ 2145.675699] LustreError: 94290:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2170.246619] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2170.248899] Lustre: Skipped 1 previous similar message
[ 2185.719146] LustreError: 94290:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2185.721351] Lustre: 94290:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e52dd9d880 x1868940988109440/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363096 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2189.274911] LustreError: 93889:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2213.766818] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2213.769439] Lustre: Skipped 3 previous similar messages
[ 2229.319108] LustreError: 93889:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2229.322693] Lustre: 93889:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e408deca80 x1868940988111488/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363140 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2231.310481] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2231.312918] Lustre: Skipped 1 previous similar message
[ 2231.314329] LustreError: 95000:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2256.262525] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2256.264811] Lustre: Skipped 3 previous similar messages
[ 2271.359083] LustreError: 95000:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2271.362500] Lustre: 95000:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e50560d880 x1868940988113536/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363182 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2271.622360] LustreError: 93889:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2311.663097] LustreError: 93889:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2311.665243] Lustre: 93889:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e4066cbb80 x1868940988115328/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363222 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2316.786722] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 00:54:07 (1782363247)
[ 2317.781603] LustreError: 97727:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2317.785912] LustreError: 97727:0:(osd_handler.c:720:osd_ro()) Skipped 15 previous similar messages
[ 2318.086952] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2319.321247] Lustre: Failing over lustre-MDT0000
[ 2319.448782] Lustre: server umount lustre-MDT0000 complete
[ 2321.905461] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2321.908103] LustreError: Skipped 15 previous similar messages
[ 2322.008770] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2322.008872] Lustre: *** cfs_fail_loc=712, val=0***
[ 2322.013105] Lustre: Skipped 34 previous similar messages
[ 2322.014273] LustreError: 6591:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@ffff95e52e4bc000 x1868940993249536/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295
[ 2322.047344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2322.049110] Lustre: Skipped 17 previous similar messages
[ 2322.074054] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2322.074126] Lustre: lustre-MDT0000: Aborting client recovery
[ 2322.075932] Lustre: Skipped 26 previous similar messages
[ 2322.077206] LustreError: 98342:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2322.078481] Lustre: 98391:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2322.083432] Lustre: 98391:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2322.085827] Lustre: 98391:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f4cf112e-56b0-4499-aacf-a956cfcc6c06@<unknown>
[ 2322.088635] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 2322.100526] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0]
[ 2322.135387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2945)
[ 2322.135511] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2913)
[ 2323.456109] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2327.293329] Lustre: Failing over lustre-MDT0000
[ 2327.447838] Lustre: server umount lustre-MDT0000 complete
[ 2341.254529] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2342.279840] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2342.282257] Lustre: Skipped 10 previous similar messages
[ 2342.301031] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2342.303684] Lustre: Skipped 12 previous similar messages
[ 2342.321434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2977)
[ 2342.321460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2945)
[ 2344.949757] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2345.445107] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2348.706347] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 00:54:39 (1782363279)
[ 2351.731337] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 00:54:42 (1782363282)
[ 2352.063219] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 2352.064575] LustreError: 99255:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff95e52dd9d880 x1868940988165120/t0(0) o700->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:64/0 lens 264/248 e 0 to 0 dl 1782363294 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295
[ 2352.096666] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 2352.098410] Lustre: Skipped 34 previous similar messages
[ 2368.390591] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2368.393012] Lustre: Skipped 4 previous similar messages
[ 2369.476224] Lustre: Failing over lustre-MDT0000
[ 2369.745717] Lustre: server umount lustre-MDT0000 complete
[ 2381.727126] Lustre: 3307:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782363252/real 1782363252]  req@ffff95e506f8f100 x1868940993247360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782363312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2381.734090] Lustre: 3307:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 66 previous similar messages
[ 2382.757744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2979 to 0x240000400:3009)
[ 2382.757757] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2947 to 0x280000400:2977)
[ 2383.487936] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2387.125924] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2387.710583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2391.726864] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:55:22 (1782363322)
[ 2392.554180] Lustre: Failing over lustre-OST0000
[ 2392.591985] Lustre: server umount lustre-OST0000 complete
[ 2392.967541] LustreError: 6589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2392.973092] LustreError: 6589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 2397.663763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 2402.783903] LustreError: 82415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2402.789177] LustreError: 82415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages
[ 2406.997744] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2410.752325] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2411.284130] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2476.554184] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:56:47 (1782363407)
[ 2477.917043] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2478.641240] Lustre: Failing over lustre-MDT0000
[ 2478.900650] Lustre: server umount lustre-MDT0000 complete
[ 2492.851776] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2577.295523] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2577.298636] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 4 previous similar messages
[ 2587.527098] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2587.532139] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 1 previous similar message
[ 2608.008311] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2608.014968] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 3 previous similar messages
[ 2648.970856] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2648.976316] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages
[ 2675.500170] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports
[ 2675.502625] Lustre: 103925:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp
[ 2675.507039] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 2675.508973] Lustre: 103925:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout
[ 2675.511818] Lustre: 103925:0:(ldlm_lib.c:2380:target_recovery_overseer()) Skipped 1 previous similar message
[ 2675.514323] Lustre: 103925:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2675.517238] Lustre: 103925:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2675.532371] Lustre: lustre-MDT0000-osd: cancel update llog [0x20001a210:0x1:0x0]
[ 2675.576148] Lustre: 103925:0:(ldlm_lib.c:2937:target_recovery_thread()) too long recovery - read logs
[ 2675.581656] LustreError: dumping log to /tmp/lustre-log.1782363606.103925
[ 2675.651831] Lustre: *** cfs_fail_loc=216, val=0***
[ 2675.651899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3029 to 0x240000400:3073)
[ 2675.654824] LustreError: 103913:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 2676.704263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2997 to 0x280000400:3041)
[ 2680.855976] Lustre: DEBUG MARKER: replay-single test_48: @@@@@@ FAIL: client_up failed