[ 1861.528988] Lustre: Failing over lustre-MDT0000
[ 1861.693309] Lustre: server umount lustre-MDT0000 complete
[ 1864.167384] Lustre: lustre-MDT0000: Aborting client recovery
[ 1864.168805] LustreError: 86275:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 1864.171215] Lustre: 86321:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 1864.173442] Lustre: 86321:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 1864.175335] Lustre: 86321:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f4cf112e-56b0-4499-aacf-a956cfcc6c06@<unknown>
[ 1864.178131] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 1864.188995] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 1864.222145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1544 to 0x280000400:1633)
[ 1864.222145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1571 to 0x240000400:1633)
[ 1865.590585] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1871.284606] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:46:42 (1782362802)
[ 1877.558071] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1878.151693] Lustre: Failing over lustre-MDT0000
[ 1878.399542] Lustre: server umount lustre-MDT0000 complete
[ 1892.575860] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1894.571348] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2034 to 0x280000400:2049)
[ 1894.571563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2034 to 0x240000400:2049)
[ 1896.339342] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1896.862658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1904.946062] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:47:15 (1782362835)
[ 1910.167929] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1912.931220] Lustre: Failing over lustre-MDT0000
[ 1913.234815] Lustre: server umount lustre-MDT0000 complete
[ 1927.179559] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1927.371290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2450 to 0x240000400:2465)
[ 1927.371340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2450 to 0x280000400:2465)
[ 1930.894652] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1931.459236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1939.515963] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 00:47:50 (1782362870)
[ 1940.218918] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 1940.558268] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1940.561454] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 1942.731877] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 00:47:53 (1782362873)
[ 1947.552073] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 1950.469922] Lustre: Failing over lustre-OST0000
[ 1950.524756] Lustre: server umount lustre-OST0000 complete
[ 1951.199834] LustreError: 87951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1952.137141] LustreError: 10734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1956.319580] LustreError: 83739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1961.439700] LustreError: 87951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1961.444026] LustreError: 87951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 1964.702236] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2009.892925] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 00:49:00 (1782362940)
[ 2011.213725] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2012.108099] Lustre: Failing over lustre-MDT0000
[ 2012.305375] Lustre: server umount lustre-MDT0000 complete
[ 2026.284924] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2028.268772] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 2028.268810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2913)
[ 2030.007274] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2030.577236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2044.144271] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 00:49:34 (1782362974)
[ 2044.895270] LustreError: 93915:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 2044.896031] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2045.820177] LustreError: 93891:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2045.919489] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2881)
[ 2051.039118] LustreError: 93891:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2051.041789] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2051.656726] LustreError: 93890:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2056.671108] LustreError: 93890:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2056.674780] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2057.254710] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2062.303140] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2062.305381] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2062.898132] LustreError: 93891:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2067.935121] LustreError: 93891:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2068.505681] LustreError: 95000:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2073.567167] LustreError: 95000:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2073.570064] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2073.573207] Lustre: Skipped 1 previous similar message
[ 2079.824690] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2079.828418] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message
[ 2085.343157] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2085.345688] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message
[ 2090.975186] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2090.978048] Lustre: Skipped 2 previous similar messages
[ 2097.201950] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2097.203892] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages
[ 2102.239218] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2102.241564] LustreError: 93889:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages
[ 2104.965504] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 00:50:35 (1782363035)
[ 2105.610484] LustreError: 93891:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2114.950767] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2120.070630] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2125.190654] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2130.310747] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2130.313092] Lustre: Skipped 1 previous similar message
[ 2135.430583] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2145.655152] LustreError: 93891:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2145.658828] Lustre: 93891:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e40a4caa00 x1868940988105472/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363056 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2145.672153] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2145.674577] Lustre: Skipped 3 previous similar messages
[ 2145.675699] LustreError: 94290:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2170.246619] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2170.248899] Lustre: Skipped 1 previous similar message
[ 2185.719146] LustreError: 94290:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2185.721351] Lustre: 94290:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e52dd9d880 x1868940988109440/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363096 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2189.274911] LustreError: 93889:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2213.766818] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2213.769439] Lustre: Skipped 3 previous similar messages
[ 2229.319108] LustreError: 93889:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2229.322693] Lustre: 93889:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e408deca80 x1868940988111488/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363140 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2231.310481] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2231.312918] Lustre: Skipped 1 previous similar message
[ 2231.314329] LustreError: 95000:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2256.262525] Lustre: lustre-MDT0000: Export ffff95e5315a4000 already connecting from 192.168.202.24@tcp
[ 2256.264811] Lustre: Skipped 3 previous similar messages
[ 2271.359083] LustreError: 95000:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2271.362500] Lustre: 95000:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e50560d880 x1868940988113536/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363182 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2271.622360] LustreError: 93889:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2311.663097] LustreError: 93889:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2311.665243] Lustre: 93889:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff95e4066cbb80 x1868940988115328/t0(0) o38->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363222 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2316.786722] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 00:54:07 (1782363247)
[ 2317.781603] LustreError: 97727:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2317.785912] LustreError: 97727:0:(osd_handler.c:720:osd_ro()) Skipped 15 previous similar messages
[ 2318.086952] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2319.321247] Lustre: Failing over lustre-MDT0000
[ 2319.448782] Lustre: server umount lustre-MDT0000 complete
[ 2321.905461] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2321.908103] LustreError: Skipped 15 previous similar messages
[ 2322.008770] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2322.008872] Lustre: *** cfs_fail_loc=712, val=0***
[ 2322.013105] Lustre: Skipped 34 previous similar messages
[ 2322.014273] LustreError: 6591:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@ffff95e52e4bc000 x1868940993249536/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295
[ 2322.047344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2322.049110] Lustre: Skipped 17 previous similar messages
[ 2322.074054] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2322.074126] Lustre: lustre-MDT0000: Aborting client recovery
[ 2322.075932] Lustre: Skipped 26 previous similar messages
[ 2322.077206] LustreError: 98342:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2322.078481] Lustre: 98391:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2322.083432] Lustre: 98391:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2322.085827] Lustre: 98391:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f4cf112e-56b0-4499-aacf-a956cfcc6c06@<unknown>
[ 2322.088635] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 2322.100526] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0]
[ 2322.135387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2945)
[ 2322.135511] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2913)
[ 2323.456109] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2327.293329] Lustre: Failing over lustre-MDT0000
[ 2327.447838] Lustre: server umount lustre-MDT0000 complete
[ 2341.254529] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2342.279840] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2342.282257] Lustre: Skipped 10 previous similar messages
[ 2342.301031] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2342.303684] Lustre: Skipped 12 previous similar messages
[ 2342.321434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2977)
[ 2342.321460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2945)
[ 2344.949757] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2345.445107] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2348.706347] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 00:54:39 (1782363279)
[ 2351.731337] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 00:54:42 (1782363282)
[ 2352.063219] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 2352.064575] LustreError: 99255:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff95e52dd9d880 x1868940988165120/t0(0) o700->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:64/0 lens 264/248 e 0 to 0 dl 1782363294 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295
[ 2352.096666] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 2352.098410] Lustre: Skipped 34 previous similar messages
[ 2368.390591] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnecting
[ 2368.393012] Lustre: Skipped 4 previous similar messages
[ 2369.476224] Lustre: Failing over lustre-MDT0000
[ 2369.745717] Lustre: server umount lustre-MDT0000 complete
[ 2381.727126] Lustre: 3307:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782363252/real 1782363252]  req@ffff95e506f8f100 x1868940993247360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782363312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2381.734090] Lustre: 3307:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 66 previous similar messages
[ 2382.757744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2979 to 0x240000400:3009)
[ 2382.757757] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2947 to 0x280000400:2977)
[ 2383.487936] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2387.125924] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2387.710583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2391.726864] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:55:22 (1782363322)
[ 2392.554180] Lustre: Failing over lustre-OST0000
[ 2392.591985] Lustre: server umount lustre-OST0000 complete
[ 2392.967541] LustreError: 6589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2392.973092] LustreError: 6589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 2397.663763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 2402.783903] LustreError: 82415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2402.789177] LustreError: 82415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages
[ 2406.997744] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2410.752325] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2411.284130] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2476.554184] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:56:47 (1782363407)
[ 2477.917043] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2478.641240] Lustre: Failing over lustre-MDT0000
[ 2478.900650] Lustre: server umount lustre-MDT0000 complete
[ 2492.851776] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2577.295523] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2577.298636] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 4 previous similar messages
[ 2587.527098] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2587.532139] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 1 previous similar message
[ 2608.008311] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2608.014968] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 3 previous similar messages
[ 2648.970856] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2648.976316] Lustre: 103925:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages
[ 2675.500170] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports
[ 2675.502625] Lustre: 103925:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp
[ 2675.507039] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 2675.508973] Lustre: 103925:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout
[ 2675.511818] Lustre: 103925:0:(ldlm_lib.c:2380:target_recovery_overseer()) Skipped 1 previous similar message
[ 2675.514323] Lustre: 103925:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2675.517238] Lustre: 103925:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2675.532371] Lustre: lustre-MDT0000-osd: cancel update llog [0x20001a210:0x1:0x0]
[ 2675.576148] Lustre: 103925:0:(ldlm_lib.c:2937:target_recovery_thread()) too long recovery - read logs
[ 2675.581656] LustreError: dumping log to /tmp/lustre-log.1782363606.103925
[ 2675.651831] Lustre: *** cfs_fail_loc=216, val=0***
[ 2675.651899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3029 to 0x240000400:3073)
[ 2675.654824] LustreError: 103913:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 2676.704263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2997 to 0x280000400:3041)
[ 2680.855976] Lustre: DEBUG MARKER: replay-single test_48: @@@@@@ FAIL: client_up failed
[ 2687.332781] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 01:00:17 (1782363617)
[ 2688.314846] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2697.083228] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 01:00:27 (1782363627)
[ 2698.873136] Lustre: Failing over lustre-MDT0000
[ 2699.060956] Lustre: server umount lustre-MDT0000 complete
[ 2715.546942] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 2715.550929] LustreError: 106054:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff95e5046ece00 x1868940988250368/t0(0) o101->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:427/0 lens 328/344 e 0 to 0 dl 1782363657 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295
[ 2715.556041] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2730.886620] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnected, waiting for 1 clients in recovery for 1:25
[ 2730.922249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3043 to 0x280000400:3073)
[ 2730.922451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3029 to 0x240000400:3105)
[ 2733.386996] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2734.083607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2738.676032] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 01:01:09 (1782363669)
[ 2740.137354] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2741.799107] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2742.513234] Lustre: Failing over lustre-MDT0000
[ 2742.678980] Lustre: server umount lustre-MDT0000 complete
[ 2756.526090] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3105)
[ 2756.526101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3029 to 0x240000400:3137)
[ 2757.768395] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2762.092595] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2762.725591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2766.858289] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 01:01:37 (1782363697)
[ 2767.338386] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2770.314223] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2771.188921] Lustre: Failing over lustre-MDT0000
[ 2771.308272] Lustre: server umount lustre-MDT0000 complete
[ 2785.815165] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2792.864135] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3137)
[ 2792.864138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3169)
[ 2794.789244] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2795.361296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2799.041129] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 01:02:09 (1782363729)
[ 2799.435483] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2802.180359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2802.913886] Lustre: Failing over lustre-MDT0000
[ 2803.011760] Lustre: server umount lustre-MDT0000 complete
[ 2816.977552] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2825.118948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3139 to 0x280000400:3169)
[ 2825.118948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3201)
[ 2829.462639] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 01:02:40 (1782363760)
[ 2830.823911] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2830.826387] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2830.828230] LustreError: 110994:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff95e52edf9f80 x1868940988294912/t257698037777(0) o35->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:542/0 lens 392/456 e 0 to 0 dl 1782363772 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0
[ 2831.712762] Lustre: Failing over lustre-MDT0000
[ 2831.979203] Lustre: server umount lustre-MDT0000 complete
[ 2845.597713] Lustre: 112231:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff95e405c89180 x1868940988294912/t257698037777(0) o35->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:557/0 lens 392/456 e 0 to 0 dl 1782363787 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0
[ 2845.603259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3203 to 0x240000400:3233)
[ 2845.603291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3139 to 0x280000400:3201)
[ 2845.881960] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2849.538275] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2850.075525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2853.520972] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 01:03:04 (1782363784)
[ 2853.868984] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2853.870588] LustreError: 112228:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff95e5047f6680 x1868940988306432/t261993005072(0) o36->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:565/0 lens 504/448 e 0 to 0 dl 1782363795 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 2856.277675] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2856.887323] Lustre: Failing over lustre-MDT0000
[ 2856.988253] Lustre: server umount lustre-MDT0000 complete
[ 2870.176492] Lustre: 113857:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff95e506f8f100 x1868940988306432/t261993005072(0) o36->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:582/0 lens 504/2880 e 0 to 0 dl 1782363812 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 2870.179078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3235 to 0x240000400:3265)
[ 2870.179089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3139 to 0x280000400:3233)
[ 2870.682982] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2874.352342] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2874.883519] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2878.332462] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 01:03:29 (1782363809)
[ 2878.695606] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2878.697481] LustreError: 113857:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff95e52d614a80 x1868940988318336/t266287972368(0) o36->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:590/0 lens 504/448 e 0 to 0 dl 1782363820 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 2879.968474] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2881.156240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2881.743751] Lustre: Failing over lustre-MDT0000
[ 2881.842988] Lustre: server umount lustre-MDT0000 complete
[ 2894.236383] Lustre: 115505:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff95e50560dc00 x1868940988318336/t266287972368(0) o36->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:606/0 lens 504/2880 e 0 to 0 dl 1782363836 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 2894.241187] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3235 to 0x280000400:3265)
[ 2894.241354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3235 to 0x240000400:3297)
[ 2894.243476] Lustre: 115505:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 2895.353760] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2900.312797] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 01:03:51 (1782363831)
[ 2900.678741] LustreError: 115482:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff95e53db63b80 x1868940988329856/t270582939664(0) o36->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:612/0 lens 504/448 e 0 to 0 dl 1782363842 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 2900.685403] LustreError: 115482:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message
[ 2901.975071] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2901.976972] Lustre: Skipped 1 previous similar message
[ 2903.444476] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2904.064113] Lustre: Failing over lustre-MDT0000
[ 2904.181412] Lustre: server umount lustre-MDT0000 complete
[ 2918.108713] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2926.489089] Lustre: 116961:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff95e506f8d180 x1868940988329856/t270582939664(0) o36->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:638/0 lens 504/2880 e 0 to 0 dl 1782363868 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 2926.493620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3235 to 0x240000400:3329)
[ 2926.493657] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3267 to 0x280000400:3297)
[ 2930.720522] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 01:04:21 (1782363861)
[ 2931.071676] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2932.346580] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2932.347891] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2932.349106] Lustre: Skipped 2 previous similar messages
[ 2934.201802] LustreError: 117818:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2934.206267] LustreError: 117818:0:(osd_handler.c:720:osd_ro()) Skipped 7 previous similar messages
[ 2934.488122] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2935.082838] Lustre: Failing over lustre-MDT0000
[ 2935.178424] Lustre: server umount lustre-MDT0000 complete
[ 2947.307082] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2947.310550] LustreError: Skipped 11 previous similar messages
[ 2947.435775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2947.441644] Lustre: Skipped 24 previous similar messages
[ 2947.484470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2947.486422] Lustre: Skipped 12 previous similar messages
[ 2947.512272] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2947.514329] Lustre: Skipped 14 previous similar messages
[ 2948.794500] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2949.129542] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2949.134305] Lustre: Skipped 11 previous similar messages
[ 2949.153838] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2949.156517] Lustre: Skipped 11 previous similar messages
[ 2949.171157] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3267 to 0x280000400:3329)
[ 2949.171157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3331 to 0x240000400:3361)
[ 2949.173212] Lustre: 118368:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff95e504631f80 x1868940988341760/t274877906960(0) o35->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:661/0 lens 392/456 e 0 to 0 dl 1782363891 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0
[ 2952.673471] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 2952.675527] Lustre: Skipped 24 previous similar messages
[ 2953.695655] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 01:04:44 (1782363884)
[ 2954.017217] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 2954.019295] LustreError: 118365:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff95e52d614a80 x1868940988351232/t279172874255(0) o101->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:665/0 lens 664/608 e 0 to 0 dl 1782363895 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0
[ 2954.024634] LustreError: 118365:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message
[ 2970.504140] Lustre: 119115:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff95e52e4bc000 x1868940988351232/t279172874255(0) o101->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:682/0 lens 664/3488 e 0 to 0 dl 1782363912 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0
[ 2972.656249] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 01:05:03 (1782363903)
[ 2973.876950] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2974.491806] Lustre: Failing over lustre-MDT0000
[ 2974.595571] Lustre: server umount lustre-MDT0000 complete
[ 2988.115281] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2990.021868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3331 to 0x240000400:3393)
[ 2990.023172] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3331 to 0x280000400:3361)
[ 2991.665974] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2992.187412] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2994.143134] Lustre: 3309:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782363909/real 1782363909]  req@ffff95e52e4bdf80 x1868940993466112/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782363925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2994.153200] Lustre: 3309:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
[ 3005.642358] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 01:05:36 (1782363936)
[ 3007.122283] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3007.729834] Lustre: Failing over lustre-MDT0000
[ 3007.915439] Lustre: server umount lustre-MDT0000 complete
[ 3021.421891] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3021.733499] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3363 to 0x280000400:3393)
[ 3021.733502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3331 to 0x240000400:3425)
[ 3024.924140] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3025.422062] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3027.648360] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 3032.453056] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 01:06:03 (1782363963)
[ 3040.112680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3040.661699] Lustre: Failing over lustre-MDT0000
[ 3040.735647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3040.737446] Lustre: Skipped 1 previous similar message
[ 3041.038744] Lustre: server umount lustre-MDT0000 complete
[ 3054.510458] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3056.456796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4676 to 0x240000400:4705)
[ 3056.456812] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4644 to 0x280000400:4673)
[ 3058.148973] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3058.628665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3074.486941] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 01:06:45 (1782364005)
[ 3076.233296] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3076.978380] Lustre: Failing over lustre-MDT0000
[ 3077.136295] Lustre: server umount lustre-MDT0000 complete
[ 3090.366258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4676 to 0x240000400:4737)
[ 3090.366582] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4675 to 0x280000400:4705)
[ 3091.705723] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3095.370133] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3096.078491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3099.663852] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 3100.216843] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 3102.362339] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 01:07:13 (1782364033)
[ 3107.930574] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 3124.873263] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 3124.874524] Lustre: Skipped 1 previous similar message
[ 3124.875621] LustreError: 125422:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff95e52cb9d850 x1868940991118592/t296352743435(0) o36->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:81/0 lens 66040/440 e 0 to 0 dl 1782364066 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0
[ 3141.000466] Lustre: 125421:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff95e5372cd500 x1868940991118592/t296352743435(0) o36->f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp:97/0 lens 66040/440 e 0 to 0 dl 1782364082 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0
[ 3143.819642] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 3144.302768] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 01:07:55 (1782364075)
[ 3147.149959] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3147.964706] Lustre: Failing over lustre-MDT0000
[ 3148.257386] Lustre: server umount lustre-MDT0000 complete
[ 3161.748086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4839 to 0x240000400:4865)
[ 3161.748448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4806 to 0x280000400:4833)
[ 3162.389755] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3166.507856] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3167.227684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3171.903605] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 01:08:22 (1782364102)
[ 3178.157794] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3181.948560] Lustre: Failing over lustre-OST0000
[ 3181.961126] Lustre: lustre-OST0000: Not available for connect from 192.168.202.24@tcp (stopping)
[ 3181.996533] Lustre: server umount lustre-OST0000 complete
[ 3182.598105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107
[ 3182.601264] LustreError: Skipped 4 previous similar messages
[ 3182.603309] LustreError: 82418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3182.609629] LustreError: 82418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 3186.655993] LustreError: 34033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3191.775614] LustreError: 10734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3191.780074] LustreError: 10734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 3196.208477] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3207.453730] Lustre: Failing over lustre-OST0000
[ 3207.484893] Lustre: server umount lustre-OST0000 complete
[ 3207.559238] LustreError: 6590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3207.563359] LustreError: 6590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 3221.800387] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3225.421775] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3225.972249] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3259.749863] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 01:09:50 (1782364190)
[ 3260.619801] Lustre: Failing over lustre-MDT0000
[ 3260.831705] Lustre: server umount lustre-MDT0000 complete
[ 3274.148559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5281)
[ 3274.148597] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5249)
[ 3274.576203] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3285.995154] Lustre: Failing over lustre-MDT0000
[ 3286.090454] Lustre: server umount lustre-MDT0000 complete
[ 3299.749985] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3299.750644] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5281)
[ 3299.750646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5313)
[ 3303.458203] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3304.001406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3307.399704] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 01:10:38 (1782364238)
[ 3318.732836] Lustre: Failing over lustre-OST0000
[ 3318.767540] Lustre: server umount lustre-OST0000 complete
[ 3319.263731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3319.263953] LustreError: 82415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3319.267734] LustreError: Skipped 2 previous similar messages
[ 3319.273832] LustreError: 82415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages
[ 3333.588679] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3337.882435] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3338.612923] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3343.091207] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 01:11:13 (1782364273)
[ 3343.770723] Lustre: Failing over lustre-MDT0000
[ 3344.067524] Lustre: server umount lustre-MDT0000 complete
[ 3347.681359] Lustre: *** cfs_fail_loc=605, val=0***
[ 3347.684608] LustreError: 136013:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc1268b60 failed: rc = -95
[ 3347.692151] LustreError: 136013:0:(obd_config.c:845:class_setup()) setup MGS failed (-95)
[ 3347.696548] LustreError: 136013:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95
[ 3347.700897] LustreError: 136013:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered
[ 3347.705474] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 3347.709149] LustreError: 136013:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000
[ 3347.734777] Lustre: server umount lustre-MDT0000 complete
[ 3347.736634] LustreError: 136013:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 3350.955901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5283 to 0x280000400:5313)
[ 3350.955902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5315 to 0x240000400:5345)
[ 3351.371892] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3354.447135] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 01:11:25 (1782364285)
[ 3356.192090] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3357.592787] Lustre: Failing over lustre-MDT0000
[ 3357.826573] Lustre: server umount lustre-MDT0000 complete
[ 3371.400918] Lustre: *** cfs_fail_loc=707, val=0***
[ 3372.259421] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3386.766057] Lustre: lustre-MDT0000: Client f4cf112e-56b0-4499-aacf-a956cfcc6c06 (at 192.168.202.24@tcp) reconnected, waiting for 1 clients in recovery for 0:54
[ 3458.453028] Lustre: 137638:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 3458.457893] Lustre: 137638:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 6 previous similar messages
[ 3468.678993] Lustre: 137638:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 3468.684023] Lustre: 137638:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 1 previous similar message
[ 3489.159167] Lustre: 137638:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 3489.163562] Lustre: 137638:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 3 previous similar messages
[ 3530.153448] Lustre: 137638:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 3530.160339] Lustre: 137638:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages
[ 3551.500222] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports
[ 3551.504772] Lustre: 137638:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f4cf112e-56b0-4499-aacf-a956cfcc6c06@192.168.202.24@tcp
[ 3551.512601] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 3551.516586] Lustre: 137638:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout
[ 3551.521043] Lustre: 137638:0:(ldlm_lib.c:2380:target_recovery_overseer()) Skipped 1 previous similar message
[ 3551.525045] Lustre: 137638:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3551.528430] Lustre: 137638:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 1 previous similar message
[ 3551.544684] Lustre: lustre-MDT0000-osd: cancel update llog [0x20001b980:0x1:0x0]
[ 3551.574794] Lustre: 137638:0:(ldlm_lib.c:2937:target_recovery_thread()) too long recovery - read logs
[ 3551.578175] LustreError: dumping log to /tmp/lustre-log.1782364482.137638
[ 3551.624575] Lustre: lustre-MDT0000: Recovery over after 3:00, of 1 clients 0 recovered and 1 was evicted.
[ 3551.628653] Lustre: Skipped 11 previous similar messages
[ 3551.646894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5322 to 0x280000400:5345)
[ 3551.646925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5355 to 0x240000400:5377)
[ 3562.798208] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3563.378868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3565.447189] Lustre: DEBUG MARKER: replay-single test_62: @@@@@@ FAIL: unlinkmany /mnt/lustre/d62.replay-single/f62.replay-single failed