[ 2042.530190] Lustre: Failing over lustre-MDT0000
[ 2042.770277] Lustre: server umount lustre-MDT0000 complete
[ 2046.849478] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2046.850684] LDISKFS-fs (dm-0): recovery complete
[ 2046.853431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2046.978861] Lustre: lustre-MDT0000: Aborting client recovery
[ 2046.980611] LustreError: 107216:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2046.983291] Lustre: 107249:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2046.985545] Lustre: 107249:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2046.987494] Lustre: 107249:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@<unknown>
[ 2046.990217] Lustre: 107249:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message
[ 2046.992297] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 2046.994699] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 2046.999971] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0]
[ 2047.020053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633)
[ 2047.020182] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633)
[ 2048.333573] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2052.067241] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2059.820313] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:49:54 (1782362994)
[ 2067.642127] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2068.274514] Lustre: Failing over lustre-MDT0000
[ 2068.493287] Lustre: server umount lustre-MDT0000 complete
[ 2082.549072] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2082.550757] LDISKFS-fs (dm-0): recovery complete
[ 2082.553441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2083.924668] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2087.945801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049)
[ 2087.945830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049)
[ 2089.726426] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2090.355498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2098.551368] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:50:33 (1782363033)
[ 2106.237338] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2108.720995] Lustre: Failing over lustre-MDT0000
[ 2109.072196] Lustre: server umount lustre-MDT0000 complete
[ 2123.011464] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2123.013124] LDISKFS-fs (dm-0): recovery complete
[ 2123.015997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2124.416829] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2128.920319] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465)
[ 2128.920339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465)
[ 2130.739090] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2131.280718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2139.781692] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 00:51:14 (1782363074)
[ 2140.463740] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 2140.788707] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2140.792257] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 2142.968801] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 00:51:17 (1782363077)
[ 2148.779590] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 2151.582416] Lustre: Failing over lustre-OST0000
[ 2151.636419] Lustre: server umount lustre-OST0000 complete
[ 2165.629065] LDISKFS-fs (dm-2): 3 truncates cleaned up
[ 2165.630283] LDISKFS-fs (dm-2): recovery complete
[ 2165.632955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2167.506499] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2212.731595] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 00:52:27 (1782363147)
[ 2215.341096] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2216.254062] Lustre: Failing over lustre-MDT0000
[ 2216.361102] Lustre: server umount lustre-MDT0000 complete
[ 2216.928577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2216.932894] Lustre: Skipped 74 previous similar messages
[ 2230.321790] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2230.322942] LDISKFS-fs (dm-0): recovery complete
[ 2230.325584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2230.475553] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2230.477588] Lustre: Skipped 27 previous similar messages
[ 2231.955630] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2235.877942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 2235.880252] Lustre: Skipped 78 previous similar messages
[ 2235.907095] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 2235.907138] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881)
[ 2237.629835] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2238.169832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2251.657700] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 00:53:06 (1782363186)
[ 2252.257089] Lustre: 116571:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782363171/real 1782363171]  req@ffff8cba85195500 x1868940962485120/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1782363187 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0 projid:4294967295
[ 2252.265619] LustreError: 116571:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 2252.267237] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2253.281590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913)
[ 2253.316394] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2258.400092] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2258.402372] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2258.992199] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2264.032102] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2264.034991] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2264.614267] LustreError: 6486:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2269.664108] LustreError: 6486:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2269.667453] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2270.258126] LustreError: 8841:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2275.296102] LustreError: 8841:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2275.887522] LustreError: 10064:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2280.928120] LustreError: 10064:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2280.930388] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2280.932702] Lustre: Skipped 1 previous similar message
[ 2287.147506] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2287.149823] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message
[ 2292.192215] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2292.194661] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message
[ 2297.824142] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2297.828321] Lustre: Skipped 2 previous similar messages
[ 2304.030852] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2304.035141] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages
[ 2309.088154] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2309.090787] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages
[ 2311.840291] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 00:54:06 (1782363246)
[ 2312.427482] LustreError: 6487:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2321.835932] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2326.955729] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2332.077902] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2337.195644] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2337.197880] Lustre: Skipped 1 previous similar message
[ 2342.316926] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2352.472192] LustreError: 6487:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2352.475481] Lustre: 6487:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cbbaec44380 x1868940954647680/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363267 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2352.555915] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2352.560799] Lustre: Skipped 3 previous similar messages
[ 2352.562990] LustreError: 12612:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2377.131862] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2377.141909] Lustre: Skipped 1 previous similar message
[ 2392.608105] LustreError: 12612:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2392.610378] Lustre: 12612:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cbbb3e4f480 x1868940954652416/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363307 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2396.168991] LustreError: 6486:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2420.651823] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2420.653969] Lustre: Skipped 3 previous similar messages
[ 2436.216107] LustreError: 6486:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2436.218518] Lustre: 6486:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cba83aba300 x1868940954656000/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363351 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2438.113820] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2438.116409] Lustre: Skipped 1 previous similar message
[ 2438.117541] LustreError: 6487:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2462.635760] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2462.638283] Lustre: Skipped 3 previous similar messages
[ 2478.160061] LustreError: 6487:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2478.162983] Lustre: 6487:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cbbaec44e00 x1868940954659456/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363393 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2480.014370] LustreError: 8841:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2520.056110] LustreError: 8841:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2520.059844] Lustre: 8841:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cbb89130a80 x1868940954662912/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363435 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2523.488165] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 00:57:38 (1782363458)
[ 2527.297577] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2528.436243] Lustre: Failing over lustre-MDT0000
[ 2528.537261] Lustre: server umount lustre-MDT0000 complete
[ 2532.462505] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2532.463784] LDISKFS-fs (dm-0): recovery complete
[ 2532.466419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2532.503360] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2532.506752] LustreError: Skipped 17 previous similar messages
[ 2532.567392] Lustre: *** cfs_fail_loc=712, val=0***
[ 2532.569373] LustreError: 8397:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@ffff8cbbb51da680 x1868940962630016/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295
[ 2532.616751] Lustre: lustre-MDT0000: Aborting client recovery
[ 2532.618040] LustreError: 121412:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2532.620298] Lustre: 121445:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2532.622702] Lustre: 121445:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2532.624802] Lustre: 121445:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@<unknown>
[ 2532.627850] Lustre: 121445:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message
[ 2532.629982] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 2532.632624] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0]
[ 2532.640191] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0]
[ 2532.660068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945)
[ 2532.660096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913)
[ 2533.854185] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2537.953484] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2543.643873] Lustre: Failing over lustre-MDT0000
[ 2543.811195] Lustre: server umount lustre-MDT0000 complete
[ 2548.192699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2548.195707] LustreError: Skipped 16 previous similar messages
[ 2556.684628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2558.156104] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2560.939721] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2560.944109] Lustre: Skipped 13 previous similar messages
[ 2562.025049] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 2562.028905] Lustre: Skipped 14 previous similar messages
[ 2562.048024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977)
[ 2562.048024] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945)
[ 2563.759316] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2564.302917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2567.666295] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 00:58:22 (1782363502)
[ 2567.713135] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2567.715815] Lustre: Skipped 2 previous similar messages
[ 2570.778257] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 00:58:25 (1782363505)
[ 2571.085338] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 2571.086814] LustreError: 99125:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff8cbb90342c50 x1868940954717696/t0(0) o700->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:287/0 lens 264/248 e 0 to 0 dl 1782363517 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295
[ 2588.618196] Lustre: Failing over lustre-MDT0000
[ 2588.880818] Lustre: server umount lustre-MDT0000 complete
[ 2591.596556] LustreError: 12612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2591.603641] LustreError: 12612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 68 previous similar messages
[ 2601.709052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2601.821076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2601.823148] Lustre: Skipped 10 previous similar messages
[ 2603.181730] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2607.091057] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977)
[ 2607.091065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009)
[ 2608.788264] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2609.283726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2613.127799] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:59:08 (1782363548)
[ 2613.966856] Lustre: Failing over lustre-OST0000
[ 2614.027282] Lustre: server umount lustre-OST0000 complete
[ 2626.593443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2628.420207] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2631.929352] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2632.452773] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2697.580320] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 01:00:32 (1782363632)
[ 2700.087988] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2700.814507] Lustre: Failing over lustre-MDT0000
[ 2701.041346] Lustre: server umount lustre-MDT0000 complete
[ 2715.076555] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2715.077868] LDISKFS-fs (dm-0): recovery complete
[ 2715.081708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2716.549788] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2761.136517] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2761.148545] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 4 previous similar messages
[ 2771.373084] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2791.851806] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2791.856584] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 3 previous similar messages
[ 2832.812146] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2832.815976] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages
[ 2899.500099] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports
[ 2899.502441] Lustre: 127574:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp
[ 2899.505565] Lustre: 127574:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message
[ 2899.508372] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 2899.510078] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2899.514110] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 13 previous similar messages
[ 2899.517486] Lustre: 127574:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout
[ 2899.520884] Lustre: 127574:0:(ldlm_lib.c:2380:target_recovery_overseer()) Skipped 1 previous similar message
[ 2899.524727] Lustre: 127574:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2899.527625] Lustre: 127574:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2899.533654] Lustre: lustre-MDT0000-osd: cancel update llog [0x20001a210:0x1:0x0]
[ 2899.540705] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ec:0x1:0x0]
[ 2899.550041] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 2899.550190] Lustre: 127574:0:(ldlm_lib.c:2937:target_recovery_thread()) too long recovery - read logs
[ 2899.553068] Lustre: Skipped 21 previous similar messages
[ 2899.555898] LustreError: dumping log to /tmp/lustre-log.1782363834.127574
[ 2899.614162] Lustre: *** cfs_fail_loc=216, val=0***
[ 2899.614178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3029 to 0x280000401:3073)
[ 2899.615584] LustreError: 127562:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 2900.640557] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2997 to 0x2c0000401:3041)
[ 2905.158591] Lustre: DEBUG MARKER: replay-single test_48: @@@@@@ FAIL: client_up failed