[ 2042.530190] Lustre: Failing over lustre-MDT0000
[ 2042.770277] Lustre: server umount lustre-MDT0000 complete
[ 2046.849478] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2046.850684] LDISKFS-fs (dm-0): recovery complete
[ 2046.853431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2046.978861] Lustre: lustre-MDT0000: Aborting client recovery
[ 2046.980611] LustreError: 107216:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2046.983291] Lustre: 107249:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2046.985545] Lustre: 107249:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2046.987494] Lustre: 107249:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@<unknown>
[ 2046.990217] Lustre: 107249:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message
[ 2046.992297] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 2046.994699] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 2046.999971] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0]
[ 2047.020053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633)
[ 2047.020182] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633)
[ 2048.333573] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2052.067241] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2059.820313] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:49:54 (1782362994)
[ 2067.642127] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2068.274514] Lustre: Failing over lustre-MDT0000
[ 2068.493287] Lustre: server umount lustre-MDT0000 complete
[ 2082.549072] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2082.550757] LDISKFS-fs (dm-0): recovery complete
[ 2082.553441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2083.924668] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2087.945801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049)
[ 2087.945830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049)
[ 2089.726426] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2090.355498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2098.551368] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:50:33 (1782363033)
[ 2106.237338] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2108.720995] Lustre: Failing over lustre-MDT0000
[ 2109.072196] Lustre: server umount lustre-MDT0000 complete
[ 2123.011464] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2123.013124] LDISKFS-fs (dm-0): recovery complete
[ 2123.015997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2124.416829] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2128.920319] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465)
[ 2128.920339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465)
[ 2130.739090] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2131.280718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2139.781692] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 00:51:14 (1782363074)
[ 2140.463740] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 2140.788707] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2140.792257] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 2142.968801] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 00:51:17 (1782363077)
[ 2148.779590] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 2151.582416] Lustre: Failing over lustre-OST0000
[ 2151.636419] Lustre: server umount lustre-OST0000 complete
[ 2165.629065] LDISKFS-fs (dm-2): 3 truncates cleaned up
[ 2165.630283] LDISKFS-fs (dm-2): recovery complete
[ 2165.632955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2167.506499] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2212.731595] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 00:52:27 (1782363147)
[ 2215.341096] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2216.254062] Lustre: Failing over lustre-MDT0000
[ 2216.361102] Lustre: server umount lustre-MDT0000 complete
[ 2216.928577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2216.932894] Lustre: Skipped 74 previous similar messages
[ 2230.321790] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2230.322942] LDISKFS-fs (dm-0): recovery complete
[ 2230.325584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2230.475553] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2230.477588] Lustre: Skipped 27 previous similar messages
[ 2231.955630] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2235.877942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 2235.880252] Lustre: Skipped 78 previous similar messages
[ 2235.907095] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 2235.907138] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881)
[ 2237.629835] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2238.169832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2251.657700] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 00:53:06 (1782363186)
[ 2252.257089] Lustre: 116571:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782363171/real 1782363171]  req@ffff8cba85195500 x1868940962485120/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1782363187 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0 projid:4294967295
[ 2252.265619] LustreError: 116571:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 2252.267237] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2253.281590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913)
[ 2253.316394] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2258.400092] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2258.402372] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2258.992199] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2264.032102] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2264.034991] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2264.614267] LustreError: 6486:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2269.664108] LustreError: 6486:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2269.667453] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2270.258126] LustreError: 8841:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2275.296102] LustreError: 8841:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2275.887522] LustreError: 10064:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2280.928120] LustreError: 10064:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2280.930388] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2280.932702] Lustre: Skipped 1 previous similar message
[ 2287.147506] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2287.149823] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message
[ 2292.192215] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2292.194661] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message
[ 2297.824142] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2297.828321] Lustre: Skipped 2 previous similar messages
[ 2304.030852] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping
[ 2304.035141] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages
[ 2309.088154] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 2309.090787] LustreError: 12612:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages
[ 2311.840291] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 00:54:06 (1782363246)
[ 2312.427482] LustreError: 6487:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2321.835932] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2326.955729] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2332.077902] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2337.195644] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2337.197880] Lustre: Skipped 1 previous similar message
[ 2342.316926] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2352.472192] LustreError: 6487:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2352.475481] Lustre: 6487:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cbbaec44380 x1868940954647680/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363267 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2352.555915] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2352.560799] Lustre: Skipped 3 previous similar messages
[ 2352.562990] LustreError: 12612:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2377.131862] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2377.141909] Lustre: Skipped 1 previous similar message
[ 2392.608105] LustreError: 12612:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2392.610378] Lustre: 12612:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cbbb3e4f480 x1868940954652416/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363307 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2396.168991] LustreError: 6486:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2420.651823] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2420.653969] Lustre: Skipped 3 previous similar messages
[ 2436.216107] LustreError: 6486:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2436.218518] Lustre: 6486:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cba83aba300 x1868940954656000/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363351 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2438.113820] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2438.116409] Lustre: Skipped 1 previous similar message
[ 2438.117541] LustreError: 6487:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2462.635760] Lustre: lustre-MDT0000: Export ffff8cbbb3e1b000 already connecting from 192.168.204.40@tcp
[ 2462.638283] Lustre: Skipped 3 previous similar messages
[ 2478.160061] LustreError: 6487:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2478.162983] Lustre: 6487:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cbbaec44e00 x1868940954659456/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363393 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2480.014370] LustreError: 8841:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 2520.056110] LustreError: 8841:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 2520.059844] Lustre: 8841:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff8cbb89130a80 x1868940954662912/t0(0) o38->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1782363435 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295
[ 2523.488165] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 00:57:38 (1782363458)
[ 2527.297577] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2528.436243] Lustre: Failing over lustre-MDT0000
[ 2528.537261] Lustre: server umount lustre-MDT0000 complete
[ 2532.462505] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2532.463784] LDISKFS-fs (dm-0): recovery complete
[ 2532.466419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2532.503360] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2532.506752] LustreError: Skipped 17 previous similar messages
[ 2532.567392] Lustre: *** cfs_fail_loc=712, val=0***
[ 2532.569373] LustreError: 8397:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@ffff8cbbb51da680 x1868940962630016/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295
[ 2532.616751] Lustre: lustre-MDT0000: Aborting client recovery
[ 2532.618040] LustreError: 121412:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2532.620298] Lustre: 121445:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2532.622702] Lustre: 121445:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2532.624802] Lustre: 121445:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@<unknown>
[ 2532.627850] Lustre: 121445:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message
[ 2532.629982] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 2532.632624] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0]
[ 2532.640191] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0]
[ 2532.660068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945)
[ 2532.660096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913)
[ 2533.854185] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2537.953484] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2543.643873] Lustre: Failing over lustre-MDT0000
[ 2543.811195] Lustre: server umount lustre-MDT0000 complete
[ 2548.192699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2548.195707] LustreError: Skipped 16 previous similar messages
[ 2556.684628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2558.156104] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2560.939721] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2560.944109] Lustre: Skipped 13 previous similar messages
[ 2562.025049] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 2562.028905] Lustre: Skipped 14 previous similar messages
[ 2562.048024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977)
[ 2562.048024] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945)
[ 2563.759316] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2564.302917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2567.666295] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 00:58:22 (1782363502)
[ 2567.713135] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 2567.715815] Lustre: Skipped 2 previous similar messages
[ 2570.778257] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 00:58:25 (1782363505)
[ 2571.085338] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 2571.086814] LustreError: 99125:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff8cbb90342c50 x1868940954717696/t0(0) o700->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:287/0 lens 264/248 e 0 to 0 dl 1782363517 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295
[ 2588.618196] Lustre: Failing over lustre-MDT0000
[ 2588.880818] Lustre: server umount lustre-MDT0000 complete
[ 2591.596556] LustreError: 12612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2591.603641] LustreError: 12612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 68 previous similar messages
[ 2601.709052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2601.821076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2601.823148] Lustre: Skipped 10 previous similar messages
[ 2603.181730] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2607.091057] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977)
[ 2607.091065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009)
[ 2608.788264] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2609.283726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2613.127799] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:59:08 (1782363548)
[ 2613.966856] Lustre: Failing over lustre-OST0000
[ 2614.027282] Lustre: server umount lustre-OST0000 complete
[ 2626.593443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2628.420207] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2631.929352] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2632.452773] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2697.580320] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 01:00:32 (1782363632)
[ 2700.087988] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2700.814507] Lustre: Failing over lustre-MDT0000
[ 2701.041346] Lustre: server umount lustre-MDT0000 complete
[ 2715.076555] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 2715.077868] LDISKFS-fs (dm-0): recovery complete
[ 2715.081708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2716.549788] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2761.136517] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2761.148545] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 4 previous similar messages
[ 2771.373084] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2791.851806] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2791.856584] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 3 previous similar messages
[ 2832.812146] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2832.815976] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages
[ 2899.500099] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports
[ 2899.502441] Lustre: 127574:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp
[ 2899.505565] Lustre: 127574:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message
[ 2899.508372] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 2899.510078] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 2899.514110] Lustre: 127574:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 13 previous similar messages
[ 2899.517486] Lustre: 127574:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout
[ 2899.520884] Lustre: 127574:0:(ldlm_lib.c:2380:target_recovery_overseer()) Skipped 1 previous similar message
[ 2899.524727] Lustre: 127574:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2899.527625] Lustre: 127574:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2899.533654] Lustre: lustre-MDT0000-osd: cancel update llog [0x20001a210:0x1:0x0]
[ 2899.540705] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ec:0x1:0x0]
[ 2899.550041] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 2899.550190] Lustre: 127574:0:(ldlm_lib.c:2937:target_recovery_thread()) too long recovery - read logs
[ 2899.553068] Lustre: Skipped 21 previous similar messages
[ 2899.555898] LustreError: dumping log to /tmp/lustre-log.1782363834.127574
[ 2899.614162] Lustre: *** cfs_fail_loc=216, val=0***
[ 2899.614178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3029 to 0x280000401:3073)
[ 2899.615584] LustreError: 127562:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 2900.640557] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2997 to 0x2c0000401:3041)
[ 2905.158591] Lustre: DEBUG MARKER: replay-single test_48: @@@@@@ FAIL: client_up failed
[ 2906.958476] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 01:04:01 (1782363841)
[ 2907.615091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2907.623027] Lustre: Skipped 19 previous similar messages
[ 2907.626097] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2907.632054] Lustre: Skipped 2 previous similar messages
[ 2915.677252] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 01:04:10 (1782363850)
[ 2917.060387] Lustre: Failing over lustre-MDT0000
[ 2917.235168] Lustre: server umount lustre-MDT0000 complete
[ 2930.593073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2930.705606] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2930.709695] Lustre: Skipped 7 previous similar messages
[ 2932.044344] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2935.785078] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 2935.787503] LustreError: 129226:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff8cba839ced80 x1868940954808832/t0(0) o101->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:695/0 lens 328/344 e 0 to 0 dl 1782363925 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295
[ 2994.604276] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnected, waiting for 2 clients in recovery for 1:03
[ 2994.640495] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2997 to 0x2c0000401:3073)
[ 2994.640527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3075 to 0x280000401:3105)
[ 2996.625134] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2997.206984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3001.188642] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 01:05:36 (1782363936)
[ 3002.711507] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 3006.116443] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3007.028671] Lustre: Failing over lustre-MDT0000
[ 3007.292920] Lustre: server umount lustre-MDT0000 complete
[ 3021.910875] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3021.913712] LDISKFS-fs (dm-0): recovery complete
[ 3021.919791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3023.859119] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3027.443713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105)
[ 3027.443799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3075 to 0x280000401:3137)
[ 3029.726716] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3030.444423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3034.167276] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 01:06:09 (1782363969)
[ 3034.565553] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3038.957546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3039.796041] Lustre: Failing over lustre-MDT0000
[ 3039.904784] Lustre: server umount lustre-MDT0000 complete
[ 3054.432511] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3054.433728] LDISKFS-fs (dm-0): recovery complete
[ 3054.435892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3055.711911] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3059.698574] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137)
[ 3059.698609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3075 to 0x280000401:3169)
[ 3061.389171] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3061.910901] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3065.702925] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 01:06:40 (1782364000)
[ 3066.045425] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3070.431180] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3071.084647] Lustre: Failing over lustre-MDT0000
[ 3071.189861] Lustre: server umount lustre-MDT0000 complete
[ 3085.826937] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3085.828015] LDISKFS-fs (dm-0): recovery complete
[ 3085.830673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3087.260659] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3091.441737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201)
[ 3091.441748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3169)
[ 3096.250230] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 01:07:11 (1782364031)
[ 3097.773146] Lustre: *** cfs_fail_loc=13b, val=315***
[ 3097.776046] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 3097.779167] LustreError: 6488:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff8cbb8835a050 x1868940954859136/t257698037777(0) o35->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:59/0 lens 392/456 e 0 to 0 dl 1782364044 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0
[ 3099.067587] Lustre: Failing over lustre-MDT0000
[ 3099.325446] Lustre: server umount lustre-MDT0000 complete
[ 3113.198347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3115.055746] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3118.586246] Lustre: 6489:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8cbbbffb5500 x1868940954859136/t257698037777(0) o35->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:79/0 lens 392/456 e 0 to 0 dl 1782364064 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0
[ 3118.595905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3233)
[ 3118.595927] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201)
[ 3121.017598] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3121.753581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3126.055984] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 01:07:40 (1782364060)
[ 3126.562571] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 3126.565407] LustreError: 6485:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff8cbbbb529500 x1868940954872064/t261993005072(0) o36->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:131/0 lens 504/448 e 0 to 0 dl 1782364116 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 3131.386079] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3132.311845] Lustre: Failing over lustre-MDT0000
[ 3132.433029] Lustre: server umount lustre-MDT0000 complete
[ 3146.857488] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3146.859582] LDISKFS-fs (dm-0): recovery complete
[ 3146.863266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3146.899382] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3146.901989] LustreError: Skipped 8 previous similar messages
[ 3148.278644] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3152.376302] Lustre: 119426:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8cbbbff79c00 x1868940954872064/t261993005072(0) o36->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:157/0 lens 504/2880 e 0 to 0 dl 1782364142 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 3152.381371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265)
[ 3152.381509] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233)
[ 3154.430980] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3154.989116] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3158.604609] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 01:08:13 (1782364093)
[ 3159.145671] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 3159.148965] LustreError: 6485:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff8cbbc217fb80 x1868940954885760/t266287972368(0) o36->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:164/0 lens 504/448 e 0 to 0 dl 1782364149 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 3160.580903] Lustre: *** cfs_fail_loc=13b, val=315***
[ 3163.271893] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3163.887712] Lustre: Failing over lustre-MDT0000
[ 3163.977712] Lustre: server umount lustre-MDT0000 complete
[ 3167.712924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3167.717725] LustreError: Skipped 6 previous similar messages
[ 3178.535591] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3178.538281] LDISKFS-fs (dm-0): recovery complete
[ 3178.543414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3180.144100] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3180.459788] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 3180.463540] Lustre: Skipped 9 previous similar messages
[ 3184.112599] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 3184.118836] Lustre: Skipped 9 previous similar messages
[ 3184.125492] Lustre: 119426:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8cba833bf480 x1868940954885760/t266287972368(0) o36->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:189/0 lens 504/2880 e 0 to 0 dl 1782364174 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 3184.144671] Lustre: 119426:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 3184.147730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297)
[ 3184.147731] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265)
[ 3188.570908] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 01:08:43 (1782364123)
[ 3188.903445] LustreError: 10064:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff8cbbbb574a80 x1868940954898688/t270582939664(0) o36->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:194/0 lens 504/448 e 0 to 0 dl 1782364179 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 3188.911584] LustreError: 10064:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message
[ 3190.296478] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 3190.299580] Lustre: Skipped 1 previous similar message
[ 3193.570177] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3194.384091] Lustre: Failing over lustre-MDT0000
[ 3194.514076] Lustre: server umount lustre-MDT0000 complete
[ 3195.820870] LustreError: 10064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3195.826252] LustreError: 10064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 134 previous similar messages
[ 3209.212543] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3209.215198] LDISKFS-fs (dm-0): recovery complete
[ 3209.220567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3209.370977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3209.377636] Lustre: Skipped 9 previous similar messages
[ 3211.046975] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3214.835840] Lustre: 6487:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8cbbbac2f100 x1868940954898688/t270582939664(0) o36->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:220/0 lens 504/2880 e 0 to 0 dl 1782364205 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295
[ 3214.842541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3329)
[ 3214.842564] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297)
[ 3219.065601] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 01:09:13 (1782364153)
[ 3219.589804] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3221.020307] Lustre: *** cfs_fail_loc=13b, val=315***
[ 3221.023242] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 3221.026061] Lustre: Skipped 2 previous similar messages
[ 3221.028818] LustreError: 26726:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff8cbbb3eb2300 x1868940954911104/t274877906960(0) o35->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:182/0 lens 392/456 e 0 to 0 dl 1782364167 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0
[ 3225.242240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3225.942878] Lustre: Failing over lustre-MDT0000
[ 3226.041391] Lustre: server umount lustre-MDT0000 complete
[ 3240.590942] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3240.593249] LDISKFS-fs (dm-0): recovery complete
[ 3240.598192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3242.447023] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3246.072873] Lustre: 6489:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8cbbbb420700 x1868940954911104/t274877906960(0) o35->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:207/0 lens 392/456 e 0 to 0 dl 1782364192 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0
[ 3246.085494] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329)
[ 3246.086379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3361)
[ 3250.705440] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 01:09:45 (1782364185)
[ 3251.032297] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 3311.022951] Lustre: 6485:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8cbbb3d9d880 x1868940954921344/t279172874255(0) o101->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:316/0 lens 664/3488 e 0 to 0 dl 1782364301 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0
[ 3313.663547] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 01:10:48 (1782364248)
[ 3316.816963] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3317.758190] Lustre: Failing over lustre-MDT0000
[ 3317.858621] Lustre: server umount lustre-MDT0000 complete
[ 3333.109722] LDISKFS-fs (dm-0): 4 truncates cleaned up
[ 3333.111168] LDISKFS-fs (dm-0): recovery complete
[ 3333.114102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3334.568793] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3338.764396] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3361)
[ 3338.764531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393)
[ 3340.737151] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3341.325094] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3354.912079] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 01:11:29 (1782364289)
[ 3357.822712] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3358.478665] Lustre: Failing over lustre-MDT0000
[ 3358.626065] Lustre: server umount lustre-MDT0000 complete
[ 3373.170315] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3373.173342] LDISKFS-fs (dm-0): recovery complete
[ 3373.178370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3374.697574] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3378.689556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425)
[ 3378.689557] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3393)
[ 3380.678469] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3381.342871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3383.825187] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 3389.024216] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 01:12:03 (1782364323)
[ 3399.883189] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3400.716526] Lustre: Failing over lustre-MDT0000
[ 3401.007499] Lustre: server umount lustre-MDT0000 complete
[ 3415.910404] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3415.912473] LDISKFS-fs (dm-0): recovery complete
[ 3415.918151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3417.301652] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3421.250311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705)
[ 3421.250314] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673)
[ 3422.812876] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3423.303102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3439.423228] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 01:12:54 (1782364374)
[ 3442.801142] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3443.404827] Lustre: Failing over lustre-MDT0000
[ 3443.607820] Lustre: server umount lustre-MDT0000 complete
[ 3457.714653] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3457.715987] LDISKFS-fs (dm-0): recovery complete
[ 3457.718668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3459.197493] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3463.166279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737)
[ 3463.166296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705)
[ 3464.945606] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3465.469324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3468.681841] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 3469.177626] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 3471.089845] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 01:13:26 (1782364406)
[ 3477.019257] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 3537.344874] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnecting
[ 3537.350916] Lustre: Skipped 2 previous similar messages
[ 3538.324416] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 3538.327358] Lustre: Skipped 1 previous similar message
[ 3538.329680] LustreError: 6487:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply  req@ffff8cbb88fcbc50 x1868940957711104/t296352743435(0) o36->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:543/0 lens 66040/440 e 0 to 0 dl 1782364528 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0
[ 3538.339601] LustreError: 6487:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message
[ 3597.741857] Lustre: 6486:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8cbbbff9a680 x1868940957711104/t296352743435(0) o36->9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp:603/0 lens 66040/440 e 0 to 0 dl 1782364588 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0
[ 3600.778987] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 3601.294306] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 01:15:36 (1782364536)
[ 3605.574205] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3606.305827] Lustre: Failing over lustre-MDT0000
[ 3606.497489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3606.501057] Lustre: Skipped 53 previous similar messages
[ 3606.502606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3606.504148] Lustre: Skipped 7 previous similar messages
[ 3606.571282] Lustre: server umount lustre-MDT0000 complete
[ 3620.186465] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3620.187747] LDISKFS-fs (dm-0): recovery complete
[ 3620.189838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3620.320976] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3620.323381] Lustre: Skipped 12 previous similar messages
[ 3621.476634] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3625.441952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo)
[ 3625.443649] Lustre: Skipped 54 previous similar messages
[ 3625.588616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865)
[ 3625.588638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833)
[ 3627.127400] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3627.594559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3630.999954] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 01:16:06 (1782364566)
[ 3636.032119] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3638.967114] Lustre: Failing over lustre-OST0000
[ 3639.009931] Lustre: server umount lustre-OST0000 complete
[ 3652.686658] LDISKFS-fs (dm-2): 3 truncates cleaned up
[ 3652.687981] LDISKFS-fs (dm-2): recovery complete
[ 3652.690566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3654.415461] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3665.537736] Lustre: Failing over lustre-OST0000
[ 3665.576359] Lustre: server umount lustre-OST0000 complete
[ 3678.049537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3679.804413] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3683.401927] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3683.913605] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3717.563835] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 01:17:32 (1782364652)
[ 3718.381318] Lustre: Failing over lustre-MDT0000
[ 3718.654533] Lustre: server umount lustre-MDT0000 complete
[ 3731.133973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3732.435070] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3736.576110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897)
[ 3736.576170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865)
[ 3743.728747] Lustre: Failing over lustre-MDT0000
[ 3743.903897] Lustre: server umount lustre-MDT0000 complete
[ 3756.523733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3756.559995] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3756.563115] LustreError: Skipped 9 previous similar messages
[ 3757.868608] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3761.650167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929)
[ 3761.650181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897)
[ 3763.216733] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3763.700415] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3767.024280] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 01:18:22 (1782364702)
[ 3778.295884] Lustre: Failing over lustre-OST0000
[ 3778.328047] Lustre: server umount lustre-OST0000 complete
[ 3782.113343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3782.116137] LustreError: Skipped 8 previous similar messages
[ 3791.620667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3792.299496] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 3792.301645] Lustre: Skipped 11 previous similar messages
[ 3792.868485] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 3792.871155] Lustre: Skipped 11 previous similar messages
[ 3793.415060] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3797.284336] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3797.880287] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3802.309802] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 01:18:57 (1782364737)
[ 3802.956098] Lustre: Failing over lustre-MDT0000
[ 3803.308083] Lustre: server umount lustre-MDT0000 complete
[ 3807.201076] LustreError: 47576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3807.208213] LustreError: 47576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 148 previous similar messages
[ 3807.427503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3807.466337] Lustre: *** cfs_fail_loc=605, val=0***
[ 3807.467565] LustreError: 164570:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc093cb60 failed: rc = -95
[ 3807.470449] LustreError: 164570:0:(obd_config.c:845:class_setup()) setup MGS failed (-95)
[ 3807.472286] LustreError: 164570:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95
[ 3807.474112] LustreError: 164570:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered
[ 3807.476015] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 3807.477583] LustreError: 164570:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000
[ 3807.481804] Lustre: server umount lustre-MDT0000 complete
[ 3807.482964] LustreError: 164570:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 3810.225199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3810.335366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3810.338851] Lustre: Skipped 11 previous similar messages
[ 3811.790213] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3814.630265] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 01:19:09 (1782364749)
[ 3815.428812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961)
[ 3815.428933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929)
[ 3818.149613] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3819.099897] Lustre: Failing over lustre-MDT0000
[ 3819.404719] Lustre: server umount lustre-MDT0000 complete
[ 3834.105684] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 3834.106808] LDISKFS-fs (dm-0): recovery complete
[ 3834.109473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3835.553735] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3838.381365] Lustre: *** cfs_fail_loc=707, val=0***
[ 3897.778157] Lustre: lustre-MDT0000: Client 9661445f-bd76-42fd-9522-a80d974f5132 (at 192.168.204.40@tcp) reconnected, waiting for 2 clients in recovery for 0:55
[ 3897.788600] Lustre: 166737:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 3897.791416] Lustre: 166737:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 2 previous similar messages
[ 3918.252170] Lustre: 166737:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 3918.255087] Lustre: 166737:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 4 previous similar messages
[ 3959.212059] Lustre: 166737:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1
[ 3959.216826] Lustre: 166737:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages
[ 4018.500161] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports
[ 4018.503339] Lustre: 166737:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9661445f-bd76-42fd-9522-a80d974f5132@192.168.204.40@tcp
[ 4018.510515] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 4018.514332] Lustre: 166737:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout
[ 4018.519340] Lustre: 166737:0:(ldlm_lib.c:2380:target_recovery_overseer()) Skipped 1 previous similar message
[ 4018.523026] Lustre: 166737:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 4018.527980] Lustre: 166737:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 1 previous similar message
[ 4018.532547] LustreError: 166737:0:(ldlm_lib.c:1920:abort_lock_replay_queue()) @@@ aborted:  req@ffff8cba8510b800 x1868940964538112/t0(0) o101->lustre-MDT0001-mdtlov_UUID@0@lo:241/0 lens 328/0 e 8 to 0 dl 1782364981 ref 1 fl Complete:/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295
[ 4018.544528] Lustre: lustre-MDT0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 2 known clients (0 recovered, 0 in progress, and 2 evicted) already passed deadline 0:00
[ 4018.545582] Lustre: lustre-MDT0000-osd: cancel update llog [0x20001b980:0x1:0x0]
[ 4018.561109] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ed:0x1:0x0]
[ 4018.574118] Lustre: 166737:0:(ldlm_lib.c:2937:target_recovery_thread()) too long recovery - read logs
[ 4018.579809] LustreError: dumping log to /tmp/lustre-log.1782364953.166737
[ 4018.652681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4968 to 0x280000401:4993)
[ 4018.652681] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4936 to 0x2c0000401:4961)
[ 4022.734833] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4023.277472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4023.778579] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4025.796102] Lustre: DEBUG MARKER: replay-single test_62: @@@@@@ FAIL: unlinkmany /mnt/lustre/d62.replay-single/f62.replay-single failed