[ 1430.752194] Lustre: Failing over lustre-MDT0000
[ 1430.829564] Lustre: server umount lustre-MDT0000 complete
[ 1431.812829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1431.816931] Lustre: Skipped 82 previous similar messages
[ 1433.884060] LDISKFS-fs (dm-0): recovery complete
[ 1433.885930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1433.970716] Lustre: lustre-MDT0000: Aborting client recovery
[ 1433.972454] LustreError: 31929:0:(ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 1433.975007] Lustre: 31958:0:(ldlm_lib.c:2309:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 1433.978033] Lustre: 31958:0:(ldlm_lib.c:2309:target_recovery_overseer()) Skipped 2 previous similar messages
[ 1433.980803] Lustre: 31958:0:(genops.c:1506:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@<unknown>
[ 1433.983741] Lustre: 31958:0:(genops.c:1506:class_disconnect_stale_exports()) Skipped 1 previous similar message
[ 1433.986358] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 1433.989379] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 1433.994695] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0]
[ 1434.012936] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1633)
[ 1434.012938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633)
[ 1434.741003] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1438.967054] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1446.616996] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 17:13:59 (1743455639)
[ 1453.002325] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1453.511782] Lustre: Failing over lustre-MDT0000
[ 1453.592847] Lustre: server umount lustre-MDT0000 complete
[ 1466.501893] LDISKFS-fs (dm-0): recovery complete
[ 1466.503183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1466.528699] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1466.530839] LustreError: Skipped 20 previous similar messages
[ 1467.348005] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1467.976222] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1467.980305] Lustre: Skipped 16 previous similar messages
[ 1471.634469] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 1471.638319] Lustre: Skipped 16 previous similar messages
[ 1471.651559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049)
[ 1471.651561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049)
[ 1472.398921] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1472.753473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1479.200935] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 17:14:32 (1743455672)
[ 1483.905677] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1485.915704] Lustre: Failing over lustre-MDT0000
[ 1485.997065] Lustre: server umount lustre-MDT0000 complete
[ 1498.811446] LDISKFS-fs (dm-0): recovery complete
[ 1498.813229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1499.661833] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1504.398178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465)
[ 1504.398196] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465)
[ 1505.147703] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1505.528921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1511.980375] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 17:15:05 (1743455705)
[ 1512.490245] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 1512.742827] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1512.745432] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 1514.153603] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 17:15:07 (1743455707)
[ 1519.012395] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 1521.298081] Lustre: Failing over lustre-OST0000
[ 1521.326920] Lustre: server umount lustre-OST0000 complete
[ 1522.408315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107
[ 1534.212903] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 1534.291765] LDISKFS-fs (dm-2): recovery complete
[ 1534.293486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 1534.314549] Lustre: 8686:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.142@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 1535.563156] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1579.864411] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 17:16:13 (1743455773)
[ 1581.933897] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1582.636012] Lustre: Failing over lustre-MDT0000
[ 1582.711950] Lustre: server umount lustre-MDT0000 complete
[ 1595.578283] LDISKFS-fs (dm-0): recovery complete
[ 1595.580065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1596.418844] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1600.683145] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 1600.684947] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881)
[ 1601.429745] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1601.788269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1614.422140] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 17:16:47 (1743455807)
[ 1615.700652] LustreError: 24844:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_race id 701 sleeping
[ 1616.682502] Lustre: 11324:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743455794/real 1743455794]  req@ffff88008b2c6300 x1828144258143360/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1743455810 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0
[ 1616.692698] LustreError: 11324:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 1616.693248] LustreError: 9553:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_fail_race id 701 waking
[ 1616.693278] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1616.693297] LustreError: 24844:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_fail_race id 701 awake: rc=4011
[ 1617.170004] LustreError: 24844:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_race id 701 sleeping
[ 1617.700827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913)
[ 1622.172503] LustreError: 24844:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1622.174934] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnecting
[ 1622.177200] Lustre: Skipped 1 previous similar message
[ 1622.637102] LustreError: 7208:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_race id 701 sleeping
[ 1627.640445] LustreError: 7208:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1627.643764] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnecting
[ 1628.123187] LustreError: 7208:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_race id 701 sleeping
[ 1633.125435] LustreError: 7208:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1633.128003] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnecting
[ 1633.587664] LustreError: 7208:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_race id 701 sleeping
[ 1638.589435] LustreError: 7208:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1644.045488] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnecting
[ 1644.048230] Lustre: Skipped 1 previous similar message
[ 1644.506761] LustreError: 7209:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_race id 701 sleeping
[ 1644.509234] LustreError: 7209:0:(ldlm_lib.c:1089:target_handle_connect()) Skipped 1 previous similar message
[ 1649.511479] LustreError: 7209:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1649.513599] LustreError: 7209:0:(ldlm_lib.c:1089:target_handle_connect()) Skipped 1 previous similar message
[ 1660.447546] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnecting
[ 1660.450396] Lustre: Skipped 2 previous similar messages
[ 1660.932254] LustreError: 7209:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_race id 701 sleeping
[ 1660.934289] LustreError: 7209:0:(ldlm_lib.c:1089:target_handle_connect()) Skipped 2 previous similar messages
[ 1665.936437] LustreError: 7209:0:(ldlm_lib.c:1089:target_handle_connect()) cfs_fail_race id 701 awake: rc=0
[ 1665.939002] LustreError: 7209:0:(ldlm_lib.c:1089:target_handle_connect()) Skipped 2 previous similar messages
[ 1667.875185] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 17:17:41 (1743455861)
[ 1668.356094] LustreError: 7207:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1678.356437] Lustre: lustre-MDT0000: Export ffff88012b779800 already connecting from 192.168.202.42@tcp
[ 1683.368178] Lustre: lustre-MDT0000: Export ffff88012b779800 already connecting from 192.168.202.42@tcp
[ 1688.376285] Lustre: lustre-MDT0000: Export ffff88012b779800 already connecting from 192.168.202.42@tcp
[ 1693.384145] Lustre: lustre-MDT0000: Export ffff88012b779800 already connecting from 192.168.202.42@tcp
[ 1698.392371] Lustre: lustre-MDT0000: Export ffff88012b779800 already connecting from 192.168.202.42@tcp
[ 1708.358479] LustreError: 7207:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1708.360909] Lustre: 7207:0:(service.c:2362:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff88012e493800 x1828144254797696/t0(0) o38->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1743455881 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0
[ 1708.408455] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnecting
[ 1708.411251] Lustre: Skipped 2 previous similar messages
[ 1708.881208] LustreError: 7208:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1718.881411] Lustre: lustre-MDT0000: Export ffff88012b779800 already connecting from 192.168.202.42@tcp
[ 1718.884042] Lustre: Skipped 1 previous similar message
[ 1738.920512] Lustre: lustre-MDT0000: Export ffff88012b779800 already connecting from 192.168.202.42@tcp
[ 1738.922588] Lustre: Skipped 4 previous similar messages
[ 1748.886483] LustreError: 7208:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1748.889083] Lustre: 7208:0:(service.c:2362:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff880078575f80 x1828144254803072/t0(0) o38->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1743455922 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0
[ 1748.936538] LustreError: 24844:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1773.936486] Lustre: lustre-MDT0000: Export ffff88012b779800 already connecting from 192.168.202.42@tcp
[ 1773.939490] Lustre: Skipped 1 previous similar message
[ 1788.938496] LustreError: 24844:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1788.940852] Lustre: 24844:0:(service.c:2362:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff880139bf2a00 x1828144254806400/t0(0) o38->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1743455962 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0
[ 1788.968469] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnecting
[ 1788.971816] Lustre: Skipped 2 previous similar messages
[ 1788.973696] LustreError: 8753:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1828.976480] LustreError: 8753:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1828.981142] Lustre: 8753:0:(service.c:2362:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff880084f1c700 x1828144254809472/t0(0) o38->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1743456002 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0
[ 1829.000876] LustreError: 11795:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1854.000461] Lustre: lustre-MDT0000: Export ffff88012b779800 already connecting from 192.168.202.42@tcp
[ 1854.002646] Lustre: Skipped 5 previous similar messages
[ 1869.100452] LustreError: 11795:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 awake
[ 1869.102965] Lustre: 11795:0:(service.c:2362:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff88006e054e00 x1828144254812544/t0(0) o38->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:0/0 lens 520/416 e 0 to 0 dl 1743456042 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0
[ 1873.065473] LustreError: 7207:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1893.468508] LustreError: 7207:0:(ldlm_lib.c:1342:target_handle_connect()) cfs_fail_timeout interrupted
[ 1894.474810] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 17:21:27 (1743456087)
[ 1896.519567] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1897.476933] Lustre: Failing over lustre-MDT0000
[ 1897.546637] Lustre: server umount lustre-MDT0000 complete
[ 1900.994882] LDISKFS-fs (dm-0): recovery complete
[ 1900.996315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1901.064633] Lustre: *** cfs_fail_loc=712, val=0***
[ 1901.066069] LustreError: 20985:0:(service.c:1231:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@ffff88008cfbbb80 x1828144258292736/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0
[ 1901.080356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1901.082507] Lustre: Skipped 11 previous similar messages
[ 1901.089499] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1901.089562] Lustre: lustre-MDT0000: Aborting client recovery
[ 1901.089567] LustreError: 17443:0:(ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 1901.094860] Lustre: Skipped 20 previous similar messages
[ 1901.096256] Lustre: 17472:0:(ldlm_lib.c:2309:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 1901.098533] Lustre: 17472:0:(ldlm_lib.c:2309:target_recovery_overseer()) Skipped 2 previous similar messages
[ 1901.100875] Lustre: 17472:0:(genops.c:1506:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@<unknown>
[ 1901.103947] Lustre: 17472:0:(genops.c:1506:class_disconnect_stale_exports()) Skipped 1 previous similar message
[ 1901.105968] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 1901.108439] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0]
[ 1901.113265] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0]
[ 1901.130970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945)
[ 1901.130971] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913)
[ 1901.982126] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1906.090667] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1911.792848] Lustre: Failing over lustre-MDT0000
[ 1911.906046] Lustre: server umount lustre-MDT0000 complete
[ 1924.091636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1924.970386] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1929.221388] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945)
[ 1929.221543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977)
[ 1930.208729] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1930.779926] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1934.073324] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 17:22:07 (1743456127)
[ 1934.095730] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnecting
[ 1934.098241] Lustre: Skipped 2 previous similar messages
[ 1936.559215] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 17:22:09 (1743456129)
[ 1936.803870] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 1936.805629] LustreError: 7216:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff88012362ca80 x1828144254869760/t0(0) o700->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:326/0 lens 264/248 e 0 to 0 dl 1743456141 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0
[ 1953.531025] Lustre: Failing over lustre-MDT0000
[ 1953.599177] Lustre: server umount lustre-MDT0000 complete
[ 1953.817209] LustreError: 11795:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1953.824370] LustreError: 11795:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 96 previous similar messages
[ 1965.545613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1966.359268] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1970.665036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009)
[ 1970.668631] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977)
[ 1971.423607] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1971.793795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1975.034932] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 17:22:48 (1743456168)
[ 1975.647293] Lustre: Failing over lustre-OST0000
[ 1975.652958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping)
[ 1975.654702] Lustre: Skipped 1 previous similar message
[ 1975.683208] Lustre: server umount lustre-OST0000 complete
[ 1988.320324] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 1988.326544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 1988.361874] Lustre: 22222:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.142@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 1989.988120] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1992.812235] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 1993.326359] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2058.194228] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 17:24:11 (1743456251)
[ 2060.842380] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2061.656901] Lustre: Failing over lustre-MDT0000
[ 2061.755802] Lustre: server umount lustre-MDT0000 complete
[ 2063.524879] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2063.531386] Lustre: Skipped 32 previous similar messages
[ 2075.927264] LDISKFS-fs (dm-0): recovery complete
[ 2075.930011] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2075.984076] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2075.992471] LustreError: Skipped 5 previous similar messages
[ 2077.227723] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2079.016913] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2079.020484] Lustre: Skipped 6 previous similar messages
[ 2081.077894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to  (at 0@lo)
[ 2081.080830] Lustre: Skipped 37 previous similar messages
[ 2081.138089] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 2081.140581] Lustre: Skipped 6 previous similar messages
[ 2081.155187] Lustre: *** cfs_fail_loc=216, val=0***
[ 2081.155197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041)
[ 2081.161381] LustreError: 24786:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 2082.163928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073)
[ 2142.699922] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 17:25:35 (1743456335)
[ 2149.676763] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 17:25:42 (1743456342)
[ 2150.384745] Lustre: Failing over lustre-MDT0000
[ 2150.455656] Lustre: server umount lustre-MDT0000 complete
[ 2162.961020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2164.185515] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2168.090997] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 2168.093764] LustreError: 26778:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff88012b21c380 x1828144254989440/t0(0) o101->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:597/0 lens 328/344 e 0 to 0 dl 1743456412 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0
[ 2224.162917] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnected, waiting for 2 clients in recovery for 0:54
[ 2224.186159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105)
[ 2224.186160] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073)
[ 2225.098730] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2225.563615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2229.234619] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 17:27:02 (1743456422)
[ 2230.572139] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2232.823454] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2233.545291] Lustre: Failing over lustre-MDT0000
[ 2233.651235] Lustre: server umount lustre-MDT0000 complete
[ 2234.197031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2247.519899] LDISKFS-fs (dm-0): recovery complete
[ 2247.522565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2248.808155] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2252.664101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137)
[ 2252.664106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105)
[ 2253.679025] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2254.181881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2257.134672] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 17:27:30 (1743456450)
[ 2257.406800] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2260.949985] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2261.447014] Lustre: Failing over lustre-MDT0000
[ 2261.518133] Lustre: server umount lustre-MDT0000 complete
[ 2262.660906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2274.962768] LDISKFS-fs (dm-0): recovery complete
[ 2274.964121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2276.052488] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2280.088105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169)
[ 2280.088121] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3137)
[ 2281.010901] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2281.434465] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2284.327894] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 17:27:57 (1743456477)
[ 2284.651752] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2288.144507] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2288.859804] Lustre: Failing over lustre-MDT0000
[ 2288.951289] Lustre: server umount lustre-MDT0000 complete
[ 2302.690326] LDISKFS-fs (dm-0): recovery complete
[ 2302.691608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2303.799402] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2307.813512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201)
[ 2307.820545] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3169)
[ 2312.264766] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 17:28:25 (1743456505)
[ 2313.681471] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2313.685504] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2313.689135] LustreError: 7211:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff880094ba6680 x1828144255040128/t257698037777(0) o35->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:703/0 lens 392/456 e 0 to 0 dl 1743456518 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2314.703136] Lustre: Failing over lustre-MDT0000
[ 2314.810110] Lustre: server umount lustre-MDT0000 complete
[ 2327.131719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2328.121257] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2332.237409] Lustre: 7211:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8800a878dc00 x1828144255040128/t257698037777(0) o35->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:721/0 lens 392/456 e 0 to 0 dl 1743456536 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2332.248179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233)
[ 2332.248192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3201)
[ 2332.971196] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2333.350255] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2336.097745] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 17:28:49 (1743456529)
[ 2336.401642] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2336.403451] LustreError: 8753:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff88006ed66300 x1828144255053184/t261993005072(0) o36->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:14/0 lens 504/448 e 0 to 0 dl 1743456584 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2339.953982] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2340.595762] Lustre: Failing over lustre-MDT0000
[ 2340.680776] Lustre: server umount lustre-MDT0000 complete
[ 2342.244982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2353.652735] LDISKFS-fs (dm-0): recovery complete
[ 2353.653983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2354.582995] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2358.755432] Lustre: 29372:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff880139bf0700 x1828144255053184/t261993005072(0) o36->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:37/0 lens 504/2880 e 0 to 0 dl 1743456607 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2358.769304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265)
[ 2358.769323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3233)
[ 2359.826696] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2360.355632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2363.297168] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 17:29:16 (1743456556)
[ 2363.632834] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2363.636048] LustreError: 29372:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff88012e493800 x1828144255065728/t266287972368(0) o36->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:42/0 lens 504/448 e 0 to 0 dl 1743456612 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2364.961590] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2367.315697] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2367.944712] Lustre: Failing over lustre-MDT0000
[ 2368.032221] Lustre: server umount lustre-MDT0000 complete
[ 2381.261745] LDISKFS-fs (dm-0): recovery complete
[ 2381.264318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2382.404906] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2386.386628] Lustre: 8753:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8801366f4380 x1828144255065728/t266287972368(0) o36->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:64/0 lens 504/2880 e 0 to 0 dl 1743456634 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2386.398328] Lustre: 8753:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 2386.403677] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265)
[ 2386.403683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297)
[ 2390.929094] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 17:29:44 (1743456584)
[ 2391.237938] LustreError: 8753:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a7e0f480 x1828144255078144/t270582939664(0) o36->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:69/0 lens 504/448 e 0 to 0 dl 1743456639 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2391.248039] LustreError: 8753:0:(ldlm_lib.c:3271:target_send_reply_msg()) Skipped 1 previous similar message
[ 2392.571645] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2392.574023] Lustre: Skipped 1 previous similar message
[ 2395.278983] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2395.816457] Lustre: Failing over lustre-MDT0000
[ 2395.932507] Lustre: server umount lustre-MDT0000 complete
[ 2409.200992] LDISKFS-fs (dm-0): recovery complete
[ 2409.202835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2410.318113] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2414.318695] Lustre: 29372:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88009fbf3b80 x1828144255078144/t270582939664(0) o36->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:92/0 lens 504/2880 e 0 to 0 dl 1743456662 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0
[ 2414.330038] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297)
[ 2414.330076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3329)
[ 2418.358005] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 17:30:11 (1743456611)
[ 2418.646500] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2419.925133] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2419.927170] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2419.928267] Lustre: Skipped 2 previous similar messages
[ 2423.408858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2423.908241] Lustre: Failing over lustre-MDT0000
[ 2423.982115] Lustre: server umount lustre-MDT0000 complete
[ 2424.324842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2437.543978] LDISKFS-fs (dm-0): recovery complete
[ 2437.546462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2438.340955] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2442.659199] Lustre: 7211:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff880130710380 x1828144255090304/t274877906960(0) o35->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:77/0 lens 392/456 e 0 to 0 dl 1743456647 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0
[ 2442.676228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361)
[ 2442.676249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329)
[ 2446.803660] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 17:30:39 (1743456639)
[ 2447.136904] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 2447.139586] LustreError: 11795:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a5817100 x1828144255100160/t279172874255(0) o101->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:125/0 lens 664/608 e 0 to 0 dl 1743456695 ref 1 fl Interpret:/200/0 rc 301/0 job:'touch.0' uid:0 gid:0
[ 2447.152254] LustreError: 11795:0:(ldlm_lib.c:3271:target_send_reply_msg()) Skipped 1 previous similar message
[ 2507.159079] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnecting
[ 2507.164708] Lustre: Skipped 4 previous similar messages
[ 2507.169732] Lustre: 7209:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88006e661f80 x1828144255100160/t279172874255(0) o101->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:185/0 lens 664/3488 e 0 to 0 dl 1743456755 ref 1 fl Interpret:/202/0 rc 0/0 job:'touch.0' uid:0 gid:0
[ 2509.493711] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 17:31:42 (1743456702)
[ 2511.907257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2512.433028] Lustre: Failing over lustre-MDT0000
[ 2512.508637] Lustre: server umount lustre-MDT0000 complete
[ 2525.804368] LDISKFS-fs (dm-0): recovery complete
[ 2525.807260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2525.916403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2525.920234] Lustre: Skipped 13 previous similar messages
[ 2525.929424] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2525.933301] Lustre: Skipped 15 previous similar messages
[ 2526.956391] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2530.971199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393)
[ 2530.973713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3361)
[ 2531.994963] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2532.573226] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2546.007166] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 17:32:19 (1743456739)
[ 2548.787776] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2549.424167] Lustre: Failing over lustre-MDT0000
[ 2549.514165] Lustre: server umount lustre-MDT0000 complete
[ 2555.973003] LustreError: 11417:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2555.981730] LustreError: 11417:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 181 previous similar messages
[ 2563.432744] LDISKFS-fs (dm-0): recovery complete
[ 2563.435737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2564.666882] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2568.603505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3425)
[ 2568.603526] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393)
[ 2569.767705] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2570.295466] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2572.274923] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2577.650925] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 17:32:50 (1743456770)
[ 2589.812070] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2590.553261] Lustre: Failing over lustre-MDT0000
[ 2590.688062] Lustre: server umount lustre-MDT0000 complete
[ 2604.609333] LDISKFS-fs (dm-0): recovery complete
[ 2604.612175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2605.728683] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2609.799761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705)
[ 2609.799764] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673)
[ 2610.638731] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2611.140968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2627.737051] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 17:33:40 (1743456820)
[ 2631.116246] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2631.873094] Lustre: Failing over lustre-MDT0000
[ 2631.981924] Lustre: server umount lustre-MDT0000 complete
[ 2646.281390] LDISKFS-fs (dm-0): recovery complete
[ 2646.283992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2647.556092] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2651.453832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737)
[ 2651.456662] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705)
[ 2652.464432] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2653.002054] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2655.873074] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 2656.414166] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 2658.766560] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 17:34:11 (1743456851)
[ 2664.741451] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 2725.238320] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2725.240390] Lustre: Skipped 1 previous similar message
[ 2725.242796] LustreError: 7208:0:(ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply  req@ffff88012e1ff480 x1828144257890816/t296352743434(0) o36->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:403/0 lens 66040/440 e 0 to 0 dl 1743456973 ref 1 fl Interpret:/200/0 rc 0/0 job:'setfattr.0' uid:0 gid:0
[ 2785.252114] Lustre: 7207:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88012e216c50 x1828144257890816/t296352743434(0) o36->8ccb3a61-1507-4888-8ab7-179c8ff271ea@192.168.202.42@tcp:463/0 lens 66040/440 e 0 to 0 dl 1743457033 ref 1 fl Interpret:/202/0 rc 0/0 job:'setfattr.0' uid:0 gid:0
[ 2787.947205] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 2788.318713] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 17:36:21 (1743456981)
[ 2792.053973] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2792.668203] Lustre: Failing over lustre-MDT0000
[ 2792.762135] Lustre: server umount lustre-MDT0000 complete
[ 2796.644831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2796.648015] Lustre: Skipped 55 previous similar messages
[ 2805.775760] LDISKFS-fs (dm-0): recovery complete
[ 2805.777448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2805.802798] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2805.805149] LustreError: Skipped 13 previous similar messages
[ 2806.831877] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2810.296199] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2810.298454] Lustre: Skipped 13 previous similar messages
[ 2810.869684] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2810.871396] Lustre: Skipped 57 previous similar messages
[ 2810.993629] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 2810.995824] Lustre: Skipped 13 previous similar messages
[ 2811.008899] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833)
[ 2811.012133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865)
[ 2811.802040] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2812.162239] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2815.036198] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 17:36:48 (1743457008)
[ 2819.236442] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 2821.520757] Lustre: Failing over lustre-OST0000
[ 2821.546782] Lustre: server umount lustre-OST0000 complete
[ 2825.400397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107
[ 2825.402366] LustreError: Skipped 3 previous similar messages
[ 2834.257246] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2834.291083] LDISKFS-fs (dm-2): recovery complete
[ 2834.292579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2834.308634] Lustre: 29979:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.142@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 2835.443275] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2846.385075] Lustre: Failing over lustre-OST0000
[ 2846.414650] Lustre: server umount lustre-OST0000 complete
[ 2858.529451] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2858.533216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2858.552701] Lustre: 31176:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.142@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 2859.780078] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2862.446029] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2862.819379] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2895.986114] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 17:38:09 (1743457089)
[ 2896.677754] Lustre: Failing over lustre-MDT0000
[ 2896.743295] Lustre: server umount lustre-MDT0000 complete
[ 2909.175703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2910.188604] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2914.298981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865)
[ 2914.300506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897)
[ 2921.243782] Lustre: Failing over lustre-MDT0000
[ 2921.318992] Lustre: server umount lustre-MDT0000 complete
[ 2933.237576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2934.040646] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2938.345973] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897)
[ 2938.346018] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929)
[ 2939.115475] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2939.527197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2942.220586] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 17:38:55 (1743457135)
[ 2953.064762] Lustre: Failing over lustre-OST0000
[ 2953.091028] Lustre: server umount lustre-OST0000 complete
[ 2965.533482] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2965.540208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2965.574209] Lustre: 3247:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.142@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 2966.887673] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2969.675589] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2970.221471] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2973.519121] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 17:39:26 (1743457166)
[ 2974.085768] Lustre: Failing over lustre-MDT0000
[ 2974.180874] Lustre: server umount lustre-MDT0000 complete
[ 2977.089000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2977.128783] Lustre: *** cfs_fail_loc=605, val=0***
[ 2977.131248] LustreError: 5105:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffa05c23d0 failed: rc = -95
[ 2977.136371] LustreError: 5105:0:(obd_config.c:782:class_setup()) setup MGS failed (-95)
[ 2977.139848] LustreError: 5105:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95
[ 2977.143404] LustreError: 5105:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered
[ 2977.147054] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 2977.150419] LustreError: 5105:0:(tgt_mount.c:1758:server_put_super()) no obd lustre-MDT0000
[ 2977.155692] Lustre: server umount lustre-MDT0000 complete
[ 2977.157439] LustreError: 5105:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 2979.584311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2980.529265] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2982.570551] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 17:39:35 (1743457175)
[ 2984.716347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929)
[ 2984.716370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961)
[ 2987.261590] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2988.254932] Lustre: Failing over lustre-MDT0000
[ 2988.337748] Lustre: server umount lustre-MDT0000 complete
[ 3001.666163] LDISKFS-fs (dm-0): recovery complete
[ 3001.668085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3002.535741] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3005.609176] Lustre: *** cfs_fail_loc=707, val=0***
[ 3065.641776] Lustre: lustre-MDT0000: Client 8ccb3a61-1507-4888-8ab7-179c8ff271ea (at 192.168.202.42@tcp) reconnected, waiting for 2 clients in recovery for 0:54
[ 3065.747220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993)
[ 3065.747222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961)
[ 3066.650030] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3067.130341] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3070.103726] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 17:41:03 (1743457263)
[ 3092.740693] LustreError: 10091:0:(service.c:2341:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms
[ 3093.442244] Lustre: DEBUG MARKER: replay-single test_65a: @@@@@@ FAIL: No early reply