[ 1477.269114] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1477.316853] Lustre: Failing over lustre-MDT0000 [ 1480.324004] LDISKFS-fs (dm-0): recovery complete [ 1480.326212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.353996] Lustre: server umount lustre-MDT0000 complete [ 1480.401379] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.401398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.401408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.412748] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.427848] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.432480] Lustre: lustre-MDT0000: Aborting client recovery [ 1480.433912] LustreError: 28198:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1480.437545] Lustre: 28228:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1480.440817] Lustre: 28228:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1480.444513] Lustre: 28228:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9f0681d6-1720-457c-8da4-97bb22560eb5@ [ 1480.448540] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1480.449785] Lustre: 28228:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1481.190289] Lustre: 28228:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1485.424570] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1491.562936] LustreError: 167-0: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1497.753253] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:27:59 (1717558079) [ 1498.232456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1498.290845] Lustre: Failing over lustre-MDT0000 [ 1500.446223] Lustre: server umount lustre-MDT0000 complete [ 1500.446248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.446300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.446323] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.446397] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.446400] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.446402] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.465632] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.172333] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.454254] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.454319] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.454323] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.458569] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.177789] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.462213] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.462245] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.462248] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.462294] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.214012] LDISKFS-fs (dm-0): recovery complete [ 1511.215367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.240693] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.032101] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1512.185345] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1516.861430] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1517.224135] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1523.669206] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1528.414024] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:28:31 (1717558111) [ 1530.386285] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1530.439707] Lustre: Failing over lustre-MDT0000 [ 1531.342100] Lustre: server umount lustre-MDT0000 complete [ 1531.342143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.342175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.342181] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.342247] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.342255] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.342258] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.367870] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.221550] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.350152] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.350188] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.350217] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.350279] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.225711] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.358157] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.358195] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.358251] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.358257] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.233200] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1543.342559] LDISKFS-fs (dm-0): recovery complete [ 1543.344117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.369280] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.140924] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1547.241626] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1549.407690] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1549.773136] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1556.289408] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1556.776770] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 23:29:03 (1717558143) [ 1557.021615] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1557.023912] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.028498] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1558.401900] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1563.306686] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 23:29:05 (1717558145) [ 1565.668040] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1565.687008] Lustre: Failing over lustre-OST0000 [ 1567.057450] Lustre: server umount lustre-OST0000 complete [ 1567.059696] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1567.064627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1567.282833] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.478246] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.478327] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.483196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.281861] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1573.486215] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1573.486216] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1577.289604] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.494320] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.494323] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.738172] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.742614] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1578.821992] LDISKFS-fs (dm-2): recovery complete [ 1578.823542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1580.000256] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1580.090784] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1623.916649] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 23:30:11 (1717558211) [ 1626.090926] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1626.841102] Lustre: Failing over lustre-MDT0000 [ 1626.888850] Lustre: server umount lustre-MDT0000 complete [ 1627.375298] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.942520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.942522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.942621] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.942658] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.942674] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.942846] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.988908] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1632.378215] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.950330] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.950457] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.950499] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.950512] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.385768] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.958198] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.958200] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.958205] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.958276] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.084571] LDISKFS-fs (dm-0): recovery complete [ 1640.086053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1640.111183] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.846147] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1642.394139] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1645.702739] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1646.057633] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1658.581559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1659.761873] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 23:30:46 (1717558246) [ 1661.201965] LustreError: 11320:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1661.205146] Lustre: 7395:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558233/real 1717558233] req@ffff88007ccf4700 x1800988648117440/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1717558249 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre [ 1661.213829] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.214019] LustreError: 7395:0:(/home/green/git/lustre-release/lustre/osp/osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 1661.214037] LustreError: 9984:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:210:cfs_race_loc()) cfs_fail_race id 701 waking [ 1661.214082] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1661.214090] LustreError: 11320:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=3551 [ 1661.653355] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1666.654887] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1666.658497] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1667.125029] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1672.126932] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1672.130420] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1672.580833] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1677.582908] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1677.586155] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1678.062396] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1683.064933] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1683.068140] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1683.535127] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1688.538944] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1688.544042] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1689.014867] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1694.017943] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1694.021044] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1694.506209] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1699.509914] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1699.513684] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1699.973866] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1704.977972] LustreError: 7122:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1704.980839] LustreError: 7122:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1705.447528] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1710.450940] LustreError: 7122:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1710.454946] LustreError: 7122:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1712.457252] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1712.947676] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 23:31:39 (1717558299) [ 1712.951826] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1722.947764] LustreError: 11320:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1727.961647] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1732.969928] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1737.977703] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1742.985717] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1747.993869] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1752.954947] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1752.958126] LustreError: 11320:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1753.001743] Lustre: 11320:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff88008f0d2680 x1800988643812544/t0(0) o38->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558321 ref 1 fl Complete:H/200/0 rc 0/0 job:'lc [ 1753.481226] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1753.484115] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1763.481689] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1768.489909] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1773.498155] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1774.060412] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1778.505908] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1783.514312] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1788.522065] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1793.485911] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1793.490672] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1793.530092] Lustre: 7120:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff880136a7a680 x1800988643814976/t0(0) o38->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558361 ref 1 fl Complete:H/200/0 rc 0/0 job:'lc [ 1793.546148] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1818.530492] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1823.546131] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1828.554058] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1833.552879] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1833.555075] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1833.561392] Lustre: 7120:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff88008b598380 x1800988643816576/t0(0) o38->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558401 ref 1 fl Complete:H/200/0 rc 0/0 job:'kw [ 1833.568439] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1858.562460] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1863.578373] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1868.586501] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1873.569936] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1873.574732] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1873.593954] Lustre: 7120:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8800840e7b80 x1800988643818112/t0(0) o38->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558441 ref 1 fl Complete:H/200/0 rc 0/0 job:'kw [ 1873.610740] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1898.593976] LustreError: 11441:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1903.609753] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1908.617656] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1913.616014] Lustre: lustre-MDT0000: Export ffff880082896000 already connecting from 192.168.202.55@tcp [ 1913.621145] LustreError: 11441:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1913.625920] Lustre: 11441:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff88009ac06d80 x1800988643819648/t0(0) o38->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558481 ref 1 fl Complete:H/200/0 rc 0/0 job:'kw [ 1913.625927] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1938.624935] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1938.632367] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:148:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 1938.632618] Lustre: 7120:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/5s); client may timeout req@ffff88009869ad80 x1800988643821184/t0(0) o38->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558521 ref 1 fl Complete:H/200/0 rc 0/0 job:'kwo [ 1940.291315] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1943.059276] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 23:35:27 (1717558527) [ 1944.495671] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1944.574715] Lustre: Failing over lustre-MDT0000 [ 1945.678314] Lustre: server umount lustre-MDT0000 complete [ 1945.682015] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1945.688864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.670326] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.670502] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.670534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.670549] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.670740] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.670762] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.670766] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.654902] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1949.108648] LDISKFS-fs (dm-0): recovery complete [ 1949.111194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.155740] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1949.227447] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1949.244129] LustreError: 18666:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:1236:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff88007ba70380 x1800988648190528/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 [ 1949.253092] Lustre: lustre-MDT0000: Aborting client recovery [ 1949.254935] LustreError: 13483:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1949.260607] Lustre: 13512:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1949.265913] Lustre: 13512:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1949.271641] Lustre: 13512:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9f0681d6-1720-457c-8da4-97bb22560eb5@ [ 1949.277718] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1949.279541] Lustre: 13512:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1950.413516] Lustre: 13512:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1954.238509] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1961.992156] LustreError: 167-0: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1962.090125] Lustre: Failing over lustre-MDT0000 [ 1963.682945] Lustre: server umount lustre-MDT0000 complete [ 1964.254518] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.254520] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.254578] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.254656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.254774] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.294443] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.294688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.310770] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1968.682257] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.262434] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.262514] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.262530] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.262545] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.689993] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.271321] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.271333] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.271338] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.271382] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.866507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.896378] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.979407] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.698074] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1980.748793] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1981.312042] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1984.590211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1984.613130] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 23:36:12 (1717558572) [ 1984.646195] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1987.108305] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 1987.343751] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 23:36:14 (1717558574) [ 2003.353779] LustreError: 21409:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff880099a0f480 x1800988643848512/t0(0) o700->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:26/0 lens 264/248 e 0 to 0 dl 1717558586 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 [ 2004.036180] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 2004.085533] Lustre: Failing over lustre-MDT0000 [ 2004.361800] Lustre: server umount lustre-MDT0000 complete [ 2005.006316] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.006319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.006330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.006337] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.006422] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.006442] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.006452] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.031191] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.369774] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.014293] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.014297] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.014336] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.014338] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.377719] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.022069] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.022138] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.022143] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.022146] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.036514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.064922] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.831354] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.385637] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2021.919143] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2022.519933] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2025.662132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2026.534071] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:36:53 (1717558613) [ 2026.578663] Lustre: Failing over lustre-OST0000 [ 2029.411304] Lustre: server umount lustre-OST0000 complete [ 2031.150352] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.150389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.150534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.161764] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.410328] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.158636] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.158649] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.418105] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.554970] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.562046] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2039.567326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2039.643714] mount.lustre (18103) used greatest stack depth: 10016 bytes left [ 2041.062609] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2041.469143] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2043.684383] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2044.116356] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2108.627530] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:38:16 (1717558696) [ 2110.854913] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2111.534370] Lustre: Failing over lustre-MDT0000 [ 2111.587527] Lustre: server umount lustre-MDT0000 complete [ 2114.543861] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.750655] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.750675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.750896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.751073] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.751097] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.751146] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.796470] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.545788] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.758198] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.758297] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.758328] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.758331] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.553213] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.712568] LDISKFS-fs (dm-0): recovery complete [ 2124.713935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.738171] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.547498] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.561630] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2129.994020] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2191.592738] LustreError: 20610:0:(/home/green/git/lustre-release/lustre/osp/osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2192.046455] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 23:39:39 (1717558779) [ 2192.053757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.282308] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2192.285058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.690122] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2199.681498] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 23:39:46 (1717558786) [ 2199.743104] Lustre: Failing over lustre-MDT0000 [ 2199.902138] Lustre: server umount lustre-MDT0000 complete [ 2199.902234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.902254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.902284] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.902355] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.902360] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.902364] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.928890] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.686365] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.910291] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.910368] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.910400] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.910402] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.690056] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.918619] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.918632] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.918640] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.918921] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.251910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.304649] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.241159] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2214.698094] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2217.409383] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2274.722015] LustreError: 22594:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88008f0d1500 x1800988643908096/t0(0) o101->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:298/0 lens 328/344 e 0 to 0 dl 1717558858 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 [ 2275.488402] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 2276.001220] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2279.946841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2284.112265] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 23:41:07 (1717558867) [ 2284.726234] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2284.766336] Lustre: Failing over lustre-MDT0000 [ 2284.768073] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.800173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.812310] Lustre: server umount lustre-MDT0000 complete [ 2287.518553] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.518632] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.518732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.518934] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.518946] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.528447] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.528711] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.818224] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.526451] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.526534] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.526654] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.526663] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.826395] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.534175] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.534177] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.534186] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.534228] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.030518] LDISKFS-fs (dm-0): recovery complete [ 2298.032069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.060681] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.881929] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.834064] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2303.770186] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2304.174690] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2307.803586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2311.924757] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 23:41:35 (1717558895) [ 2312.531538] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2312.588028] Lustre: Failing over lustre-MDT0000 [ 2313.150122] Lustre: server umount lustre-MDT0000 complete [ 2313.150185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.150220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.150249] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.150347] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.150352] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.150357] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.171948] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.860361] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.158216] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.158279] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.158283] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.158284] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.865479] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.166184] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.166201] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.166212] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.166329] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.873198] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.525208] LDISKFS-fs (dm-0): recovery complete [ 2325.526453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.549638] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.315029] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2329.881616] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2331.179909] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2331.539757] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2334.182583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2337.767666] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 23:42:01 (1717558921) [ 2338.303067] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2338.350163] Lustre: Failing over lustre-MDT0000 [ 2339.900062] Lustre: server umount lustre-MDT0000 complete [ 2340.638132] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.638160] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2340.638176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.638260] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.638269] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.638311] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.638426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.664974] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.668911] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.906027] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.646428] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.646438] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.646454] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.646477] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.913361] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.654136] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.654144] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.654147] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.654169] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.439719] LDISKFS-fs (dm-0): recovery complete [ 2351.442092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.480593] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.266622] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.921920] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2360.243294] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2361.508951] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 23:42:27 (1717558947) [ 2362.210183] LustreError: 7124:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800819a5f80 x1800988643933440/t257698037777(0) o35->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:400/0 lens 392/456 e 0 to 0 dl 1717558960 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2362.267146] Lustre: Failing over lustre-MDT0000 [ 2364.945886] Lustre: server umount lustre-MDT0000 complete [ 2366.558251] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.558259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.558372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.558427] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.558507] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.564419] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.590990] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2366.591217] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.601276] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.945798] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.566176] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.566291] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.566297] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.566314] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.157351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.183190] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.953700] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.991799] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2379.269966] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2379.819722] Lustre: 7124:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88007e5fb800 x1800988643933440/t257698037777(0) o35->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:418/0 lens 392/456 e 0 to 0 dl 1717558978 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2380.308050] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2383.744327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2384.113302] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 23:42:51 (1717558971) [ 2387.449068] LustreError: 7122:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88013682b800 x1800988643939776/t261993005072(0) o36->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:467/0 lens 504/448 e 0 to 0 dl 1717559027 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2387.926419] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2387.977913] Lustre: Failing over lustre-MDT0000 [ 2389.278367] Lustre: server umount lustre-MDT0000 complete [ 2389.278369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.278441] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.278484] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2389.278490] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.278526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.278575] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.278577] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.325139] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.980425] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.286334] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.286335] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.286348] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.289667] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.985717] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.294085] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.294149] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.294263] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.294268] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.993479] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.833160] LDISKFS-fs (dm-0): recovery complete [ 2400.834667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.862731] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.641533] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2405.002090] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2405.942328] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2406.500496] Lustre: 11320:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88012e938380 x1800988643939776/t261993005072(0) o36->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:489/0 lens 504/2880 e 0 to 0 dl 1717559049 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2406.878798] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2409.690917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2410.022533] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 23:43:17 (1717558997) [ 2411.278050] LustreError: 11320:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009ac30000 x1800988643946560/t266287972368(0) o36->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:493/0 lens 504/448 e 0 to 0 dl 1717559053 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2413.164099] LustreError: 7124:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800985e8700 x1800988643946688/t266287972369(0) o35->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:450/0 lens 392/456 e 0 to 0 dl 1717559010 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2413.656635] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2413.711175] Lustre: Failing over lustre-MDT0000 [ 2415.021181] Lustre: server umount lustre-MDT0000 complete [ 2415.950043] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.950155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.950207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.950243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.950245] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.950297] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.950311] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.950319] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.984921] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2420.025702] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.958282] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.958299] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.958302] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.958340] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.033386] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.966114] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.966158] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.970500] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.970506] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.500083] LDISKFS-fs (dm-0): recovery complete [ 2426.502232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.531584] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.333652] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.041848] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2431.605674] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2431.608015] Lustre: 7124:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88007e5fb480 x1800988643946688/t266287972369(0) o35->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:470/0 lens 392/456 e 0 to 0 dl 1717559030 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2435.350632] Lustre: 7121:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88007e5fb800 x1800988643946560/t266287972368(0) o36->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:514/0 lens 504/2880 e 0 to 0 dl 1717559074 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2435.608756] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 23:43:42 (1717559022) [ 2439.042194] LustreError: 8533:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88008b59ad80 x1800988643952704/t270582939664(0) o36->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:518/0 lens 504/448 e 0 to 0 dl 1717559078 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2439.602037] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2439.663626] Lustre: Failing over lustre-MDT0000 [ 2440.061950] Lustre: server umount lustre-MDT0000 complete [ 2441.614076] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.614102] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.614106] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2441.614116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.614143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.614234] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.614253] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.614261] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.642040] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.065748] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.622109] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.622188] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.622214] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.622221] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.073587] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.630011] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.630076] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.630114] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.630116] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.542108] LDISKFS-fs (dm-0): recovery complete [ 2452.543295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.567146] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.375820] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.081865] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2457.637100] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2461.133533] Lustre: 11320:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880136a27b80 x1800988643952704/t270582939664(0) o36->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:540/0 lens 504/2880 e 0 to 0 dl 1717559100 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2462.616992] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 23:44:08 (1717559048) [ 2465.478361] LustreError: 7124:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff880081b6c000 x1800988643958784/t274877906960(0) o35->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:501/0 lens 392/456 e 0 to 0 dl 1717559061 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2465.985224] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2466.036046] Lustre: Failing over lustre-MDT0000 [ 2467.646269] Lustre: server umount lustre-MDT0000 complete [ 2467.646283] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.646310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.646312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.646372] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2467.646408] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.646417] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.646441] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.670175] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.108552] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.654333] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.654387] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.654430] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.654432] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.113714] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.662067] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.662114] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.662163] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.662174] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.310959] LDISKFS-fs (dm-0): recovery complete [ 2479.312469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.335968] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.121515] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2480.212946] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2484.405610] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2488.385995] Lustre: 7124:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88012c170a80 x1800988643958784/t274877906960(0) o35->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:523/0 lens 392/456 e 0 to 0 dl 1717559083 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2488.626019] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 23:44:35 (1717559075) [ 2548.652300] LustreError: 11441:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88012d41b800 x1800988643963712/t279172874255(0) o101->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:571/0 lens 664/608 e 0 to 0 dl 1717559131 ref 1 fl Interpret:/200/0 rc 301/0 job:'touch.0' uid:0 gid:0 [ 2548.673898] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 2550.275855] Lustre: 11441:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88013052e300 x1800988643963712/t279172874255(0) o101->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:631/0 lens 664/3488 e 0 to 0 dl 1717559191 ref 1 fl Interpret:/202/0 rc 0/0 job:'touch.0' uid:0 gid:0 [ 2552.134002] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 23:45:37 (1717559137) [ 2552.611280] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2552.656874] Lustre: Failing over lustre-MDT0000 [ 2553.660900] Lustre: server umount lustre-MDT0000 complete [ 2554.510211] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.510217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.510248] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.510258] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.514896] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.515060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.515069] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.515090] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.541016] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.666021] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.518197] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.518202] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.518205] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.518212] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.673252] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.526104] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.530747] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.530783] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.530799] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.499425] LDISKFS-fs (dm-0): recovery complete [ 2565.500943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.526598] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.288976] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2568.681775] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2571.134911] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2571.496993] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2584.098491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2586.182225] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 23:46:11 (1717559171) [ 2586.660069] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2586.711765] Lustre: Failing over lustre-MDT0000 [ 2588.718931] Lustre: server umount lustre-MDT0000 complete [ 2590.622258] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.622260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.622276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.622410] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.622413] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.622423] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.622498] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.647432] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.721782] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.630230] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.630235] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.630245] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.630280] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.729377] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.569203] LDISKFS-fs (dm-0): recovery complete [ 2599.570576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2599.596096] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.344887] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.737750] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2605.231171] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2605.580883] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2607.428692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2612.176151] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2623.802457] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 23:46:39 (1717559199) [ 2624.520322] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2624.630544] Lustre: Failing over lustre-MDT0000 [ 2624.686039] Lustre: server umount lustre-MDT0000 complete [ 2624.686069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.686102] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.686132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.686144] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.686174] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.686193] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.687958] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.726066] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2628.782796] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.694345] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.695860] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.695922] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.695940] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.786514] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.703746] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.703910] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.703932] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.703978] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.437173] LDISKFS-fs (dm-0): recovery complete [ 2638.438670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.469142] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.793450] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.223332] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2644.458602] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2644.963131] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2663.343246] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2666.189093] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 23:47:30 (1717559250) [ 2666.901505] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2666.962676] Lustre: Failing over lustre-MDT0000 [ 2668.574327] Lustre: server umount lustre-MDT0000 complete [ 2668.574406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.574451] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.574484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.574584] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.574591] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.574602] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.612529] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.701526] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.705833] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.582150] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.582175] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.582201] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.582208] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.705599] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.705601] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.590096] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.590225] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.590284] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.590300] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.713726] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.713729] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.993835] LDISKFS-fs (dm-0): recovery complete [ 2679.995723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.022354] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.786620] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.722041] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2685.774007] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2686.333237] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2688.430959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2688.787943] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2690.151274] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2755.787636] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 23:47:57 (1717559277) [ 2756.284309] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 2816.311306] LustreError: 7122:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009ac31f80 x1800988645442880/t296352743434(0) o36->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:84/0 lens 66040/440 e 0 to 0 dl 1717559399 ref 1 fl Interpret:/200/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 [ 2816.331018] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnecting [ 2819.034514] Lustre: 21050:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a50c6850 x1800988645442880/t296352743434(0) o36->9f0681d6-1720-457c-8da4-97bb22560eb5@192.168.202.55@tcp:144/0 lens 66040/440 e 0 to 0 dl 1717559459 ref 1 fl Interpret:/202/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 [ 2819.588261] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 2823.160888] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 23:50:06 (1717559406) [ 2823.869236] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2823.921259] Lustre: Failing over lustre-MDT0000 [ 2825.310380] Lustre: server umount lustre-MDT0000 complete [ 2825.310413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.310416] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.310529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.310583] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.310590] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.310756] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.358537] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.335971] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.318155] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.318180] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.318186] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.318224] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.338362] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.326215] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.326296] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.326301] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.326334] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.345450] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.049172] LDISKFS-fs (dm-0): recovery complete [ 2837.050725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.077511] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.854490] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.353885] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2842.779591] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2843.155703] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2846.188029] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2851.684485] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 23:50:33 (1717559433) [ 2854.196377] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2854.220804] Lustre: Failing over lustre-OST0000 [ 2856.383386] Lustre: server umount lustre-OST0000 complete [ 2857.166189] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.166195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.166299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.177422] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.386108] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.174245] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.174250] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.393523] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.182195] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.182201] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.327729] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.334256] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2867.381621] LDISKFS-fs (dm-2): recovery complete [ 2867.383940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2868.640158] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2868.792933] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2879.623445] Lustre: Failing over lustre-OST0000 [ 2879.645211] Lustre: server umount lustre-OST0000 complete [ 2881.423077] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.446417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.451276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.451395] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.458037] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.426694] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.454259] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.433966] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.069601] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.079184] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2892.086063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2893.376224] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2893.495738] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2895.708347] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2896.083537] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2929.076542] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 23:51:56 (1717559516) [ 2929.778118] Lustre: Failing over lustre-MDT0000 [ 2929.830525] Lustre: server umount lustre-MDT0000 complete [ 2931.506009] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.206227] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.206254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.206290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.206346] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.206384] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.206945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.207052] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.244393] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.505685] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.214170] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.214239] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.214273] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.513395] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.721859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.750030] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.536492] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.521419] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2953.486213] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2953.544340] Lustre: Failing over lustre-MDT0000 [ 2956.541040] Lustre: server umount lustre-MDT0000 complete [ 2956.830170] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.830306] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.830358] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.830361] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2956.830388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.831698] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.834733] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.834867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.862191] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.545629] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.838142] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.838159] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.838192] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.838196] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2965.513524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.540328] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.309030] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2966.553248] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2971.168953] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2971.528691] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2974.226877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2985.036902] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 23:52:41 (1717559561) [ 2985.051353] Lustre: Failing over lustre-OST0000 [ 2985.630140] Lustre: server umount lustre-OST0000 complete [ 2985.630142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.630237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.638848] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2986.590611] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.638368] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.638376] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2991.593795] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.646358] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.646371] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.601513] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.293349] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.297234] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2997.300079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2998.461119] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2998.552293] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3000.945765] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3001.453038] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3004.458105] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 23:53:11 (1717559591) [ 3004.953337] Lustre: Failing over lustre-MDT0000 [ 3005.007752] Lustre: server umount lustre-MDT0000 complete [ 3005.678293] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3005.683098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.689102] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.625835] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.342041] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.342129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.342227] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.342229] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.342236] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.342291] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.462383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.487443] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.492150] LustreError: 32256:0:(/home/green/git/lustre-release/lustre/obdclass/llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffa0558350 failed: rc = -95 [ 3007.495842] LustreError: 32256:0:(/home/green/git/lustre-release/lustre/obdclass/obd_config.c:797:class_setup()) setup MGS failed (-95) [ 3007.501213] LustreError: 32256:0:(/home/green/git/lustre-release/lustre/obdclass/obd_mount.c:216:lustre_start_simple()) MGS setup error -95 [ 3007.507101] LustreError: 32256:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_mount.c:135:server_deregister_mount()) MGS not registered [ 3007.512288] LustreError: 0-0: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3007.514428] LustreError: 32256:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_mount.c:1751:server_put_super()) no obd lustre-MDT0000 [ 3007.519548] LustreError: 32256:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_mount.c:135:server_deregister_mount()) lustre-MDT0000 not registered [ 3007.522675] Lustre: server umount lustre-MDT0000 complete [ 3009.547754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.577575] LustreError: 32256:0:(/home/green/git/lustre-release/lustre/llite/super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3010.360260] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3011.625474] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3012.269516] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3016.482406] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 23:53:19 (1717559599) [ 3017.206511] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3017.256655] Lustre: Failing over lustre-MDT0000 [ 3019.662168] Lustre: server umount lustre-MDT0000 complete [ 3019.662173] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.662249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.662295] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3019.662344] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.662418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.681242] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.681405] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.689657] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3021.644035] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.670087] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.670137] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.670157] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.670161] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.649613] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3029.678065] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3029.678113] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3029.678117] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3029.678154] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.073530] LDISKFS-fs (dm-0): recovery complete [ 3030.075737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.104363] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.915563] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3031.657540] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3091.695017] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3092.701829] Lustre: lustre-MDT0000: Client 9f0681d6-1720-457c-8da4-97bb22560eb5 (at 192.168.202.55@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 3093.091923] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3097.300320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3120.586659] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 23:54:44 (1717559684) [ 3131.590957] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3132.737750] LustreError: 7120:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake