[ 3195.672878] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3195.687948] LustreError: Skipped 1 previous similar message
[ 3195.856491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up)
[ 3196.183837] LustreError: 90933:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 3196.189594] LustreError: 90933:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3196.202086] Lustre: 90981:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3196.213441] Lustre: 90981:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3196.219597] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 3196.339794] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 3196.490399] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1507 to 0x0:1569
[ 3196.501375] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1512 to 0x0:1601
[ 3200.678596] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3213.213624] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:17:42 (1776183462)
[ 3238.723896] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3266.192066] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3266.215185] LustreError: Skipped 1 previous similar message
[ 3267.451068] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1970 to 0x0:1985
[ 3267.453322] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2002 to 0x0:2017
[ 3271.011934] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3280.685958] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3282.223149] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3301.733039] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:19:11 (1776183551)
[ 3320.021421] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3351.970782] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3357.199683] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2418 to 0x0:2433
[ 3357.224758] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2386 to 0x0:2401
[ 3363.198627] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3365.237953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3386.497365] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 12:20:36 (1776183636)
[ 3387.868696] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 3389.596583] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 12:20:38 (1776183638)
[ 3391.523859] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 3392.382307] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3392.386353] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 3392.398434] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2386 to 0x0:2433
[ 3397.663111] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 12:20:47 (1776183647)
[ 3414.187427] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3423.720054] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -19
[ 3424.551453] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3429.649766] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3429.669216] LustreError: Skipped 1 previous similar message
[ 3434.767846] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3434.791697] LustreError: Skipped 1 previous similar message
[ 3445.637814] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2835 to 0x0:2881
[ 3447.485770] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3503.582644] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 12:22:32 (1776183752)
[ 3507.737867] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3527.603639] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3527.613236] Lustre: Skipped 21 previous similar messages
[ 3529.117135] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 3529.119289] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2834 to 0x0:2849
[ 3531.897332] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3536.351385] LustreError: 99121:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 3536.361295] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3537.380724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2835 to 0x0:2913
[ 3542.801740] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3544.773740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3564.169578] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 12:23:33 (1776183813)
[ 3568.678441] LustreError: 99097:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3573.727206] LustreError: 99097:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3573.731930] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnecting
[ 3573.768682] LustreError: 36297:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3575.291449] LustreError: 99097:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3580.383209] LustreError: 99097:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3580.386242] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnecting
[ 3580.394206] LustreError: 99098:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3580.398859] LustreError: 99098:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message
[ 3581.431569] LustreError: 99097:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3586.527517] LustreError: 99097:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3586.533143] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnecting
[ 3586.540716] Lustre: Skipped 1 previous similar message
[ 3588.088624] LustreError: 99808:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3593.183193] LustreError: 99808:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3595.043458] LustreError: 100463:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3600.351709] LustreError: 100463:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3600.360171] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnecting
[ 3600.368963] Lustre: Skipped 1 previous similar message
[ 3609.063412] LustreError: 99808:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3609.068866] LustreError: 99808:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 3614.178459] LustreError: 99808:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3614.186878] LustreError: 99808:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 3621.343319] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnecting
[ 3621.353478] Lustre: Skipped 2 previous similar messages
[ 3631.103135] LustreError: 99808:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3631.113600] LustreError: 99808:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages
[ 3636.191129] LustreError: 99808:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3636.201603] LustreError: 99808:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 3643.235586] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 12:24:52 (1776183892)
[ 3644.904174] LustreError: 100463:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 3651.343619] Lustre: lustre-MDT0000: Export 00000000d6821d19 already connecting from 192.168.202.54@tcp
[ 3652.864060] Lustre: lustre-MDT0000: Export 00000000d6821d19 already connecting from 192.168.202.54@tcp
[ 3654.714237] Lustre: lustre-MDT0000: Export 00000000d6821d19 already connecting from 192.168.202.54@tcp
[ 3658.319820] Lustre: lustre-MDT0000: Export 00000000d6821d19 already connecting from 192.168.202.54@tcp
[ 3658.335550] Lustre: Skipped 2 previous similar messages
[ 3663.924047] Lustre: lustre-MDT0000: Export 00000000d6821d19 already connecting from 192.168.202.54@tcp
[ 3663.936832] Lustre: Skipped 3 previous similar messages
[ 3668.321959] LustreError: 100463:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 3668.335978] Lustre: 100463:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/4s); client may timeout  req@000000003421b49c x1862460025342016/t0(0) o38->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:0/0 lens 520/416 e 0 to 0 dl 1776183914 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0'
[ 3671.821830] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnecting
[ 3671.830490] Lustre: Skipped 3 previous similar messages
[ 3672.680906] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:25:21 (1776183921)
[ 3676.563739] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3687.295747] Lustre: MGC192.168.202.154@tcp: Connection restored to  (at 0@lo)
[ 3687.305180] Lustre: Skipped 32 previous similar messages
[ 3687.512151] Lustre: *** cfs_fail_loc=712, val=0***
[ 3687.514073] LustreError: 36298:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@00000000d0bd4809 x1862460031829632/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 3687.523363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3687.673812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3687.676931] Lustre: Skipped 10 previous similar messages
[ 3687.742581] LustreError: 103393:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 3687.747836] LustreError: 103393:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3687.753807] Lustre: 103442:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3687.758892] Lustre: 103442:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3687.762955] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 3687.834517] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0]
[ 3687.956749] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2834 to 0x0:2881
[ 3687.960619] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2835 to 0x0:2945
[ 3692.182775] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3710.303202] Lustre: 2992:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183953/real 1776183953]  req@00000000f6df3037 x1862460031834496/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776183960 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 3710.324971] Lustre: 2992:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 66 previous similar messages
[ 3728.674635] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0xe74272de27369d2c to 0xe74272de2736a0eb
[ 3728.689060] Lustre: Skipped 7 previous similar messages
[ 3729.337129] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2835 to 0x0:2977
[ 3729.344273] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2834 to 0x0:2913
[ 3732.774584] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3740.550266] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3742.046729] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3749.185200] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:26:38 (1776183998)
[ 3749.316104] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnecting
[ 3756.646796] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:26:45 (1776184005)
[ 3757.556934] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 3757.563720] LustreError: 104431:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000ac7300b6 x1862460025365056/t0(0) o700->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:458/0 lens 264/248 e 0 to 0 dl 1776184013 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 3767.599527] Lustre: Failing over lustre-MDT0000
[ 3767.601915] Lustre: Skipped 7 previous similar messages
[ 3767.823058] Lustre: server umount lustre-MDT0000 complete
[ 3767.826384] Lustre: Skipped 7 previous similar messages
[ 3776.863310] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3776.879487] Lustre: Skipped 16 previous similar messages
[ 3776.927594] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3776.943359] LustreError: Skipped 6 previous similar messages
[ 3794.979323] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3794.993206] Lustre: Skipped 5 previous similar messages
[ 3795.122142] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3795.130768] Lustre: Skipped 5 previous similar messages
[ 3795.155883] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2979 to 0x0:3009
[ 3795.156766] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2915 to 0x0:2945
[ 3798.029555] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3807.267576] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3809.173110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3817.475665] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:27:46 (1776184066)
[ 3824.408214] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3824.423951] LustreError: Skipped 3 previous similar messages
[ 3824.609918] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3839.965241] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3020 to 0x0:3041
[ 3842.887598] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3852.344550] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3854.438131] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3926.892610] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:29:36 (1776184176)
[ 3930.296653] LustreError: 108689:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3930.300878] LustreError: 108689:0:(osd_handler.c:698:osd_ro()) Skipped 6 previous similar messages
[ 3931.085558] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3961.377298] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2966 to 0x0:3009
[ 3961.381336] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3052 to 0x0:3073
[ 3965.178412] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4033.112329] Lustre: *** cfs_fail_loc=216, val=0***
[ 4039.167788] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:31:28 (1776184288)
[ 4041.013975] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 4041.023462] Lustre: Skipped 2 previous similar messages
[ 4041.031385] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3084 to 0x0:3105
[ 4042.146906] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3084 to 0x0:3137
[ 4054.249763] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:31:43 (1776184303)
[ 4089.563662] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4089.570403] LustreError: 111002:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000f4081a73 x1862460025415104/t0(0) o101->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:35/0 lens 328/344 e 0 to 0 dl 1776184345 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 4093.842973] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4096.786212] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 4096.881529] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3084 to 0x0:3169
[ 4096.884353] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3021 to 0x0:3041
[ 4105.717536] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4107.930612] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4119.014753] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:32:47 (1776184367)
[ 4121.637260] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4127.416585] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4158.567589] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4158.577290] Lustre: Skipped 8 previous similar messages
[ 4163.899528] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4170.760583] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3171 to 0x0:3201
[ 4170.764334] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3021 to 0x0:3073
[ 4177.599788] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4179.706389] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4189.869744] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:33:58 (1776184438)
[ 4191.099965] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4197.091927] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4232.776887] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4243.427832] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3021 to 0x0:3105
[ 4243.429152] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3233
[ 4249.850463] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4251.262513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4259.128556] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:35:08 (1776184508)
[ 4260.041660] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4265.841496] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4294.132817] Lustre: MGC192.168.202.154@tcp: Connection restored to  (at 0@lo)
[ 4294.148443] Lustre: Skipped 24 previous similar messages
[ 4294.740196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4294.743064] Lustre: Skipped 7 previous similar messages
[ 4297.299248] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3265
[ 4297.308136] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137
[ 4299.505528] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4311.919658] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:36:00 (1776184560)
[ 4314.540848] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4314.547357] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 4314.558290] LustreError: 116290:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000f513ab47 x1862460025440000/t261993005072(0) o35->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:260/0 lens 392/456 e 0 to 0 dl 1776184570 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4327.265387] Lustre: 2989:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184570/real 1776184570]  req@000000003421b49c x1862460031914816/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776184577 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 4327.296737] Lustre: 2989:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 44 previous similar messages
[ 4344.810080] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0xe74272de2736d6f3 to 0xe74272de2736dd0c
[ 4344.824994] Lustre: Skipped 6 previous similar messages
[ 4350.602650] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4354.893659] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3169
[ 4354.894495] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3297
[ 4354.895141] Lustre: 117637:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000ea98c7ba x1862460025440000/t261993005072(0) o35->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:300/0 lens 392/456 e 0 to 0 dl 1776184610 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4360.842726] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4362.818645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4374.563114] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:37:02 (1776184622)
[ 4376.425890] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4376.431616] LustreError: 117634:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000005a0c13 x1862460025447040/t266287972368(0) o36->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:322/0 lens 504/448 e 0 to 0 dl 1776184632 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4383.510289] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnecting
[ 4383.524800] Lustre: Skipped 4 previous similar messages
[ 4383.550398] Lustre: 117633:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000f4674d2c x1862460025447040/t266287972368(0) o36->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:329/0 lens 504/448 e 0 to 0 dl 1776184639 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4383.983824] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4386.257870] Lustre: Failing over lustre-MDT0000
[ 4386.263983] Lustre: Skipped 7 previous similar messages
[ 4386.273159] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4386.291421] Lustre: Skipped 16 previous similar messages
[ 4386.313629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4386.566887] Lustre: server umount lustre-MDT0000 complete
[ 4386.570277] Lustre: Skipped 7 previous similar messages
[ 4398.559422] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4398.569588] LustreError: Skipped 6 previous similar messages
[ 4422.458060] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4428.563702] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4428.573378] Lustre: Skipped 7 previous similar messages
[ 4428.667950] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4428.680698] Lustre: Skipped 7 previous similar messages
[ 4428.758458] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3329
[ 4428.759332] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3201
[ 4435.979621] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4438.084218] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4447.945618] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:38:16 (1776184696)
[ 4449.501464] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4449.513370] LustreError: 119384:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000016955d84 x1862460025454784/t270582939664(0) o36->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:395/0 lens 504/448 e 0 to 0 dl 1776184705 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4451.878533] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4456.352566] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4456.737390] Lustre: 119386:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000056d4b1c5 x1862460025454784/t270582939664(0) o36->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:402/0 lens 504/448 e 0 to 0 dl 1776184712 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4456.767332] Lustre: 119386:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 4459.493120] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4459.504565] LustreError: Skipped 5 previous similar messages
[ 4494.649303] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4501.899044] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 4501.899338] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3233
[ 4509.045981] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:39:18 (1776184758)
[ 4510.350734] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4510.352994] Lustre: Skipped 1 previous similar message
[ 4510.368543] LustreError: 121137:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000093088abc x1862460025461504/t274877906960(0) o36->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:456/0 lens 504/448 e 0 to 0 dl 1776184766 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4510.392018] LustreError: 121137:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message
[ 4512.383382] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4512.388497] Lustre: Skipped 1 previous similar message
[ 4516.659102] Lustre: 121136:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000002ba484d7 x1862460025461504/t274877906960(0) o36->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:462/0 lens 504/448 e 0 to 0 dl 1776184772 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4517.736684] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4546.322475] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (not set up)
[ 4546.329872] Lustre: Skipped 1 previous similar message
[ 4547.932297] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3363 to 0x0:3393
[ 4547.934930] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3265
[ 4552.439534] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4565.308207] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:40:14 (1776184814)
[ 4566.588513] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4568.884876] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4568.888324] LustreError: 122727:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000613c7d72 x1862460025468160/t279172874256(0) o35->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:514/0 lens 392/456 e 0 to 0 dl 1776184824 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4573.880958] LustreError: 123616:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4573.889262] LustreError: 123616:0:(osd_handler.c:698:osd_ro()) Skipped 6 previous similar messages
[ 4574.039640] Lustre: 122727:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000a3a71ff3 x1862460025468160/t279172874256(0) o35->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:520/0 lens 392/456 e 0 to 0 dl 1776184830 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4574.857745] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4607.661405] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4619.329105] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425
[ 4619.332895] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3297
[ 4626.886603] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:41:15 (1776184875)
[ 4628.313486] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 4628.318964] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4628.329648] Lustre: Skipped 1 previous similar message
[ 4628.332128] LustreError: 124223:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000004b7da6c3 x1862460025473280/t283467841549(0) o101->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:574/0 lens 664/600 e 0 to 0 dl 1776184884 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0'
[ 4634.402983] Lustre: 124224:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000a40bd027 x1862460025473280/t283467841549(0) o101->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:580/0 lens 664/3424 e 0 to 0 dl 1776184890 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 4643.279091] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:41:31 (1776184891)
[ 4647.628148] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4678.413720] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4678.427631] LustreError: Skipped 1 previous similar message
[ 4679.924759] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3457
[ 4679.939300] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3329
[ 4685.204376] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4696.450564] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4698.525655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4718.707753] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:42:47 (1776184967)
[ 4724.069895] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4761.440527] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4769.672132] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3489
[ 4769.672781] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 4777.043374] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4778.577840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4785.758879] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 4798.639216] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:44:08 (1776185048)
[ 4843.139733] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4846.355445] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping)
[ 4876.965602] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4876.974108] Lustre: Skipped 9 previous similar messages
[ 4879.577842] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4612 to 0x0:4641
[ 4879.583944] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4740 to 0x0:4769
[ 4882.017146] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4892.325189] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4893.954467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4966.162615] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:46:55 (1776185215)
[ 4970.114994] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4971.806359] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping)
[ 4971.822555] Lustre: Skipped 3 previous similar messages
[ 4979.167589] Lustre: 2991:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185222/real 1776185222]  req@0000000002636e58 x1862460032317376/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185229 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0'
[ 4979.194149] Lustre: 2991:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
[ 4996.581069] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0xe74272de273859ac to 0xe74272de273b0a99
[ 4996.593894] Lustre: Skipped 7 previous similar messages
[ 4996.600105] Lustre: MGC192.168.202.154@tcp: Connection restored to  (at 0@lo)
[ 4996.603967] Lustre: Skipped 26 previous similar messages
[ 4997.055639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4997.069120] Lustre: Skipped 13 previous similar messages
[ 4997.170862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4997.175563] Lustre: Skipped 8 previous similar messages
[ 5001.883350] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5003.627746] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4612 to 0x0:4673
[ 5003.630972] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4771 to 0x0:4801
[ 5010.578320] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5012.076273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5024.692407] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 5026.200388] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 5033.686221] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:48:02 (1776185282)
[ 5040.760757] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 5084.963854] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnecting
[ 5084.980341] Lustre: Skipped 4 previous similar messages
[ 5088.106696] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 5088.110674] LustreError: 131892:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000ecba6131 x1862460026950272/t300647710728(0) o36->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:314/0 lens 66040/440 e 0 to 0 dl 1776185379 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 5130.020178] Lustre: 131893:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000007e90e730 x1862460026950272/t300647710728(0) o36->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:356/0 lens 66040/440 e 0 to 0 dl 1776185421 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 5138.988599] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 5140.735845] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:49:49 (1776185389)
[ 5150.541313] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5152.940618] Lustre: Failing over lustre-MDT0000
[ 5152.949263] Lustre: Skipped 7 previous similar messages
[ 5153.314933] Lustre: server umount lustre-MDT0000 complete
[ 5153.319567] Lustre: Skipped 7 previous similar messages
[ 5162.911903] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5162.936886] LustreError: Skipped 7 previous similar messages
[ 5180.229672] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5180.242236] Lustre: Skipped 7 previous similar messages
[ 5181.593973] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5181.612862] Lustre: Skipped 7 previous similar messages
[ 5181.679302] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4902 to 0x0:4929
[ 5181.689090] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4775 to 0x0:4801
[ 5185.347984] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5194.365418] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5195.924907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5205.433325] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:50:54 (1776185454)
[ 5224.151965] LustreError: 135674:0:(osd_handler.c:698:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only ***
[ 5224.166337] LustreError: 135674:0:(osd_handler.c:698:osd_ro()) Skipped 5 previous similar messages
[ 5225.114654] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5241.313935] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5241.320325] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5251.554517] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5251.569824] LustreError: Skipped 1 previous similar message
[ 5261.037468] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5275.617293] LustreError: 137007:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery
[ 5275.622601] Lustre: 136488:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 5275.629158] Lustre: 136488:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 5275.635779] Lustre: 136488:0:(ldlm_lib.c:1804:abort_req_replay_queue()) @@@ aborted:  req@000000001c7cac1b x1862460032368960/t0(17179870573) o6->lustre-MDT0000-mdtlov_UUID@0@lo:486/0 lens 544/0 e 0 to 0 dl 1776185551 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'osp-syn-0-0.0'
[ 5275.663104] LustreError: 136488:0:(ofd_obd.c:1309:ofd_iocontrol()) lustre-OST0000: not supported cmd = 0xc00866c1
[ 5275.664502] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -19
[ 5275.672176] Lustre: 136488:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6
[ 5275.676995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping)
[ 5280.751854] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5296.622504] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5299.763481] LustreError: 2988:0:(client.c:3161:ptlrpc_replay_interpret()) @@@ status 0, old was -19  req@0000000030b84acf x1862460032368960/t17179870573(17179870573) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 1776185588 ref 2 fl Interpret:RQU/4/0 rc 0/0 job:'osp-syn-0-0.0'
[ 5300.396223] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5330 to 0x0:5345
[ 5304.854640] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5306.424859] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5347.309292] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:53:16 (1776185596)
[ 5350.370170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5350.378026] Lustre: Skipped 1 previous similar message
[ 5381.504098] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5202 to 0x0:5217
[ 5384.777245] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5428.004401] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (not set up)
[ 5429.289748] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5202 to 0x0:5249
[ 5429.295547] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5330 to 0x0:5377
[ 5433.796774] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5443.047488] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5444.561398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5454.915252] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:55:03 (1776185703)
[ 5474.077215] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5474.095134] LustreError: Skipped 2 previous similar messages
[ 5474.284664] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5486.983015] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 5486.992031] Lustre: Skipped 6 previous similar messages
[ 5488.431451] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5379 to 0x0:5409
[ 5491.559980] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5502.371792] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5504.208766] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5513.332605] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:56:02 (1776185762)
[ 5523.477751] Lustre: *** cfs_fail_loc=605, val=0***
[ 5523.479392] LustreError: 142994:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=000000001c44d4c5 failed: rc = -95
[ 5523.482678] LustreError: 142994:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 5523.484957] LustreError: 142994:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 5523.487416] LustreError: 142994:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 5523.489792] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 5523.492325] LustreError: 142994:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 5523.559589] LustreError: 142994:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 5530.394998] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (not set up)
[ 5532.587195] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5251 to 0x0:5281
[ 5532.587941] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5379 to 0x0:5441
[ 5535.382844] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5543.381599] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:56:32 (1776185792)
[ 5547.950340] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5583.560990] Lustre: *** cfs_fail_loc=707, val=0***
[ 5589.058394] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5597.090678] Lustre: 2990:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185806/real 1776185806]  req@0000000086c35aee x1862460032464064/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776185847 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0'
[ 5597.127104] Lustre: 2990:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 59 previous similar messages
[ 5625.642444] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnected, waiting for 1 clients in recovery for 0:57
[ 5626.484037] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5295 to 0x0:5313
[ 5626.484354] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5454 to 0x0:5473
[ 5633.070991] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5634.344718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5642.099520] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:58:11 (1776185891)
[ 5671.506808] LustreError: 144702:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 5677.567141] LustreError: 144702:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5694.034361] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:59:03 (1776185943)
[ 5723.084097] LustreError: 6312:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms
[ 5729.127252] LustreError: 6312:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake
[ 5737.236127] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:59:46 (1776185986)
[ 5764.859203] LustreError: 144701:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms
[ 5769.879654] LustreError: 144701:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5771.857708] LustreError: 144702:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms
[ 5781.871167] LustreError: 144702:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5804.063396] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 13:00:53 (1776186053)
[ 5861.760291] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 13:01:50 (1776186110)
[ 5890.728540] LustreError: 144702:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 5891.159193] LustreError: 144702:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5899.010780] LustreError: 144704:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 5899.038488] LustreError: 144704:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 22 previous similar messages
[ 5899.471428] LustreError: 144704:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5899.483520] LustreError: 144704:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 22 previous similar messages
[ 5915.063385] LustreError: 145152:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 5915.074503] LustreError: 145152:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 45 previous similar messages
[ 5915.511553] LustreError: 145152:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5915.516924] LustreError: 145152:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 45 previous similar messages
[ 5925.506189] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 13:02:54 (1776186174)
[ 5959.859519] Lustre: DEBUG MARKER: phase 2
[ 5969.049864] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 13:03:38 (1776186218)
[ 6053.316520] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 13:05:02 (1776186302)
[ 6054.627775] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1
[ 6056.514855] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 13:05:05 (1776186305)
[ 6060.595896] Lustre: DEBUG MARKER: Started rundbench load pid=131389 ...
[ 6064.972741] LustreError: 151502:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6064.980197] LustreError: 151502:0:(osd_handler.c:698:osd_ro()) Skipped 1 previous similar message
[ 6065.903863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6068.879607] Lustre: DEBUG MARKER: test_70b fail mds1 1 times
[ 6070.769808] Lustre: Failing over lustre-MDT0000
[ 6070.778838] Lustre: Skipped 7 previous similar messages
[ 6070.865597] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping)
[ 6071.144827] Lustre: server umount lustre-MDT0000 complete
[ 6071.146983] Lustre: Skipped 8 previous similar messages
[ 6080.480266] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6080.487657] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6080.492661] LustreError: Skipped 4 previous similar messages
[ 6080.523045] Lustre: Skipped 13 previous similar messages
[ 6097.898524] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0xe74272de273c5712 to 0xe74272de273caeb5
[ 6097.921931] Lustre: Skipped 5 previous similar messages
[ 6097.940244] Lustre: MGC192.168.202.154@tcp: Connection restored to  (at 0@lo)
[ 6097.964441] Lustre: Skipped 19 previous similar messages
[ 6099.130108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6099.134387] Lustre: Skipped 8 previous similar messages
[ 6099.288178] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6099.308785] Lustre: Skipped 2 previous similar messages
[ 6105.275736] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6114.078703] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6114.090864] Lustre: Skipped 7 previous similar messages
[ 6114.702880] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6114.720629] Lustre: Skipped 7 previous similar messages
[ 6114.817219] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5394 to 0x0:5409
[ 6114.818105] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5576 to 0x0:5601
[ 6120.864983] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6122.551891] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6129.628376] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6132.394230] Lustre: DEBUG MARKER: test_70b fail mds1 2 times
[ 6168.735125] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5433 to 0x0:5473
[ 6168.737120] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5624 to 0x0:5665
[ 6168.907841] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6179.244241] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6180.839200] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6214.202437] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 13:07:43 (1776186463)
[ 6340.115465] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6352.104287] Lustre: DEBUG MARKER: test_70c fail mds1 1 times
[ 6364.511193] Lustre: 2990:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186607/real 1776186607]  req@00000000bd9a0854 x1862460032806784/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776186614 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 6364.548352] Lustre: 2990:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
[ 6388.486763] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6395.627941] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6156 to 0x0:6177
[ 6395.671511] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6348 to 0x0:6369
[ 6403.494221] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6405.416767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6497.965750] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 13:12:27 (1776186747)
[ 6499.478484] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs
[ 6501.085650] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 13:12:30 (1776186750)
[ 6502.832350] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs
[ 6504.992520] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 13:12:33 (1776186753)
[ 6512.708373] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 6515.366626] Lustre: DEBUG MARKER: test_70f failing OST 1 times
[ 6518.549719] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6518.578218] LustreError: Skipped 5 previous similar messages
[ 6518.763689] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 6528.786172] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6528.816509] LustreError: Skipped 4 previous similar messages
[ 6538.935759] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6679 to 0x0:6721
[ 6543.118710] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6555.526749] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6557.336776] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6568.768870] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 6571.889711] Lustre: DEBUG MARKER: test_70f failing OST 2 times
[ 6574.804470] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6574.820126] LustreError: Skipped 3 previous similar messages
[ 6575.071619] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 6594.114132] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6679 to 0x0:6753
[ 6596.611263] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6607.033919] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6608.818988] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6619.915497] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 13:14:29 (1776186869)
[ 6621.358624] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs
[ 6622.743640] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:14:32 (1776186872)
[ 6626.765750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6665.901507] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6671.125517] Lustre: *** cfs_fail_loc=302, val=2147483648***
[ 6678.318351] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 6678.522924] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6487 to 0x0:6529
[ 6678.522924] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6755 to 0x0:6785
[ 6685.215478] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6687.045967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6696.283894] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:15:45 (1776186945)
[ 6700.277861] LustreError: 161783:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6700.285171] LustreError: 161783:0:(osd_handler.c:698:osd_ro()) Skipped 5 previous similar messages
[ 6701.097754] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6703.864913] Lustre: Failing over lustre-MDT0000
[ 6703.866717] Lustre: Skipped 5 previous similar messages
[ 6704.339890] Lustre: server umount lustre-MDT0000 complete
[ 6704.342846] Lustre: Skipped 5 previous similar messages
[ 6711.647627] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6711.663811] Lustre: Skipped 9 previous similar messages
[ 6711.775299] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6711.788535] LustreError: Skipped 3 previous similar messages
[ 6728.165420] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0xe74272de27450cca to 0xe74272de27451185
[ 6728.192253] Lustre: Skipped 3 previous similar messages
[ 6728.220495] Lustre: MGC192.168.202.154@tcp: Connection restored to  (at 0@lo)
[ 6728.222946] Lustre: Skipped 13 previous similar messages
[ 6728.890929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6728.895259] Lustre: Skipped 5 previous similar messages
[ 6728.965042] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6728.978618] Lustre: Skipped 5 previous similar messages
[ 6730.509670] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6730.516892] Lustre: Skipped 5 previous similar messages
[ 6730.555871] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 6730.560416] LustreError: 162441:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000002ff12aac x1862460031720832/t339302416387(339302416387) o101->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:411/0 lens 592/600 e 0 to 0 dl 1776186986 ref 1 fl Interpret:/4/0 rc 301/0 job:'multiop.0'
[ 6733.724752] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6737.715173] Lustre: lustre-MDT0000: Client e454ce83-8c28-4138-b5ac-db708cacd723 (at 192.168.202.54@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 6737.735270] Lustre: 162440:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000073a1995b x1862460031720832/t339302416387(339302416387) o101->e454ce83-8c28-4138-b5ac-db708cacd723@192.168.202.54@tcp:418/0 lens 592/3424 e 0 to 0 dl 1776186993 ref 1 fl Interpret:/6/0 rc 0/0 job:'multiop.0'
[ 6737.894296] Lustre: lustre-MDT0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted.
[ 6737.907862] Lustre: Skipped 5 previous similar messages
[ 6737.944556] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6755 to 0x0:6817
[ 6737.944851] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6561
[ 6742.762661] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6744.082250] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6752.194421] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:16:41 (1776187001)
[ 6758.368554] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 6758.374818] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6758.405331] LustreError: Skipped 7 previous similar messages
[ 6789.264851] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6593
[ 6794.139124] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6801.686768] Lustre: lustre-OST0000: Denying connection for new client d027a081-796c-4e6e-8797-45082fc2735b (at 192.168.202.54@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59
[ 6801.696805] Lustre: Skipped 11 previous similar messages
[ 6803.316052] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6755 to 0x0:6849
[ 6805.870375] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6818.121477] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:17:46 (1776187066)
[ 6819.404587] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs
[ 6821.648662] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:17:50 (1776187070)
[ 6823.963665] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs
[ 6826.386730] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:17:55 (1776187075)
[ 6828.855607] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs
[ 6830.709946] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:17:59 (1776187079)
[ 6832.322299] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs
[ 6834.085666] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:18:03 (1776187083)
[ 6835.624296] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs
[ 6837.317805] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:18:06 (1776187086)
[ 6838.795672] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs
[ 6840.549152] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:18:09 (1776187089)
[ 6841.984909] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs
[ 6844.159485] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:18:12 (1776187092)
[ 6845.822719] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs
[ 6847.302373] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:18:16 (1776187096)
[ 6848.658408] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs
[ 6850.539429] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:18:19 (1776187099)
[ 6852.176751] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs
[ 6853.888556] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:18:23 (1776187103)
[ 6855.828671] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs
[ 6858.235357] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:18:26 (1776187106)
[ 6860.781676] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs
[ 6862.813893] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:18:31 (1776187111)
[ 6865.328844] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs
[ 6867.917837] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:18:36 (1776187116)
[ 6869.598811] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs
[ 6871.880608] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:18:40 (1776187120)
[ 6873.620487] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs
[ 6874.619387] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:18:44 (1776187124)
[ 6875.785233] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs
[ 6877.515355] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:18:46 (1776187126)
[ 6879.560221] Lustre: 167064:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting d027a081-796c-4e6e-8797-45082fc2735b at adminstrative request
[ 6890.324224] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:18:59 (1776187139)
[ 6896.630503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6896.634092] Lustre: Skipped 1 previous similar message
[ 6931.195971] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6940.211911] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6901 to 0x0:6945
[ 6940.215576] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6645 to 0x0:6689
[ 6947.930919] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6949.906222] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6958.671153] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:20:07 (1776187207)
[ 6974.747153] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6974.765636] LustreError: Skipped 5 previous similar messages
[ 6992.149687] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7046 to 0x0:7073
[ 6995.540362] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7007.837266] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 7009.753899] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7020.817870] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:21:09 (1776187269)
[ 7038.636134] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6645 to 0x0:6721
[ 7038.643392] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7046 to 0x0:7105
[ 7038.918738] Lustre: 2990:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187281/real 1776187281]  req@00000000ffb3158c x1862460033107904/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776187288 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0'
[ 7038.946659] Lustre: 2990:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
[ 7045.374387] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7057.429437] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 13:21:45 (1776187305)
[ 7063.393948] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 7069.155737] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 7087.566723] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7107 to 0x0:7137
[ 7090.959414] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7100.912154] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 7103.269719] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7112.831627] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:22:41 (1776187361)
[ 7118.563140] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 7123.445223] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 7123.460748] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7123.476731] LustreError: Skipped 13 previous similar messages
[ 7142.779682] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7107 to 0x0:7169
[ 7146.179184] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7155.467902] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 7157.235572] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7164.885311] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:23:34 (1776187414)
[ 7168.512300] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 7172.302304] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7181.026524] LustreError: 5497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187431 with bad export cookie 16664007869691424745
[ 7220.213961] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7222.609961] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6763 to 0x0:6785
[ 7246.760869] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7252.126480] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7107 to 0x0:7201
[ 7264.203647] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:25:13 (1776187513)
[ 7315.863654] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7327.924572] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7330.133343] Lustre: lustre-OST0000: Denying connection for new client 02f12d5d-3200-4fc5-902f-0d65acd867d8 (at 192.168.202.54@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02
[ 7330.155724] Lustre: Skipped 1 previous similar message
[ 7350.544162] Lustre: lustre-OST0000: Denying connection for new client 02f12d5d-3200-4fc5-902f-0d65acd867d8 (at 192.168.202.54@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:42
[ 7350.575877] Lustre: Skipped 3 previous similar messages
[ 7386.384218] Lustre: lustre-OST0000: Denying connection for new client 02f12d5d-3200-4fc5-902f-0d65acd867d8 (at 192.168.202.54@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:06
[ 7386.401781] Lustre: Skipped 6 previous similar messages
[ 7393.000141] Lustre: lustre-OST0000: recovery is timed out, evict stale exports
[ 7393.008990] Lustre: lustre-OST0000: disconnecting 1 stale clients
[ 7393.077193] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.154@tcp (at 0@lo)
[ 7393.077897] Lustre: lustre-OST0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted.
[ 7393.083177] Lustre: Skipped 19 previous similar messages
[ 7393.088090] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7212 to 0x0:7233
[ 7393.089179] Lustre: Skipped 8 previous similar messages
[ 7399.084324] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 59 sec
[ 7412.754664] Lustre: DEBUG MARKER: free_before: 7518208 free_after: 7518208
[ 7418.508228] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:27:47 (1776187667)
[ 7511.455364] Lustre: DEBUG MARKER: replay-single test_90: @@@@@@ FAIL: wait_update OSTs up on MDT0000 failed