[ 3466.688426] LDISKFS-fs (dm-0): recovery complete
[ 3466.690971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3472.273959] LustreError: 112568:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 3472.277839] LustreError: 112568:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3472.282408] Lustre: 112601:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3472.285407] Lustre: 112601:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3472.289370] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 3472.294274] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 3472.310365] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0]
[ 3472.351070] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1603 to 0x0:1665
[ 3472.351070] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1665
[ 3476.298103] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3477.474263] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3496.304720] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:27:01 (1776184021)
[ 3526.380554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3528.671833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3528.678601] Lustre: Skipped 2 previous similar messages
[ 3551.785146] LDISKFS-fs (dm-0): recovery complete
[ 3551.787943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3563.661891] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3564.179406] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2066 to 0x0:2081
[ 3564.184085] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2066 to 0x0:2081
[ 3573.921426] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3575.906916] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3597.012369] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:28:42 (1776184122)
[ 3619.566169] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3627.441621] LustreError: 3362:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@000000002d8f0df0 x1862460356233984/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-0-0.0'
[ 3653.030451] LDISKFS-fs (dm-0): recovery complete
[ 3653.033936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3654.114498] Lustre: MGC192.168.202.149@tcp: Connection restored to  (at 0@lo)
[ 3654.124097] Lustre: Skipped 53 previous similar messages
[ 3658.826459] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3663.710869] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2482 to 0x0:2497
[ 3663.724026] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2482 to 0x0:2497
[ 3669.958541] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3671.928456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3694.183422] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 12:30:19 (1776184219)
[ 3696.405569] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 3699.345736] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 12:30:24 (1776184224)
[ 3701.670703] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 3702.795981] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3702.811720] Lustre: Skipped 37 previous similar messages
[ 3702.816819] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3702.828660] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 3702.861836] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2482 to 0x0:2529
[ 3709.984759] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 12:30:35 (1776184235)
[ 3735.819876] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3750.879784] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3750.893050] LustreError: Skipped 8 previous similar messages
[ 3774.274704] LDISKFS-fs (dm-2): recovery complete
[ 3774.280815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3774.509962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3774.518298] Lustre: Skipped 10 previous similar messages
[ 3776.718307] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:2945
[ 3776.731617] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65
[ 3780.058539] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3839.260437] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 12:32:44 (1776184364)
[ 3847.367227] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3850.369298] Lustre: Failing over lustre-MDT0000
[ 3850.372261] Lustre: Skipped 8 previous similar messages
[ 3850.526326] Lustre: server umount lustre-MDT0000 complete
[ 3850.531186] Lustre: Skipped 9 previous similar messages
[ 3874.635327] LDISKFS-fs (dm-0): recovery complete
[ 3874.637655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3877.673272] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 3877.682507] Lustre: Skipped 4 previous similar messages
[ 3880.037937] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3880.514174] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 3880.525593] Lustre: Skipped 4 previous similar messages
[ 3880.581234] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 3880.588497] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:2945
[ 3887.071659] LustreError: 122425:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 3887.078295] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3888.102525] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:2977
[ 3891.108663] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3892.880844] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3914.181770] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 12:33:59 (1776184439)
[ 3919.748774] LustreError: 6263:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3924.959887] LustreError: 6263:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3924.975780] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnecting
[ 3925.098333] LustreError: 44536:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3927.640851] LustreError: 6263:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3933.152086] LustreError: 6263:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3933.160340] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnecting
[ 3935.429753] LustreError: 6264:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3940.831771] LustreError: 6264:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3940.847248] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnecting
[ 3942.744153] LustreError: 6264:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3948.005372] LustreError: 6264:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3949.639992] LustreError: 6262:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3954.655267] LustreError: 6262:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3954.663590] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnecting
[ 3954.670126] Lustre: Skipped 1 previous similar message
[ 3963.660720] LustreError: 6262:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3963.664062] LustreError: 6262:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 3968.994466] LustreError: 6262:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3969.001325] LustreError: 6262:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 3976.159164] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnecting
[ 3976.162244] Lustre: Skipped 2 previous similar messages
[ 3984.609407] LustreError: 6262:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3984.615160] LustreError: 6262:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages
[ 3989.983365] LustreError: 6262:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3989.992566] LustreError: 6262:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 3998.973860] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 12:35:24 (1776184524)
[ 4000.975105] LustreError: 9709:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 4007.193862] Lustre: lustre-MDT0000: Export 00000000197261e0 already connecting from 192.168.202.49@tcp
[ 4009.413474] Lustre: lustre-MDT0000: Export 00000000197261e0 already connecting from 192.168.202.49@tcp
[ 4011.085716] Lustre: lustre-MDT0000: Export 00000000197261e0 already connecting from 192.168.202.49@tcp
[ 4014.477475] Lustre: lustre-MDT0000: Export 00000000197261e0 already connecting from 192.168.202.49@tcp
[ 4014.490698] Lustre: Skipped 2 previous similar messages
[ 4019.999461] Lustre: lustre-MDT0000: Export 00000000197261e0 already connecting from 192.168.202.49@tcp
[ 4020.008460] Lustre: Skipped 3 previous similar messages
[ 4025.881619] LustreError: 9709:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 4025.892738] Lustre: 9709:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/5s); client may timeout  req@00000000a89ebb25 x1862460338940608/t0(0) o38->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1776184547 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0'
[ 4027.180697] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnecting
[ 4027.190902] Lustre: Skipped 3 previous similar messages
[ 4031.538374] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:35:56 (1776184556)
[ 4040.545219] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4045.815591] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4045.827310] LustreError: Skipped 238 previous similar messages
[ 4052.895488] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184572/real 1776184572]  req@000000000534b08f x1862460356461504/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184579 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 4052.929120] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 4052.940640] LustreError: 166-1: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4052.954923] LustreError: Skipped 7 previous similar messages
[ 4058.622713] LDISKFS-fs (dm-0): recovery complete
[ 4058.630888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4059.115056] Lustre: Evicted from MGS (at 192.168.202.149@tcp) after server handle changed from 0xe5f53f049ede4d8e to 0xe5f53f049ede572e
[ 4059.129866] Lustre: Skipped 6 previous similar messages
[ 4059.294603] Lustre: *** cfs_fail_loc=712, val=0***
[ 4059.307905] LustreError: 8232:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@00000000efcd9505 x1862460356466176/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 4059.329880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4059.391207] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4059.394933] LustreError: 127105:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 4059.395276] Lustre: Skipped 7 previous similar messages
[ 4059.408705] LustreError: 127105:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 4059.414782] Lustre: 127139:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 4059.420404] Lustre: 127139:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 4059.426440] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 4059.432182] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0]
[ 4059.448499] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0]
[ 4059.490624] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:2977
[ 4059.500876] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:3009
[ 4064.399327] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4064.739269] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4101.440741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4115.008229] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:3041
[ 4115.009058] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:3009
[ 4115.900614] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4127.230482] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4129.651608] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4139.889789] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:37:44 (1776184664)
[ 4140.013915] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnecting
[ 4149.447907] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:37:54 (1776184674)
[ 4150.425759] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 4150.432316] LustreError: 6272:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000005d093b38 x1862460338967936/t0(0) o700->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:372/0 lens 264/248 e 0 to 0 dl 1776184682 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 4182.987180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4195.626742] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4196.391647] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3011 to 0x0:3041
[ 4196.392081] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3043 to 0x0:3073
[ 4205.864676] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4207.673385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4218.175389] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:39:03 (1776184743)
[ 4241.059758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4243.423426] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3084 to 0x0:3105
[ 4243.429694] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97
[ 4246.115371] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4256.656266] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4258.593823] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4332.319511] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:40:57 (1776184857)
[ 4340.534414] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4343.786923] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4343.813389] Lustre: Skipped 22 previous similar messages
[ 4371.267771] LDISKFS-fs (dm-0): recovery complete
[ 4371.281174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4373.475144] Lustre: MGC192.168.202.149@tcp: Connection restored to 192.168.202.149@tcp (at 0@lo)
[ 4373.492358] Lustre: Skipped 31 previous similar messages
[ 4373.787561] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (not set up)
[ 4379.911896] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137
[ 4379.913393] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3062 to 0x0:3105
[ 4381.556592] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4457.407621] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:43:01 (1776184981)
[ 4459.950982] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 4459.956407] Lustre: Skipped 2 previous similar messages
[ 4459.960836] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169
[ 4461.034589] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3201
[ 4474.455304] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:43:19 (1776184999)
[ 4479.050030] Lustre: Failing over lustre-MDT0000
[ 4479.057632] Lustre: Skipped 5 previous similar messages
[ 4479.298603] Lustre: server umount lustre-MDT0000 complete
[ 4479.300750] Lustre: Skipped 5 previous similar messages
[ 4501.011196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4505.342440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4505.352654] Lustre: Skipped 6 previous similar messages
[ 4505.881133] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4505.890167] Lustre: Skipped 4 previous similar messages
[ 4510.708342] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4510.714883] LustreError: 135260:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000002492d72 x1862460339024192/t0(0) o101->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:7/0 lens 328/344 e 0 to 0 dl 1776185072 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 4511.869560] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4548.384647] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnected, waiting for 2 clients in recovery for 0:56
[ 4548.464689] Lustre: lustre-MDT0000: Recovery over after 0:43, of 2 clients 2 recovered and 0 were evicted.
[ 4548.496283] Lustre: Skipped 4 previous similar messages
[ 4548.614687] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3233
[ 4548.618631] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3116 to 0x0:3137
[ 4555.936399] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4557.542910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4567.588918] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:44:52 (1776185092)
[ 4569.930526] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4579.236887] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4608.353540] LDISKFS-fs (dm-0): recovery complete
[ 4608.361313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4627.948495] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4629.156064] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3169
[ 4629.165612] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3265
[ 4639.826214] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4641.879861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4652.541382] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:46:17 (1776185177)
[ 4654.428063] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4665.257775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4669.231409] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4669.248048] LustreError: Skipped 256 previous similar messages
[ 4677.599401] Lustre: 3363:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185196/real 1776185196]  req@000000008c8735ea x1862460356631296/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185203 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 4677.621154] Lustre: 3363:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 4677.636592] LustreError: 166-1: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4677.649663] LustreError: Skipped 5 previous similar messages
[ 4691.574224] LDISKFS-fs (dm-0): recovery complete
[ 4691.581633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4693.989581] Lustre: Evicted from MGS (at 192.168.202.149@tcp) after server handle changed from 0xe5f53f049ede8858 to 0xe5f53f049ede8ecc
[ 4693.998228] Lustre: Skipped 5 previous similar messages
[ 4694.173978] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4694.181909] Lustre: Skipped 8 previous similar messages
[ 4698.123338] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4699.675588] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3297
[ 4699.675595] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3201
[ 4707.990798] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4709.630590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4719.400768] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:47:24 (1776185244)
[ 4720.580558] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4722.620976] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4730.245639] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4731.165755] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnecting
[ 4731.182408] Lustre: Skipped 2 previous similar messages
[ 4735.459698] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4735.465438] LustreError: Skipped 2 previous similar messages
[ 4756.733693] LDISKFS-fs (dm-0): recovery complete
[ 4756.737898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4764.846258] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3329
[ 4764.850228] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233
[ 4765.599234] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4777.652758] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:48:22 (1776185302)
[ 4780.023432] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4780.025917] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 4780.031534] LustreError: 6266:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000069cb412e x1862460339057280/t261993005072(0) o35->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:247/0 lens 392/456 e 0 to 0 dl 1776185312 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4803.560772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4815.158676] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4815.399350] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 4815.402249] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3265
[ 4815.424433] Lustre: 6266:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000ca8436b9 x1862460339057280/t261993005072(0) o35->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:282/0 lens 392/456 e 0 to 0 dl 1776185347 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4826.038847] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4828.192830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4838.106855] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:49:23 (1776185363)
[ 4839.314461] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4839.316632] LustreError: 9617:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000b6da7c64 x1862460339066112/t266287972368(0) o36->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:340/0 lens 504/448 e 0 to 0 dl 1776185405 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4848.113277] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4873.794022] LDISKFS-fs (dm-0): recovery complete
[ 4873.797300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4880.317025] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4880.419918] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3393
[ 4880.420041] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3297
[ 4880.493366] Lustre: 9709:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000004c4130f6 x1862460339066112/t266287972368(0) o36->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:381/0 lens 504/448 e 0 to 0 dl 1776185446 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4890.529534] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4892.226784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4901.772993] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:50:26 (1776185426)
[ 4902.917697] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4902.920922] LustreError: 6263:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000008f8527d x1862460339074944/t270582939664(0) o36->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:404/0 lens 504/448 e 0 to 0 dl 1776185469 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4904.838445] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4911.910985] Lustre: 6262:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000039501694 x1862460339074944/t270582939664(0) o36->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:413/0 lens 504/448 e 0 to 0 dl 1776185478 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4911.927919] Lustre: 6262:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 4912.086099] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4939.013707] LDISKFS-fs (dm-0): recovery complete
[ 4939.018281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4944.379679] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4945.424599] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3425
[ 4945.424666] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4958.002975] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:51:23 (1776185483)
[ 4959.119116] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4959.122213] Lustre: Skipped 1 previous similar message
[ 4959.125100] LustreError: 6262:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000004486c02c x1862460339082880/t274877906960(0) o36->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:460/0 lens 504/448 e 0 to 0 dl 1776185525 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4959.149326] LustreError: 6262:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message
[ 4961.297634] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4968.737653] Lustre: 6263:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000f3fa5a7f x1862460339082880/t274877906960(0) o36->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:470/0 lens 504/448 e 0 to 0 dl 1776185535 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4970.318424] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4976.096314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4976.109441] Lustre: Skipped 34 previous similar messages
[ 4996.009082] LDISKFS-fs (dm-0): recovery complete
[ 4996.013407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5000.688040] Lustre: MGC192.168.202.149@tcp: Connection restored to  (at 0@lo)
[ 5000.696308] Lustre: Skipped 41 previous similar messages
[ 5004.349341] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5006.405097] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 5006.412829] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3457
[ 5013.323531] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:52:18 (1776185538)
[ 5014.373396] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 5016.192881] Lustre: *** cfs_fail_loc=13b, val=315***
[ 5016.199641] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 5016.209456] LustreError: 6266:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000210e09ee x1862460339090752/t279172874256(0) o35->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:483/0 lens 392/456 e 0 to 0 dl 1776185548 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 5022.516674] Lustre: 6266:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000066d78841 x1862460339090752/t279172874256(0) o35->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:489/0 lens 392/456 e 0 to 0 dl 1776185554 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 5023.106161] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5047.878828] LDISKFS-fs (dm-0): recovery complete
[ 5047.882264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5055.601529] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5057.109056] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3393
[ 5057.113751] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3489
[ 5068.582148] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:53:13 (1776185593)
[ 5069.797106] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 5069.821181] LustreError: 6262:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000005a0cc70e x1862460339097152/t283467841549(0) o101->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:571/0 lens 664/600 e 0 to 0 dl 1776185636 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0'
[ 5113.667431] Lustre: 6264:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000078e8e255 x1862460339097152/t283467841549(0) o101->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:614/0 lens 664/3424 e 0 to 0 dl 1776185679 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 5121.991425] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:54:06 (1776185646)
[ 5131.114135] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5133.531684] Lustre: Failing over lustre-MDT0000
[ 5133.534169] Lustre: Skipped 8 previous similar messages
[ 5133.659589] Lustre: server umount lustre-MDT0000 complete
[ 5133.662593] Lustre: Skipped 8 previous similar messages
[ 5159.660597] LDISKFS-fs (dm-0): recovery complete
[ 5159.666229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5163.257981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5163.261290] Lustre: Skipped 8 previous similar messages
[ 5167.813845] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5167.898711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5167.912374] Lustre: Skipped 8 previous similar messages
[ 5168.664291] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 5168.668977] Lustre: Skipped 8 previous similar messages
[ 5168.691539] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3425
[ 5168.696662] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521
[ 5178.892983] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5181.123278] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5200.972608] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:55:26 (1776185726)
[ 5210.688103] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5238.054196] LDISKFS-fs (dm-0): recovery complete
[ 5238.060331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5244.026769] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5245.074210] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3553
[ 5245.077782] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3427 to 0x0:3457
[ 5254.272051] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5255.991788] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5261.200033] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 5274.855784] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:56:39 (1776185799)
[ 5332.737746] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5337.059923] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5337.092660] LustreError: Skipped 346 previous similar messages
[ 5344.224085] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185863/real 1776185863]  req@000000003eccc121 x1862460356820800/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185870 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 5344.255985] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 5344.266349] LustreError: 166-1: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5344.276391] LustreError: Skipped 8 previous similar messages
[ 5359.146203] LDISKFS-fs (dm-0): recovery complete
[ 5359.152908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5361.632991] Lustre: Evicted from MGS (at 192.168.202.149@tcp) after server handle changed from 0xe5f53f049edec115 to 0xe5f53f049ee01c21
[ 5361.641235] Lustre: Skipped 8 previous similar messages
[ 5361.875495] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5361.883499] Lustre: Skipped 8 previous similar messages
[ 5365.699878] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5367.634370] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4708 to 0x0:4737
[ 5367.643436] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4804 to 0x0:4833
[ 5374.557590] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5376.252909] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5445.745712] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:59:31 (1776185971)
[ 5454.159375] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5477.460451] LDISKFS-fs (dm-0): recovery complete
[ 5477.462520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5487.406574] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5489.213434] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4804 to 0x0:4865
[ 5489.213715] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4739 to 0x0:4769
[ 5495.940954] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5497.564313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5508.963333] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 5511.197878] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 5519.188514] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 13:00:44 (1776186044)
[ 5528.249264] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 5569.304830] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnecting
[ 5569.308671] Lustre: Skipped 4 previous similar messages
[ 5573.705393] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 5573.706925] Lustre: Skipped 1 previous similar message
[ 5573.708225] LustreError: 9709:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000284b0598 x1862460340588480/t300647710728(0) o36->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:320/0 lens 66040/440 e 0 to 0 dl 1776186140 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 5617.446199] Lustre: 6263:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000736a04e5 x1862460340588480/t300647710728(0) o36->a6de8253-4f8a-49e7-8045-8b0d1751c978@192.168.202.49@tcp:363/0 lens 66040/440 e 0 to 0 dl 1776186183 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 5632.469621] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 5634.543094] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 13:02:39 (1776186159)
[ 5651.498597] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5658.080883] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5658.085579] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5658.116304] Lustre: Skipped 22 previous similar messages
[ 5658.129067] LustreError: Skipped 3 previous similar messages
[ 5680.148612] LDISKFS-fs (dm-0): recovery complete
[ 5680.157536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5680.355170] Lustre: MGC192.168.202.149@tcp: Connection restored to  (at 0@lo)
[ 5680.364060] Lustre: Skipped 29 previous similar messages
[ 5686.006970] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5687.493160] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4967 to 0x0:4993
[ 5687.494728] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4870 to 0x0:4897
[ 5695.746425] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5697.329279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5707.880848] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 13:03:52 (1776186232)
[ 5733.354603] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5748.060952] Lustre: Failing over lustre-OST0000
[ 5748.064829] Lustre: Skipped 4 previous similar messages
[ 5748.091030] Lustre: server umount lustre-OST0000 complete
[ 5748.092987] Lustre: Skipped 4 previous similar messages
[ 5773.870066] LDISKFS-fs (dm-2): recovery complete
[ 5773.873588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5773.999285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5774.004829] Lustre: Skipped 4 previous similar messages
[ 5775.097867] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5775.112413] Lustre: Skipped 4 previous similar messages
[ 5775.645623] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 5775.656724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5409
[ 5775.657390] Lustre: Skipped 4 previous similar messages
[ 5775.684444] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129
[ 5778.426974] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5813.635476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5815.689531] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161
[ 5815.725332] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5441
[ 5820.900919] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5832.787812] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5834.982670] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5876.979160] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 13:06:42 (1776186402)
[ 5900.585441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5914.707164] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5298 to 0x0:5313
[ 5914.707300] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5473
[ 5915.931923] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5940.192167] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5940.220496] LustreError: Skipped 186 previous similar messages
[ 5956.368675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5965.403656] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5505
[ 5965.411882] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5298 to 0x0:5345
[ 5967.390720] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5979.058794] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5981.286615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5991.642312] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 13:08:36 (1776186516)
[ 6027.581301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6027.957081] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 6027.961049] Lustre: Skipped 6 previous similar messages
[ 6030.040786] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:193
[ 6030.054414] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5507 to 0x0:5537
[ 6033.460704] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6045.764386] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6047.893573] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6059.236716] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 13:09:44 (1776186584)
[ 6071.263386] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186590/real 1776186590]  req@00000000fabcd514 x1862460357400128/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776186597 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0'
[ 6071.299638] Lustre: 3362:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6071.317969] LustreError: 166-1: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6071.342334] LustreError: Skipped 4 previous similar messages
[ 6073.601571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6073.750787] Lustre: *** cfs_fail_loc=605, val=0***
[ 6073.752589] LustreError: 172500:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=00000000f6dfead1 failed: rc = -95
[ 6073.767650] LustreError: 172500:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 6073.773901] LustreError: 172500:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 6073.785159] LustreError: 172500:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 6073.793456] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 6073.800715] LustreError: 172500:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 6073.812811] LustreError: 172500:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 6082.379340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6088.676673] Lustre: Evicted from MGS (at 192.168.202.149@tcp) after server handle changed from 0xe5f53f049ee41264 to 0xe5f53f049ee419b8
[ 6088.704261] Lustre: Skipped 4 previous similar messages
[ 6094.403821] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5347 to 0x0:5377
[ 6094.406192] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5507 to 0x0:5569
[ 6094.663862] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6107.388156] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 13:10:31 (1776186631)
[ 6117.480187] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6145.770944] LDISKFS-fs (dm-0): recovery complete
[ 6145.774812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6149.936487] Lustre: *** cfs_fail_loc=707, val=0***
[ 6154.793234] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6190.934672] Lustre: lustre-MDT0000: Client a6de8253-4f8a-49e7-8045-8b0d1751c978 (at 192.168.202.49@tcp) reconnected, waiting for 2 clients in recovery for 0:58
[ 6191.577924] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5390 to 0x0:5409
[ 6191.578394] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5583 to 0x0:5601
[ 6198.867981] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6201.258837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6212.445896] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 13:12:17 (1776186737)
[ 6243.297570] LustreError: 10847:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 6247.305879] Lustre: DEBUG MARKER: replay-single test_65a: @@@@@@ FAIL: No early reply
[ 6249.320089] LustreError: 10847:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake