[ 3334.442454] LDISKFS-fs (dm-0): recovery complete
[ 3334.446489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3335.955029] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (not set up)
[ 3336.223450] LustreError: 112548:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 3336.230889] LustreError: 112548:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3336.236848] Lustre: 112581:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3336.242745] Lustre: 112581:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3336.249265] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 3336.256885] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 3336.279460] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0]
[ 3336.385328] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1697
[ 3336.415930] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1571 to 0x0:1633
[ 3341.281781] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3342.777199] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3365.709688] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:52:22 (1776181942)
[ 3390.104935] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3415.618242] LDISKFS-fs (dm-0): recovery complete
[ 3415.626964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3417.387263] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3417.394921] Lustre: Skipped 20 previous similar messages
[ 3421.144516] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3422.775816] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113
[ 3422.776020] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2034 to 0x0:2049
[ 3429.223616] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3430.661060] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3448.554915] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:53:44 (1776182024)
[ 3464.294503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3471.188576] LustreError: 3360:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@00000000ff294737 x1862458318820160/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-1-0.0'
[ 3497.625073] LDISKFS-fs (dm-0): recovery complete
[ 3497.630816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3518.529922] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3522.563385] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to  (at 0@lo)
[ 3522.582307] Lustre: Skipped 53 previous similar messages
[ 3522.636135] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2465
[ 3522.636451] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2514 to 0x0:2529
[ 3528.888699] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3530.476778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3547.743473] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 11:55:24 (1776182124)
[ 3548.977781] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 3550.561465] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:55:27 (1776182127)
[ 3552.419192] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 3553.219073] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3553.227222] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 3553.249407] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2497
[ 3559.347965] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:55:35 (1776182135)
[ 3578.205838] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3607.260791] LDISKFS-fs (dm-2): recovery complete
[ 3607.263807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3607.384133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3607.394886] Lustre: Skipped 9 previous similar messages
[ 3608.824416] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65
[ 3608.828704] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:2977
[ 3611.638557] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3666.959801] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:57:23 (1776182243)
[ 3673.464592] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3676.142978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3700.712439] LDISKFS-fs (dm-0): recovery complete
[ 3700.715931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3709.300378] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3710.049287] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 3718.478251] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3719.969496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3738.099442] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:58:34 (1776182314)
[ 3743.231037] LustreError: 8252:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3748.319220] LustreError: 8252:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3748.327609] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnecting
[ 3748.377644] LustreError: 44538:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3749.793692] LustreError: 8252:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3754.975222] LustreError: 8252:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3754.987148] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnecting
[ 3756.963265] LustreError: 8252:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3757.024277] LustreError: 122333:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 3757.027301] LustreError: 44534:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3757.058978] LustreError: 44534:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message
[ 3757.062133] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3757.064591] LustreError: 8252:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=4908
[ 3758.060447] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3009
[ 3764.191234] LustreError: 8252:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3764.195397] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnecting
[ 3764.200942] Lustre: Skipped 1 previous similar message
[ 3766.071689] LustreError: 8252:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3766.082472] LustreError: 8252:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 3771.359202] LustreError: 8252:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3773.110733] LustreError: 8252:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3778.527346] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnecting
[ 3778.542423] Lustre: Skipped 1 previous similar message
[ 3785.471448] LustreError: 6260:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3785.483979] LustreError: 8252:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=1
[ 3785.493439] LustreError: 8252:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 3787.376523] LustreError: 6261:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3787.382867] LustreError: 6261:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 3800.039435] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnecting
[ 3800.051955] Lustre: Skipped 3 previous similar messages
[ 3807.203584] LustreError: 6260:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3807.214864] LustreError: 6260:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 3807.228302] LustreError: 6260:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3814.684975] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 11:59:51 (1776182391)
[ 3816.033857] LustreError: 14685:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 3821.307827] Lustre: lustre-MDT0000: Export 00000000e1cae628 already connecting from 192.168.202.41@tcp
[ 3823.146004] Lustre: lustre-MDT0000: Export 00000000e1cae628 already connecting from 192.168.202.41@tcp
[ 3825.040711] Lustre: lustre-MDT0000: Export 00000000e1cae628 already connecting from 192.168.202.41@tcp
[ 3828.450633] Lustre: lustre-MDT0000: Export 00000000e1cae628 already connecting from 192.168.202.41@tcp
[ 3828.466089] Lustre: Skipped 2 previous similar messages
[ 3833.845292] Lustre: lustre-MDT0000: Export 00000000e1cae628 already connecting from 192.168.202.41@tcp
[ 3833.862160] Lustre: Skipped 3 previous similar messages
[ 3840.024078] LustreError: 14685:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 3840.027203] Lustre: 14685:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/4s); client may timeout  req@000000003c7e7255 x1862458304694720/t0(0) o38->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:0/0 lens 520/416 e 0 to 0 dl 1776182413 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0'
[ 3841.797331] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnecting
[ 3841.806695] Lustre: Skipped 3 previous similar messages
[ 3844.929789] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:00:21 (1776182421)
[ 3851.571739] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3855.287767] Lustre: Failing over lustre-MDT0000
[ 3855.296892] Lustre: Skipped 7 previous similar messages
[ 3855.504654] Lustre: server umount lustre-MDT0000 complete
[ 3855.508859] Lustre: Skipped 7 previous similar messages
[ 3858.400830] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3858.417018] LustreError: Skipped 9 previous similar messages
[ 3858.421462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3858.435558] Lustre: Skipped 31 previous similar messages
[ 3866.249506] LDISKFS-fs (dm-0): recovery complete
[ 3866.255816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3866.340249] LustreError: 166-1: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3866.345150] LustreError: Skipped 6 previous similar messages
[ 3866.349476] Lustre: Evicted from MGS (at 192.168.202.141@tcp) after server handle changed from 0xeaa259be4fcdd173 to 0xeaa259be4fcddaaa
[ 3866.359558] Lustre: Skipped 6 previous similar messages
[ 3866.474343] Lustre: *** cfs_fail_loc=712, val=0***
[ 3866.476550] LustreError: 44542:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@00000000d3c81eeb x1862458319039552/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 3866.498565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3866.551782] LustreError: 127006:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 3866.554753] LustreError: 127006:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3866.557738] Lustre: 127040:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3866.560700] Lustre: 127040:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3866.563128] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 3866.566704] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0]
[ 3866.575874] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0]
[ 3866.608253] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3041
[ 3866.614636] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2913
[ 3870.258286] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3871.712394] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3887.072872] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3887.079346] LustreError: Skipped 224 previous similar messages
[ 3894.242910] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182464/real 1776182464]  req@000000001f18b262 x1862458319048000/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182471 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 3894.272041] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3902.932457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3912.450941] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 3912.455731] Lustre: Skipped 4 previous similar messages
[ 3916.080442] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3916.311743] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 3916.334441] Lustre: Skipped 4 previous similar messages
[ 3916.423179] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3073
[ 3916.423287] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2945
[ 3927.080904] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3928.250378] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3935.685087] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:01:52 (1776182512)
[ 3935.764370] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnecting
[ 3944.451182] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:02:00 (1776182520)
[ 3945.573603] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 3945.578247] LustreError: 6269:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000769e3092 x1862458304721152/t0(0) o700->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:484/0 lens 264/248 e 0 to 0 dl 1776182529 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 3976.871988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3987.641695] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3988.039658] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2947 to 0x0:2977
[ 3988.044092] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3075 to 0x0:3105
[ 3998.484434] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4000.813831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4011.528838] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:03:07 (1776182587)
[ 4032.905595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4033.062938] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 4033.067080] Lustre: Skipped 8 previous similar messages
[ 4034.434701] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97
[ 4034.442586] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137
[ 4038.190802] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4046.981877] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4048.289967] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4119.491744] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:04:55 (1776182695)
[ 4127.434283] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4155.675561] LDISKFS-fs (dm-0): recovery complete
[ 4155.693150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4170.216905] Lustre: MGC192.168.202.141@tcp: Connection restored to 192.168.202.141@tcp (at 0@lo)
[ 4170.238987] Lustre: Skipped 27 previous similar messages
[ 4175.181741] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4176.457547] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169
[ 4176.459980] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2998 to 0x0:3041
[ 4250.838094] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:07:06 (1776182826)
[ 4252.857779] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 4252.862172] Lustre: Skipped 2 previous similar messages
[ 4252.866632] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201
[ 4253.465492] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3233
[ 4265.885303] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:07:21 (1776182841)
[ 4290.629181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4296.996437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4297.005714] Lustre: Skipped 6 previous similar messages
[ 4302.323077] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4302.331066] LustreError: 135169:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000a62c2c9f x1862458304780928/t0(0) o101->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:118/0 lens 328/344 e 0 to 0 dl 1776182918 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 4302.486988] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4342.023526] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnected, waiting for 2 clients in recovery for 0:56
[ 4342.154847] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3265
[ 4342.155807] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3052 to 0x0:3073
[ 4350.065810] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4351.709343] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4361.624211] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:08:57 (1776182937)
[ 4364.140440] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4373.200995] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4399.279287] LDISKFS-fs (dm-0): recovery complete
[ 4399.283197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4407.663555] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4408.906906] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3297
[ 4418.030887] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4419.984312] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4430.641962] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:10:06 (1776183006)
[ 4431.994513] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4442.406676] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4468.927907] LDISKFS-fs (dm-0): recovery complete
[ 4468.935633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4469.104881] LustreError: 11-0: MGC192.168.202.141@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107
[ 4469.108026] LustreError: 166-1: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4469.115920] LustreError: Skipped 5 previous similar messages
[ 4469.127329] LustreError: Skipped 5 previous similar messages
[ 4469.139322] LustreError: 3358:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting
[ 4469.144102] Lustre: Evicted from MGS (at 192.168.202.141@tcp) after server handle changed from 0xeaa259be4fce0b64 to 0xeaa259be4fce10e3
[ 4469.172097] Lustre: Skipped 5 previous similar messages
[ 4474.947505] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3075 to 0x0:3105
[ 4474.959101] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4476.799736] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4487.668031] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4489.962031] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4499.763566] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:11:15 (1776183075)
[ 4501.466934] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4503.471668] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4510.997914] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnecting
[ 4511.011687] Lustre: Skipped 2 previous similar messages
[ 4511.688362] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4513.784271] Lustre: Failing over lustre-MDT0000
[ 4513.786456] Lustre: Skipped 7 previous similar messages
[ 4513.913475] Lustre: server umount lustre-MDT0000 complete
[ 4513.916246] Lustre: Skipped 7 previous similar messages
[ 4515.823476] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4515.833634] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4515.847989] Lustre: Skipped 33 previous similar messages
[ 4515.878210] LustreError: Skipped 244 previous similar messages
[ 4522.975304] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183093/real 1776183093]  req@00000000009d4a64 x1862458319216256/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183100 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 4523.001684] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 4536.862819] LDISKFS-fs (dm-0): recovery complete
[ 4536.866034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4541.447243] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4541.452732] Lustre: Skipped 6 previous similar messages
[ 4545.513192] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4546.121883] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 4546.126707] Lustre: Skipped 6 previous similar messages
[ 4546.153468] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3361
[ 4546.153616] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137
[ 4557.697854] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:12:14 (1776183134)
[ 4559.559546] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4559.560935] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 4559.562272] LustreError: 6262:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000773c861f x1862458304814016/t261993005072(0) o35->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:343/0 lens 392/456 e 0 to 0 dl 1776183143 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4582.189458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4586.941861] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4587.603101] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3393
[ 4587.603730] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3169
[ 4587.607923] Lustre: 6262:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000004a4bd47c x1862458304814016/t261993005072(0) o35->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:371/0 lens 392/456 e 0 to 0 dl 1776183171 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4596.219311] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4598.040620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4606.711767] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:13:03 (1776183183)
[ 4607.726593] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4607.742667] LustreError: 8252:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000001eb4bc1a x1862458304821760/t266287972368(0) o36->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:426/0 lens 504/448 e 0 to 0 dl 1776183226 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4617.406821] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4646.132358] LDISKFS-fs (dm-0): recovery complete
[ 4646.136808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4647.298275] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4647.315389] Lustre: Skipped 6 previous similar messages
[ 4651.932883] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4652.624678] Lustre: 127818:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000017416cb x1862458304821760/t266287972368(0) o36->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:471/0 lens 504/448 e 0 to 0 dl 1776183271 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4652.656236] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3201
[ 4662.922713] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4665.201906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4674.340348] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:14:10 (1776183250)
[ 4675.471304] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4675.474296] LustreError: 127818:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000007df8afc5 x1862458304831296/t270582939664(0) o36->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:494/0 lens 504/448 e 0 to 0 dl 1776183294 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4677.411191] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4684.553837] Lustre: 6263:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000dcb4ee7a x1862458304831424/t270582939665(0) o35->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:468/0 lens 392/456 e 0 to 0 dl 1776183268 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4684.575214] Lustre: 6263:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 4685.222164] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4711.708811] LDISKFS-fs (dm-0): recovery complete
[ 4711.710709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4716.137489] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4717.594455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3425
[ 4717.597330] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233
[ 4727.041089] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:15:03 (1776183303)
[ 4727.946627] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4727.948819] Lustre: Skipped 1 previous similar message
[ 4727.955743] LustreError: 8252:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000a7a1edb6 x1862458304839296/t274877906960(0) o36->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:546/0 lens 504/448 e 0 to 0 dl 1776183346 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4727.976484] LustreError: 8252:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message
[ 4729.597657] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4736.556752] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4736.773080] Lustre: 6261:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000008ab13563 x1862458304839296/t274877906960(0) o36->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:555/0 lens 504/448 e 0 to 0 dl 1776183355 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4759.468312] LDISKFS-fs (dm-0): recovery complete
[ 4759.478282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4770.310685] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4772.330481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to  (at 0@lo)
[ 4772.348895] Lustre: Skipped 42 previous similar messages
[ 4772.432187] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3265
[ 4772.433678] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3457
[ 4779.918083] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:15:56 (1776183356)
[ 4780.910951] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4782.459744] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4782.461402] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 4782.464354] LustreError: 6262:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000863a4ef3 x1862458304847488/t279172874256(0) o35->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:566/0 lens 392/456 e 0 to 0 dl 1776183366 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4790.509860] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4790.541529] Lustre: 6262:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000a2547291 x1862458304847488/t279172874256(0) o35->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:574/0 lens 392/456 e 0 to 0 dl 1776183374 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4818.450470] LDISKFS-fs (dm-0): recovery complete
[ 4818.454378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4826.224882] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4827.768043] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3297
[ 4836.435304] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:16:53 (1776183413)
[ 4837.278910] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 4837.287111] LustreError: 6260:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000008a13c800 x1862458304854464/t283467841549(0) o101->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:654/0 lens 664/600 e 0 to 0 dl 1776183454 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0'
[ 4878.612730] Lustre: 8252:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000b4031e3e x1862458304854464/t283467841549(0) o101->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:696/0 lens 664/3424 e 0 to 0 dl 1776183496 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 4885.381945] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:17:41 (1776183461)
[ 4892.001409] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4917.500609] LDISKFS-fs (dm-0): recovery complete
[ 4917.507993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4917.957353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4917.962984] Lustre: Skipped 8 previous similar messages
[ 4922.576766] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4923.522337] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3489
[ 4923.523028] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4932.240755] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4933.960396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4953.087986] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:18:49 (1776183529)
[ 4961.181921] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4985.742986] LDISKFS-fs (dm-0): recovery complete
[ 4985.745902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4991.759792] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4993.647626] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 4993.648649] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3521
[ 5001.416606] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5003.012843] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5009.175289] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 5020.659592] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:19:57 (1776183597)
[ 5058.461087] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5060.072984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5060.092322] Lustre: Skipped 3 previous similar messages
[ 5072.351279] LustreError: 166-1: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5072.360976] LustreError: Skipped 8 previous similar messages
[ 5081.659310] LDISKFS-fs (dm-0): recovery complete
[ 5081.662416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5088.747571] Lustre: Evicted from MGS (at 192.168.202.141@tcp) after server handle changed from 0xeaa259be4fce4126 to 0xeaa259be4fcf9c32
[ 5088.754328] Lustre: Skipped 8 previous similar messages
[ 5094.087941] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5094.792870] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4772 to 0x0:4801
[ 5094.794758] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4612 to 0x0:4641
[ 5104.587849] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5106.253158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5162.954349] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:22:19 (1776183739)
[ 5170.667746] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5172.436741] Lustre: Failing over lustre-MDT0000
[ 5172.440214] Lustre: Skipped 8 previous similar messages
[ 5172.612225] Lustre: server umount lustre-MDT0000 complete
[ 5172.615283] Lustre: Skipped 8 previous similar messages
[ 5174.558729] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5174.577150] LustreError: Skipped 326 previous similar messages
[ 5176.291547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5176.315263] Lustre: Skipped 35 previous similar messages
[ 5182.431178] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183753/real 1776183753]  req@00000000d8a493c0 x1862458319724928/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183760 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 5182.455170] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5194.853284] LDISKFS-fs (dm-0): recovery complete
[ 5194.860712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5200.821358] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 5200.832447] Lustre: Skipped 8 previous similar messages
[ 5203.952511] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5205.643068] Lustre: lustre-MDT0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted.
[ 5205.660821] Lustre: Skipped 8 previous similar messages
[ 5205.716495] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4643 to 0x0:4673
[ 5205.720272] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4772 to 0x0:4833
[ 5214.361816] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5216.467342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5228.093944] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 5229.464515] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 5234.954877] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:23:31 (1776183811)
[ 5242.226199] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 5284.096914] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnecting
[ 5284.111096] Lustre: Skipped 4 previous similar messages
[ 5285.997437] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 5286.008915] Lustre: Skipped 1 previous similar message
[ 5286.015689] LustreError: 6261:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000d0b57994 x1862458306345216/t300647710728(0) o36->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:348/0 lens 66040/440 e 0 to 0 dl 1776183903 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 5329.168051] Lustre: 8252:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000a8e76e2b x1862458306345216/t300647710728(0) o36->5bb5d51e-f557-4e34-b837-13b1acb38e11@192.168.202.41@tcp:391/0 lens 66040/440 e 0 to 0 dl 1776183946 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 5338.155147] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 5339.431404] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:25:16 (1776183916)
[ 5351.367746] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5353.955337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5353.961539] Lustre: Skipped 3 previous similar messages
[ 5380.231873] LDISKFS-fs (dm-0): recovery complete
[ 5380.234986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5384.171397] Lustre: MGC192.168.202.141@tcp: Connection restored to  (at 0@lo)
[ 5384.185229] Lustre: Skipped 28 previous similar messages
[ 5384.525468] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5384.540122] Lustre: Skipped 7 previous similar messages
[ 5389.632458] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5391.019961] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4775 to 0x0:4801
[ 5391.021767] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4934 to 0x0:4961
[ 5397.940484] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5399.386429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5407.380658] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:26:23 (1776183983)
[ 5424.557680] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5456.957505] LDISKFS-fs (dm-2): recovery complete
[ 5456.960308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5459.529403] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129
[ 5459.533814] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5362 to 0x0:5377
[ 5460.426710] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5474.783532] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5474.794110] LustreError: Skipped 9 previous similar messages
[ 5492.226583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5493.607709] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161
[ 5493.610929] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5362 to 0x0:5409
[ 5496.814805] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5505.301946] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5506.956503] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5547.092826] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:28:43 (1776184123)
[ 5569.592117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5577.893226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5577.896764] Lustre: Skipped 6 previous similar messages
[ 5581.883806] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5583.383940] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5202 to 0x0:5217
[ 5583.390626] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5362 to 0x0:5441
[ 5617.803926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5628.972100] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5362 to 0x0:5473
[ 5628.972166] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5202 to 0x0:5249
[ 5629.543366] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5642.460192] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5644.227490] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5653.232345] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:30:29 (1776184229)
[ 5691.637336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5693.993701] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5475 to 0x0:5505
[ 5694.002775] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:193
[ 5696.823538] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5707.255721] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5709.015901] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5720.037615] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:31:36 (1776184296)
[ 5733.090704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5733.322640] Lustre: *** cfs_fail_loc=605, val=0***
[ 5733.328852] LustreError: 172332:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=00000000fff12d02 failed: rc = -95
[ 5733.355608] LustreError: 172332:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 5733.363404] LustreError: 172332:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 5733.372842] LustreError: 172332:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 5733.380817] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 5733.387505] LustreError: 172332:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 5733.404739] LustreError: 172332:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 5734.879349] LustreError: 166-1: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5734.897779] LustreError: Skipped 4 previous similar messages
[ 5743.841191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5752.290569] Lustre: Evicted from MGS (at 192.168.202.141@tcp) after server handle changed from 0xeaa259be4fd39099 to 0xeaa259be4fd397e6
[ 5752.309684] Lustre: Skipped 4 previous similar messages
[ 5758.013535] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5475 to 0x0:5537
[ 5758.015307] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5251 to 0x0:5281
[ 5758.318592] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5770.419431] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:32:26 (1776184346)
[ 5778.804605] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5782.057459] Lustre: Failing over lustre-MDT0000
[ 5782.068599] Lustre: Skipped 7 previous similar messages
[ 5782.266774] Lustre: server umount lustre-MDT0000 complete
[ 5782.272224] Lustre: Skipped 8 previous similar messages
[ 5783.529986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5783.531863] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5783.551702] Lustre: Skipped 25 previous similar messages
[ 5783.578198] LustreError: Skipped 251 previous similar messages
[ 5790.559332] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184361/real 1776184361]  req@00000000db282570 x1862458319957632/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184368 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 5790.576713] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5808.065976] LDISKFS-fs (dm-0): recovery complete
[ 5808.068551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5824.255761] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5824.267369] Lustre: Skipped 7 previous similar messages
[ 5824.274530] Lustre: *** cfs_fail_loc=707, val=0***
[ 5827.871678] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5865.745283] Lustre: lustre-MDT0000: Client 5bb5d51e-f557-4e34-b837-13b1acb38e11 (at 192.168.202.41@tcp) reconnected, waiting for 2 clients in recovery for 0:58
[ 5866.269662] Lustre: lustre-MDT0000: Recovery over after 0:42, of 2 clients 2 recovered and 0 were evicted.
[ 5866.280038] Lustre: Skipped 7 previous similar messages
[ 5866.316238] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5550 to 0x0:5569
[ 5866.324440] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5295 to 0x0:5313
[ 5873.762777] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5876.599629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5887.564578] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:34:23 (1776184463)
[ 5918.241051] LustreError: 14685:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 5924.279572] LustreError: 14685:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5939.735265] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:35:16 (1776184516)
[ 5971.062307] LustreError: 43149:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms
[ 5977.119274] LustreError: 43149:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake
[ 5987.884834] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:36:04 (1776184564)
[ 6018.080356] LustreError: 6261:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms
[ 6023.095115] LustreError: 6261:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 6025.481417] LustreError: 14685:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms
[ 6035.499794] LustreError: 14685:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 6050.629677] Lustre: DEBUG MARKER: replay-single test_66a: @@@@@@ FAIL: Current 31 should be less than worst 31