[ 3290.975270] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181980/real 1776181980] req@000000000c162039 x1862458443482112/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181987 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 3290.990114] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 3290.994454] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3291.001014] LustreError: Skipped 11 previous similar messages [ 3293.656323] LDISKFS-fs (dm-0): recovery complete [ 3293.660978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.478384] LustreError: 112536:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3297.490412] LustreError: 112536:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3297.503932] Lustre: 112570:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3297.512235] Lustre: 112570:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3297.518506] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3297.525246] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3297.543183] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3297.603324] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1576 to 0x0:1697 [ 3301.169779] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3302.884323] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3322.759600] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:53:37 (1776182017) [ 3353.819177] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3359.200335] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3359.206466] LustreError: Skipped 11 previous similar messages [ 3376.937956] LDISKFS-fs (dm-0): recovery complete [ 3376.945329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.315422] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3388.484817] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1603 to 0x0:1665 [ 3388.486776] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2498 to 0x0:2529 [ 3394.147795] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3395.409808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3414.810611] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:55:09 (1776182109) [ 3436.740226] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3444.322100] LustreError: 3350:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000049e3386d x1862458443607104/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-1-0.0' [ 3464.805440] LDISKFS-fs (dm-0): recovery complete [ 3464.810773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.574493] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3473.583072] Lustre: Skipped 20 previous similar messages [ 3477.148553] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3482.564138] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2066 to 0x0:2081 [ 3482.566211] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2930 to 0x0:2945 [ 3487.828823] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3489.296799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3509.698532] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 11:56:44 (1776182204) [ 3510.949800] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 3512.797996] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:56:47 (1776182207) [ 3514.949316] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3516.124525] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3516.129516] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3516.139847] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2066 to 0x0:2113 [ 3523.038467] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:56:58 (1776182218) [ 3544.268909] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3580.325756] LDISKFS-fs (dm-2): recovery complete [ 3580.334427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3580.577523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.591720] Lustre: Skipped 10 previous similar messages [ 3582.788937] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 3582.792922] Lustre: Skipped 50 previous similar messages [ 3582.794933] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3582.799702] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3347 to 0x0:3393 [ 3585.537777] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3643.405894] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:58:58 (1776182338) [ 3650.463778] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3674.144976] LDISKFS-fs (dm-0): recovery complete [ 3674.149976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.377380] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (not set up) [ 3674.385039] Lustre: Skipped 3 previous similar messages [ 3678.036634] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3679.801764] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3679.805918] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3347 to 0x0:3425 [ 3685.512509] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3686.650315] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3686.879586] LustreError: 122381:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3686.881931] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3687.903822] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2514 to 0x0:2529 [ 3704.481710] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:59:59 (1776182399) [ 3709.069847] LustreError: 6252:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3714.527121] LustreError: 6252:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3714.530956] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 3714.601828] LustreError: 8238:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3716.941493] LustreError: 9653:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3722.207504] LustreError: 9653:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3722.219651] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 3724.769823] LustreError: 6251:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3729.887131] LustreError: 6251:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3729.907452] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 3731.962878] LustreError: 6251:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3737.055289] LustreError: 6251:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3739.112794] LustreError: 9238:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3744.223121] LustreError: 9238:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3744.230987] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 3744.238944] Lustre: Skipped 1 previous similar message [ 3753.246639] LustreError: 6252:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3753.258638] LustreError: 6252:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message [ 3758.559126] LustreError: 6252:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3758.565073] LustreError: 6252:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message [ 3765.225241] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 3765.233627] Lustre: Skipped 2 previous similar messages [ 3774.083466] LustreError: 6251:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3774.089217] LustreError: 6251:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages [ 3779.551196] LustreError: 6251:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3779.564695] LustreError: 6251:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages [ 3788.494124] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 12:01:23 (1776182483) [ 3790.439811] LustreError: 9238:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3795.710803] Lustre: lustre-MDT0000: Export 00000000b4ddeb49 already connecting from 192.168.203.18@tcp [ 3797.471107] Lustre: lustre-MDT0000: Export 00000000b4ddeb49 already connecting from 192.168.203.18@tcp [ 3799.321972] Lustre: lustre-MDT0000: Export 00000000b4ddeb49 already connecting from 192.168.203.18@tcp [ 3803.260861] Lustre: lustre-MDT0000: Export 00000000b4ddeb49 already connecting from 192.168.203.18@tcp [ 3803.271294] Lustre: Skipped 2 previous similar messages [ 3808.794600] Lustre: lustre-MDT0000: Export 00000000b4ddeb49 already connecting from 192.168.203.18@tcp [ 3808.811547] Lustre: Skipped 3 previous similar messages [ 3815.599683] LustreError: 9238:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3815.633002] Lustre: 9238:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/5s); client may timeout req@0000000082453b6c x1862458426324352/t0(0) o38->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:0/0 lens 520/416 e 0 to 0 dl 1776182507 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0' [ 3816.208035] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 3816.228429] Lustre: Skipped 3 previous similar messages [ 3822.256398] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:01:56 (1776182516) [ 3829.904381] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3833.436856] Lustre: Failing over lustre-MDT0000 [ 3833.442643] Lustre: Skipped 7 previous similar messages [ 3833.658205] Lustre: server umount lustre-MDT0000 complete [ 3833.660666] Lustre: Skipped 7 previous similar messages [ 3835.367961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.394963] Lustre: Skipped 31 previous similar messages [ 3844.944924] LDISKFS-fs (dm-0): recovery complete [ 3844.949680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3848.837241] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (not set up) [ 3848.895600] Lustre: *** cfs_fail_loc=712, val=0*** [ 3848.909859] LustreError: 8237:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery req@000000006ddbe7c6 x1862458443814336/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 3848.943148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3849.081519] LustreError: 127067:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3849.087372] LustreError: 127067:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3849.092969] Lustre: 127101:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3849.108292] Lustre: 127101:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3849.114885] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3849.123693] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0] [ 3849.157035] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3849.253839] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2514 to 0x0:2561 [ 3849.255535] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3347 to 0x0:3457 [ 3853.446055] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3854.312963] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3869.666932] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3869.688637] LustreError: Skipped 176 previous similar messages [ 3888.381069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.346970] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xa141263f70c2ddcc to 0xa141263f70c2e21e [ 3898.354120] Lustre: Skipped 5 previous similar messages [ 3898.362409] LustreError: 128234:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.118@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000416eef0a) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3899.133029] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3899.137476] Lustre: Skipped 4 previous similar messages [ 3903.485755] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3904.003207] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3904.022441] Lustre: Skipped 4 previous similar messages [ 3904.075103] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2514 to 0x0:2593 [ 3904.082977] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3347 to 0x0:3489 [ 3914.251955] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3915.927754] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3924.791036] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:03:39 (1776182619) [ 3925.060045] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 3934.033618] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:03:48 (1776182628) [ 3935.459214] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3935.465080] LustreError: 6261:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000d7386f84 x1862458426351360/t0(0) o700->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:593/0 lens 264/248 e 0 to 0 dl 1776182638 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0' [ 3957.206152] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182646/real 1776182646] req@0000000085855da1 x1862458443843072/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182653 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 3957.226679] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 3957.229135] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3957.244071] LustreError: Skipped 5 previous similar messages [ 3963.776671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.187876] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3980.342053] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2595 to 0x0:2625 [ 3980.342993] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521 [ 3989.283315] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3990.952560] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4000.445623] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:04:55 (1776182695) [ 4021.843969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4023.678695] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3532 to 0x0:3553 [ 4023.687522] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97 [ 4026.919946] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4039.057839] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4042.162331] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4116.024395] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:06:50 (1776182810) [ 4124.152868] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4128.739727] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4128.759490] LustreError: Skipped 5 previous similar messages [ 4152.584738] LDISKFS-fs (dm-0): recovery complete [ 4152.589338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.280114] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4155.282550] Lustre: Skipped 8 previous similar messages [ 4159.871690] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4161.160208] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2646 to 0x0:2689 [ 4161.161334] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3564 to 0x0:3585 [ 4232.547259] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:08:47 (1776182927) [ 4234.618560] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4234.628885] Lustre: Skipped 2 previous similar messages [ 4234.632143] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4234.634951] Lustre: Skipped 30 previous similar messages [ 4234.642250] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3617 [ 4235.399455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3649 [ 4246.315641] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:09:01 (1776182941) [ 4267.328534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.013050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.020424] Lustre: Skipped 6 previous similar messages [ 4282.807752] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4283.376972] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4283.389368] LustreError: 135219:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000bf25e028 x1862458426409664/t0(0) o101->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:216/0 lens 328/344 e 0 to 0 dl 1776183016 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0' [ 4323.066087] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnected, waiting for 2 clients in recovery for 0:56 [ 4323.116213] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2701 to 0x0:2721 [ 4323.116455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3681 [ 4328.404954] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4329.757585] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4337.582756] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:10:32 (1776183032) [ 4339.596295] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4347.239549] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4370.645154] LDISKFS-fs (dm-0): recovery complete [ 4370.649188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4379.671729] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4380.765139] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2723 to 0x0:2753 [ 4380.765723] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3713 [ 4388.023143] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4389.546550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4397.446392] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:11:32 (1776183092) [ 4398.496104] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4406.788131] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4429.116731] LDISKFS-fs (dm-0): recovery complete [ 4429.119485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.365625] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4441.610385] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2755 to 0x0:2785 [ 4441.610831] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3745 [ 4447.093480] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4448.388331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4456.079426] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:12:31 (1776183151) [ 4456.867179] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4464.535862] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4466.173088] Lustre: Failing over lustre-MDT0000 [ 4466.184360] Lustre: Skipped 7 previous similar messages [ 4466.288944] Lustre: server umount lustre-MDT0000 complete [ 4466.291452] Lustre: Skipped 7 previous similar messages [ 4467.167832] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4467.185617] Lustre: Skipped 31 previous similar messages [ 4472.058939] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.074902] LustreError: Skipped 255 previous similar messages [ 4486.325811] LDISKFS-fs (dm-0): recovery complete [ 4486.328953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.713215] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4496.405526] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3777 [ 4496.406097] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2787 to 0x0:2817 [ 4505.350440] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:13:20 (1776183200) [ 4507.620785] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4507.622799] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4507.627064] LustreError: 26481:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000a6803e33 x1862458426440064/t261993005073(0) o35->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:410/0 lens 392/456 e 0 to 0 dl 1776183210 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4527.946082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4535.266364] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xa141263f70c31bbb to 0xa141263f70c3224b [ 4535.276172] Lustre: Skipped 6 previous similar messages [ 4535.548472] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4535.554100] Lustre: Skipped 7 previous similar messages [ 4538.719733] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4540.915192] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4540.922614] Lustre: Skipped 7 previous similar messages [ 4540.947932] Lustre: 26481:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000001c1d08d4 x1862458426440064/t261993005073(0) o35->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:443/0 lens 392/456 e 0 to 0 dl 1776183243 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4540.967978] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2819 to 0x0:2849 [ 4540.974555] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3809 [ 4546.678346] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4547.959263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4555.000462] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:14:10 (1776183250) [ 4556.208224] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4556.212616] LustreError: 13431:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000000fa8b229 x1862458426448128/t266287972368(0) o36->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:492/0 lens 504/448 e 0 to 0 dl 1776183292 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4565.676982] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4578.783299] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183268/real 1776183268] req@00000000525e62aa x1862458444018368/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183275 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 4578.806876] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 4578.812119] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4578.825037] LustreError: Skipped 6 previous similar messages [ 4590.011428] LDISKFS-fs (dm-0): recovery complete [ 4590.014857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4601.917592] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2851 to 0x0:2881 [ 4601.919148] Lustre: 6251:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000000a5094df x1862458426448128/t266287972368(0) o36->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:538/0 lens 504/448 e 0 to 0 dl 1776183338 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4601.920375] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3841 [ 4603.186144] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4612.799635] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4614.354283] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4622.705319] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:15:18 (1776183318) [ 4623.670577] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4623.679084] LustreError: 6251:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000fe506126 x1862458426457728/t270582939664(0) o36->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:560/0 lens 504/448 e 0 to 0 dl 1776183360 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4625.557167] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4632.110177] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4632.829253] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 4632.859330] Lustre: Skipped 2 previous similar messages [ 4632.879075] Lustre: 6254:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000ad5fa675 x1862458426457856/t270582939665(0) o35->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:535/0 lens 392/456 e 0 to 0 dl 1776183335 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4632.896822] Lustre: 6254:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4657.063147] LDISKFS-fs (dm-0): recovery complete [ 4657.068060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.637681] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4667.991453] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3873 [ 4667.992064] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2883 to 0x0:2913 [ 4678.478826] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:16:13 (1776183373) [ 4679.583377] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4679.585708] Lustre: Skipped 1 previous similar message [ 4679.591772] LustreError: 6253:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000d9421dac x1862458426465728/t274877906960(0) o36->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:616/0 lens 504/448 e 0 to 0 dl 1776183416 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4679.614482] LustreError: 6253:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 4681.258749] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4681.260979] Lustre: Skipped 1 previous similar message [ 4687.913625] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4688.664777] Lustre: 6251:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000006e412618 x1862458426465728/t274877906960(0) o36->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:625/0 lens 504/448 e 0 to 0 dl 1776183425 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4710.486388] LDISKFS-fs (dm-0): recovery complete [ 4710.489713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.773134] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4723.770228] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3905 [ 4730.349483] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:17:05 (1776183425) [ 4731.243360] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4732.949073] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4732.953660] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4732.956133] LustreError: 6254:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000003a027d04 x1862458426473600/t279172874256(0) o35->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:635/0 lens 392/456 e 0 to 0 dl 1776183435 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4739.672676] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4762.333063] LDISKFS-fs (dm-0): recovery complete [ 4762.336630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.715351] LustreError: 151190:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.118@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000bb714e29) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 4767.938844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4767.943747] Lustre: Skipped 8 previous similar messages [ 4771.463603] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4773.387792] Lustre: 6254:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000a9397e1a x1862458426473600/t279172874256(0) o35->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:676/0 lens 392/456 e 0 to 0 dl 1776183476 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4773.406969] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3907 to 0x0:3937 [ 4773.408422] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2915 to 0x0:2945 [ 4783.006442] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:17:58 (1776183478) [ 4783.859705] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4783.867932] LustreError: 9238:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000089f64b32 x1862458426480448/t283467841550(0) o101->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:720/0 lens 664/600 e 0 to 0 dl 1776183520 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0' [ 4826.888362] Lustre: 6251:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000b6c72753 x1862458426480448/t283467841550(0) o101->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:8/0 lens 664/3424 e 0 to 0 dl 1776183563 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0' [ 4832.728726] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:18:48 (1776183528) [ 4838.716177] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4845.030857] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4845.043480] LustreError: Skipped 6 previous similar messages [ 4863.017331] LDISKFS-fs (dm-0): recovery complete [ 4863.020292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4868.593774] Lustre: MGC192.168.203.118@tcp: Connection restored to (at 0@lo) [ 4868.604784] Lustre: Skipped 46 previous similar messages [ 4873.571405] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4874.323824] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3907 to 0x0:3969 [ 4874.324922] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2947 to 0x0:2977 [ 4883.779746] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4885.415348] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4903.468808] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:19:58 (1776183598) [ 4910.883161] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4932.951400] LDISKFS-fs (dm-0): recovery complete [ 4932.954470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.880931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.886463] Lustre: Skipped 9 previous similar messages [ 4943.539210] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4944.440955] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3907 to 0x0:4001 [ 4944.442122] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2979 to 0x0:3009 [ 4953.274992] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4954.898212] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4960.066837] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4971.610786] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:21:06 (1776183666) [ 5012.073750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5036.339532] LDISKFS-fs (dm-0): recovery complete [ 5036.344094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.162681] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5045.427335] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4260 to 0x0:4289 [ 5045.433815] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5252 to 0x0:5281 [ 5051.423969] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5053.002734] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5119.285067] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:23:34 (1776183814) [ 5127.192131] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5128.978652] Lustre: Failing over lustre-MDT0000 [ 5128.984376] Lustre: Skipped 8 previous similar messages [ 5129.151450] Lustre: server umount lustre-MDT0000 complete [ 5129.161075] Lustre: Skipped 8 previous similar messages [ 5131.009110] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.022478] LustreError: Skipped 335 previous similar messages [ 5132.256153] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.267268] Lustre: Skipped 36 previous similar messages [ 5152.397502] LDISKFS-fs (dm-0): recovery complete [ 5152.406844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.835411] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xa141263f70c4a15a to 0xa141263f70c752ef [ 5156.845970] Lustre: Skipped 7 previous similar messages [ 5157.360475] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5157.372252] Lustre: Skipped 7 previous similar messages [ 5160.220254] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5162.536250] Lustre: lustre-MDT0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 5162.546668] Lustre: Skipped 7 previous similar messages [ 5162.594405] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4291 to 0x0:4321 [ 5162.595686] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5252 to 0x0:5313 [ 5167.586687] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5169.161613] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5177.892073] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5179.127380] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5184.258740] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:24:39 (1776183879) [ 5191.315312] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5233.413709] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 5233.426213] Lustre: Skipped 2 previous similar messages [ 5235.752564] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5235.754430] Lustre: Skipped 1 previous similar message [ 5235.761904] LustreError: 6253:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000a332f53c x1862458427970688/t300647710728(0) o36->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:417/0 lens 66040/440 e 0 to 0 dl 1776183972 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0' [ 5277.450945] Lustre: 9653:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000008b1d7e24 x1862458427970688/t300647710728(0) o36->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:459/0 lens 66040/440 e 0 to 0 dl 1776184014 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0' [ 5285.793985] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5287.474221] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:26:22 (1776183982) [ 5300.620703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5328.482824] LDISKFS-fs (dm-0): recovery complete [ 5328.487367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.574529] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.585993] LustreError: Skipped 7 previous similar messages [ 5328.591600] LustreError: 163404:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting [ 5328.651423] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (not set up) [ 5333.746180] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5335.047450] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4422 to 0x0:4449 [ 5335.049600] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5415 to 0x0:5441 [ 5341.915480] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5343.504597] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5352.602588] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:27:27 (1776184047) [ 5372.809221] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5406.627963] LDISKFS-fs (dm-2): recovery complete [ 5406.631299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5406.779807] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5406.792657] Lustre: Skipped 5 previous similar messages [ 5408.484738] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129 [ 5408.490075] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5842 to 0x0:5857 [ 5410.644694] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5442.037634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5443.883641] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161 [ 5443.893474] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5842 to 0x0:5889 [ 5446.982225] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5455.814605] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5457.072336] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5495.816912] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:29:51 (1776184191) [ 5498.849779] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5498.855199] LustreError: Skipped 5 previous similar messages [ 5512.167945] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184200/real 1776184200] req@000000001620fbf6 x1862458444681664/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184207 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 5512.186980] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 5517.445705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5528.554425] Lustre: MGC192.168.203.118@tcp: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 5528.566136] Lustre: Skipped 28 previous similar messages [ 5532.500835] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5534.251425] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5842 to 0x0:5921 [ 5534.253167] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4850 to 0x0:4865 [ 5564.423523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.380546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5574.385246] Lustre: Skipped 6 previous similar messages [ 5579.046500] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5579.818901] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4850 to 0x0:4897 [ 5579.819647] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5842 to 0x0:5953 [ 5588.075323] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5589.359427] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5595.890695] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:31:31 (1776184291) [ 5627.598856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5629.540721] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5955 to 0x0:5985 [ 5629.543192] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:193 [ 5631.758795] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5639.566859] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5640.981143] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5648.802139] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:32:24 (1776184344) [ 5657.552901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.631526] Lustre: *** cfs_fail_loc=605, val=0*** [ 5657.633727] LustreError: 172402:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=00000000c19c78cd failed: rc = -95 [ 5657.639695] LustreError: 172402:0:(obd_config.c:774:class_setup()) setup MGS failed (-95) [ 5657.644468] LustreError: 172402:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95 [ 5657.649319] LustreError: 172402:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered [ 5657.654122] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5657.658807] LustreError: 172402:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 5657.668734] LustreError: 172402:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5663.985644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5666.787211] LustreError: 172809:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.118@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000006a45fc9a) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5670.284190] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5672.498856] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5955 to 0x0:6017 [ 5672.499329] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4899 to 0x0:4929 [ 5676.182320] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:32:51 (1776184371) [ 5681.330196] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5683.984150] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 5705.193326] LDISKFS-fs (dm-0): recovery complete [ 5705.196163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5712.653485] Lustre: *** cfs_fail_loc=707, val=0*** [ 5715.804933] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5756.686649] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 5757.073110] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6031 to 0x0:6049 [ 5757.073767] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4942 to 0x0:4961 [ 5762.244012] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5763.747114] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5771.807341] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:34:27 (1776184467) [ 5800.009874] LustreError: 9653:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 5806.055438] LustreError: 9653:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5822.688687] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:35:17 (1776184517) [ 5851.208515] LustreError: 8241:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms [ 5857.255414] LustreError: 8241:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 5865.511272] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:36:00 (1776184560) [ 5893.669928] LustreError: 6253:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5898.687245] LustreError: 6253:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5900.709638] LustreError: 13431:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 5910.735165] LustreError: 13431:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5928.041192] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 12:37:03 (1776184623) [ 5987.166928] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 12:38:02 (1776184682) [ 6015.467883] LustreError: 13431:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6015.903292] LustreError: 13431:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6023.604684] LustreError: 6255:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6023.618350] LustreError: 6255:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 6024.047151] LustreError: 6255:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6024.057263] LustreError: 6255:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 6039.898835] LustreError: 6252:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6039.912495] LustreError: 6252:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 65 previous similar messages [ 6040.335228] LustreError: 6252:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6040.346798] LustreError: 6252:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 64 previous similar messages [ 6048.060806] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 12:39:03 (1776184743) [ 6081.739624] Lustre: DEBUG MARKER: phase 2 [ 6091.880201] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 12:39:46 (1776184786) [ 6177.874953] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 12:41:12 (1776184872) [ 6179.757090] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6181.271564] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 12:41:16 (1776184876) [ 6185.855127] Lustre: DEBUG MARKER: Started rundbench load pid=138606 ... [ 6195.678712] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6198.509984] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6200.554505] Lustre: Failing over lustre-MDT0000 [ 6200.558260] Lustre: Skipped 8 previous similar messages [ 6200.748070] Lustre: server umount lustre-MDT0000 complete [ 6200.755119] Lustre: Skipped 9 previous similar messages [ 6203.359748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6203.364036] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6203.370773] Lustre: Skipped 28 previous similar messages [ 6203.397487] LustreError: Skipped 247 previous similar messages [ 6211.103259] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184900/real 1776184900] req@000000009ea7d077 x1862458444891136/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184907 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 6211.134753] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6211.150597] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6211.160705] LustreError: Skipped 4 previous similar messages [ 6223.915072] LDISKFS-fs (dm-0): recovery complete [ 6223.918737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.937089] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xa141263f70c8a382 to 0xa141263f70c996ec [ 6227.945586] Lustre: Skipped 5 previous similar messages [ 6227.951509] Lustre: MGC192.168.203.118@tcp: Connection restored to (at 0@lo) [ 6227.959482] Lustre: Skipped 21 previous similar messages [ 6228.209240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6228.216691] Lustre: Skipped 3 previous similar messages [ 6228.265064] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6228.278829] Lustre: Skipped 6 previous similar messages [ 6229.514602] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6229.527784] Lustre: Skipped 8 previous similar messages [ 6232.235867] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6233.645883] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6233.657545] Lustre: Skipped 8 previous similar messages [ 6233.696277] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6118 to 0x0:6145 [ 6233.696798] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5010 to 0x0:5025 [ 6241.383614] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6243.130918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6252.677883] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6255.285136] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6257.277306] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [ 6259.168422] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6259.178362] LustreError: Skipped 3 previous similar messages [ 6280.070683] LDISKFS-fs (dm-1): recovery complete [ 6280.073715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6284.213460] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6286.923092] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:156 to 0x2c0000400:193 [ 6286.923379] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:351 to 0x280000400:385 [ 6293.664301] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6295.561235] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6305.667322] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6308.364535] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 6333.601122] LDISKFS-fs (dm-0): recovery complete [ 6333.605555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6344.075575] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6345.347652] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6118 to 0x0:6177 [ 6345.353210] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5010 to 0x0:5057 [ 6355.904505] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6357.838659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6367.019426] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 12:44:22 (1776185062) [ 6495.000073] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6506.941477] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 6508.685334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6530.572247] LDISKFS-fs (dm-0): recovery complete [ 6530.575395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.928175] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6540.817889] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5341 to 0x0:5377 [ 6540.819079] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6461 to 0x0:6497 [ 6546.013803] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6547.502069] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6599.371234] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 12:48:14 (1776185294) [ 6727.055914] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6739.041726] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 6740.585307] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 6762.533522] LDISKFS-fs (dm-0): recovery complete [ 6762.536389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.905854] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6774.121206] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6608 to 0x0:6625 [ 6774.125232] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5488 to 0x0:5505 [ 6781.033860] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6783.160431] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6791.381904] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 12:51:26 (1776185486) [ 6920.214452] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6933.084356] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 6935.248111] Lustre: Failing over lustre-MDT0000 [ 6935.254062] Lustre: Skipped 4 previous similar messages [ 6935.298693] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 6935.494521] Lustre: server umount lustre-MDT0000 complete [ 6935.502792] Lustre: Skipped 4 previous similar messages [ 6936.032536] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.036580] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6936.048523] Lustre: Skipped 19 previous similar messages [ 6936.065296] LustreError: Skipped 159 previous similar messages [ 6942.625812] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185632/real 1776185632] req@000000008a222947 x1862458446898752/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185639 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 6942.664541] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6942.682046] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6942.704258] LustreError: Skipped 3 previous similar messages [ 6960.774754] LDISKFS-fs (dm-0): recovery complete [ 6960.785525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.546887] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xa141263f70d30d16 to 0xa141263f70d6dd1f [ 6976.564261] Lustre: Skipped 3 previous similar messages [ 6976.575261] Lustre: MGC192.168.203.118@tcp: Connection restored to (at 0@lo) [ 6976.584789] Lustre: Skipped 22 previous similar messages [ 6976.887682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6976.895186] Lustre: Skipped 4 previous similar messages [ 6976.944632] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6976.954247] Lustre: Skipped 4 previous similar messages [ 6977.664296] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6977.672129] Lustre: Skipped 4 previous similar messages [ 6981.503442] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6989.114195] Lustre: lustre-MDT0000: Recovery over after 0:12, of 2 clients 2 recovered and 0 were evicted. [ 6989.129151] Lustre: Skipped 4 previous similar messages [ 6989.187278] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5991 to 0x0:6017 [ 6989.188416] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7112 to 0x0:7137 [ 6995.898976] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6997.947778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7006.658992] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 12:55:01 (1776185701) [ 7017.768445] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7020.242322] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7021.310741] Lustre: lustre-OST0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnecting [ 7021.331522] Lustre: Skipped 1 previous similar message [ 7044.956796] LDISKFS-fs (dm-2): recovery complete [ 7044.964624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7047.151403] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:857 to 0x280000400:929 [ 7047.166490] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7184 to 0x0:7201 [ 7049.677947] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7061.073476] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7063.224714] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7074.125641] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 12:56:09 (1776185769) [ 7204.087071] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7212.593776] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7225.049578] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 7227.238381] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 7232.077680] LustreError: 6237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776185928 with bad export cookie 11619610567510514975 [ 7232.095662] LustreError: 6237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7258.011111] LDISKFS-fs (dm-0): recovery complete [ 7258.034509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7271.730558] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7299.342806] LDISKFS-fs (dm-1): recovery complete [ 7299.352034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7299.804736] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 7299.819894] LustreError: Skipped 2 previous similar messages [ 7305.052264] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7306.340838] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:857 to 0x280000400:961 [ 7306.342334] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:666 to 0x2c0000400:737 [ 7312.869237] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7184 to 0x0:7233 [ 7312.869930] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6064 to 0x0:6081 [ 7319.271223] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7321.166658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7323.001152] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7332.732929] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:00:27 (1776186027) [ 7339.648405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7371.526090] LDISKFS-fs (dm-0): recovery complete [ 7371.542276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.164475] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7393.052205] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7394.575912] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnected, waiting for 2 clients in recovery for 0:59 [ 7394.632014] Lustre: 197932:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000706e2c4c x1862458434281984/t343597384264(343597384264) o101->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:277/0 lens 648/3424 e 0 to 0 dl 1776186097 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7394.742224] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6064 to 0x0:6113 [ 7394.746912] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7235 to 0x0:7265 [ 7402.296546] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7403.866810] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7410.575474] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:01:45 (1776186105) [ 7416.656782] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7418.853589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7418.864215] Lustre: Skipped 3 previous similar messages [ 7445.289517] LDISKFS-fs (dm-0): recovery complete [ 7445.293079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.496369] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7448.498718] LustreError: 197479:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000007c7bbfcc x1862458434281984/t343597384264(343597384264) o101->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:331/0 lens 648/600 e 0 to 0 dl 1776186151 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0' [ 7454.060787] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7455.537173] Lustre: lustre-MDT0000: Client 425ca546-5bd1-4add-8704-f85a5f984239 (at 192.168.203.18@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 7455.576243] Lustre: 197932:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000000becf1f2 x1862458434281984/t343597384264(343597384264) o101->425ca546-5bd1-4add-8704-f85a5f984239@192.168.203.18@tcp:338/0 lens 648/3424 e 0 to 0 dl 1776186158 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7455.739040] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7235 to 0x0:7297 [ 7455.745792] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6145 [ 7464.109466] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7465.805110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7473.660960] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:02:48 (1776186168) [ 7499.949118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7513.128715] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6177 [ 7513.537658] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7521.727592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.302948] Lustre: lustre-OST0000: Denying connection for new client 84e9dc5e-009a-45d5-a67d-9fdee8de725f (at 192.168.203.18@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7523.319502] Lustre: Skipped 12 previous similar messages [ 7523.689638] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7235 to 0x0:7329 [ 7523.699940] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:857 to 0x280000400:993 [ 7526.470849] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7537.666729] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:03:52 (1776186232) [ 7539.298375] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7539.304844] LustreError: 187065:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000eba1a4b0 x1862458448136768/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:422/0 lens 1056/4320 e 0 to 0 dl 1776186242 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7546.335380] Lustre: 198496:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186236/real 1776186236] req@00000000fcf76a48 x1862458448136768/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1056/4320 e 0 to 1 dl 1776186243 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'osp_up0-1.0' [ 7546.368712] Lustre: 198496:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 7546.376603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7546.394771] Lustre: Skipped 21 previous similar messages [ 7546.408370] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7548.918490] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7551.759237] Lustre: Failing over lustre-MDT0000 [ 7551.760995] Lustre: Skipped 7 previous similar messages [ 7552.124181] Lustre: server umount lustre-MDT0000 complete [ 7552.127263] Lustre: Skipped 7 previous similar messages [ 7556.578333] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.611983] LustreError: Skipped 240 previous similar messages [ 7563.743164] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7563.750404] LustreError: Skipped 4 previous similar messages [ 7575.935032] LDISKFS-fs (dm-0): recovery complete [ 7575.937991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.161960] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xa141263f70db7860 to 0xa141263f70db7f36 [ 7581.174340] Lustre: Skipped 4 previous similar messages [ 7581.182387] Lustre: MGC192.168.203.118@tcp: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 7581.185478] Lustre: Skipped 30 previous similar messages [ 7581.387313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7581.393545] Lustre: Skipped 7 previous similar messages [ 7581.425274] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7581.427588] Lustre: Skipped 7 previous similar messages [ 7581.447557] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7581.465820] Lustre: Skipped 7 previous similar messages [ 7585.507785] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7586.890398] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6209 [ 7586.892461] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7361 [ 7594.390656] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7595.702544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7604.322352] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:04:59 (1776186299) [ 7605.507672] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7605.514205] LustreError: 187065:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000f3685c0a x1862458448158976/t365072220170(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:488/0 lens 2200/4320 e 0 to 0 dl 1776186308 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7612.907660] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7614.329962] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7623.161638] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7627.526052] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [ 7654.119355] LDISKFS-fs (dm-1): recovery complete [ 7654.126469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.493079] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7660.041158] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7660.052771] Lustre: Skipped 8 previous similar messages [ 7660.091575] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1025 [ 7660.091840] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:748 to 0x2c0000400:769 [ 7666.335154] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7667.661603] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7675.017079] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:06:10 (1776186370) [ 7675.862120] LustreError: 197483:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000001810bc02 x1862458448179712/t365072220189(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:558/0 lens 2200/4320 e 0 to 0 dl 1776186378 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7682.735353] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7683.043554] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7689.384859] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7711.624721] LDISKFS-fs (dm-0): recovery complete [ 7711.633028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.865534] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7723.710149] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6241 [ 7723.710702] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7393 [ 7730.555354] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7732.104273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7756.484899] LDISKFS-fs (dm-1): recovery complete [ 7756.490770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7759.804315] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7761.956138] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:801 [ 7761.956864] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1057 [ 7768.373612] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7770.078897] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7778.659620] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:07:53 (1776186473) [ 7779.588431] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7779.592575] Lustre: Skipped 1 previous similar message [ 7786.977083] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7788.540613] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7794.878167] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7800.636557] LustreError: 6237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186497 with bad export cookie 11619610567510824494 [ 7800.645763] LustreError: 6237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7823.034726] LDISKFS-fs (dm-0): recovery complete [ 7823.040639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.356566] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7859.631195] LDISKFS-fs (dm-1): recovery complete [ 7859.635503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7865.268587] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7865.473338] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1068 to 0x280000400:1089 [ 7865.474939] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:812 to 0x2c0000400:833 [ 7865.709574] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6273 [ 7865.714455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7425 [ 7876.454629] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7878.252296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7879.917477] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7889.033711] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:09:44 (1776186584) [ 7890.195577] LustreError: 216459:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000012c9ad6d x1862458435214976/t25769803846(0) o36->84e9dc5e-009a-45d5-a67d-9fdee8de725f@192.168.203.18@tcp:17/0 lens 560/448 e 0 to 0 dl 1776186592 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 7890.215757] LustreError: 216459:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 7897.352341] Lustre: lustre-MDT0001: Client 84e9dc5e-009a-45d5-a67d-9fdee8de725f (at 192.168.203.18@tcp) reconnecting [ 7897.377513] Lustre: 216006:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000dc7e7ede x1862458435214976/t25769803846(0) o36->84e9dc5e-009a-45d5-a67d-9fdee8de725f@192.168.203.18@tcp:25/0 lens 560/448 e 0 to 0 dl 1776186600 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 7902.438906] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7931.329422] LDISKFS-fs (dm-0): recovery complete [ 7931.334245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7952.348569] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7952.988915] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6305 [ 7952.989130] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7457 [ 7961.768296] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7963.370000] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7972.470365] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:11:07 (1776186667) [ 7980.674475] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7980.813956] Lustre: lustre-MDT0001: Client 84e9dc5e-009a-45d5-a67d-9fdee8de725f (at 192.168.203.18@tcp) reconnecting [ 7980.827867] Lustre: 216008:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000002bb234ff x1862458435233536/t25769803916(0) o36->84e9dc5e-009a-45d5-a67d-9fdee8de725f@192.168.203.18@tcp:108/0 lens 560/448 e 0 to 0 dl 1776186683 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8005.146597] LDISKFS-fs (dm-1): recovery complete [ 8005.151656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.936127] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8010.792157] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:854 to 0x2c0000400:897 [ 8010.792265] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1110 to 0x280000400:1153 [ 8019.649594] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8021.278658] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8029.451407] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:12:04 (1776186724) [ 8037.627472] Lustre: lustre-MDT0001: Client 84e9dc5e-009a-45d5-a67d-9fdee8de725f (at 192.168.203.18@tcp) reconnecting [ 8037.638086] Lustre: 216008:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000005574e157 x1862458435249984/t30064771142(0) o36->84e9dc5e-009a-45d5-a67d-9fdee8de725f@192.168.203.18@tcp:165/0 lens 560/448 e 0 to 0 dl 1776186740 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8039.593672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8045.638834] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8051.682697] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8051.690181] LustreError: Skipped 7 previous similar messages [ 8068.033687] LDISKFS-fs (dm-0): recovery complete [ 8068.041584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.903419] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8082.060458] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6337 [ 8082.061402] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7489 [ 8089.857158] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8091.699543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8122.236972] LDISKFS-fs (dm-1): recovery complete [ 8122.240750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.182603] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8128.038922] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:908 to 0x2c0000400:929 [ 8128.038938] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1164 to 0x280000400:1185 [ 8137.055819] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8138.893121] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8149.405433] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:14:03 (1776186843) [ 8150.588096] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8150.590455] Lustre: Skipped 3 previous similar messages [ 8150.594911] LustreError: 216007:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000e8fe8683 x1862458435272448/t34359738438(0) o36->84e9dc5e-009a-45d5-a67d-9fdee8de725f@192.168.203.18@tcp:278/0 lens 560/448 e 0 to 0 dl 1776186853 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 8150.620687] LustreError: 216007:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 2 previous similar messages [ 8157.962777] Lustre: lustre-MDT0001: Client 84e9dc5e-009a-45d5-a67d-9fdee8de725f (at 192.168.203.18@tcp) reconnecting [ 8157.984352] Lustre: 216007:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000007c524fb8 x1862458435272448/t34359738438(0) o36->84e9dc5e-009a-45d5-a67d-9fdee8de725f@192.168.203.18@tcp:285/0 lens 560/448 e 0 to 0 dl 1776186860 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8161.792843] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8170.354637] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8172.827898] Lustre: Failing over lustre-MDT0000 [ 8172.830372] Lustre: Skipped 9 previous similar messages [ 8173.059788] Lustre: server umount lustre-MDT0000 complete [ 8173.063122] Lustre: Skipped 9 previous similar messages [ 8173.318369] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8173.338391] LustreError: Skipped 273 previous similar messages [ 8174.048390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.058996] Lustre: Skipped 39 previous similar messages [ 8177.540340] LustreError: 9231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186874 with bad export cookie 11619610567510838165 [ 8177.545796] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8177.550532] LustreError: 9231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8177.563015] LustreError: Skipped 4 previous similar messages [ 8185.247360] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186875/real 1776186875] req@000000005f9e7dd4 x1862458448333184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776186882 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 8185.266338] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [ 8202.495523] LDISKFS-fs (dm-0): recovery complete [ 8202.499093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.918176] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xa141263f70dbcb95 to 0xa141263f70dbda7c [ 8210.926253] Lustre: Skipped 4 previous similar messages [ 8210.931213] Lustre: MGC192.168.203.118@tcp: Connection restored to (at 0@lo) [ 8210.936776] Lustre: Skipped 41 previous similar messages [ 8211.243250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8211.255117] Lustre: Skipped 9 previous similar messages [ 8211.302920] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8211.319567] Lustre: Skipped 9 previous similar messages [ 8216.730701] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8225.533303] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8225.544780] Lustre: Skipped 9 previous similar messages [ 8238.871398] LDISKFS-fs (dm-1): recovery complete [ 8238.876917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8244.461982] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:940 to 0x2c0000400:961 [ 8244.462952] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1196 to 0x280000400:1217 [ 8244.611311] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7521 [ 8244.614768] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6369 [ 8244.663299] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8257.655236] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8260.593688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8263.851170] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8276.035775] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:16:10 (1776186970) [ 8285.153666] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8286.441066] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8311.403078] LDISKFS-fs (dm-1): recovery complete [ 8311.407943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8311.548239] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (not set up) [ 8311.554944] Lustre: Skipped 4 previous similar messages [ 8315.068986] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8316.922634] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8316.932440] Lustre: Skipped 10 previous similar messages [ 8316.972155] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:972 to 0x2c0000400:993 [ 8316.975068] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1249 [ 8324.204265] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8325.503065] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8334.311033] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:17:08 (1776187028) [ 8342.496796] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8344.611746] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8371.357119] LDISKFS-fs (dm-0): recovery complete [ 8371.359807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.304106] LustreError: 233035:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.118@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000003e238fa3) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8380.717223] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8382.096536] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6401 [ 8382.096982] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7553 [ 8391.344877] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8393.354983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8402.311324] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:18:17 (1776187097) [ 8411.105587] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8411.449288] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8418.891600] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8448.865093] LDISKFS-fs (dm-0): recovery complete [ 8448.871294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.746882] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6433 [ 8454.746888] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8454.747111] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7585 [ 8464.021366] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8465.628460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8493.385184] LDISKFS-fs (dm-1): recovery complete [ 8493.395366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8498.290930] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8499.246027] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:972 to 0x2c0000400:1025 [ 8499.262364] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1281 [ 8507.863876] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8509.625478] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8518.657646] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:20:13 (1776187213) [ 8526.305250] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8528.294811] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8539.283920] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8546.676032] LustreError: 6239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187243 with bad export cookie 11619610567510848476 [ 8546.686548] LustreError: 6239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8571.523601] LDISKFS-fs (dm-0): recovery complete [ 8571.531638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.782716] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8609.126854] LDISKFS-fs (dm-1): recovery complete [ 8609.135974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.536153] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8615.707396] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1313 [ 8615.716137] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:972 to 0x2c0000400:1057 [ 8615.881830] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6465 [ 8615.882286] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7617 [ 8625.876567] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8627.630703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8629.597051] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8638.760598] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:22:13 (1776187333) [ 8646.938727] Lustre: lustre-MDT0001: Client 84e9dc5e-009a-45d5-a67d-9fdee8de725f (at 192.168.203.18@tcp) reconnecting [ 8646.963465] Lustre: 239650:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000028d32f25 x1862458435330880/t51539607554(0) o36->84e9dc5e-009a-45d5-a67d-9fdee8de725f@192.168.203.18@tcp:19/0 lens 496/2888 e 0 to 0 dl 1776187349 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0' [ 8651.563293] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8656.868246] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8656.891478] LustreError: Skipped 9 previous similar messages [ 8680.461854] LDISKFS-fs (dm-0): recovery complete [ 8680.469729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8689.829833] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8690.250393] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7649 [ 8690.251783] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6497 [ 8699.381614] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8701.087465] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8708.377674] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:23:23 (1776187403) [ 8709.866205] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8709.870102] Lustre: Skipped 5 previous similar messages [ 8709.880806] LustreError: 239648:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000002f69be63 x1862458435339584/t51539607556(0) o36->84e9dc5e-009a-45d5-a67d-9fdee8de725f@192.168.203.18@tcp:82/0 lens 496/456 e 0 to 0 dl 1776187412 ref 1 fl Interpret:/0/0 rc 0/0 job:'rmdir.0' [ 8709.942741] LustreError: 239648:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 5 previous similar messages [ 8717.718367] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8742.782887] LDISKFS-fs (dm-1): recovery complete [ 8742.786871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.440131] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8748.587325] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:972 to 0x2c0000400:1089 [ 8748.587934] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1345 [ 8758.221760] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8760.514477] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8770.810794] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:24:25 (1776187465) [ 8782.243760] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8791.311111] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8793.292924] Lustre: Failing over lustre-MDT0000 [ 8793.304891] Lustre: Skipped 9 previous similar messages [ 8793.650867] Lustre: server umount lustre-MDT0000 complete [ 8793.657253] Lustre: Skipped 9 previous similar messages [ 8794.080757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.090910] Lustre: Skipped 32 previous similar messages [ 8794.096025] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8794.105268] LustreError: Skipped 211 previous similar messages [ 8801.631221] Lustre: 3349:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187491/real 1776187491] req@0000000051d62f6c x1862458448504960/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187498 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 8801.673232] Lustre: 3349:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 8801.687324] LustreError: 166-1: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8801.699704] LustreError: Skipped 4 previous similar messages [ 8817.976695] LDISKFS-fs (dm-0): recovery complete [ 8817.980945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.176372] Lustre: Evicted from MGS (at 192.168.203.118@tcp) after server handle changed from 0xa141263f70dc0635 to 0xa141263f70dc111e [ 8819.203618] Lustre: Skipped 4 previous similar messages [ 8819.220585] Lustre: MGC192.168.203.118@tcp: Connection restored to (at 0@lo) [ 8819.227610] Lustre: Skipped 41 previous similar messages [ 8819.481959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8819.486948] Lustre: Skipped 9 previous similar messages [ 8819.528640] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8819.531454] Lustre: Skipped 9 previous similar messages [ 8824.519974] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8824.934604] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6529 [ 8824.936956] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7681 [ 8833.344603] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8835.063555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8862.001493] LDISKFS-fs (dm-1): recovery complete [ 8862.005038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8862.466044] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8862.476222] Lustre: Skipped 10 previous similar messages [ 8866.647377] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8867.371538] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:972 to 0x2c0000400:1121 [ 8867.375771] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1377 [ 8875.312986] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8876.988525] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8884.942339] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:26:19 (1776187579) [ 8893.690653] Lustre: lustre-MDT0001: Client 84e9dc5e-009a-45d5-a67d-9fdee8de725f (at 192.168.203.18@tcp) reconnecting [ 8893.703669] Lustre: Skipped 2 previous similar messages [ 8893.978824] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8901.864648] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8908.134469] LustreError: 6239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187604 with bad export cookie 11619610567510855966 [ 8908.162726] LustreError: 6239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8908.256370] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8908.261140] Lustre: Skipped 4 previous similar messages [ 8931.527200] LDISKFS-fs (dm-0): recovery complete [ 8931.530994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8945.756405] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8969.416552] LDISKFS-fs (dm-1): recovery complete [ 8969.425302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8974.510627] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8974.980514] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8974.985920] Lustre: Skipped 9 previous similar messages [ 8975.054315] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:972 to 0x2c0000400:1153 [ 8975.058194] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1409 [ 8975.170960] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6115 to 0x0:6561 [ 8975.171309] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7331 to 0x0:7713 [ 8984.138815] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8985.908036] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8987.407283] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8996.360471] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:28:11 (1776187691) [ 8998.413466] Lustre: 253524:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 84e9dc5e-009a-45d5-a67d-9fdee8de725f at adminstrative request [ 9009.542876] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:28:24 (1776187704) [ 9036.479581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9049.232947] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9050.317301] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6613 to 0x0:6657 [ 9057.751972] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9059.437588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9067.430559] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:29:22 (1776187762) [11123.703369] Lustre: DEBUG MARKER: replay-single test_85b: @@@@@@ FAIL: unused locks (0) should be zero