[ 3068.649772] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3076.481700] LustreError: 90841:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 3076.490721] LustreError: 90841:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3076.496492] Lustre: 90888:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3076.502315] Lustre: 90888:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3076.507635] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 3076.547834] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 3076.637539] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1539 to 0x0:1601
[ 3076.647691] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1603 to 0x0:1697
[ 3080.590625] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3093.937102] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:40:09 (1776181209)
[ 3114.918218] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3116.577922] Lustre: Failing over lustre-MDT0000
[ 3116.579741] Lustre: Skipped 12 previous similar messages
[ 3116.845172] Lustre: server umount lustre-MDT0000 complete
[ 3116.852481] Lustre: Skipped 12 previous similar messages
[ 3127.647287] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3127.659382] Lustre: Skipped 24 previous similar messages
[ 3145.198764] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3148.612475] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3160.790404] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3160.796374] Lustre: Skipped 7 previous similar messages
[ 3160.889308] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3160.895154] Lustre: Skipped 7 previous similar messages
[ 3160.928889] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2002 to 0x0:2017
[ 3160.930138] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113
[ 3165.493554] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3166.851515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3183.213678] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:41:39 (1776181299)
[ 3196.633602] LustreError: 93967:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3196.638167] LustreError: 93967:0:(osd_handler.c:698:osd_ro()) Skipped 11 previous similar messages
[ 3197.227907] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3214.239299] LustreError: 166-1: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3214.249687] LustreError: Skipped 12 previous similar messages
[ 3230.695563] Lustre: Evicted from MGS (at 192.168.202.150@tcp) after server handle changed from 0x3b0b3e0ffe7513f0 to 0x3b0b3e0ffe761d17
[ 3230.720026] Lustre: Skipped 12 previous similar messages
[ 3236.509352] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3237.645223] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2418 to 0x0:2433
[ 3237.645413] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2514 to 0x0:2529
[ 3245.674724] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3247.139627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3266.741200] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 11:43:02 (1776181382)
[ 3268.376678] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 3270.038831] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:43:05 (1776181385)
[ 3272.463031] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 3273.116660] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3273.119242] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 3273.139204] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2418 to 0x0:2465
[ 3278.564790] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:43:14 (1776181394)
[ 3297.598395] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3308.282320] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3308.295557] LustreError: Skipped 1 previous similar message
[ 3309.550481] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3318.491634] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3318.499602] LustreError: Skipped 3 previous similar messages
[ 3329.249186] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:2977
[ 3330.513343] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3390.001950] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:45:05 (1776181505)
[ 3394.274966] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3398.114581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3398.121379] Lustre: Skipped 1 previous similar message
[ 3427.263772] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3427.269022] Lustre: Skipped 20 previous similar messages
[ 3428.730909] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 3428.732658] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2881
[ 3431.321348] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3439.753656] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3441.045654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3459.763275] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:46:15 (1776181575)
[ 3464.195103] LustreError: 98972:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3469.279150] LustreError: 98972:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3469.286093] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnecting
[ 3469.381471] LustreError: 36268:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3471.201557] LustreError: 98972:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3474.399344] LustreError: 98993:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 3474.406818] LustreError: 36268:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3474.422730] LustreError: 36268:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message
[ 3474.434150] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3474.436721] LustreError: 98972:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=1776
[ 3475.425561] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3009
[ 3476.258176] LustreError: 98972:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3481.567609] LustreError: 98972:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3481.573595] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnecting
[ 3481.584719] Lustre: Skipped 1 previous similar message
[ 3483.784325] LustreError: 98971:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3488.994495] LustreError: 98972:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3489.006929] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnecting
[ 3489.006932] LustreError: 98971:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=1
[ 3489.006952] Lustre: lustre-MDT0000: Export 0000000046afcfca already connecting from 192.168.202.50@tcp
[ 3491.421056] LustreError: 98972:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3496.927130] LustreError: 98972:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3504.095315] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnecting
[ 3504.112292] Lustre: Skipped 1 previous similar message
[ 3506.218865] LustreError: 99427:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3506.225951] LustreError: 99427:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 3511.264289] LustreError: 99427:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3511.273767] LustreError: 99427:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 3524.575215] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnecting
[ 3524.586761] Lustre: Skipped 2 previous similar messages
[ 3526.342353] LustreError: 98971:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3526.344734] LustreError: 98971:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages
[ 3531.743439] LustreError: 98971:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3531.761663] LustreError: 98971:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 3540.064221] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 11:47:36 (1776181656)
[ 3541.678718] LustreError: 99427:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 3546.857197] Lustre: lustre-MDT0000: Export 0000000046afcfca already connecting from 192.168.202.50@tcp
[ 3548.903969] Lustre: lustre-MDT0000: Export 0000000046afcfca already connecting from 192.168.202.50@tcp
[ 3551.202443] Lustre: lustre-MDT0000: Export 0000000046afcfca already connecting from 192.168.202.50@tcp
[ 3555.250204] Lustre: lustre-MDT0000: Export 0000000046afcfca already connecting from 192.168.202.50@tcp
[ 3555.253639] Lustre: Skipped 2 previous similar messages
[ 3563.553506] Lustre: lustre-MDT0000: Export 0000000046afcfca already connecting from 192.168.202.50@tcp
[ 3563.558232] Lustre: Skipped 5 previous similar messages
[ 3564.839083] LustreError: 99427:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 3564.849190] Lustre: 99427:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/3s); client may timeout  req@00000000bddf3a84 x1862457868411712/t0(0) o38->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:0/0 lens 520/416 e 0 to 0 dl 1776181679 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0'
[ 3567.318583] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnecting
[ 3567.332476] Lustre: Skipped 2 previous similar messages
[ 3569.766739] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 11:48:05 (1776181685)
[ 3573.134024] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3576.802438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3576.809113] Lustre: Skipped 1 previous similar message
[ 3583.747912] Lustre: MGC192.168.202.150@tcp: Connection restored to  (at 0@lo)
[ 3583.759531] Lustre: Skipped 26 previous similar messages
[ 3583.970657] Lustre: *** cfs_fail_loc=712, val=0***
[ 3583.977725] LustreError: 34475:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@000000008d571d17 x1862457885386816/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 3584.008537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3584.149897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3584.153647] Lustre: Skipped 8 previous similar messages
[ 3584.316725] LustreError: 103266:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 3584.322254] LustreError: 103266:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3584.332823] Lustre: 103313:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3584.339910] Lustre: 103313:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3584.348670] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 3584.475715] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0]
[ 3584.590181] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3041
[ 3587.940282] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3605.471195] Lustre: 3029:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181715/real 1776181715]  req@00000000e10d0f61 x1862457885391360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776181722 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 3605.490319] Lustre: 3029:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 33 previous similar messages
[ 3627.403085] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3635.046393] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2913
[ 3635.048384] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3073
[ 3639.293480] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3640.379792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3647.026471] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 11:49:23 (1776181763)
[ 3647.105685] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnecting
[ 3652.397494] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 11:49:28 (1776181768)
[ 3653.186269] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 3653.191729] LustreError: 104284:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000002e5bd19c x1862457868434688/t0(0) o700->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:486/0 lens 264/248 e 0 to 0 dl 1776181776 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 3697.411071] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3705.707915] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2915 to 0x0:2945
[ 3705.708588] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3075 to 0x0:3105
[ 3711.909288] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3713.493623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3722.918360] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:50:38 (1776181838)
[ 3725.137059] Lustre: Failing over lustre-OST0000
[ 3725.142033] Lustre: Skipped 6 previous similar messages
[ 3725.208444] Lustre: server umount lustre-OST0000 complete
[ 3725.210383] Lustre: Skipped 6 previous similar messages
[ 3726.050982] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3726.077356] LustreError: Skipped 3 previous similar messages
[ 3726.309133] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3745.995154] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137
[ 3748.634943] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3756.736614] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3758.329489] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3828.959757] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:52:24 (1776181944)
[ 3831.715577] LustreError: 108546:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3831.723902] LustreError: 108546:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages
[ 3832.643494] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3842.912596] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3842.934191] Lustre: Skipped 15 previous similar messages
[ 3842.975340] LustreError: 166-1: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3842.985670] LustreError: Skipped 4 previous similar messages
[ 3860.454821] Lustre: Evicted from MGS (at 192.168.202.150@tcp) after server handle changed from 0x3b0b3e0ffe779844 to 0x3b0b3e0ffe77a71d
[ 3860.468071] Lustre: Skipped 4 previous similar messages
[ 3862.238073] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3862.255725] Lustre: Skipped 6 previous similar messages
[ 3863.075433] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3863.083568] Lustre: Skipped 6 previous similar messages
[ 3863.126919] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169
[ 3863.130501] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2966 to 0x0:3009
[ 3865.796130] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3937.849793] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 11:54:13 (1776182053)
[ 3939.482695] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3939.488504] Lustre: Skipped 2 previous similar messages
[ 3939.496052] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201
[ 3940.303634] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3233
[ 3951.424991] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 11:54:27 (1776182067)
[ 3984.104541] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 3984.106798] LustreError: 110875:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000023360a6c x1862457868486016/t0(0) o101->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:62/0 lens 328/344 e 0 to 0 dl 1776182107 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 3987.022814] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3991.258515] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 3991.518552] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3020 to 0x0:3041
[ 3991.520836] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3265
[ 3997.597208] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3999.081649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4006.933281] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 11:55:23 (1776182123)
[ 4009.129414] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4012.833412] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4044.266559] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4044.274315] Lustre: Skipped 8 previous similar messages
[ 4048.759297] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4050.798639] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3043 to 0x0:3073
[ 4050.801058] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3297
[ 4056.443411] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4057.605716] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4064.916769] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 11:56:20 (1776182180)
[ 4066.090455] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4071.402666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4103.498233] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4105.980488] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3043 to 0x0:3105
[ 4105.981673] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4112.894605] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4114.080205] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4122.183946] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 11:57:18 (1776182238)
[ 4123.238830] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4128.201615] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4155.448719] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4155.480086] LustreError: Skipped 7 previous similar messages
[ 4160.618227] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4163.319408] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3043 to 0x0:3137
[ 4163.326713] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 4172.972397] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 11:58:08 (1776182288)
[ 4175.208195] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4175.215571] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 4175.217779] LustreError: 116124:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000c0004ee9 x1862457868512704/t261993005073(0) o35->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:253/0 lens 392/456 e 0 to 0 dl 1776182298 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4204.768650] Lustre: MGC192.168.202.150@tcp: Connection restored to  (at 0@lo)
[ 4204.772098] Lustre: Skipped 27 previous similar messages
[ 4205.322382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4205.326737] Lustre: Skipped 8 previous similar messages
[ 4210.054725] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4215.587543] Lustre: 117467:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000a04e6d5e x1862457868512704/t261993005073(0) o35->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:294/0 lens 392/456 e 0 to 0 dl 1776182339 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4215.603323] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3393
[ 4215.611916] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3169
[ 4221.067261] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4222.607832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4230.293822] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 11:59:06 (1776182346)
[ 4231.124027] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4231.126060] LustreError: 117463:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000006b8ee077 x1862457868521024/t266287972368(0) o36->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:309/0 lens 504/448 e 0 to 0 dl 1776182354 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4236.071151] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4247.519273] Lustre: 3026:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182358/real 1776182358]  req@0000000018bdc843 x1862457885479040/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776182365 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 4247.550430] Lustre: 3026:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
[ 4270.726239] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4271.380301] Lustre: 119223:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000ff53196a x1862457868521024/t266287972368(0) o36->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:349/0 lens 504/448 e 0 to 0 dl 1776182394 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4271.383469] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425
[ 4271.396313] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3201
[ 4278.411613] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4279.699376] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4286.630589] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:00:02 (1776182402)
[ 4287.770402] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4287.772661] LustreError: 119221:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000091f4e13e x1862457868528768/t270582939664(0) o36->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:366/0 lens 504/448 e 0 to 0 dl 1776182411 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4289.700929] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4293.397720] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4293.858335] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnecting
[ 4293.873701] Lustre: Skipped 1 previous similar message
[ 4293.896196] Lustre: 119222:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000f1ce2620 x1862457868528768/t270582939664(0) o36->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:372/0 lens 504/448 e 0 to 0 dl 1776182417 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4293.920906] Lustre: 119222:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 4323.676808] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3457
[ 4323.678226] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3233
[ 4324.278149] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4334.710785] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:00:50 (1776182450)
[ 4335.771649] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4335.777314] Lustre: Skipped 1 previous similar message
[ 4335.782931] LustreError: 120964:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000f841ed9d x1862457868535744/t274877906960(0) o36->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:414/0 lens 504/448 e 0 to 0 dl 1776182459 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4335.803792] LustreError: 120964:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message
[ 4337.400952] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4337.403246] Lustre: Skipped 1 previous similar message
[ 4340.591468] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4341.983088] Lustre: 120964:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000002ce74188 x1862457868535744/t274877906960(0) o36->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:420/0 lens 504/448 e 0 to 0 dl 1776182465 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4342.090836] Lustre: Failing over lustre-MDT0000
[ 4342.092789] Lustre: Skipped 8 previous similar messages
[ 4342.351961] Lustre: server umount lustre-MDT0000 complete
[ 4342.353350] Lustre: Skipped 8 previous similar messages
[ 4360.403804] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (not set up)
[ 4362.450619] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3489
[ 4362.452573] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3265
[ 4364.561349] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4374.343305] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:01:30 (1776182490)
[ 4375.573929] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4377.357404] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4377.359494] LustreError: 122559:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000425cf9a1 x1862457868542848/t279172874256(0) o35->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:455/0 lens 392/456 e 0 to 0 dl 1776182500 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4381.502427] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4381.920640] Lustre: 122559:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000d1457e6d x1862457868542848/t279172874256(0) o35->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:460/0 lens 392/456 e 0 to 0 dl 1776182505 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4412.096685] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521
[ 4412.103892] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3297
[ 4416.347858] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4430.362844] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:02:25 (1776182545)
[ 4431.416226] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 4431.426418] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4431.433665] Lustre: Skipped 1 previous similar message
[ 4431.438108] LustreError: 124048:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000007c4b61ac x1862457868548928/t283467841549(0) o101->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:509/0 lens 664/600 e 0 to 0 dl 1776182554 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0'
[ 4438.765552] Lustre: 124048:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000008e0671b0 x1862457868548928/t283467841549(0) o101->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:517/0 lens 664/3424 e 0 to 0 dl 1776182562 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 4445.518610] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:02:41 (1776182561)
[ 4448.610147] LustreError: 125189:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4448.619918] LustreError: 125189:0:(osd_handler.c:698:osd_ro()) Skipped 7 previous similar messages
[ 4449.676253] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4462.559287] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4462.559693] LustreError: 166-1: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4462.579383] Lustre: Skipped 21 previous similar messages
[ 4462.598224] LustreError: Skipped 9 previous similar messages
[ 4479.904224] Lustre: Evicted from MGS (at 192.168.202.150@tcp) after server handle changed from 0x3b0b3e0ffe77e7dd to 0x3b0b3e0ffe77edc5
[ 4479.908787] Lustre: Skipped 9 previous similar messages
[ 4484.131957] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4494.036895] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4494.049076] Lustre: Skipped 9 previous similar messages
[ 4494.124209] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4494.135428] Lustre: Skipped 9 previous similar messages
[ 4494.202825] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4494.230769] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3553
[ 4498.878614] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4499.942253] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4517.678177] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:03:53 (1776182633)
[ 4521.544756] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4555.876469] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4564.796374] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 4564.797027] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3585
[ 4570.921129] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4572.610963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4578.407846] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 4590.495748] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:05:06 (1776182706)
[ 4631.366345] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4633.584539] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4633.596472] LustreError: Skipped 1 previous similar message
[ 4654.792791] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4654.804919] Lustre: Skipped 9 previous similar messages
[ 4659.664772] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4660.289657] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4865
[ 4660.291122] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4612 to 0x0:4641
[ 4671.257446] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4673.100399] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4756.047541] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:07:51 (1776182871)
[ 4760.933965] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4793.363742] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4643 to 0x0:4673
[ 4793.364038] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4897
[ 4797.369991] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4807.520908] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4809.090934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4821.114493] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 4822.748450] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 4829.153953] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:09:04 (1776182944)
[ 4836.880130] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 4881.115233] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnecting
[ 4881.129409] Lustre: Skipped 3 previous similar messages
[ 4884.454340] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4884.469503] LustreError: 131710:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000009c87f06b x1862457870029312/t300647710728(0) o36->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:243/0 lens 66040/440 e 0 to 0 dl 1776183043 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 4930.297159] Lustre: 132321:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000ce9bde6f x1862457870029312/t300647710728(0) o36->660f15a4-e118-452c-9768-966c8cf666f4@192.168.202.50@tcp:289/0 lens 66040/440 e 0 to 0 dl 1776183089 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 4942.283771] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 4944.042304] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:10:59 (1776183059)
[ 4955.381438] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4958.557850] Lustre: Failing over lustre-MDT0000
[ 4958.559499] Lustre: Skipped 5 previous similar messages
[ 4959.043625] Lustre: server umount lustre-MDT0000 complete
[ 4959.045129] Lustre: Skipped 5 previous similar messages
[ 4968.415811] Lustre: 3026:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183078/real 1776183078]  req@00000000149f9a15 x1862457885891584/t0(0) o400->MGC192.168.202.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183085 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 4968.440832] Lustre: 3026:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 50 previous similar messages
[ 4984.804251] Lustre: MGC192.168.202.150@tcp: Connection restored to  (at 0@lo)
[ 4984.820589] Lustre: Skipped 26 previous similar messages
[ 4985.259859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4985.263688] Lustre: Skipped 8 previous similar messages
[ 4987.127570] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4998 to 0x0:5025
[ 4987.128940] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4775 to 0x0:4801
[ 4989.938936] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4999.704440] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5001.317717] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5010.973200] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:12:06 (1776183126)
[ 5028.680571] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5041.637963] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5061.050338] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5441
[ 5062.519911] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5079.010796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5079.025958] Lustre: Skipped 10 previous similar messages
[ 5097.988960] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5097.998428] Lustre: Skipped 5 previous similar messages
[ 5098.311711] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 5098.323909] Lustre: Skipped 5 previous similar messages
[ 5098.343974] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5473
[ 5101.099798] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5111.305292] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5112.982667] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5153.918891] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:14:29 (1776183269)
[ 5175.325788] LustreError: 166-1: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5175.339232] LustreError: Skipped 4 previous similar messages
[ 5175.349230] Lustre: Evicted from MGS (at 192.168.202.150@tcp) after server handle changed from 0x3b0b3e0ffe7c2ab0 to 0x3b0b3e0ffe7d3a91
[ 5175.358193] Lustre: Skipped 4 previous similar messages
[ 5179.705323] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5180.343098] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5505
[ 5180.343098] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5202 to 0x0:5217
[ 5222.360728] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5537
[ 5224.196222] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5231.707492] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5232.898558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5239.259359] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:15:55 (1776183355)
[ 5256.159556] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5271.353206] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 5271.362451] Lustre: Skipped 6 previous similar messages
[ 5273.498652] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5540 to 0x0:5569
[ 5276.351968] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5285.653876] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5287.116189] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5295.922492] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:16:51 (1776183411)
[ 5304.606616] Lustre: *** cfs_fail_loc=605, val=0***
[ 5304.609058] LustreError: 142799:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=000000008a4cdbad failed: rc = -95
[ 5304.612731] LustreError: 142799:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 5304.616414] LustreError: 142799:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 5304.620668] LustreError: 142799:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 5304.625528] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 5304.629625] LustreError: 142799:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 5304.702592] LustreError: 142799:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 5313.411713] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5540 to 0x0:5601
[ 5313.412463] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5202 to 0x0:5249
[ 5313.913490] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5322.093070] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:17:17 (1776183437)
[ 5325.044995] LustreError: 143848:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5325.051620] LustreError: 143848:0:(osd_handler.c:698:osd_ro()) Skipped 5 previous similar messages
[ 5326.033798] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5329.638048] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping)
[ 5352.889689] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5361.373224] Lustre: *** cfs_fail_loc=707, val=0***
[ 5405.439169] Lustre: lustre-MDT0000: Client 660f15a4-e118-452c-9768-966c8cf666f4 (at 192.168.202.50@tcp) reconnected, waiting for 1 clients in recovery for 0:55
[ 5405.922398] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5615 to 0x0:5633
[ 5405.923141] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5262 to 0x0:5281
[ 5411.735571] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5413.393601] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5421.956509] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:18:57 (1776183537)
[ 5452.255849] LustreError: 7686:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 5455.597392] Lustre: DEBUG MARKER: replay-single test_65a: @@@@@@ FAIL: No early reply