[ 3282.916904] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3290.730314] LustreError: 91101:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 3290.732926] LustreError: 91101:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3290.735834] Lustre: 91148:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3290.739926] Lustre: 91148:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3290.743212] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 3290.835615] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1603 to 0x0:1665
[ 3290.849342] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1512 to 0x0:1633
[ 3295.122466] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3307.922255] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:02:32 (1776204152)
[ 3332.040103] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3362.051807] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3362.054684] Lustre: Skipped 21 previous similar messages
[ 3362.527746] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2034 to 0x0:2049
[ 3362.527815] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2066 to 0x0:2081
[ 3367.002865] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3375.932045] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3377.784599] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3396.773794] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:04:01 (1776204241)
[ 3412.639698] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3442.296562] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3450.309341] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2482 to 0x0:2497
[ 3455.775438] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3457.493627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3476.355139] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 18:05:20 (1776204320)
[ 3477.661946] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 3479.460169] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 18:05:23 (1776204323)
[ 3481.290079] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 3482.044199] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3482.048076] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 3482.060849] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2465
[ 3488.244036] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 18:05:32 (1776204332)
[ 3508.548916] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3522.035263] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3522.044417] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3528.174820] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3528.191537] LustreError: Skipped 2 previous similar messages
[ 3540.786797] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:2945
[ 3542.835757] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3601.995542] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 18:07:26 (1776204446)
[ 3605.865051] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3610.091409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3610.100464] Lustre: Skipped 2 previous similar messages
[ 3629.872794] Lustre: MGC192.168.202.160@tcp: Connection restored to 192.168.202.160@tcp (at 0@lo)
[ 3629.887292] Lustre: Skipped 31 previous similar messages
[ 3635.843419] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 3635.852605] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2881
[ 3635.989958] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3646.172677] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3647.689324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3667.755262] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 18:08:32 (1776204512)
[ 3673.720203] LustreError: 99228:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3679.199268] LustreError: 99228:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3679.202077] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 3679.247692] LustreError: 99228:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3680.225122] Lustre: 99252:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204482/real 1776204482]  req@00000000e83eeab9 x1862481641923712/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1776204526 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'osp-pre-0-0.0'
[ 3680.250509] Lustre: 99252:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 50 previous similar messages
[ 3680.255717] LustreError: 36314:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3680.258519] LustreError: 36314:0:(libcfs_fail.h:180:cfs_race()) Skipped 2 previous similar messages
[ 3680.259263] LustreError: 99252:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 3680.260806] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3680.260813] Lustre: Skipped 1 previous similar message
[ 3681.292666] LustreError: 99227:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3681.321160] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:2977
[ 3686.368076] LustreError: 99227:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3686.376382] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 3688.707640] LustreError: 99228:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3694.047513] LustreError: 99228:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3694.061966] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 3694.066855] LustreError: 100397:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3696.372536] LustreError: 99228:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3701.727841] LustreError: 99228:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3703.678583] LustreError: 99228:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3708.895274] LustreError: 99228:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3708.909640] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 3708.914677] Lustre: Skipped 2 previous similar messages
[ 3716.068550] LustreError: 100397:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3717.653799] LustreError: 99228:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3717.659036] LustreError: 99228:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 3722.720277] LustreError: 99228:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3722.734895] LustreError: 99228:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 3730.399603] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 3730.406749] Lustre: Skipped 3 previous similar messages
[ 3730.420706] LustreError: 99226:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3739.458389] LustreError: 100397:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3739.465195] LustreError: 100397:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages
[ 3744.736108] LustreError: 100397:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3744.744362] LustreError: 100397:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 3753.510318] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 18:09:58 (1776204598)
[ 3755.050993] LustreError: 100395:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 3761.148679] Lustre: lustre-MDT0000: Export 00000000c8ea4910 already connecting from 192.168.202.60@tcp
[ 3762.994397] Lustre: lustre-MDT0000: Export 00000000c8ea4910 already connecting from 192.168.202.60@tcp
[ 3764.557584] Lustre: lustre-MDT0000: Export 00000000c8ea4910 already connecting from 192.168.202.60@tcp
[ 3767.947065] Lustre: lustre-MDT0000: Export 00000000c8ea4910 already connecting from 192.168.202.60@tcp
[ 3767.950087] Lustre: Skipped 1 previous similar message
[ 3772.409535] Lustre: lustre-MDT0000: Export 00000000c8ea4910 already connecting from 192.168.202.60@tcp
[ 3772.416276] Lustre: Skipped 2 previous similar messages
[ 3776.801561] LustreError: 100395:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 3776.813819] Lustre: 100395:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout  req@00000000c78e70d3 x1862481624960448/t0(0) o38->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:0/0 lens 520/416 e 0 to 0 dl 1776204621 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0'
[ 3781.108246] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 3781.113608] Lustre: Skipped 4 previous similar messages
[ 3781.298345] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 18:10:26 (1776204626)
[ 3784.738379] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3787.586963] Lustre: Failing over lustre-MDT0000
[ 3787.590935] Lustre: Skipped 7 previous similar messages
[ 3787.744135] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3787.756802] Lustre: Skipped 16 previous similar messages
[ 3787.765598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3787.771955] Lustre: Skipped 1 previous similar message
[ 3787.950845] Lustre: server umount lustre-MDT0000 complete
[ 3787.953974] Lustre: Skipped 7 previous similar messages
[ 3794.685677] Lustre: *** cfs_fail_loc=712, val=0***
[ 3794.688560] LustreError: 36307:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@00000000f050772b x1862481641940416/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 3794.718513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3794.866710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3794.872779] Lustre: Skipped 9 previous similar messages
[ 3794.925970] LustreError: 103526:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 3794.930561] LustreError: 103526:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3794.934906] Lustre: 103574:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3794.939125] Lustre: 103574:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3794.943218] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 3795.001251] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2913
[ 3795.001426] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:3009
[ 3798.882621] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3833.569607] Lustre: Evicted from MGS (at 192.168.202.160@tcp) after server handle changed from 0x4dcdbeabfe79fdf9 to 0x4dcdbeabfe7a01c6
[ 3833.574730] Lustre: Skipped 6 previous similar messages
[ 3838.565403] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3847.797097] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2945
[ 3847.800423] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:3041
[ 3852.961879] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3854.525707] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3861.441101] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 18:11:46 (1776204706)
[ 3861.609706] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 3869.057027] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 18:11:53 (1776204713)
[ 3869.979758] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 3869.987363] LustreError: 104553:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000004f819988 x1862481624983232/t0(0) o700->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:27/0 lens 264/248 e 0 to 0 dl 1776204722 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 3888.607136] LustreError: 166-1: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3888.616983] LustreError: Skipped 6 previous similar messages
[ 3909.562339] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3921.389851] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3921.395307] Lustre: Skipped 5 previous similar messages
[ 3921.484623] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3921.491597] Lustre: Skipped 5 previous similar messages
[ 3921.552497] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2947 to 0x0:2977
[ 3921.555489] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3043 to 0x0:3073
[ 3928.082632] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3929.442927] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3937.375300] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:13:01 (1776204781)
[ 3941.886895] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3941.920364] LustreError: Skipped 3 previous similar messages
[ 3942.372262] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3959.427818] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3084 to 0x0:3105
[ 3962.958453] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3974.246920] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3976.319063] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4046.577357] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:14:51 (1776204891)
[ 4049.252195] LustreError: 108830:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4049.255484] LustreError: 108830:0:(osd_handler.c:698:osd_ro()) Skipped 6 previous similar messages
[ 4050.059309] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4080.174114] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4080.178262] Lustre: Skipped 9 previous similar messages
[ 4083.626386] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4094.854517] Lustre: *** cfs_fail_loc=216, val=0***
[ 4094.856289] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2998 to 0x0:3041
[ 4094.863813] LustreError: 109455:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 4095.904628] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137
[ 4161.725458] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 18:16:46 (1776205006)
[ 4163.372429] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 4163.375681] Lustre: Skipped 2 previous similar messages
[ 4163.382359] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169
[ 4164.041759] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3201
[ 4175.899742] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 18:17:00 (1776205020)
[ 4211.601357] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4221.436385] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4221.442748] LustreError: 111176:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000333cb8ad x1862481625035968/t0(0) o101->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:378/0 lens 328/344 e 0 to 0 dl 1776205073 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 4228.584946] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 4228.735268] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3233
[ 4228.737073] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3053 to 0x0:3073
[ 4233.957620] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4235.380314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4243.753525] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 18:18:08 (1776205088)
[ 4245.880066] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4249.591982] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4278.240937] Lustre: MGC192.168.202.160@tcp: Connection restored to  (at 0@lo)
[ 4278.253551] Lustre: Skipped 22 previous similar messages
[ 4282.336944] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4285.986978] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3075 to 0x0:3105
[ 4285.994919] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3265
[ 4290.428313] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4291.546667] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4298.046249] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 18:19:03 (1776205143)
[ 4299.055675] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4303.909890] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4316.511301] Lustre: 3026:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205155/real 1776205155]  req@000000007d5d9981 x1862481642009536/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776205162 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 4316.528856] Lustre: 3026:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 29 previous similar messages
[ 4333.033834] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4333.041815] LustreError: Skipped 6 previous similar messages
[ 4338.096366] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4339.345139] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3297
[ 4339.352410] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137
[ 4348.135038] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4349.616427] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4359.104804] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 18:20:03 (1776205203)
[ 4360.268806] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4365.918839] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4396.191849] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3169
[ 4396.196960] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4398.975462] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4411.198895] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 18:20:55 (1776205255)
[ 4413.292318] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4413.303878] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 4413.311653] LustreError: 116453:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000008e9ac858 x1862481625061888/t261993005072(0) o35->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:570/0 lens 392/456 e 0 to 0 dl 1776205265 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4416.227615] Lustre: Failing over lustre-MDT0000
[ 4416.235591] Lustre: Skipped 8 previous similar messages
[ 4416.545577] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping)
[ 4416.844108] Lustre: server umount lustre-MDT0000 complete
[ 4416.848711] Lustre: Skipped 8 previous similar messages
[ 4427.103246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4427.128029] Lustre: Skipped 19 previous similar messages
[ 4444.645583] Lustre: Evicted from MGS (at 192.168.202.160@tcp) after server handle changed from 0x4dcdbeabfe7a3796 to 0x4dcdbeabfe7a3da8
[ 4444.659242] Lustre: Skipped 6 previous similar messages
[ 4445.150559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4445.155658] Lustre: Skipped 8 previous similar messages
[ 4448.315750] Lustre: 117800:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000064591c12 x1862481625061888/t261993005072(0) o35->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:605/0 lens 392/456 e 0 to 0 dl 1776205300 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4448.317906] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 4448.323868] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3201
[ 4450.692260] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4460.981773] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4462.905715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4471.463331] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 18:21:56 (1776205316)
[ 4472.479325] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4472.484945] LustreError: 117797:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000ce119a79 x1862481625069632/t266287972368(0) o36->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:629/0 lens 504/448 e 0 to 0 dl 1776205324 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4477.738494] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4479.988796] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 4479.993348] Lustre: Skipped 2 previous similar messages
[ 4493.221438] LustreError: 166-1: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4493.239241] LustreError: Skipped 6 previous similar messages
[ 4516.773787] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4521.097106] Lustre: 119539:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000045a9b083 x1862481625069632/t266287972368(0) o36->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:678/0 lens 504/448 e 0 to 0 dl 1776205373 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4521.109637] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3393
[ 4521.111307] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233
[ 4526.319352] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4527.965513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4537.190778] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 18:23:01 (1776205381)
[ 4538.251235] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4538.261412] LustreError: 119998:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@0000000049a555b6 x1862481625078208/t270582939664(0) o36->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:695/0 lens 504/448 e 0 to 0 dl 1776205390 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4540.456040] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4544.196692] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4545.537425] Lustre: 119540:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000015a0c4e3 x1862481625078208/t270582939664(0) o36->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:702/0 lens 504/448 e 0 to 0 dl 1776205397 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4545.558562] Lustre: 119540:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 4574.189735] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4574.197678] Lustre: Skipped 8 previous similar messages
[ 4574.397305] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4574.410493] Lustre: Skipped 8 previous similar messages
[ 4574.455407] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425
[ 4574.455749] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3265
[ 4577.503297] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4589.216586] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 18:23:53 (1776205433)
[ 4590.447378] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4590.455494] Lustre: Skipped 1 previous similar message
[ 4590.460374] LustreError: 121781:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000ba2dbc2e x1862481625085120/t274877906960(0) o36->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:747/0 lens 504/448 e 0 to 0 dl 1776205442 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4590.493705] LustreError: 121781:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message
[ 4592.456927] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4592.458941] Lustre: Skipped 1 previous similar message
[ 4597.026907] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4597.757140] Lustre: 121278:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000003918c5d4 x1862481625085120/t274877906960(0) o36->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:754/0 lens 504/448 e 0 to 0 dl 1776205449 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4632.561774] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4642.996266] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3457
[ 4643.007082] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3297
[ 4648.792241] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 18:24:53 (1776205493)
[ 4649.852891] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4651.544885] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4651.553226] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 4651.557069] LustreError: 122868:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000a2528597 x1862481625092672/t279172874256(0) o35->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:53/0 lens 392/456 e 0 to 0 dl 1776205503 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4655.043224] LustreError: 123760:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 4655.051120] LustreError: 123760:0:(osd_handler.c:698:osd_ro()) Skipped 6 previous similar messages
[ 4655.789239] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4656.129234] Lustre: 122868:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000008bf134ef x1862481625092672/t279172874256(0) o35->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:58/0 lens 392/456 e 0 to 0 dl 1776205508 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4683.783030] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4683.789613] Lustre: Skipped 8 previous similar messages
[ 4685.960420] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3489
[ 4685.960423] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3329
[ 4687.502105] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4700.578960] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 18:25:44 (1776205544)
[ 4701.618533] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 4701.630841] LustreError: 124370:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000000c6e2661 x1862481625098432/t283467841549(0) o101->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:103/0 lens 664/600 e 0 to 0 dl 1776205553 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0'
[ 4708.856572] Lustre: 124368:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000002859792 x1862481625098432/t283467841549(0) o101->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:111/0 lens 664/3424 e 0 to 0 dl 1776205561 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 4716.059201] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 18:26:00 (1776205560)
[ 4720.780550] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4748.892285] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3361
[ 4748.899483] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521
[ 4751.442368] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4760.856878] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4762.872536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4782.025922] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 18:27:06 (1776205626)
[ 4787.217833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4789.748140] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping)
[ 4819.396280] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4822.380585] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3393
[ 4822.383245] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3553
[ 4829.075812] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4830.697121] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4837.069588] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 4848.968116] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 18:28:13 (1776205693)
[ 4888.185406] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4916.712109] Lustre: MGC192.168.202.160@tcp: Connection restored to  (at 0@lo)
[ 4916.724905] Lustre: Skipped 29 previous similar messages
[ 4918.418089] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4804 to 0x0:4833
[ 4918.423882] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4644 to 0x0:4673
[ 4924.537921] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4936.095352] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4936.543208] Lustre: 3024:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205738/real 1776205738]  req@00000000a66948f6 x1862481642088960/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776205782 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 4936.569089] Lustre: 3024:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 51 previous similar messages
[ 4937.782446] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5007.977503] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 18:30:52 (1776205852)
[ 5012.105808] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5042.348029] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5042.365355] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4835 to 0x0:4865
[ 5042.370713] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4644 to 0x0:4705
[ 5043.680313] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5043.705302] Lustre: Skipped 15 previous similar messages
[ 5051.313278] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5052.822396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5062.870680] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 5064.250032] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 5069.948920] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 18:31:54 (1776205914)
[ 5076.796203] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 5118.461896] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 5118.478111] Lustre: Skipped 4 previous similar messages
[ 5120.664839] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 5120.678672] Lustre: Skipped 1 previous similar message
[ 5120.684595] LustreError: 132021:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000004841c560 x1862481626576960/t300647710728(0) o36->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:557/0 lens 66040/440 e 0 to 0 dl 1776206007 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 5163.523251] Lustre: 132020:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000008e31b535 x1862481626576960/t300647710728(0) o36->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:600/0 lens 66040/440 e 0 to 0 dl 1776206050 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 5174.920638] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 5176.404295] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 18:33:41 (1776206021)
[ 5186.184671] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5189.034573] Lustre: Failing over lustre-MDT0000
[ 5189.038253] Lustre: Skipped 8 previous similar messages
[ 5189.107979] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping)
[ 5189.324598] Lustre: server umount lustre-MDT0000 complete
[ 5189.329623] Lustre: Skipped 8 previous similar messages
[ 5199.199297] LustreError: 166-1: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5199.208895] LustreError: Skipped 7 previous similar messages
[ 5216.747705] Lustre: Evicted from MGS (at 192.168.202.160@tcp) after server handle changed from 0x4dcdbeabfe7e6b04 to 0x4dcdbeabfe7e98a0
[ 5216.757180] Lustre: Skipped 8 previous similar messages
[ 5217.316223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5217.323379] Lustre: Skipped 8 previous similar messages
[ 5221.820779] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5221.837830] Lustre: Skipped 6 previous similar messages
[ 5222.353828] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5223.007641] Lustre: lustre-MDT0000: Recovery over after 0:02, of 1 clients 1 recovered and 0 were evicted.
[ 5223.019746] Lustre: Skipped 6 previous similar messages
[ 5223.059458] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4806 to 0x0:4833
[ 5223.061284] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4967 to 0x0:4993
[ 5231.790877] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5233.474367] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5243.463848] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 18:34:48 (1776206088)
[ 5260.634557] LustreError: 135796:0:(osd_handler.c:698:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only ***
[ 5260.643076] LustreError: 135796:0:(osd_handler.c:698:osd_ro()) Skipped 5 previous similar messages
[ 5261.553109] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5273.086143] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5273.107372] LustreError: Skipped 1 previous similar message
[ 5278.189173] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5278.205512] LustreError: Skipped 1 previous similar message
[ 5288.444976] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5288.460627] LustreError: Skipped 3 previous similar messages
[ 5290.275742] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 5290.289087] Lustre: Skipped 5 previous similar messages
[ 5292.584489] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5409
[ 5294.291401] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5308.933701] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5308.946024] LustreError: Skipped 1 previous similar message
[ 5327.772982] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5441
[ 5329.828273] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5339.640992] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5341.435870] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5381.869533] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 18:37:06 (1776206226)
[ 5411.571173] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5249
[ 5411.571327] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5473
[ 5413.865069] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5457.596764] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5281
[ 5457.597624] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5505
[ 5462.383562] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5471.803175] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5473.105409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5481.323705] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 18:38:46 (1776206326)
[ 5496.359494] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5496.372125] LustreError: Skipped 6 previous similar messages
[ 5517.373641] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.160@tcp (at 0@lo)
[ 5517.393046] Lustre: Skipped 16 previous similar messages
[ 5517.402865] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5507 to 0x0:5537
[ 5520.806222] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5531.534852] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5533.409110] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5544.026749] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 18:39:48 (1776206388)
[ 5555.857170] Lustre: *** cfs_fail_loc=605, val=0***
[ 5555.862041] LustreError: 143118:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=000000003b0609d6 failed: rc = -95
[ 5555.872791] LustreError: 143118:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 5555.882482] LustreError: 143118:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 5555.895931] LustreError: 143118:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 5555.908412] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 5555.917508] LustreError: 143118:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 5556.045673] LustreError: 143118:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 5558.240087] Lustre: 3024:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206397/real 1776206397]  req@000000002c08d050 x1862481642555200/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776206404 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 5558.265654] Lustre: 3024:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 72 previous similar messages
[ 5566.039472] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5507 to 0x0:5569
[ 5566.039492] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5283 to 0x0:5313
[ 5569.643747] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5578.394045] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 18:40:22 (1776206422)
[ 5582.076701] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5585.381725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5616.125369] Lustre: *** cfs_fail_loc=707, val=0***
[ 5618.484888] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5659.169351] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnected, waiting for 1 clients in recovery for 0:56
[ 5659.684165] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5583 to 0x0:5601
[ 5659.692733] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5326 to 0x0:5345
[ 5665.335507] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5667.345569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5676.233644] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 18:42:00 (1776206520)
[ 5706.306547] LustreError: 145282:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 5712.351283] LustreError: 145282:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5730.107714] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 18:42:54 (1776206574)
[ 5759.936675] LustreError: 6322:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms
[ 5765.983118] LustreError: 6322:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake
[ 5775.637428] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 18:43:39 (1776206619)
[ 5808.768868] LustreError: 144819:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms
[ 5813.815117] LustreError: 144819:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5816.846177] LustreError: 144820:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms
[ 5826.887254] LustreError: 144820:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5848.859508] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 18:44:52 (1776206692)
[ 5907.454136] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 18:45:52 (1776206752)
[ 5935.999682] LustreError: 144820:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 5936.415693] LustreError: 144820:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5944.195238] LustreError: 144822:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 5944.203676] LustreError: 144822:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 22 previous similar messages
[ 5944.631423] LustreError: 144822:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5944.637273] LustreError: 144822:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 22 previous similar messages
[ 5960.596194] LustreError: 144820:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 5960.607679] LustreError: 144820:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 53 previous similar messages
[ 5961.039168] LustreError: 144820:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5961.052030] LustreError: 144820:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 53 previous similar messages
[ 5972.393863] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 18:46:56 (1776206816)
[ 6006.996461] Lustre: DEBUG MARKER: phase 2
[ 6017.657584] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 18:47:42 (1776206862)
[ 6103.494828] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 18:49:08 (1776206948)
[ 6105.247906] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1
[ 6107.069291] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 18:49:11 (1776206951)
[ 6112.474296] Lustre: DEBUG MARKER: Started rundbench load pid=131417 ...
[ 6118.078357] LustreError: 151639:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 6118.083211] LustreError: 151639:0:(osd_handler.c:698:osd_ro()) Skipped 1 previous similar message
[ 6119.592405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6122.760543] Lustre: DEBUG MARKER: test_70b fail mds1 1 times
[ 6125.284133] Lustre: Failing over lustre-MDT0000
[ 6125.288673] Lustre: Skipped 7 previous similar messages
[ 6125.806814] Lustre: server umount lustre-MDT0000 complete
[ 6125.812645] Lustre: Skipped 8 previous similar messages
[ 6134.751194] LustreError: 166-1: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6134.760454] LustreError: Skipped 4 previous similar messages
[ 6151.136471] Lustre: Evicted from MGS (at 192.168.202.160@tcp) after server handle changed from 0x4dcdbeabfe7fb6b2 to 0x4dcdbeabfe8009bd
[ 6151.146426] Lustre: Skipped 4 previous similar messages
[ 6151.151342] Lustre: MGC192.168.202.160@tcp: Connection restored to  (at 0@lo)
[ 6151.153819] Lustre: Skipped 6 previous similar messages
[ 6152.004250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6152.013572] Lustre: Skipped 7 previous similar messages
[ 6152.161843] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6152.169711] Lustre: Skipped 6 previous similar messages
[ 6157.288500] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6157.299775] Lustre: Skipped 13 previous similar messages
[ 6159.064821] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6167.147780] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6167.153973] Lustre: Skipped 7 previous similar messages
[ 6167.860827] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6167.870824] Lustre: Skipped 7 previous similar messages
[ 6167.953889] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5424 to 0x0:5441
[ 6167.958553] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5702 to 0x0:5729
[ 6171.487116] Lustre: 3023:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206973/real 1776206973]  req@000000009f0d53f7 x1862481642636224/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776207017 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 6171.503799] Lustre: 3023:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 6174.483632] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6176.397402] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6184.948247] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6188.037038] Lustre: DEBUG MARKER: test_70b fail mds1 2 times
[ 6222.191979] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5753 to 0x0:5793
[ 6222.192791] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5465 to 0x0:5505
[ 6224.186628] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6235.671870] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6238.374282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6263.444956] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 18:51:47 (1776207107)
[ 6389.711647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6402.279096] Lustre: DEBUG MARKER: test_70c fail mds1 1 times
[ 6437.394183] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6443.335322] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6454 to 0x0:6497
[ 6443.336557] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6167 to 0x0:6209
[ 6449.515504] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6450.905097] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6490.664438] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 18:55:35 (1776207335)
[ 6492.445995] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs
[ 6494.414569] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 18:55:38 (1776207338)
[ 6496.373682] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs
[ 6498.602502] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 18:55:42 (1776207342)
[ 6507.201757] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 6510.050190] Lustre: DEBUG MARKER: test_70f failing OST 1 times
[ 6513.637900] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6513.654118] LustreError: Skipped 7 previous similar messages
[ 6523.873453] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6523.880836] LustreError: Skipped 2 previous similar messages
[ 6531.548849] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6503 to 0x0:6529
[ 6534.713285] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6546.641176] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6548.947035] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6562.232942] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 6565.661911] Lustre: DEBUG MARKER: test_70f failing OST 2 times
[ 6566.895590] Lustre: lustre-OST0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnecting
[ 6566.905868] Lustre: Skipped 1 previous similar message
[ 6570.982941] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6570.994130] LustreError: Skipped 3 previous similar messages
[ 6588.575681] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6503 to 0x0:6561
[ 6592.257079] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6603.544263] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6605.287947] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6616.090499] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 18:57:40 (1776207460)
[ 6617.769336] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs
[ 6619.807832] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 18:57:44 (1776207464)
[ 6623.562379] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6652.927386] Lustre: *** cfs_fail_loc=302, val=2147483648***
[ 6657.422462] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6659.070298] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 6659.277282] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6563 to 0x0:6593
[ 6659.281139] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6216 to 0x0:6241
[ 6667.584701] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6669.365643] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6678.344964] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 18:58:42 (1776207522)
[ 6682.773902] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6686.515144] LustreError: 162029:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000fc42a68c) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 6712.348275] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 6712.354861] LustreError: 162485:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000bc733e40 x1862481630109888/t339302416387(339302416387) o101->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:604/0 lens 592/600 e 0 to 0 dl 1776207564 ref 1 fl Interpret:/4/0 rc 301/0 job:'multiop.0'
[ 6716.466872] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6719.502508] Lustre: lustre-MDT0000: Client a542bddd-5b62-47b6-ad60-b918c5d278e1 (at 192.168.202.60@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 6719.523958] Lustre: 162484:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000e2215352 x1862481630109888/t339302416387(339302416387) o101->a542bddd-5b62-47b6-ad60-b918c5d278e1@192.168.202.60@tcp:611/0 lens 592/3424 e 0 to 0 dl 1776207571 ref 1 fl Interpret:/6/0 rc 0/0 job:'multiop.0'
[ 6719.685489] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6273
[ 6719.693395] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6563 to 0x0:6625
[ 6727.233380] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6729.095530] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6739.092628] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 18:59:42 (1776207582)
[ 6743.389172] Lustre: Failing over lustre-OST0000
[ 6743.390871] Lustre: Skipped 6 previous similar messages
[ 6743.517888] Lustre: server umount lustre-OST0000 complete
[ 6743.519802] Lustre: Skipped 6 previous similar messages
[ 6745.569218] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 6745.580259] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6745.603048] LustreError: Skipped 6 previous similar messages
[ 6758.879199] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6758.882029] LustreError: 166-1: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6758.901078] Lustre: Skipped 11 previous similar messages
[ 6758.925132] LustreError: Skipped 4 previous similar messages
[ 6776.290208] Lustre: Evicted from MGS (at 192.168.202.160@tcp) after server handle changed from 0x4dcdbeabfe85f97a to 0x4dcdbeabfe85fd55
[ 6776.296255] Lustre: Skipped 4 previous similar messages
[ 6776.301914] Lustre: MGC192.168.202.160@tcp: Connection restored to  (at 0@lo)
[ 6776.314797] Lustre: Skipped 16 previous similar messages
[ 6776.588858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6776.592947] Lustre: Skipped 6 previous similar messages
[ 6776.735466] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6305
[ 6781.343495] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6787.435613] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 6787.443481] Lustre: Skipped 6 previous similar messages
[ 6788.709947] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6788.718032] Lustre: Skipped 6 previous similar messages
[ 6788.721604] Lustre: lustre-OST0000: Denying connection for new client 8a14253a-aef8-41a4-8435-02c4e5bef3ca (at 192.168.202.60@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59
[ 6788.747394] Lustre: Skipped 11 previous similar messages
[ 6789.109532] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6789.124546] Lustre: Skipped 6 previous similar messages
[ 6789.133374] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6563 to 0x0:6657
[ 6791.754538] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6803.697516] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 19:00:48 (1776207648)
[ 6805.042441] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs
[ 6806.861664] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 19:00:51 (1776207651)
[ 6808.595264] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs
[ 6810.237558] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 19:00:54 (1776207654)
[ 6811.668236] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs
[ 6813.637063] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 19:00:57 (1776207657)
[ 6815.667799] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs
[ 6817.739574] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 19:01:01 (1776207661)
[ 6819.280649] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs
[ 6821.537340] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 19:01:05 (1776207665)
[ 6823.593907] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs
[ 6825.682679] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 19:01:09 (1776207669)
[ 6827.527065] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs
[ 6829.329122] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 19:01:13 (1776207673)
[ 6831.361856] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs
[ 6833.790403] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 19:01:17 (1776207677)
[ 6835.562721] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs
[ 6837.421938] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 19:01:21 (1776207681)
[ 6838.933172] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs
[ 6840.576959] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 19:01:25 (1776207685)
[ 6842.004632] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs
[ 6843.910587] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 19:01:28 (1776207688)
[ 6845.689584] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs
[ 6847.736809] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 19:01:32 (1776207692)
[ 6849.083273] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs
[ 6850.907529] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 19:01:35 (1776207695)
[ 6852.727923] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs
[ 6854.790452] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 19:01:39 (1776207699)
[ 6856.782592] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs
[ 6858.643792] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 19:01:43 (1776207703)
[ 6860.162434] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs
[ 6861.661196] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 19:01:46 (1776207706)
[ 6863.685895] Lustre: 167118:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 8a14253a-aef8-41a4-8435-02c4e5bef3ca at adminstrative request
[ 6874.235333] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 19:01:58 (1776207718)
[ 6891.424108] Lustre: 3026:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776207730/real 1776207730]  req@00000000a34b59db x1862481643045824/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776207737 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 6891.459947] Lustre: 3026:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 39 previous similar messages
[ 6912.019602] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6709 to 0x0:6753
[ 6912.019675] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6357 to 0x0:6401
[ 6912.402770] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6924.736625] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6926.538362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6936.385865] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 19:03:00 (1776207780)
[ 6954.483759] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 6954.494050] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6954.504259] LustreError: Skipped 5 previous similar messages
[ 6972.608474] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6854 to 0x0:6881
[ 6974.576573] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6983.964478] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6985.822061] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6995.630768] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 19:04:00 (1776207840)
[ 7008.360256] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6854 to 0x0:6913
[ 7008.364226] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6357 to 0x0:6433
[ 7012.627693] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7021.998486] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 19:04:26 (1776207866)
[ 7026.115167] LustreError: 171585:0:(osd_handler.c:698:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only ***
[ 7026.132764] LustreError: 171585:0:(osd_handler.c:698:osd_ro()) Skipped 6 previous similar messages
[ 7027.309829] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 7033.824170] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 7051.749636] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:6945
[ 7053.962831] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7063.969323] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 7065.742171] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7075.085107] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 19:05:19 (1776207919)
[ 7080.442596] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 7085.543480] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7085.567430] LustreError: Skipped 14 previous similar messages
[ 7105.191774] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.60@tcp inode [0x200028c71:0x5:0x0] object 0x0:6946 extent [0-1048575]: client csum 4a46e2c4, server csum e447639d
[ 7105.511886] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6947 to 0x0:6977
[ 7108.907245] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7120.141632] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 7121.975322] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7132.082843] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 19:06:16 (1776207976)
[ 7136.587002] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 7141.184934] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7153.103374] LustreError: 5510:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207999 with bad export cookie 5606346757065037783
[ 7189.608698] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6474 to 0x0:6497
[ 7192.804612] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7211.810705] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6947 to 0x0:7009
[ 7213.857776] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7231.633980] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 19:07:55 (1776208075)
[ 7247.848161] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 7286.081910] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7304.850818] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7306.977688] Lustre: lustre-OST0000: Denying connection for new client af799082-f7c8-444e-8455-eae8538efe73 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05
[ 7327.737783] Lustre: lustre-OST0000: Denying connection for new client af799082-f7c8-444e-8455-eae8538efe73 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:44
[ 7327.760754] Lustre: Skipped 3 previous similar messages
[ 7363.570286] Lustre: lustre-OST0000: Denying connection for new client af799082-f7c8-444e-8455-eae8538efe73 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:08
[ 7363.597550] Lustre: Skipped 6 previous similar messages
[ 7372.000342] Lustre: lustre-OST0000: recovery is timed out, evict stale exports
[ 7372.004726] Lustre: lustre-OST0000: disconnecting 1 stale clients
[ 7372.097653] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7020 to 0x0:7041
[ 7377.170867] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 59 sec
[ 7392.447665] Lustre: DEBUG MARKER: free_before: 7518208 free_after: 7518208
[ 7399.160893] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 19:10:43 (1776208243)
[ 7491.594634] Lustre: DEBUG MARKER: replay-single test_90: @@@@@@ FAIL: wait_update OSTs up on MDT0000 failed