[ 3549.535168] LDISKFS-fs (dm-0): recovery complete
[ 3549.538916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3549.976941] LustreError: 112582:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 3549.990760] LustreError: 112582:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3549.999587] Lustre: 112615:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3550.005654] Lustre: 112615:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3550.020590] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 3550.031542] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0]
[ 3550.062510] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0]
[ 3550.132129] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1697
[ 3550.133114] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1544 to 0x0:1665
[ 3555.318872] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3556.321444] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3578.228833] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:44:03 (1776181443)
[ 3603.355373] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3606.503523] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3606.512767] LustreError: Skipped 5 previous similar messages
[ 3628.907443] LDISKFS-fs (dm-0): recovery complete
[ 3628.913652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3629.479183] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3629.494294] Lustre: Skipped 21 previous similar messages
[ 3634.943664] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2066 to 0x0:2081
[ 3634.953695] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113
[ 3635.740785] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3647.173645] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3649.027429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3668.617369] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:45:34 (1776181534)
[ 3688.939881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3717.825846] LDISKFS-fs (dm-0): recovery complete
[ 3717.828651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3724.790135] Lustre: MGC192.168.202.155@tcp: Connection restored to  (at 0@lo)
[ 3724.794684] Lustre: Skipped 54 previous similar messages
[ 3725.002520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3725.006236] Lustre: Skipped 10 previous similar messages
[ 3730.258050] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3734.914790] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2482 to 0x0:2497
[ 3734.914831] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2514 to 0x0:2529
[ 3741.908448] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3744.102416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3763.323506] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 11:47:08 (1776181628)
[ 3765.122312] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 3767.029982] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:47:12 (1776181632)
[ 3768.987567] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 3769.971636] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3769.986108] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 3770.002362] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2482 to 0x0:2529
[ 3775.845283] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:47:21 (1776181641)
[ 3796.128581] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3829.214466] LDISKFS-fs (dm-2): recovery complete
[ 3829.219669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3831.741585] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65
[ 3831.756082] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:2977
[ 3833.777649] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3891.469569] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:49:16 (1776181756)
[ 3898.519906] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3914.733122] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3914.760131] LustreError: Skipped 234 previous similar messages
[ 3928.149508] LDISKFS-fs (dm-0): recovery complete
[ 3928.165664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3947.246840] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3948.144207] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 3948.145173] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:2945
[ 3955.169081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3955.180178] Lustre: Skipped 36 previous similar messages
[ 3955.183412] LustreError: 122436:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 3955.185589] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3956.259940] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3009
[ 3956.536530] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3958.288569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3977.086936] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:50:42 (1776181842)
[ 3982.630568] LustreError: 9238:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3987.936025] LustreError: 9238:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3987.950100] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 3988.058299] LustreError: 44564:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3990.041649] LustreError: 9238:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3995.106192] LustreError: 9238:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3995.117049] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 3995.153257] LustreError: 6263:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3995.158459] LustreError: 6263:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message
[ 3997.229180] LustreError: 13997:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 4002.271229] LustreError: 13997:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 4002.276114] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 4002.280877] Lustre: Skipped 1 previous similar message
[ 4004.459406] LustreError: 13997:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 4009.951549] LustreError: 13997:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 4011.625175] LustreError: 89591:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 4017.119404] LustreError: 89591:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 4017.125074] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 4017.132579] Lustre: Skipped 1 previous similar message
[ 4026.011095] LustreError: 89591:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 4026.024784] LustreError: 89591:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 4031.455161] LustreError: 89591:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 4031.464643] LustreError: 89591:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 4038.624383] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 4038.645662] Lustre: Skipped 2 previous similar messages
[ 4048.001499] LustreError: 89591:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 4048.005032] LustreError: 89591:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages
[ 4053.151251] LustreError: 6262:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 4053.156314] LustreError: 89591:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=1
[ 4053.164476] LustreError: 89591:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 4064.278713] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 11:52:09 (1776181929)
[ 4066.085117] LustreError: 13997:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 4071.578839] Lustre: lustre-MDT0000: Export 0000000020136fdf already connecting from 192.168.202.55@tcp
[ 4072.943040] Lustre: lustre-MDT0000: Export 0000000020136fdf already connecting from 192.168.202.55@tcp
[ 4074.384863] Lustre: lustre-MDT0000: Export 0000000020136fdf already connecting from 192.168.202.55@tcp
[ 4078.186375] Lustre: lustre-MDT0000: Export 0000000020136fdf already connecting from 192.168.202.55@tcp
[ 4078.193810] Lustre: Skipped 1 previous similar message
[ 4083.425859] Lustre: lustre-MDT0000: Export 0000000020136fdf already connecting from 192.168.202.55@tcp
[ 4083.438485] Lustre: Skipped 2 previous similar messages
[ 4088.535084] LustreError: 13997:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 4088.542983] Lustre: 13997:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout  req@000000004779b44c x1862457585377920/t0(0) o38->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:0/0 lens 520/416 e 0 to 0 dl 1776181952 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0'
[ 4092.058556] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 4092.063560] Lustre: Skipped 4 previous similar messages
[ 4093.170254] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 11:52:38 (1776181958)
[ 4099.779515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4102.868739] Lustre: Failing over lustre-MDT0000
[ 4102.870318] Lustre: Skipped 9 previous similar messages
[ 4102.972042] Lustre: server umount lustre-MDT0000 complete
[ 4102.974016] Lustre: Skipped 9 previous similar messages
[ 4109.791297] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181969/real 1776181969]  req@00000000389f6339 x1862457604981888/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181976 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 4109.822415] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 4109.832525] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4109.843892] LustreError: Skipped 5 previous similar messages
[ 4114.600822] LDISKFS-fs (dm-0): recovery complete
[ 4114.604651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4115.937110] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0x4792d6c2f61e5fd9 to 0x4792d6c2f61e695d
[ 4115.945940] Lustre: Skipped 5 previous similar messages
[ 4116.052935] Lustre: *** cfs_fail_loc=712, val=0***
[ 4116.057778] LustreError: 44560:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@00000000b17c59da x1862457604986496/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 4116.076373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4116.139979] LustreError: 127113:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery
[ 4116.149817] LustreError: 127113:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 4116.156116] Lustre: 127147:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 4116.158844] Lustre: 127147:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages
[ 4116.166219] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 4116.170847] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0]
[ 4116.193392] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0]
[ 4116.266909] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:2977
[ 4116.274197] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3041
[ 4121.573680] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4121.965292] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4154.872479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4159.504039] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4159.648068] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4159.653723] Lustre: Skipped 4 previous similar messages
[ 4161.020020] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 4161.033771] Lustre: Skipped 4 previous similar messages
[ 4161.073368] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:3009
[ 4161.073558] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3073
[ 4167.526786] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4169.030258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4175.638578] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 11:54:01 (1776182041)
[ 4175.777826] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 4183.056355] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 11:54:08 (1776182048)
[ 4183.859968] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 4183.862130] LustreError: 6271:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000005475da91 x1862457585403584/t0(0) o700->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:11/0 lens 264/248 e 0 to 0 dl 1776182056 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 4212.279642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4227.131107] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3075 to 0x0:3105
[ 4227.132130] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3011 to 0x0:3041
[ 4227.862841] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4236.997266] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4238.747447] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4247.602290] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:55:13 (1776182113)
[ 4251.619759] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 4251.625976] LustreError: Skipped 4 previous similar messages
[ 4268.875208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4269.092293] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 4269.111946] Lustre: Skipped 8 previous similar messages
[ 4271.021877] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97
[ 4271.033411] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137
[ 4274.681615] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4285.877887] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4287.621059] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4359.792156] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:57:05 (1776182225)
[ 4367.098523] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4391.567217] LDISKFS-fs (dm-0): recovery complete
[ 4391.570346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4391.679538] Lustre: MGC192.168.202.155@tcp: Connection restored to 192.168.202.155@tcp (at 0@lo)
[ 4391.689285] Lustre: Skipped 31 previous similar messages
[ 4391.857329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4391.861568] Lustre: Skipped 6 previous similar messages
[ 4395.805225] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4397.550028] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3062 to 0x0:3105
[ 4397.551316] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169
[ 4468.107472] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 11:58:53 (1776182333)
[ 4469.748468] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 4469.753793] Lustre: Skipped 2 previous similar messages
[ 4469.765073] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201
[ 4470.557720] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3233
[ 4481.449413] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 11:59:07 (1776182347)
[ 4504.036254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4509.433884] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4509.693498] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4509.701318] LustreError: 135258:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000004846985a x1862457585459648/t0(0) o101->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:369/0 lens 328/344 e 0 to 0 dl 1776182414 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 4549.783643] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnected, waiting for 2 clients in recovery for 0:57
[ 4549.886917] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3116 to 0x0:3137
[ 4549.889397] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3265
[ 4558.040348] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4559.782329] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4569.929551] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:00:35 (1776182435)
[ 4572.033994] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4578.202113] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4580.832928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4580.845821] Lustre: Skipped 24 previous similar messages
[ 4580.851868] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4580.861445] LustreError: Skipped 192 previous similar messages
[ 4602.880921] LDISKFS-fs (dm-0): recovery complete
[ 4602.885111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4608.715683] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4610.656406] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3297
[ 4610.658299] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3169
[ 4616.942666] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4618.361511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4626.385505] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:01:32 (1776182492)
[ 4627.573873] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4637.538359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4668.181182] LDISKFS-fs (dm-0): recovery complete
[ 4668.188661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4687.908854] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3201
[ 4687.913348] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4688.155454] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4698.151096] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4700.172761] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4709.371381] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:02:54 (1776182574)
[ 4710.725486] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4713.043939] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4721.939953] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4723.965099] Lustre: Failing over lustre-MDT0000
[ 4723.966853] Lustre: Skipped 7 previous similar messages
[ 4724.100146] Lustre: server umount lustre-MDT0000 complete
[ 4724.105953] Lustre: Skipped 7 previous similar messages
[ 4735.903150] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182595/real 1776182595]  req@00000000836656b9 x1862457605156224/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182602 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 4735.917225] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 4735.925356] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4735.935169] LustreError: Skipped 6 previous similar messages
[ 4747.258338] LDISKFS-fs (dm-0): recovery complete
[ 4747.262875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4753.379558] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0x4792d6c2f61ea0ca to 0x4792d6c2f61ea776
[ 4753.385455] Lustre: Skipped 6 previous similar messages
[ 4753.392410] LustreError: 141739:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.202.155@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000a9df901c) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4757.829495] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4759.125032] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3233
[ 4759.127350] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 4768.703487] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:03:54 (1776182634)
[ 4770.958409] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4770.960184] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 4770.972761] LustreError: 40651:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000067f30ac x1862457585493952/t261993005072(0) o35->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:598/0 lens 392/456 e 0 to 0 dl 1776182643 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4792.705763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4799.137672] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4799.145435] Lustre: Skipped 7 previous similar messages
[ 4803.150430] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4803.572496] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 4803.576792] Lustre: Skipped 7 previous similar messages
[ 4803.624064] Lustre: 40651:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000fb943950 x1862457585493952/t261993005072(0) o35->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:630/0 lens 392/456 e 0 to 0 dl 1776182675 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4803.657149] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3363 to 0x0:3393
[ 4803.666319] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3265
[ 4812.124361] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4814.109405] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4822.367861] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:04:47 (1776182687)
[ 4823.661673] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4823.668979] LustreError: 89591:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000005e9878f6 x1862457585502784/t266287972368(0) o36->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:684/0 lens 504/448 e 0 to 0 dl 1776182729 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4834.614941] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4862.378347] LDISKFS-fs (dm-0): recovery complete
[ 4862.394890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4869.684424] Lustre: 6261:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000058d85842 x1862457585502784/t266287972368(0) o36->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:730/0 lens 504/448 e 0 to 0 dl 1776182775 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4869.701228] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425
[ 4869.703358] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3297
[ 4869.798096] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4880.992560] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4883.050237] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4891.883826] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:05:57 (1776182757)
[ 4892.903027] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4892.918431] LustreError: 6263:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000002819040c x1862457585512832/t270582939664(0) o36->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:754/0 lens 504/448 e 0 to 0 dl 1776182799 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4894.852943] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4902.043929] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 4902.054115] Lustre: Skipped 3 previous similar messages
[ 4902.072940] Lustre: 6264:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000002651d2d2 x1862457585512960/t270582939665(0) o35->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:729/0 lens 392/456 e 0 to 0 dl 1776182774 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4902.098390] Lustre: 6264:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 4902.825521] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4905.446430] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4905.452293] LustreError: Skipped 7 previous similar messages
[ 4933.292434] LDISKFS-fs (dm-0): recovery complete
[ 4933.303858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4935.405941] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4935.422318] Lustre: Skipped 7 previous similar messages
[ 4940.424410] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4940.888259] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4940.890719] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3457
[ 4956.460288] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:07:00 (1776182820)
[ 4958.139271] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4958.147016] Lustre: Skipped 1 previous similar message
[ 4958.155419] LustreError: 6261:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000d4d61128 x1862457585521344/t274877906960(0) o36->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:64/0 lens 504/448 e 0 to 0 dl 1776182864 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4958.178042] LustreError: 6261:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message
[ 4960.380610] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4967.621791] Lustre: 13997:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000004c21f3c2 x1862457585521344/t274877906960(0) o36->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:73/0 lens 504/448 e 0 to 0 dl 1776182873 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4971.310610] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5000.551992] LDISKFS-fs (dm-0): recovery complete
[ 5000.561722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5016.041637] Lustre: MGC192.168.202.155@tcp: Connection restored to  (at 0@lo)
[ 5016.054476] Lustre: Skipped 41 previous similar messages
[ 5016.251239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5016.254024] Lustre: Skipped 7 previous similar messages
[ 5021.188608] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5021.809610] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3489
[ 5021.811410] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3361
[ 5032.508419] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:08:17 (1776182897)
[ 5033.724084] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 5035.746672] Lustre: *** cfs_fail_loc=13b, val=315***
[ 5035.756257] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 5035.767379] LustreError: 40651:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000037e331d3 x1862457585530816/t279172874256(0) o35->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:108/0 lens 392/456 e 0 to 0 dl 1776182908 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 5042.848642] Lustre: 40651:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000e0621c67 x1862457585530816/t279172874256(0) o35->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:115/0 lens 392/456 e 0 to 0 dl 1776182915 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 5043.205353] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5071.583621] LDISKFS-fs (dm-0): recovery complete
[ 5071.587494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5092.748119] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5092.974075] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3393
[ 5092.978322] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521
[ 5106.204865] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:09:31 (1776182971)
[ 5107.182147] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 5107.196050] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 5107.198157] LustreError: 6261:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000008cc1ebf2 x1862457585538880/t283467841549(0) o101->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:213/0 lens 664/600 e 0 to 0 dl 1776183013 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0'
[ 5148.844165] Lustre: 9238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000be992a2b x1862457585538880/t283467841549(0) o101->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:255/0 lens 664/3424 e 0 to 0 dl 1776183055 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 5156.146403] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:10:21 (1776183021)
[ 5164.062595] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5180.898170] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5180.920984] LustreError: Skipped 383 previous similar messages
[ 5190.059102] LDISKFS-fs (dm-0): recovery complete
[ 5190.062810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5198.981887] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3425
[ 5198.984203] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3523 to 0x0:3553
[ 5199.125948] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5210.001779] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5211.932098] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5231.612889] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:11:37 (1776183097)
[ 5239.612780] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5244.895824] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5244.909800] Lustre: Skipped 37 previous similar messages
[ 5265.834299] LDISKFS-fs (dm-0): recovery complete
[ 5265.837996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5273.687446] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5275.245616] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3427 to 0x0:3457
[ 5275.247893] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3523 to 0x0:3585
[ 5284.421385] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5286.558568] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5292.118372] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 5303.289706] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:12:48 (1776183168)
[ 5352.453674] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5354.632489] Lustre: Failing over lustre-MDT0000
[ 5354.636669] Lustre: Skipped 7 previous similar messages
[ 5355.055254] Lustre: server umount lustre-MDT0000 complete
[ 5355.060470] Lustre: Skipped 7 previous similar messages
[ 5364.191157] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183223/real 1776183223]  req@0000000016127eea x1862457605332544/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183230 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 5364.230656] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5364.248975] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5364.266274] LustreError: Skipped 7 previous similar messages
[ 5380.839489] LDISKFS-fs (dm-0): recovery complete
[ 5380.844635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5381.607868] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0x4792d6c2f61ed2fe to 0x4792d6c2f6202dfc
[ 5381.625111] Lustre: Skipped 7 previous similar messages
[ 5386.497510] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5387.641711] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4865
[ 5387.642280] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4708 to 0x0:4737
[ 5396.690723] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5399.017818] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5462.166878] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:15:27 (1776183327)
[ 5470.593604] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5494.166603] LDISKFS-fs (dm-0): recovery complete
[ 5494.169730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5499.215040] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 5499.220464] Lustre: Skipped 7 previous similar messages
[ 5502.741831] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5503.582103] Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted.
[ 5503.591634] Lustre: Skipped 7 previous similar messages
[ 5503.617694] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4708 to 0x0:4769
[ 5503.618299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4867 to 0x0:4897
[ 5512.042648] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5513.893413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5525.491521] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 5527.659465] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 5534.544794] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:16:39 (1776183399)
[ 5542.320774] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 5586.073505] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 5586.086431] Lustre: Skipped 3 previous similar messages
[ 5588.465553] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 5588.473319] LustreError: 9238:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000031c30271 x1862457587031168/t300647710728(0) o36->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:694/0 lens 66040/440 e 0 to 0 dl 1776183494 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 5631.157059] Lustre: 6263:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000007483c382 x1862457587031168/t300647710728(0) o36->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:737/0 lens 66040/440 e 0 to 0 dl 1776183537 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 5641.887951] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 5643.342701] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:18:29 (1776183509)
[ 5657.130600] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5682.797797] LDISKFS-fs (dm-0): recovery complete
[ 5682.801510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5685.737240] Lustre: MGC192.168.202.155@tcp: Connection restored to  (at 0@lo)
[ 5685.763441] Lustre: Skipped 29 previous similar messages
[ 5686.049145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5686.052796] Lustre: Skipped 5 previous similar messages
[ 5686.084923] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5686.092576] Lustre: Skipped 6 previous similar messages
[ 5691.332853] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5692.740981] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4870 to 0x0:4897
[ 5692.750103] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4999 to 0x0:5025
[ 5700.927703] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5702.504547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5713.623949] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:19:39 (1776183579)
[ 5736.360035] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5750.757094] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 5750.766054] LustreError: Skipped 5 previous similar messages
[ 5772.965634] LDISKFS-fs (dm-2): recovery complete
[ 5772.972606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5775.885532] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129
[ 5775.896577] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5441
[ 5777.060761] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5793.771073] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5793.791970] LustreError: Skipped 189 previous similar messages
[ 5811.682517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5813.972834] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161
[ 5813.986920] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5473
[ 5817.351399] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5828.471436] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5830.194498] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5869.798491] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:22:15 (1776183735)
[ 5872.096259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5872.109093] Lustre: Skipped 17 previous similar messages
[ 5891.006618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5901.868332] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5902.878829] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5505
[ 5902.879282] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5298 to 0x0:5313
[ 5939.719807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5946.484768] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5947.421265] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5298 to 0x0:5345
[ 5947.422060] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5537
[ 5954.443983] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5955.799317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5962.435826] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:23:48 (1776183828)
[ 5975.807971] Lustre: Failing over lustre-OST0000
[ 5975.809814] Lustre: Skipped 6 previous similar messages
[ 5975.836574] Lustre: server umount lustre-OST0000 complete
[ 5975.838747] Lustre: Skipped 6 previous similar messages
[ 5994.213374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5996.058634] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5569
[ 5996.064634] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:193
[ 5998.059675] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6007.486783] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6008.829906] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6018.133680] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:24:43 (1776183883)
[ 6028.036327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6028.119254] Lustre: *** cfs_fail_loc=605, val=0***
[ 6028.123956] LustreError: 172480:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=000000001ee7f5ef failed: rc = -95
[ 6028.134845] LustreError: 172480:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 6028.140316] LustreError: 172480:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 6028.144425] LustreError: 172480:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 6028.148777] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 6028.151975] LustreError: 172480:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 6028.160831] LustreError: 172480:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 6034.593281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6034.704753] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6034.712891] LustreError: Skipped 4 previous similar messages
[ 6034.719758] LustreError: 172876:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting
[ 6034.726871] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0x4792d6c2f62423c1 to 0x4792d6c2f6242af2
[ 6034.736561] Lustre: Skipped 4 previous similar messages
[ 6038.355330] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6040.104402] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5347 to 0x0:5377
[ 6040.120903] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5601
[ 6046.133811] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:25:11 (1776183911)
[ 6052.661519] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6078.487430] LDISKFS-fs (dm-0): recovery complete
[ 6078.490894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6080.680911] Lustre: *** cfs_fail_loc=707, val=0***
[ 6084.564730] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6122.682541] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnected, waiting for 2 clients in recovery for 0:57
[ 6122.987280] Lustre: lustre-MDT0000: Recovery over after 0:42, of 2 clients 2 recovered and 0 were evicted.
[ 6122.997223] Lustre: Skipped 7 previous similar messages
[ 6123.036342] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5614 to 0x0:5633
[ 6123.037768] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5391 to 0x0:5409
[ 6128.217139] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6129.783239] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6138.931267] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:26:44 (1776184004)
[ 6168.638198] LustreError: 6262:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 6174.703242] LustreError: 6262:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 6192.193654] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:27:37 (1776184057)
[ 6221.562379] LustreError: 8240:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms
[ 6227.608437] LustreError: 8240:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake
[ 6238.294692] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:28:23 (1776184103)
[ 6268.330225] LustreError: 13997:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms
[ 6273.368246] LustreError: 13997:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 6276.049701] LustreError: 6263:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms
[ 6286.063157] LustreError: 6263:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 6304.110713] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 12:29:29 (1776184169)
[ 6364.441738] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 12:30:29 (1776184229)
[ 6395.252436] LustreError: 6263:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 6395.673556] LustreError: 6263:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 6403.565599] LustreError: 6264:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 6403.577501] LustreError: 6264:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 26 previous similar messages
[ 6404.007141] LustreError: 6264:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 6404.013798] LustreError: 6264:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 26 previous similar messages
[ 6419.668558] LustreError: 6261:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 6419.676380] LustreError: 6261:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 66 previous similar messages
[ 6420.103191] LustreError: 6261:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 6420.106594] LustreError: 6261:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 66 previous similar messages
[ 6429.520655] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 12:31:34 (1776184294)
[ 6466.233247] Lustre: DEBUG MARKER: phase 2
[ 6478.652155] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 12:32:23 (1776184343)
[ 6567.459228] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 12:33:52 (1776184432)
[ 6569.138901] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1
[ 6571.628100] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 12:33:56 (1776184436)
[ 6576.508579] Lustre: DEBUG MARKER: Started rundbench load pid=138615 ...
[ 6586.498816] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6589.371279] Lustre: DEBUG MARKER: test_70b fail mds1 1 times
[ 6591.707086] Lustre: Failing over lustre-MDT0000
[ 6591.715056] Lustre: Skipped 2 previous similar messages
[ 6591.820167] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.55@tcp (stopping)
[ 6591.927956] Lustre: server umount lustre-MDT0000 complete
[ 6591.936885] Lustre: Skipped 3 previous similar messages
[ 6592.673514] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6592.701743] LustreError: Skipped 133 previous similar messages
[ 6594.017323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6594.037237] Lustre: Skipped 19 previous similar messages
[ 6601.183075] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184460/real 1776184460]  req@00000000cfa49dc8 x1862457606064704/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184467 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 6601.222161] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 6615.508728] LDISKFS-fs (dm-0): recovery complete
[ 6615.511832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6618.645484] Lustre: MGC192.168.202.155@tcp: Connection restored to  (at 0@lo)
[ 6618.654075] Lustre: Skipped 30 previous similar messages
[ 6619.015803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6619.026969] Lustre: Skipped 7 previous similar messages
[ 6619.121561] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6619.132451] Lustre: Skipped 7 previous similar messages
[ 6620.810535] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 6620.818165] Lustre: Skipped 8 previous similar messages
[ 6623.638232] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6625.849449] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5492 to 0x0:5537
[ 6625.849746] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5735 to 0x0:5761
[ 6633.530645] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6635.073077] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6645.271646] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 6647.964501] Lustre: DEBUG MARKER: test_70b fail mds2 2 times
[ 6649.824904] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 6649.831847] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107
[ 6649.841258] LustreError: Skipped 6 previous similar messages
[ 6673.614920] LDISKFS-fs (dm-1): recovery complete
[ 6673.617721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6677.829916] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6679.152990] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:198 to 0x280000400:225
[ 6679.175061] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33
[ 6688.925411] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 6691.404351] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6728.558432] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 12:36:33 (1776184593)
[ 6857.521870] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6869.941693] Lustre: DEBUG MARKER: test_70c fail mds1 1 times
[ 6871.832728] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.55@tcp (stopping)
[ 6871.838047] Lustre: Skipped 2 previous similar messages
[ 6880.735323] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184739/real 1776184739]  req@0000000086dffd20 x1862457606778432/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184746 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0'
[ 6880.760084] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6880.771268] LustreError: Skipped 2 previous similar messages
[ 6896.604988] LDISKFS-fs (dm-0): recovery complete
[ 6896.612238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6898.175311] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0x4792d6c2f624983d to 0x4792d6c2f62a23d8
[ 6898.195842] Lustre: Skipped 2 previous similar messages
[ 6904.421100] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6907.819715] Lustre: lustre-MDT0000: Recovery over after 0:09, of 2 clients 2 recovered and 0 were evicted.
[ 6907.835065] Lustre: Skipped 2 previous similar messages
[ 6907.896914] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6225 to 0x0:6241
[ 6907.897828] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6002 to 0x0:6017
[ 6915.691763] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6917.730105] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6975.274243] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 12:40:40 (1776184840)
[ 7106.005894] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7119.311740] Lustre: DEBUG MARKER: test_70d fail mds2 1 times
[ 7121.970022] LustreError: 3358:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@0000000038985c54 x1862457607630464/t0(0) o103->lustre-MDT0000-osp-MDT0001@0@lo:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'jbd2/dm-1-8.0'
[ 7121.996752] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.55@tcp (stopping)
[ 7155.903593] LDISKFS-fs (dm-1): recovery complete
[ 7155.920414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7162.556348] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7164.266533] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:446 to 0x2c0000400:481
[ 7164.269125] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:638 to 0x280000400:673
[ 7175.438852] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7178.276284] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7190.078521] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 12:44:14 (1776185054)
[ 7321.897111] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7335.273448] Lustre: DEBUG MARKER: test_70e fail mds2 1 times
[ 7337.294497] Lustre: Failing over lustre-MDT0001
[ 7337.298021] Lustre: Skipped 3 previous similar messages
[ 7337.308418] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107
[ 7337.321554] LustreError: Skipped 2 previous similar messages
[ 7337.324810] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7337.331711] Lustre: Skipped 11 previous similar messages
[ 7337.336715] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 7337.339117] Lustre: Skipped 4 previous similar messages
[ 7337.422965] Lustre: server umount lustre-MDT0001 complete
[ 7337.425207] Lustre: Skipped 3 previous similar messages
[ 7338.175643] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7338.185100] LustreError: Skipped 117 previous similar messages
[ 7360.046405] LDISKFS-fs (dm-1): recovery complete
[ 7360.051785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7360.325913] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 7360.353105] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 7360.368145] Lustre: Skipped 3 previous similar messages
[ 7361.498131] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 7361.510125] Lustre: Skipped 3 previous similar messages
[ 7364.795342] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7365.612407] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to  (at 0@lo)
[ 7365.630696] Lustre: Skipped 15 previous similar messages
[ 7365.695065] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:681 to 0x280000400:705
[ 7365.699392] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:490 to 0x2c0000400:513
[ 7375.412903] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7377.097439] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7386.049741] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 12:47:31 (1776185251)
[ 7398.206850] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 7399.594407] Lustre: lustre-OST0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnecting
[ 7399.604218] Lustre: Skipped 1 previous similar message
[ 7401.140621] Lustre: DEBUG MARKER: test_70f failing OST 1 times
[ 7429.733423] LDISKFS-fs (dm-2): recovery complete
[ 7429.737104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 7429.845304] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 7431.020151] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6862 to 0x0:6881
[ 7431.026213] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:681 to 0x280000400:737
[ 7434.630710] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7445.318665] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 7446.702766] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7458.848643] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 12:48:44 (1776185324)
[ 7587.652566] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7595.380463] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7608.333257] Lustre: DEBUG MARKER: fail mds2 mds1 1 times
[ 7610.982931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 7630.815337] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185490/real 1776185490]  req@0000000061d3f5d7 x1862457609016768/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185497 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0'
[ 7630.849602] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7646.663805] LDISKFS-fs (dm-1): recovery complete
[ 7646.675783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7748.596987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 7748.611619] Lustre: Skipped 3 previous similar messages
[ 7753.612991] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7777.855093] LDISKFS-fs (dm-0): recovery complete
[ 7777.869739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7785.446335] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0x4792d6c2f62a23d8 to 0x4792d6c2f63626d7
[ 7790.827382] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7791.699296] Lustre: lustre-MDT0001: Recovery over after 0:41, of 2 clients 2 recovered and 0 were evicted.
[ 7791.714181] Lustre: Skipped 3 previous similar messages
[ 7791.783184] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:490 to 0x2c0000400:545
[ 7791.786419] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:681 to 0x280000400:769
[ 7799.021433] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6862 to 0x0:6913
[ 7799.025586] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6639 to 0x0:6657
[ 7806.456250] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7808.400931] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7810.130628] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7820.145567] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 12:54:45 (1776185685)
[ 7828.226974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7858.341389] LDISKFS-fs (dm-0): recovery complete
[ 7858.344863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7872.712753] Lustre: *** cfs_fail_loc=302, val=2147483648***
[ 7877.535734] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7879.899094] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnected, waiting for 2 clients in recovery for 0:58
[ 7879.953951] Lustre: 195683:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000099d007cc x1862457593395840/t330712492747(330712492747) o101->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:687/0 lens 648/3424 e 0 to 0 dl 1776185752 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0'
[ 7880.077278] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:6945
[ 7880.078536] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6639 to 0x0:6689
[ 7888.651294] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7890.838379] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7899.920497] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 12:56:05 (1776185765)
[ 7907.545088] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7935.101721] LDISKFS-fs (dm-0): recovery complete
[ 7935.107563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7938.979587] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 7938.995889] LustreError: 197147:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000482c6897 x1862457593395840/t330712492747(330712492747) o101->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:746/0 lens 648/600 e 0 to 0 dl 1776185811 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0'
[ 7942.778819] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7945.402951] Lustre: lustre-MDT0000: Client 720c4e0e-0d1c-4bf2-89de-c0926e705ea9 (at 192.168.202.55@tcp) reconnected, waiting for 2 clients in recovery for 0:58
[ 7945.415897] Lustre: 195684:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000034f05bd4 x1862457593395840/t330712492747(330712492747) o101->720c4e0e-0d1c-4bf2-89de-c0926e705ea9@192.168.202.55@tcp:752/0 lens 648/3424 e 0 to 0 dl 1776185817 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0'
[ 7945.598724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:6977
[ 7945.602265] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6691 to 0x0:6721
[ 7956.220823] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7958.502303] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7972.367209] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 12:57:17 (1776185837)
[ 7978.405247] Lustre: Failing over lustre-OST0000
[ 7978.409634] Lustre: Skipped 5 previous similar messages
[ 7978.460379] Lustre: server umount lustre-OST0000 complete
[ 7978.463243] Lustre: Skipped 5 previous similar messages
[ 7978.982665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7978.998262] Lustre: Skipped 17 previous similar messages
[ 7979.006258] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7979.012029] LustreError: Skipped 273 previous similar messages
[ 7991.263334] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185850/real 1776185850]  req@0000000058d85842 x1862457609242560/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185857 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 7991.294105] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 8003.678703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8008.702583] Lustre: MGC192.168.202.155@tcp: Connection restored to  (at 0@lo)
[ 8008.712574] Lustre: Skipped 21 previous similar messages
[ 8009.072853] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 8009.077960] Lustre: Skipped 5 previous similar messages
[ 8013.522641] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8014.305092] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 8014.315106] Lustre: Skipped 5 previous similar messages
[ 8014.369825] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6691 to 0x0:6753
[ 8020.323549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8021.146428] Lustre: lustre-OST0000: Denying connection for new client 483324c7-9542-46a6-a92e-a2b564637f61 (at 192.168.202.55@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59
[ 8021.155539] Lustre: Skipped 12 previous similar messages
[ 8021.803635] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:681 to 0x280000400:801
[ 8021.829592] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7009
[ 8024.025404] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8033.221730] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 12:58:18 (1776185898)
[ 8034.188882] Lustre: *** cfs_fail_loc=1701, val=2147483648***
[ 8034.191524] LustreError: 8246:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000dd837e44 x1862457609259264/t347892350983(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:86/0 lens 1056/4320 e 0 to 0 dl 1776185906 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0'
[ 8040.229211] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8041.447977] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 8046.564666] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 8046.579969] LustreError: Skipped 5 previous similar messages
[ 8066.492786] LDISKFS-fs (dm-0): recovery complete
[ 8066.496059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8076.977902] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8077.428640] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6785
[ 8077.440858] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7041
[ 8086.073353] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 8087.708715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8096.360392] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 12:59:21 (1776185961)
[ 8097.528172] LustreError: 195687:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@0000000068445c59 x1862457609281472/t352187318282(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:149/0 lens 2200/4320 e 0 to 0 dl 1776185969 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0'
[ 8103.928056] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 8105.007162] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8111.640801] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8134.974765] LDISKFS-fs (dm-1): recovery complete
[ 8134.977626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8139.081555] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8140.321703] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:556 to 0x2c0000400:577
[ 8140.322423] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:812 to 0x280000400:833
[ 8147.427224] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8149.024525] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8158.213269] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:00:23 (1776186023)
[ 8166.388927] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 8166.786881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8174.825034] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8202.146327] LDISKFS-fs (dm-0): recovery complete
[ 8202.148950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8211.009645] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8212.157366] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7073
[ 8212.160812] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6817
[ 8221.265290] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 8223.260488] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8253.630422] LDISKFS-fs (dm-1): recovery complete
[ 8253.643242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8257.964178] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8259.116251] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:588 to 0x2c0000400:609
[ 8259.116498] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:844 to 0x280000400:865
[ 8267.818776] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8270.315502] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8283.919166] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:02:28 (1776186148)
[ 8285.158164] Lustre: *** cfs_fail_loc=1701, val=2147483648***
[ 8285.159826] Lustre: Skipped 2 previous similar messages
[ 8285.174679] LustreError: 195688:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000006da4a151 x1862457609341888/t356482285588(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:337/0 lens 2200/4320 e 0 to 0 dl 1776186157 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0'
[ 8285.208106] LustreError: 195688:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message
[ 8292.323820] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 8298.286383] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8305.830416] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8312.927263] LustreError: 159038:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186179 with bad export cookie 5157420656135421600
[ 8312.932255] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8312.944138] LustreError: 159038:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 8312.969717] LustreError: Skipped 5 previous similar messages
[ 8317.092950] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.55@tcp (stopping)
[ 8317.105196] Lustre: Skipped 4 previous similar messages
[ 8342.852553] LDISKFS-fs (dm-0): recovery complete
[ 8342.855494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8351.418502] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8375.203550] LDISKFS-fs (dm-1): recovery complete
[ 8375.206961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8375.664638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8375.675944] Lustre: Skipped 10 previous similar messages
[ 8381.999118] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:876 to 0x280000400:897
[ 8382.012680] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:620 to 0x2c0000400:641
[ 8382.110077] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6849
[ 8382.110471] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7105
[ 8382.441669] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8393.828418] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8395.603697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8397.415084] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8409.511662] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:04:34 (1776186274)
[ 8417.955041] Lustre: lustre-MDT0001: Client 483324c7-9542-46a6-a92e-a2b564637f61 (at 192.168.202.55@tcp) reconnecting
[ 8418.014130] Lustre: 215920:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000d1c3db55 x1862457594322176/t34359738438(0) o36->483324c7-9542-46a6-a92e-a2b564637f61@192.168.202.55@tcp:470/0 lens 560/448 e 0 to 0 dl 1776186290 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0'
[ 8423.221140] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8450.313432] LDISKFS-fs (dm-0): recovery complete
[ 8450.324595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8451.562847] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0x4792d6c2f636cdfe to 0x4792d6c2f636dc36
[ 8451.580923] Lustre: Skipped 6 previous similar messages
[ 8457.331170] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 8457.333628] Lustre: Skipped 11 previous similar messages
[ 8457.406288] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6881
[ 8457.409096] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7137
[ 8458.305766] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8469.368421] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 8471.329759] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8482.954832] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:05:47 (1776186347)
[ 8484.252128] LustreError: 214237:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000a8d46ddb x1862457594340096/t34359738508(0) o36->483324c7-9542-46a6-a92e-a2b564637f61@192.168.202.55@tcp:536/0 lens 560/448 e 0 to 0 dl 1776186356 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0'
[ 8484.283311] LustreError: 214237:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message
[ 8491.693438] Lustre: lustre-MDT0001: Client 483324c7-9542-46a6-a92e-a2b564637f61 (at 192.168.202.55@tcp) reconnecting
[ 8491.719669] Lustre: 214237:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000bd5f80dd x1862457594340096/t34359738508(0) o36->483324c7-9542-46a6-a92e-a2b564637f61@192.168.202.55@tcp:544/0 lens 560/448 e 0 to 0 dl 1776186364 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0'
[ 8492.727972] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8519.934519] LDISKFS-fs (dm-1): recovery complete
[ 8519.950155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8525.896459] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:662 to 0x2c0000400:705
[ 8525.898710] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:918 to 0x280000400:961
[ 8526.271673] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8538.960138] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8541.174694] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8552.758426] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:06:57 (1776186417)
[ 8554.300453] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 8554.304167] Lustre: Skipped 2 previous similar messages
[ 8561.318744] Lustre: lustre-MDT0001: Client 483324c7-9542-46a6-a92e-a2b564637f61 (at 192.168.202.55@tcp) reconnecting
[ 8561.335367] Lustre: 214237:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000056a228ff x1862457594357056/t38654705734(0) o36->483324c7-9542-46a6-a92e-a2b564637f61@192.168.202.55@tcp:613/0 lens 560/448 e 0 to 0 dl 1776186433 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0'
[ 8565.119436] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8573.333416] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8581.787158] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8581.802297] LustreError: Skipped 251 previous similar messages
[ 8603.549895] LDISKFS-fs (dm-0): recovery complete
[ 8603.557205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8611.815921] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 8611.835245] Lustre: Skipped 41 previous similar messages
[ 8611.942371] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8612.034824] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7169
[ 8612.036063] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6913
[ 8624.426723] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 8627.467320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8632.466132] Lustre: Failing over lustre-MDT0001
[ 8632.472529] Lustre: Skipped 10 previous similar messages
[ 8632.640291] Lustre: server umount lustre-MDT0001 complete
[ 8632.644980] Lustre: Skipped 10 previous similar messages
[ 8637.411328] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8637.423164] Lustre: Skipped 39 previous similar messages
[ 8657.157543] LDISKFS-fs (dm-1): recovery complete
[ 8657.164445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8657.585862] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 8657.592444] Lustre: Skipped 10 previous similar messages
[ 8658.592512] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 8658.613906] Lustre: Skipped 10 previous similar messages
[ 8662.386641] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8663.098548] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:993
[ 8663.099486] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:737
[ 8671.576440] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8673.230964] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8683.903615] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:09:09 (1776186549)
[ 8692.394218] Lustre: lustre-MDT0001: Client 483324c7-9542-46a6-a92e-a2b564637f61 (at 192.168.202.55@tcp) reconnecting
[ 8692.439167] Lustre: 215919:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000068500a31 x1862457594379648/t42949673030(0) o36->483324c7-9542-46a6-a92e-a2b564637f61@192.168.202.55@tcp:744/0 lens 560/448 e 0 to 0 dl 1776186564 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0'
[ 8697.869810] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8708.365385] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8713.696956] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 8713.719107] LustreError: Skipped 7 previous similar messages
[ 8715.728628] LustreError: 159038:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186582 with bad export cookie 5157420656135435229
[ 8715.746115] LustreError: 159038:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 8726.498690] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186585/real 1776186585]  req@00000000bf2b53ab x1862457609466304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776186592 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 8726.552182] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 8741.385732] LDISKFS-fs (dm-0): recovery complete
[ 8741.390508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8757.353783] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8786.243769] LDISKFS-fs (dm-1): recovery complete
[ 8786.248503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8791.588617] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8793.016229] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1025
[ 8793.017099] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:748 to 0x2c0000400:769
[ 8793.030446] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6945
[ 8793.036520] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7201
[ 8801.801968] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8803.898624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8806.058514] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8817.828620] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:11:22 (1776186682)
[ 8819.438975] LustreError: 185304:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@000000001aa617bd x1862457609490880/t373662154768(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:116/0 lens 1728/4320 e 0 to 0 dl 1776186691 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0'
[ 8819.460310] LustreError: 185304:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 2 previous similar messages
[ 8825.832895] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 8828.789972] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8830.951029] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 8830.959411] Lustre: Skipped 8 previous similar messages
[ 8854.911700] LDISKFS-fs (dm-1): recovery complete
[ 8854.926142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8860.460800] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8860.701200] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1057
[ 8860.707062] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:801
[ 8871.480473] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8873.527431] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8882.572966] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:12:27 (1776186747)
[ 8891.370156] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 8891.468779] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8919.944729] LDISKFS-fs (dm-0): recovery complete
[ 8919.955981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8941.659329] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8942.257160] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6977
[ 8942.257299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7233
[ 8953.223909] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 8956.125303] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8966.633497] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:13:51 (1776186831)
[ 8975.328892] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 8975.601921] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8984.242290] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8996.831612] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8996.838851] LustreError: Skipped 4 previous similar messages
[ 9010.596259] LDISKFS-fs (dm-0): recovery complete
[ 9010.601372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9014.469688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9014.477968] Lustre: Skipped 8 previous similar messages
[ 9018.756871] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9020.030471] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7009
[ 9020.031756] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7265
[ 9028.850361] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 9030.700448] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9061.962731] LDISKFS-fs (dm-1): recovery complete
[ 9061.972264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9067.493264] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9067.516041] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 9067.525944] Lustre: Skipped 8 previous similar messages
[ 9067.568305] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1089
[ 9067.569261] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:833
[ 9077.577449] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 9079.315455] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9091.077675] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:15:56 (1776186956)
[ 9092.655858] Lustre: *** cfs_fail_loc=1701, val=2147483648***
[ 9092.658671] Lustre: Skipped 4 previous similar messages
[ 9098.729642] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 9100.814554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 9108.896551] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 9114.530778] LustreError: 6246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186980 with bad export cookie 5157420656135445610
[ 9114.538878] LustreError: 6246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 9138.709842] LDISKFS-fs (dm-0): recovery complete
[ 9138.715987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9146.784258] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0x4792d6c2f6371c6a to 0x4792d6c2f63725c4
[ 9146.806069] Lustre: Skipped 4 previous similar messages
[ 9153.726507] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9179.428952] LDISKFS-fs (dm-1): recovery complete
[ 9179.432655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9184.865481] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9186.137980] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:865
[ 9186.159854] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1121
[ 9186.346607] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7041
[ 9186.347580] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7297
[ 9196.901436] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 9198.559273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9201.317114] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9214.365514] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:17:58 (1776187078)
[ 9223.841678] Lustre: lustre-MDT0001: Client 483324c7-9542-46a6-a92e-a2b564637f61 (at 192.168.202.55@tcp) reconnecting
[ 9223.864660] Lustre: 237878:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000008cd1bb14 x1862457594440256/t60129542146(0) o36->483324c7-9542-46a6-a92e-a2b564637f61@192.168.202.55@tcp:521/0 lens 496/2888 e 0 to 0 dl 1776187096 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0'
[ 9224.166302] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 9226.208871] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9226.216104] LustreError: Skipped 253 previous similar messages
[ 9254.324647] LDISKFS-fs (dm-0): recovery complete
[ 9254.332023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9267.225038] Lustre: MGC192.168.202.155@tcp: Connection restored to  (at 0@lo)
[ 9267.232375] Lustre: Skipped 40 previous similar messages
[ 9267.522178] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 9267.535027] Lustre: Skipped 8 previous similar messages
[ 9268.893849] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 9268.898623] Lustre: Skipped 8 previous similar messages
[ 9273.023944] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7329
[ 9273.025209] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7073
[ 9273.294854] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9284.534228] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 9286.184491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9294.582763] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:19:19 (1776187159)
[ 9304.508883] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 9306.539894] Lustre: Failing over lustre-MDT0001
[ 9306.545542] Lustre: Skipped 9 previous similar messages
[ 9306.717644] Lustre: server umount lustre-MDT0001 complete
[ 9306.720089] Lustre: Skipped 9 previous similar messages
[ 9308.641987] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9308.668057] Lustre: Skipped 36 previous similar messages
[ 9333.382755] LDISKFS-fs (dm-1): recovery complete
[ 9333.385769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9338.639390] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9338.903039] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1153
[ 9338.903511] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:897
[ 9349.274540] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 9351.785664] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9362.275080] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:20:27 (1776187227)
[ 9363.589150] LustreError: 238332:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply  req@00000000ee6ee1d7 x1862457594456896/t64424509442(0) o36->483324c7-9542-46a6-a92e-a2b564637f61@192.168.202.55@tcp:660/0 lens 496/456 e 0 to 0 dl 1776187235 ref 1 fl Interpret:/0/0 rc 0/0 job:'rmdir.0'
[ 9363.626212] LustreError: 238332:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 5 previous similar messages
[ 9370.441252] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 9378.951456] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 9384.943382] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 9384.954621] LustreError: Skipped 6 previous similar messages
[ 9392.096089] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187251/real 1776187251]  req@000000003c1e785e x1862457609645504/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187258 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 9392.141102] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
[ 9405.838254] LDISKFS-fs (dm-0): recovery complete
[ 9405.844414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9415.392777] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7105
[ 9415.394503] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7361
[ 9418.175769] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9434.812651] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 9437.975228] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9468.152838] LDISKFS-fs (dm-1): recovery complete
[ 9468.159030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9474.108860] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:929
[ 9474.115641] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1185
[ 9474.658725] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9487.422656] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 9489.489937] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9500.795318] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:22:46 (1776187366)
[ 9509.533867] Lustre: lustre-MDT0001: Client 483324c7-9542-46a6-a92e-a2b564637f61 (at 192.168.202.55@tcp) reconnecting
[ 9509.543750] Lustre: Skipped 2 previous similar messages
[ 9509.553303] Lustre: 240508:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000007373da75 x1862457594470784/t68719476738(0) o36->483324c7-9542-46a6-a92e-a2b564637f61@192.168.202.55@tcp:51/0 lens 496/2888 e 0 to 0 dl 1776187381 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0'
[ 9509.574114] Lustre: 240508:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 2 previous similar messages
[ 9509.899447] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 9517.738753] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 9524.146231] LustreError: 159038:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187390 with bad export cookie 5157420656135453051
[ 9524.156429] LustreError: 159038:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 9548.042301] LDISKFS-fs (dm-0): recovery complete
[ 9548.045844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9561.000202] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9585.415291] LDISKFS-fs (dm-1): recovery complete
[ 9585.424876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9590.786934] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9592.091614] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1217
[ 9592.093323] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:961
[ 9592.227721] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6915 to 0x0:7393
[ 9592.231790] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7137
[ 9602.478754] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 9604.205746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9606.074989] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9614.913775] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:24:40 (1776187480)
[ 9617.448433] Lustre: 251765:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 483324c7-9542-46a6-a92e-a2b564637f61 at adminstrative request
[ 9628.833861] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:24:53 (1776187493)
[ 9649.631199] LustreError: 166-1: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9649.646542] LustreError: Skipped 4 previous similar messages
[ 9658.700184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9666.263513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9666.266373] Lustre: Skipped 9 previous similar messages
[ 9670.533808] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9671.827485] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 9671.841590] Lustre: Skipped 8 previous similar messages
[ 9671.880721] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7444 to 0x0:7489
[ 9671.883922] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7189 to 0x0:7233
[ 9681.756593] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 9683.637133] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 9694.250068] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:25:59 (1776187559)
[ 9731.998758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9733.952443] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7590 to 0x0:7617
[ 9733.987591] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1249
[ 9736.144678] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9744.672142] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 9745.980882] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 9755.390244] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:27:00 (1776187620)
[ 9769.352920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9769.531979] Lustre: Evicted from MGS (at 192.168.202.155@tcp) after server handle changed from 0x4792d6c2f6376898 to 0x4792d6c2f637a1da
[ 9769.539491] Lustre: Skipped 4 previous similar messages
[ 9773.622556] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9775.149555] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7189 to 0x0:7265
[ 9781.603970] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 13:27:27 (1776187647)
[ 9789.882381] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 9826.276403] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9826.300829] LustreError: Skipped 222 previous similar messages
[ 9836.730795] LDISKFS-fs (dm-2): recovery complete
[ 9836.738517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9842.180829] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9845.800531] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7590 to 0x0:7649
[ 9845.810268] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1281
[ 9852.724960] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 9854.503947] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 9863.533372] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:28:48 (1776187728)
[ 9871.933581] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 9899.997264] LDISKFS-fs (dm-2): recovery complete
[ 9900.002576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9900.331511] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 9900.350513] Lustre: Skipped 9 previous similar messages
[ 9901.848548] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 9901.854973] Lustre: Skipped 9 previous similar messages
[ 9902.292059] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.55@tcp inode [0x20002ea31:0x5:0x0] object 0x0:7650 extent [0-4194303]: client csum 6c8fe2ab, server csum 288be91c
[ 9902.502397] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.155@tcp (at 0@lo)
[ 9902.506818] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7651 to 0x0:7681
[ 9902.516974] Lustre: Skipped 36 previous similar messages
[ 9902.531302] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1313
[ 9904.731866] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9912.789492] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 9914.499736] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 9922.156509] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:29:47 (1776187787)
[ 9928.864543] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 9935.152796] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 9941.242593] Lustre: Failing over lustre-MDT0000
[ 9941.245061] Lustre: Skipped 9 previous similar messages
[ 9941.473197] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9941.484225] Lustre: Skipped 29 previous similar messages
[ 9941.497253] Lustre: server umount lustre-MDT0000 complete
[ 9941.503126] Lustre: Skipped 9 previous similar messages
[ 9945.158593] LustreError: 159038:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187811 with bad export cookie 5157420656135479770
[ 9945.170651] LustreError: 159038:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9969.342802] LDISKFS-fs (dm-0): recovery complete
[ 9969.351236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9982.419293] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 9984.891587] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7267 to 0x0:7297
[10006.649346] LDISKFS-fs (dm-2): recovery complete
[10006.654433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10008.623804] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1345
[10011.541317] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[10027.003606] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:31:32 (1776187892)
[10041.312471] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[10041.319984] LustreError: Skipped 7 previous similar messages
[10050.020256] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187909/real 1776187909]  req@000000000c9f9a20 x1862457609833216/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187916 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[10050.052390] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[10058.211627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10071.925964] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[10080.635680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10085.589849] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[10087.771595] Lustre: lustre-OST0000: Denying connection for new client 4ca0045a-f23b-439a-aaf5-25a4c7ef2b23 (at 192.168.202.55@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04
[10087.788468] Lustre: Skipped 1 previous similar message
[10093.209768] Lustre: lustre-OST0000: Denying connection for new client 4ca0045a-f23b-439a-aaf5-25a4c7ef2b23 (at 192.168.202.55@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:58
[10098.331785] Lustre: lustre-OST0000: Denying connection for new client 4ca0045a-f23b-439a-aaf5-25a4c7ef2b23 (at 192.168.202.55@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53
[10103.454988] Lustre: lustre-OST0000: Denying connection for new client 4ca0045a-f23b-439a-aaf5-25a4c7ef2b23 (at 192.168.202.55@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:48
[10113.694251] Lustre: lustre-OST0000: Denying connection for new client 4ca0045a-f23b-439a-aaf5-25a4c7ef2b23 (at 192.168.202.55@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:38
[10113.706554] Lustre: Skipped 1 previous similar message
[10134.171249] Lustre: lustre-OST0000: Denying connection for new client 4ca0045a-f23b-439a-aaf5-25a4c7ef2b23 (at 192.168.202.55@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:17
[10134.191990] Lustre: Skipped 3 previous similar messages
[10152.000408] Lustre: lustre-OST0000: recovery is timed out, evict stale exports
[10152.005533] Lustre: lustre-OST0000: disconnecting 1 stale clients
[10152.038257] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1036 to 0x280000400:1377
[10152.054534] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7732 to 0x0:7753
[10157.627703] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 60 sec
[10166.795418] Lustre: DEBUG MARKER: free_before: 7646580 free_after: 7646580
[10172.940499] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:33:58 (1776188038)
[10266.696082] Lustre: DEBUG MARKER: replay-single test_90: @@@@@@ FAIL: wait_update OSTs up on MDT0000 failed