[ 3631.821559] Lustre: *** cfs_fail_loc=513, val=601***
[ 3633.729806] Lustre: *** cfs_fail_loc=513, val=601***
[ 3633.732235] Lustre: Skipped 3 previous similar messages
[ 3634.274155] LustreError: 14361:0:(service.c:2346:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876424851628544
[ 3635.168589] Lustre: *** cfs_fail_loc=513, val=601***
[ 3635.175706] Lustre: Skipped 13 previous similar messages
[ 3637.216204] Lustre: *** cfs_fail_loc=513, val=601***
[ 3637.224572] Lustre: Skipped 9 previous similar messages
[ 3641.825484] Lustre: *** cfs_fail_loc=513, val=601***
[ 3641.834109] Lustre: Skipped 6 previous similar messages
[ 3649.503327] Lustre: 6691:0:(service.c:1612:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@ffff98fcf480aa00 x1876424838538240/t0(0) o4->b3fd1724-4646-40f8-90ad-6ed236611c02@192.168.201.51@tcp:741/0 lens 488/448 e 1 to 0 dl 1789501741 ref 2 fl Interpret:/600/0 rc 0/0 job:'dd.60000' uid:60000 gid:60000 projid:0
[ 3650.527520] Lustre: 6693:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789501721/real 1789501721]  req@ffff98fcffaf5180 x1876424851628544/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789501737 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_002.0' uid:0 gid:0 projid:4294967295
[ 3650.530606] Lustre: *** cfs_fail_loc=513, val=601***
[ 3650.563303] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3650.564246] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting
[ 3650.564825] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 3650.576554] LustreError: 14365:0:(service.c:2346:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876424851632512
[ 3650.594542] Lustre: Skipped 22 previous similar messages
[ 3650.642548] LustreError: 14365:0:(service.c:2346:ptlrpc_server_handle_req_in()) Skipped 1 previous similar message
[ 3651.636225] LustreError: 14360:0:(service.c:2346:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876424851632768
[ 3662.816973] LustreError: 5842:0:(service.c:2346:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876424851634816
[ 3662.836076] LustreError: 5842:0:(service.c:2346:ptlrpc_server_handle_req_in()) Skipped 2 previous similar messages
[ 3666.911371] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789501737/real 1789501737]  req@ffff98fcffaf5180 x1876424851632640/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789501753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3666.947928] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3666.966270] Lustre: *** cfs_fail_loc=513, val=601***
[ 3666.968013] Lustre: Skipped 37 previous similar messages
[ 3666.970813] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting
[ 3667.000192] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 3667.935518] Lustre: 14239:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789501738/real 1789501738]  req@ffff98fcc4695f80 x1876424851632768/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789501754 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_003.0' uid:0 gid:0 projid:4294967295
[ 3667.981763] Lustre: 14239:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3670.075879] LustreError: 14353:0:(service.c:2346:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876424851637120
[ 3678.175148] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789501749/real 1789501749]  req@ffff98fbc2e4d500 x1876424851635072/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789501765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3678.177329] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3678.231772] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3678.274074] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0001_UUID (at 0@lo) reconnecting
[ 3678.288351] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 3686.367194] Lustre: 6692:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789501757/real 1789501757]  req@ffff98fcf486dc00 x1876424851637120/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789501773 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_001.0' uid:0 gid:0 projid:4294967295
[ 3686.417202] Lustre: 6692:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3686.431161] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3686.453757] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting
[ 3686.474796] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 3729.151594] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 15:50:15 (1789501815)
[ 3763.091879] Lustre: Failing over lustre-OST0000
[ 3763.173298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3763.185433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3763.206675] Lustre: server umount lustre-OST0000 complete
[ 3768.292328] LustreError: 8673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3768.318936] LustreError: 8673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages
[ 3770.060459] LustreError: 39263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3773.430731] LustreError: 39264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3776.491677] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 3776.571159] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 3777.960408] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 3778.381903] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 3778.382571] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 3784.411946] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3796.374191] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3802.317197] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3807.112312] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3812.259825] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3818.624932] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3824.697608] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3857.815880] Lustre: Failing over lustre-OST0000
[ 3857.922966] Lustre: server umount lustre-OST0000 complete
[ 3858.407101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3858.417944] LustreError: 39264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3858.425180] LustreError: 39264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 3863.527532] LustreError: 39267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3866.937285] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 3866.955095] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 3868.335201] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3868.521659] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3868.529069] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 3874.567492] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3882.435170] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3888.342348] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3894.540204] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3900.698733] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3906.576056] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3911.743648] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3950.505207] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 15:53:56 (1789502036)
[ 3988.286253] Lustre: *** cfs_fail_loc=a02, val=0***
[ 3995.578966] LustreError: 3309:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff98fcf9642000 id:60000 enforced:1 granted: 1026 pending:0 waiting:0 req:1 usage: 2052 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3995.599458] Lustre: Failing over lustre-OST0000
[ 3995.743996] Lustre: server umount lustre-OST0000 complete
[ 3996.642744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3996.648195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3996.662024] LustreError: 6685:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4004.963746] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 4004.990341] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 4005.596558] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4006.759930] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 4006.762384] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 4012.097435] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4021.503779] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 4028.053326] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 4033.619717] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 4039.943370] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 4045.827803] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 4052.048667] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 4112.848835] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 15:56:38 (1789502198)
[ 4141.454331] LustreError: 90999:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout id a03 sleeping for 10000ms
[ 4141.460533] LustreError: 90999:0:(qsd_reint.c:488:qsd_reint_main()) Skipped 1 previous similar message
[ 4143.406427] Lustre: Failing over lustre-MDT0000
[ 4143.721192] Lustre: server umount lustre-MDT0000 complete
[ 4147.063106] LustreError: 91003:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout interrupted
[ 4154.323462] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4154.735526] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4154.750331] Lustre: Skipped 1 previous similar message
[ 4155.038518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4155.184244] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4159.236165] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 4159.418220] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 4159.492808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:145 to 0x240000400:161)
[ 4159.494360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161)
[ 4159.976473] LustreError: 3306:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x2020000:0x0].0x0 (ffff98fcc2e45b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4160.002143] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 4160.800302] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4164.063546] Lustre: 3309:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789502235/real 1789502235]  req@ffff98fcffe2a680 x1876424852023424/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789502251 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4171.358460] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 4263.940617] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 4271.088662] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 4278.234693] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 4284.323310] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 4290.475530] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 4329.728224] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 16:00:15 (1789502415)
[ 4356.020505] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 4361.926789] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 4414.338986] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 16:01:40 (1789502500)
[ 4416.087726] Lustre: DEBUG MARKER: SKIP: sanity-quota test_7e needs >= 2 MDTs
[ 4418.007135] Lustre: DEBUG MARKER: == sanity-quota test 7f: Quota reintegration automatically ========================================================== 16:01:44 (1789502504)
[ 4436.874994] Lustre: *** cfs_fail_loc=a11, val=0***
[ 4443.170991] Lustre: *** cfs_fail_loc=a11, val=0***
[ 4498.401113] Lustre: *** cfs_fail_loc=a11, val=0***
[ 4498.423612] Lustre: Skipped 1 previous similar message
[ 4621.614872] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 16:05:07 (1789502707)
[ 4826.388488] Lustre: DEBUG MARKER: SKIP: sanity-quota test_9 skipping SLOW test 9
[ 4827.881604] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 16:08:34 (1789502914)
[ 4886.126366] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 16:09:32 (1789502972)
[ 4937.787704] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12a skipping SLOW test 12a
[ 4939.723821] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 16:10:25 (1789503025)
[ 4941.075199] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12b needs >= 2 MDTs
[ 4942.618704] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 16:10:28 (1789503028)
[ 5016.617685] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 16:11:42 (1789503102)
[ 5039.677134] Lustre: Failing over lustre-OST0000
[ 5039.803163] Lustre: server umount lustre-OST0000 complete
[ 5039.864050] LustreError: 101465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5039.871701] LustreError: 101465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages
[ 5040.622859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5040.627964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5044.948320] LustreError: 39264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5044.974979] LustreError: 39264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 5050.061291] LustreError: 39263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5050.100749] LustreError: 39263:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 5052.661285] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 5052.674453] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 5054.242591] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5054.551528] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 5054.551530] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 5054.571444] Lustre: Skipped 1 previous similar message
[ 5058.451126] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5097.873736] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 16:13:03 (1789503183)
[ 5127.510228] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 16:13:33 (1789503213)
[ 5152.466220] Lustre: lustre-OST0000: Client b3fd1724-4646-40f8-90ad-6ed236611c02 (at 192.168.201.51@tcp) reconnecting
[ 5183.193087] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 16:14:29 (1789503269)
[ 5184.882942] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs
[ 5187.170930] Lustre: DEBUG MARKER: == sanity-quota test 16c: lfs quota should preserve usage with an unavailable OST ========================================================== 16:14:32 (1789503272)
[ 5214.081605] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request
[ 5215.937138] Lustre: Failing over lustre-OST0000
[ 5216.032647] Lustre: server umount lustre-OST0000 complete
[ 5218.034311] LustreError: 101451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5218.050473] LustreError: 101451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message
[ 5227.693826] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976a5a6ec000.ost_server_uuid 50
[ 5229.174738] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff976a5a6ec000.ost_server_uuid in DISCONN state after 0 sec
[ 5237.388548] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 5237.407067] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 5237.977852] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5237.987879] Lustre: lustre-OST0000: Denying connection for new client b01ecd27-8a1a-4bf5-b85f-d66031e73ace (at 192.168.201.51@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59
[ 5238.946354] Lustre: lustre-OST0000: Denying connection for new client b01ecd27-8a1a-4bf5-b85f-d66031e73ace (at 192.168.201.51@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:58
[ 5243.084209] Lustre: lustre-OST0000: Denying connection for new client b01ecd27-8a1a-4bf5-b85f-d66031e73ace (at 192.168.201.51@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54
[ 5244.441462] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5247.111508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5247.121055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5247.138984] LustreError: 6690:0:(tgt_handler.c:534:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffff98fcc47a1180 x1876424853137664/t0(0) o13->lustre-MDT0000-mdtlov_UUID@0@lo:80/0 lens 224/0 e 0 to 0 dl 1789503345 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0 projid:4294967295
[ 5247.139145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_get_info to node 0@lo failed: rc = -11
[ 5247.139194] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 5247.167235] LustreError: 6690:0:(tgt_handler.c:534:tgt_filter_recovery_request()) Skipped 1 previous similar message
[ 5248.205324] Lustre: lustre-OST0000: Denying connection for new client b01ecd27-8a1a-4bf5-b85f-d66031e73ace (at 192.168.201.51@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:59
[ 5250.923506] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 5253.323674] Lustre: lustre-OST0000: Denying connection for new client b01ecd27-8a1a-4bf5-b85f-d66031e73ace (at 192.168.201.51@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:54
[ 5263.565746] Lustre: lustre-OST0000: Denying connection for new client b01ecd27-8a1a-4bf5-b85f-d66031e73ace (at 192.168.201.51@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:43
[ 5263.598401] Lustre: Skipped 1 previous similar message
[ 5284.050400] Lustre: lustre-OST0000: Denying connection for new client b01ecd27-8a1a-4bf5-b85f-d66031e73ace (at 192.168.201.51@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:23
[ 5284.067617] Lustre: Skipped 3 previous similar messages
[ 5307.500204] Lustre: lustre-OST0000: recovery is timed out, evict stale exports
[ 5307.504068] Lustre: 113083:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client b3fd1724-4646-40f8-90ad-6ed236611c02@<unknown>
[ 5307.521186] Lustre: lustre-OST0000: disconnecting 1 stale clients
[ 5319.890650] Lustre: lustre-OST0000: Denying connection for new client b01ecd27-8a1a-4bf5-b85f-d66031e73ace (at 192.168.201.51@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 1 evicted) to recover in 0:17
[ 5319.931796] Lustre: Skipped 6 previous similar messages
[ 5337.500172] Lustre: lustre-OST0000: recovery is timed out, evict stale exports
[ 5337.507242] Lustre: 113083:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client lustre-MDT0000-mdtlov_UUID@0@lo
[ 5337.522331] Lustre: lustre-OST0000: disconnecting 1 stale clients
[ 5337.550440] Lustre: lustre-OST0000: Recovery over after 1:40, of 2 clients 0 recovered and 2 were evicted.
[ 5340.136821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5340.162324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5340.183386] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 5348.197345] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
[ 5403.465540] Lustre: DEBUG MARKER: rpc test_16c: @@@@@@ FAIL: can't put import for os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid into FULL state after 50 sec, have
[ 5404.759780] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing check_logdir /tmp/test_logs/1789503434
[ 5407.708650] Lustre: DEBUG MARKER: sanity-quota test_16c: @@@@@@ FAIL: mds1: import is not in FULL state after 50