[ 2219.966967] Lustre: *** cfs_fail_loc=513, val=601***
[ 2219.970029] Lustre: Skipped 2 previous similar messages
[ 2220.235607] LustreError: 11787:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876834158759680
[ 2220.512662] Lustre: *** cfs_fail_loc=513, val=601***
[ 2220.515049] Lustre: Skipped 14 previous similar messages
[ 2225.087202] Lustre: *** cfs_fail_loc=513, val=601***
[ 2225.090437] Lustre: Skipped 8 previous similar messages
[ 2230.206686] Lustre: *** cfs_fail_loc=513, val=601***
[ 2230.208180] Lustre: Skipped 9 previous similar messages
[ 2235.326554] Lustre: *** cfs_fail_loc=513, val=601***
[ 2235.328017] Lustre: Skipped 10 previous similar messages
[ 2235.872313] Lustre: 40749:0:(service.c:1612:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@ffff8d6b01379500 x1876834156316800/t0(0) o4->90757b05-b2b9-45f6-8cf0-6729daed05cb@192.168.202.5@tcp:129/0 lens 488/448 e 1 to 0 dl 1789890709 ref 2 fl Interpret:/600/0 rc 0/0 job:'dd.60000' uid:60000 gid:60000 projid:0
[ 2236.384186] Lustre: 6568:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789890689/real 1789890689]  req@ffff8d6c09347100 x1876834158759680/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789890705 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_000.0' uid:0 gid:0 projid:4294967295
[ 2236.399770] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2236.405247] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting
[ 2236.408585] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 2237.450384] LustreError: 14224:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876834158763904
[ 2245.566989] Lustre: *** cfs_fail_loc=513, val=601***
[ 2245.571340] Lustre: Skipped 24 previous similar messages
[ 2253.792155] Lustre: 6569:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789890706/real 1789890706]  req@ffff8d6c09344a80 x1876834158763904/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789890722 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_001.0' uid:0 gid:0 projid:4294967295
[ 2253.811523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2253.826649] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting
[ 2253.834914] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 2255.817718] LustreError: 14226:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876834158767616
[ 2257.888607] LustreError: 5770:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1876834158768128
[ 2257.892774] LustreError: 5770:0:(service.c:2341:ptlrpc_server_handle_req_in()) Skipped 1 previous similar message
[ 2261.984360] Lustre: *** cfs_fail_loc=513, val=601***
[ 2261.986856] Lustre: Skipped 42 previous similar messages
[ 2271.200307] Lustre: 6570:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789890724/real 1789890724]  req@ffff8d6c09346680 x1876834158767616/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789890740 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_002.0' uid:0 gid:0 projid:4294967295
[ 2271.218845] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2271.232624] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0000_UUID (at 0@lo) reconnecting
[ 2271.239625] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 2273.249075] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789890726/real 1789890726]  req@ffff8d6c11f70a80 x1876834158768128/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789890742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 2273.249132] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2273.256847] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 2273.263675] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0001_UUID (at 0@lo) reconnecting
[ 2273.266216] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 2297.177351] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 03:52:45 (1789890765)
[ 2313.296050] Lustre: Failing over lustre-OST0000
[ 2313.353951] Lustre: server umount lustre-OST0000 complete
[ 2314.209718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2318.546953] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 2318.553876] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 2319.715737] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2319.892665] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 2319.892752] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 2320.904312] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2324.792094] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2327.347273] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2329.647379] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2331.940758] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2334.365115] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2336.866520] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2364.137517] Lustre: Failing over lustre-OST0000
[ 2364.255222] Lustre: server umount lustre-OST0000 complete
[ 2364.905557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2364.922715] LustreError: 41040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2364.948502] LustreError: 41040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages
[ 2370.019471] LustreError: 6564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2371.187843] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 2371.214841] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 2372.278613] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2372.331045] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2372.332361] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 2377.259392] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2384.285225] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2389.807916] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2396.114102] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2400.356778] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2404.797424] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2409.168437] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2435.193670] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 03:55:03 (1789890903)
[ 2460.875802] Lustre: *** cfs_fail_loc=a02, val=0***
[ 2465.608388] LustreError: 3284:0:(qsd_handler.c:318:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff8d6b04da7728 id:60000 enforced:1 granted: 1026 pending:0 waiting:0 req:1 usage: 2052 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 2465.626314] Lustre: Failing over lustre-OST0000
[ 2465.699855] Lustre: server umount lustre-OST0000 complete
[ 2465.742821] LustreError: 41038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2468.324786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2468.334496] LustreError: 41037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2470.090239] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 2470.111099] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 2470.848892] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2471.740841] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 2471.741269] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 2473.830612] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2478.889211] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2481.713845] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2484.716418] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2487.781677] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2490.415467] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2493.383050] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2534.050388] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 03:56:42 (1789891002)
[ 2556.201685] LustreError: 95460:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout id a03 sleeping for 10000ms
[ 2557.376622] Lustre: Failing over lustre-MDT0000
[ 2557.410583] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2557.464704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2557.470213] Lustre: Skipped 1 previous similar message
[ 2557.691283] Lustre: server umount lustre-MDT0000 complete
[ 2560.080145] LustreError: 95460:0:(qsd_reint.c:488:qsd_reint_main()) cfs_fail_timeout interrupted
[ 2560.085389] LustreError: 95460:0:(qsd_reint.c:488:qsd_reint_main()) Skipped 2 previous similar messages
[ 2564.092847] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2564.457996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2564.575321] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2567.723368] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2568.148657] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2568.214204] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2568.253760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:145 to 0x240000400:161)
[ 2568.253868] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161)
[ 2572.993920] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2577.701230] LustreError: 3283:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x2020000:0x0].0x0 (ffff8d6c0b258a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 2577.712463] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 2673.784155] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2676.298162] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2678.766653] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2681.395865] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2684.063262] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2707.765042] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 03:59:36 (1789891176)
[ 2723.184571] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2724.772143] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2751.444787] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 04:00:20 (1789891220)
[ 2751.953643] Lustre: DEBUG MARKER: SKIP: sanity-quota test_7e needs >= 2 MDTs
[ 2752.457247] Lustre: DEBUG MARKER: == sanity-quota test 7f: Quota reintegration automatically ========================================================== 04:00:21 (1789891221)
[ 2762.597204] Lustre: *** cfs_fail_loc=a11, val=0***
[ 2762.598924] Lustre: Skipped 2 previous similar messages
[ 2767.964958] Lustre: *** cfs_fail_loc=a11, val=0***
[ 2822.713089] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 04:01:31 (1789891291)
[ 2999.386595] Lustre: DEBUG MARKER: SKIP: sanity-quota test_9 skipping SLOW test 9
[ 3000.197904] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 04:04:28 (1789891468)
[ 3036.115085] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 04:05:05 (1789891505)
[ 3068.684732] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12a skipping SLOW test 12a
[ 3069.164651] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 04:05:38 (1789891538)
[ 3069.617332] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12b needs >= 2 MDTs
[ 3070.111110] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 04:05:39 (1789891539)
[ 3109.852306] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 04:06:18 (1789891578)
[ 3125.625213] Lustre: Failing over lustre-OST0000
[ 3125.656284] Lustre: server umount lustre-OST0000 complete
[ 3125.729497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3125.735555] Lustre: Skipped 1 previous similar message
[ 3125.738189] LustreError: 41038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3130.626657] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 3130.633143] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 3131.326993] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 3131.829265] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 3131.829422] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 3131.839043] Lustre: Skipped 1 previous similar message
[ 3132.935793] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3161.163927] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 04:07:10 (1789891630)
[ 3174.872854] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 04:07:23 (1789891643)
[ 3186.270422] Lustre: lustre-OST0000: Client 90757b05-b2b9-45f6-8cf0-6729daed05cb (at 192.168.202.5@tcp) reconnecting
[ 3202.406279] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 04:07:51 (1789891671)
[ 3202.848518] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs
[ 3203.370283] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 04:07:52 (1789891672)
[ 3216.546706] Lustre: *** cfs_fail_loc=a04, val=37***
[ 3216.548824] LustreError: 40751:0:(qsd_handler.c:318:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x11  qsd:lustre-OST0001 qtype:usr lqe: ffff8d6c277eee40 id:60000 enforced:1 granted: 0 pending:0 waiting:1 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3217.603120] Lustre: *** cfs_fail_loc=a04, val=37***
[ 3217.604592] LustreError: 6569:0:(qsd_handler.c:318:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x11  qsd:lustre-OST0001 qtype:usr lqe: ffff8d6c277eee40 id:60000 enforced:1 granted: 0 pending:0 waiting:1024 req:1 usage: 0 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3222.996320] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3223.494742] Lustre: DEBUG MARKER: sanity-quota test_17: @@@@@@ FAIL: Used space(0K) is less than 1M