[ 2726.041186] Lustre: *** cfs_fail_loc=513, val=0***
[ 2726.571423] Lustre: *** cfs_fail_loc=513, val=0***
[ 2726.573990] Lustre: Skipped 8 previous similar messages
[ 2726.899437] LustreError: 6130:0:(service.c:2127:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1864928785815744
[ 2727.903821] Lustre: *** cfs_fail_loc=513, val=0***
[ 2727.912689] Lustre: Skipped 19 previous similar messages
[ 2729.952451] Lustre: *** cfs_fail_loc=513, val=0***
[ 2729.958491] Lustre: Skipped 9 previous similar messages
[ 2734.047187] Lustre: 39121:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778537308/real 1778537308]  req@000000001786362b x1864928785815744/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1778537315 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'ll_ost_io00_005.0'
[ 2734.068193] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2734.080182] Lustre: *** cfs_fail_loc=513, val=0***
[ 2734.082404] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0001_UUID (at 0@lo) reconnecting
[ 2734.085069] Lustre: Skipped 23 previous similar messages
[ 2734.096680] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2734.164382] LustreError: 6130:0:(service.c:2127:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1864928785817984
[ 2741.215642] Lustre: 3150:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778537315/real 1778537315]  req@00000000b0c2b23b x1864928785817984/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1778537322 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'ll_ost_io00_005.0'
[ 2741.239534] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2741.256896] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0001_UUID (at 0@lo) reconnecting
[ 2741.272764] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2742.243941] Lustre: *** cfs_fail_loc=513, val=0***
[ 2742.245925] Lustre: Skipped 45 previous similar messages
[ 2743.847655] LustreError: 6130:0:(service.c:2127:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1864928785820608
[ 2750.944923] Lustre: 39121:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778537325/real 1778537325]  req@00000000c7efd191 x1864928785820608/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1778537332 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'ll_ost_io00_005.0'
[ 2750.972367] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2750.994225] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0001_UUID (at 0@lo) reconnecting
[ 2751.005354] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2754.086143] LustreError: 6130:0:(service.c:2127:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1864928785823168
[ 2758.630833] Lustre: *** cfs_fail_loc=513, val=0***
[ 2758.634977] Lustre: Skipped 81 previous similar messages
[ 2761.183155] Lustre: 39121:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778537335/real 1778537335]  req@00000000b0b95425 x1864928785823168/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1778537342 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'ll_ost_io00_005.0'
[ 2761.214334] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2761.238513] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0001_UUID (at 0@lo) reconnecting
[ 2761.248371] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2765.293236] LustreError: 6130:0:(service.c:2127:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1864928785825792
[ 2772.448366] Lustre: 39121:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778537346/real 1778537346]  req@00000000c3120012 x1864928785825792/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1778537353 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'ll_ost_io00_005.0'
[ 2772.466887] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2772.494380] Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0001_UUID (at 0@lo) reconnecting
[ 2772.500829] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2830.433413] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 18:10:10 (1778537410)
[ 2855.671328] Lustre: Failing over lustre-OST0000
[ 2855.824261] Lustre: server umount lustre-OST0000 complete
[ 2858.504611] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107
[ 2858.509965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2858.519884] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2859.490170] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2859.504366] LustreError: Skipped 1 previous similar message
[ 2860.326762] LustreError: 73820:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 2860.333146] LustreError: 73820:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 11 previous similar messages
[ 2860.602564] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2864.608491] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2864.620090] LustreError: Skipped 1 previous similar message
[ 2864.837599] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 2864.853966] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 2865.703668] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 2866.478918] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 2866.479615] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.115@tcp (at 0@lo)
[ 2866.499433] Lustre: lustre-OST0000: deleting orphan objects from 0x0:136 to 0x0:161
[ 2867.940971] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2873.427401] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2877.422923] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2881.189805] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2884.944199] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2890.195371] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2894.306940] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2922.323767] Lustre: Failing over lustre-OST0000
[ 2922.414445] Lustre: server umount lustre-OST0000 complete
[ 2922.978801] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 2922.985318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2922.993889] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2922.999929] Lustre: Skipped 2 previous similar messages
[ 2927.598548] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 2927.616638] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 2929.026657] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2929.062150] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 2929.063694] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.115@tcp (at 0@lo)
[ 2929.075052] Lustre: Skipped 1 previous similar message
[ 2929.087818] Lustre: lustre-OST0000: deleting orphan objects from 0x0:136 to 0x0:193
[ 2929.091682] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65
[ 2930.633799] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2936.383641] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2940.527041] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2944.160930] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2947.657494] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2952.729595] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 2956.358413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 2991.817299] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 18:12:51 (1778537571)
[ 3018.798276] LustreError: 82427:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3018.808066] LustreError: 82427:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 14 previous similar messages
[ 3021.839864] Lustre: *** cfs_fail_loc=a02, val=0***
[ 3025.849832] Lustre: Failing over lustre-OST0000
[ 3025.925064] Lustre: server umount lustre-OST0000 complete
[ 3026.400923] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 3026.401086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3026.407618] LustreError: Skipped 2 previous similar messages
[ 3026.425585] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3026.438115] LustreError: Skipped 4 previous similar messages
[ 3030.619919] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 3030.637372] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 3032.438882] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 3032.741513] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 3032.743556] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.204.115@tcp (at 0@lo)
[ 3032.754716] Lustre: Skipped 1 previous similar message
[ 3032.766372] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97
[ 3032.769705] Lustre: lustre-OST0000: deleting orphan objects from 0x0:196 to 0x0:225
[ 3033.475699] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3038.245211] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3041.538053] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3044.874390] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3048.485894] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3053.468641] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3057.437196] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3117.310752] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 18:14:57 (1778537697)
[ 3136.367216] LustreError: 87855:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3136.372381] LustreError: 87855:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 17 previous similar messages
[ 3139.071406] LustreError: 88045:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id a03 sleeping for 10000ms
[ 3139.079916] LustreError: 88045:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 5 previous similar messages
[ 3140.291608] Lustre: Failing over lustre-MDT0000
[ 3140.693515] Lustre: server umount lustre-MDT0000 complete
[ 3142.229426] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3142.245502] LustreError: Skipped 3 previous similar messages
[ 3143.007184] LustreError: 88045:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 3143.011672] LustreError: 11-0: lustre-MDT0000-lwp-OST0001: operation dt_index_read to node 0@lo failed: rc = -107
[ 3143.012326] LustreError: 88045:0:(fail.c:144:__cfs_fail_timeout_set()) Skipped 1 previous similar message
[ 3143.013356] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3143.028788] LustreError: Skipped 1 previous similar message
[ 3143.045193] Lustre: Skipped 2 previous similar messages
[ 3146.446093] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3146.455416] Lustre: Evicted from MGS (at 192.168.204.115@tcp) after server handle changed from 0x993efef251707360 to 0x993efef251710d21
[ 3146.472801] Lustre: MGC192.168.204.115@tcp: Connection restored to 192.168.204.115@tcp (at 0@lo)
[ 3146.478945] Lustre: Skipped 1 previous similar message
[ 3146.716753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3146.787528] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3147.303305] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 3149.602838] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3151.855540] LustreError: 3148:0:(ldlm_resource.c:1127:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x2020000:0x0].0x0 (00000000b988acea) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3151.975908] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 3152.002228] Lustre: lustre-OST0001: deleting orphan objects from 0x0:114 to 0x0:129
[ 3152.002228] Lustre: lustre-OST0000: deleting orphan objects from 0x0:196 to 0x0:257
[ 3154.306979] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3257.689380] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3261.270464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3264.649554] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3269.356364] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3272.466478] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3306.573580] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 18:18:06 (1778537886)
[ 3321.450807] LustreError: 95872:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3321.463333] LustreError: 95872:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 17 previous similar messages
[ 3326.193852] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3329.464669] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3371.698470] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 18:19:11 (1778537951)
[ 3372.945832] Lustre: DEBUG MARKER: SKIP: sanity-quota test_7e requires zpool with active userobj_accounting
[ 3373.761430] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 18:19:13 (1778537953)
[ 3388.522136] LustreError: 98378:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 3388.526925] LustreError: 98378:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 5 previous similar messages
[ 3576.674404] Lustre: DEBUG MARKER: SKIP: sanity-quota test_9 skipping SLOW test 9
[ 3577.456654] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 18:22:37 (1778538157)
[ 3622.120485] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 18:23:22 (1778538202)
[ 3667.844877] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12a skipping SLOW test 12a
[ 3668.743640] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 18:24:08 (1778538248)
[ 3727.646797] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 3728.588537] Lustre: DEBUG MARKER: Files for group (quota_usr):
[ 3730.329522] Lustre: DEBUG MARKER: Files for project (1000):
[ 3744.312062] Lustre: DEBUG MARKER: sanity-quota test_12b: @@@@@@ FAIL: rebalancing failed
[ 3767.902592] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 18:25:48 (1778538348)
[ 3781.039739] Lustre: DEBUG MARKER: Files for user (quota_usr):
[ 3782.658395] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1482).
[ 3821.785331] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 18:26:41 (1778538401)
[ 3832.613599] LustreError: 106810:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 3832.620563] LustreError: 106810:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 11 previous similar messages
[ 3840.684549] Lustre: Failing over lustre-OST0000
[ 3840.723979] Lustre: server umount lustre-OST0000 complete
[ 3842.015809] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 3842.019878] LustreError: Skipped 2 previous similar messages
[ 3842.022684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3842.029470] Lustre: Skipped 2 previous similar messages
[ 3842.033058] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3842.040762] LustreError: Skipped 8 previous similar messages
[ 3847.429898] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 3847.443166] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 3848.740536] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 3849.485128] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 3849.485441] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.115@tcp (at 0@lo)
[ 3849.497389] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1237 to 0x0:1281
[ 3849.498403] Lustre: Skipped 4 previous similar messages
[ 3849.511868] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129
[ 3849.652352] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3887.065937] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 18:27:47 (1778538467)
[ 3904.437789] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 18:28:04 (1778538484)
[ 3918.058694] Lustre: lustre-MDT0001: Client 6ac2e97b-9247-419d-97c1-02b5e3ef51ba (at 192.168.204.15@tcp) reconnecting
[ 3946.364121] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 18:28:46 (1778538526)
[ 3947.158323] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs
[ 3947.999973] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 18:28:48 (1778538528)
[ 3957.923273] Lustre: DEBUG MARKER: sanity-quota test_17: @@@@@@ FAIL: Used space(1482) for user quota_usr isn't 0.