[  955.074626] Lustre: DEBUG MARKER: sanity-quota test_6: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0.
[  960.496506] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 04:31:26 (1789893086)
[  963.981412] Lustre: DEBUG MARKER: sanity-quota test_7a: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0
[  969.229449] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 04:31:35 (1789893095)
[  972.520908] Lustre: DEBUG MARKER: sanity-quota test_7b: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0
[  977.772144] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 04:31:44 (1789893104)
[  981.097497] Lustre: DEBUG MARKER: sanity-quota test_7c: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0
[  986.440355] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 04:31:52 (1789893112)
[  995.458206] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[  997.152914] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1013.158579] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 04:32:19 (1789893139)
[ 1022.026303] Lustre: Failing over lustre-MDT0001
[ 1022.135441] Lustre: server umount lustre-MDT0001 complete
[ 1023.969939] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1023.973456] Lustre: Skipped 2 previous similar messages
[ 1027.509895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1027.662614] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1027.697769] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 1029.032525] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1029.050698] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1031.601111] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1032.678154] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1032.682684] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 1032.700723] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33)
[ 1033.316688] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1035.100803] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1036.764123] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1038.497615] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1040.298608] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1068.578812] Lustre: Failing over lustre-MDT0001
[ 1068.793967] Lustre: server umount lustre-MDT0001 complete
[ 1070.027612] LustreError: 7655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1070.037579] LustreError: 7655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages
[ 1073.193695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1073.350784] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1073.358164] Lustre: Skipped 2 previous similar messages
[ 1073.448398] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1073.484772] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 1075.113114] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1075.368583] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1078.759322] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1078.762454] Lustre: Skipped 2 previous similar messages
[ 1078.774745] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 1078.795458] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65)
[ 1078.798137] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1080.983977] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1083.865528] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1086.266846] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1088.720893] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1090.944903] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1125.624346] Lustre: DEBUG MARKER: == sanity-quota test 7f: Quota reintegration automatically ========================================================== 04:34:11 (1789893251)
[ 1130.897539] Lustre: *** cfs_fail_loc=a11, val=0***
[ 1130.898663] Lustre: Skipped 3 previous similar messages
[ 1136.325553] Lustre: *** cfs_fail_loc=a11, val=0***
[ 1165.688421] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 04:34:51 (1789893291)
[ 1233.901447] Lustre: 83587:0:(qsd_reint.c:249:qsd_reint_index()) lustre-MDT0001: index version for fid [0x200000005:0x1003:0x0] is 0, but index isn't empty (1)
[ 1317.477419] LustreError: 6518:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 1327.049506] Lustre: DEBUG MARKER: == sanity-quota test 9: Block limit larger than 4GB (b10707) ========================================================== 04:37:33 (1789893453)
[ 1327.538748] Lustre: DEBUG MARKER: OST0_SIZE: 3603208 required: 4900000
[ 1329.456930] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 04:37:35 (1789893455)
[ 1335.438886] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1335.932842] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1065092).
[ 1349.132916] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 04:37:55 (1789893475)
[ 1369.556841] Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing ======== 04:38:15 (1789893495)
[ 1375.810554] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1376.331372] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:21632).
[ 1405.738736] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 04:38:52 (1789893532)
[ 1411.829915] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1412.369535] Lustre: DEBUG MARKER: File quota isn't 0 (u:quota_usr:513).
[ 1444.900621] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 04:39:31 (1789893571)
[ 1449.600357] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1450.093030] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:34944).
[ 1469.500914] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 04:39:55 (1789893595)
[ 1479.951222] Lustre: Failing over lustre-OST0000
[ 1480.010428] Lustre: server umount lustre-OST0000 complete
[ 1481.184829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 1481.187440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1481.191590] LustreError: 83575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1483.233775] LustreError: 8880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1483.234096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 1483.241644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1485.612644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1485.695060] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1485.702664] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 1486.946290] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 1487.488465] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1487.726843] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 1487.726858] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 1487.733322] Lustre: Skipped 2 previous similar messages
[ 1500.303827] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 04:40:26 (1789893626)
[ 1508.930467] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 04:40:35 (1789893635)
[ 1513.472382] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1513.994300] Lustre: DEBUG MARKER: sanity-quota test_16a: @@@@@@ FAIL: failed to get quota after deactivate OSC
[ 1515.954887] Lustre: lustre-OST0000: Client ae9dcacf-a11d-4878-8e09-0ef4bfecbc2a (at 192.168.202.44@tcp) reconnecting
[ 1530.446279] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 04:40:56 (1789893656)
[ 1530.956112] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs
[ 1531.508514] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 04:40:57 (1789893657)
[ 1535.076160] Lustre: DEBUG MARKER: sanity-quota test_17: @@@@@@ FAIL: Used space(27776) for user quota_usr isn't 0.