[ 1053.685189] Lustre: DEBUG MARKER: sanity-quota test_6: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0.
[ 1059.042294] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 03:43:53 (1789890233)
[ 1062.444254] Lustre: DEBUG MARKER: sanity-quota test_7a: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0
[ 1067.619828] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 03:44:01 (1789890241)
[ 1070.888308] Lustre: DEBUG MARKER: sanity-quota test_7b: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0
[ 1075.980886] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 03:44:10 (1789890250)
[ 1079.245751] Lustre: DEBUG MARKER: sanity-quota test_7c: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0
[ 1084.581399] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 03:44:18 (1789890258)
[ 1095.236106] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1096.936201] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1111.547392] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 03:44:45 (1789890285)
[ 1121.672800] Lustre: Failing over lustre-MDT0001
[ 1121.784485] Lustre: server umount lustre-MDT0001 complete
[ 1122.273688] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107
[ 1122.274197] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1127.194751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1127.341209] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1127.371740] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 1128.732353] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1128.910629] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1131.246203] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1132.520283] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1132.526556] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 1132.544589] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33)
[ 1133.090901] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1134.908785] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1136.626952] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1138.377836] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1140.036477] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1160.957981] Lustre: Failing over lustre-MDT0001
[ 1161.108413] Lustre: server umount lustre-MDT0001 complete
[ 1163.233978] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1163.236521] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1163.237083] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107
[ 1163.237717] Lustre: Skipped 4 previous similar messages
[ 1163.246594] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages
[ 1163.947643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1164.124249] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1164.157527] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 1164.369817] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1165.488523] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1168.018393] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1169.379994] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1169.383343] Lustre: Skipped 2 previous similar messages
[ 1169.390372] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 1169.412302] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65)
[ 1169.772247] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1171.637100] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1173.402414] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1175.183481] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1176.872917] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1200.525639] Lustre: DEBUG MARKER: == sanity-quota test 7f: Quota reintegration automatically ========================================================== 03:46:14 (1789890374)
[ 1206.738355] Lustre: *** cfs_fail_loc=a11, val=0***
[ 1206.739490] Lustre: Skipped 3 previous similar messages
[ 1212.099996] Lustre: *** cfs_fail_loc=a11, val=0***
[ 1238.536017] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 03:46:52 (1789890412)
[ 1303.531764] Lustre: 84323:0:(qsd_reint.c:249:qsd_reint_index()) lustre-MDT0001: index version for fid [0x200000005:0x1003:0x0] is 0, but index isn't empty (1)
[ 1402.385275] Lustre: DEBUG MARKER: == sanity-quota test 9: Block limit larger than 4GB (b10707) ========================================================== 03:49:36 (1789890576)
[ 1402.902388] Lustre: DEBUG MARKER: OST0_SIZE: 3602320 required: 4900000
[ 1405.104616] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 03:49:39 (1789890579)
[ 1411.319199] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1411.843556] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1065088).
[ 1425.133991] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 03:49:59 (1789890599)
[ 1445.635488] Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing ======== 03:50:19 (1789890619)
[ 1451.770411] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1452.351383] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1049728).
[ 1479.447916] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 03:50:53 (1789890653)
[ 1485.801877] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1486.314691] Lustre: DEBUG MARKER: File quota isn't 0 (u:quota_usr:513).
[ 1519.989356] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 03:51:34 (1789890694)
[ 1524.837267] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1525.403382] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:34940).
[ 1545.555141] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 03:51:59 (1789890719)
[ 1556.119787] Lustre: Failing over lustre-OST0000
[ 1556.182142] Lustre: server umount lustre-OST0000 complete
[ 1556.960617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 1556.964189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1556.969235] LustreError: 15399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1558.497200] LustreError: 84311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1558.501214] LustreError: 84311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 1561.808638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1561.895920] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1561.907474] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 1563.041130] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 1563.604904] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 1563.605026] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 1563.609347] Lustre: Skipped 2 previous similar messages
[ 1563.886874] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1581.313412] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 03:52:35 (1789890755)
[ 1590.810141] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 03:52:45 (1789890765)
[ 1598.448996] Lustre: lustre-MDT0001: Client 779eb4ce-e66f-46ab-b22d-8560ba0c39c6 (at 192.168.204.33@tcp) reconnecting
[ 1612.916452] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 03:53:07 (1789890787)
[ 1613.668418] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs
[ 1614.585538] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 03:53:08 (1789890788)
[ 1620.748841] Lustre: DEBUG MARKER: sanity-quota test_17: @@@@@@ FAIL: Used space(27772) for user quota_usr isn't 0.