[ 1015.949240] Lustre: DEBUG MARKER: sanity-quota test_6: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0.
[ 1020.890441] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================================== 03:45:29 (1789890329)
[ 1023.884444] Lustre: DEBUG MARKER: sanity-quota test_7a: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0
[ 1028.482352] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) ========================================================== 03:45:37 (1789890337)
[ 1031.695956] Lustre: DEBUG MARKER: sanity-quota test_7b: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0
[ 1036.873990] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ========================================================== 03:45:45 (1789890345)
[ 1040.096900] Lustre: DEBUG MARKER: sanity-quota test_7c: @@@@@@ FAIL: Used space(24576) for user quota_usr isn't 0
[ 1045.226151] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ========================================================== 03:45:53 (1789890353)
[ 1054.025098] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 1055.623638] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 1070.288769] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================================== 03:46:18 (1789890378)
[ 1078.958461] Lustre: Failing over lustre-MDT0001
[ 1079.052970] Lustre: server umount lustre-MDT0001 complete
[ 1079.776776] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107
[ 1079.778955] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1081.313773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1084.245712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1084.385225] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1084.415537] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 1085.645180] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1085.697673] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1088.179898] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1089.509104] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1089.520750] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 1089.539762] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33)
[ 1089.797669] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1091.476953] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1093.116716] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1094.736899] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1096.374278] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1117.226052] Lustre: Failing over lustre-MDT0001
[ 1117.446824] Lustre: server umount lustre-MDT0001 complete
[ 1120.225902] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1120.226619] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107
[ 1120.227191] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1120.227198] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages
[ 1120.232671] Lustre: Skipped 2 previous similar messages
[ 1120.291116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1120.482421] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1120.517894] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 1120.913428] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1121.848418] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1124.477209] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1125.860363] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1125.863406] Lustre: Skipped 2 previous similar messages
[ 1125.870297] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 1125.889668] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65)
[ 1126.228123] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1127.967430] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1129.555851] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1131.305323] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 1132.952577] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475
[ 1156.891794] Lustre: DEBUG MARKER: == sanity-quota test 7f: Quota reintegration automatically ========================================================== 03:47:45 (1789890465)
[ 1161.964565] Lustre: *** cfs_fail_loc=a11, val=0***
[ 1161.966036] Lustre: Skipped 3 previous similar messages
[ 1167.378855] Lustre: *** cfs_fail_loc=a11, val=0***
[ 1196.603122] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ==== 03:48:25 (1789890505)
[ 1262.577462] Lustre: 84444:0:(qsd_reint.c:249:qsd_reint_index()) lustre-MDT0001: index version for fid [0x200000005:0x1004:0x0] is 0, but index isn't empty (1)
[ 1348.131568] LustreError: 6518:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 1365.391248] Lustre: DEBUG MARKER: == sanity-quota test 9: Block limit larger than 4GB (b10707) ========================================================== 03:51:14 (1789890674)
[ 1365.876050] Lustre: DEBUG MARKER: OST0_SIZE: 3599312 required: 4900000
[ 1367.751100] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ======== 03:51:16 (1789890676)
[ 1375.001196] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1375.534975] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1065092).
[ 1389.907531] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota ======= 03:51:38 (1789890698)
[ 1412.159785] Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing ======== 03:52:00 (1789890720)
[ 1417.950244] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1418.440705] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:1049732).
[ 1446.762785] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ======== 03:52:35 (1789890755)
[ 1453.035432] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1453.541846] Lustre: DEBUG MARKER: File quota isn't 0 (u:quota_usr:513).
[ 1498.451505] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================================== 03:53:26 (1789890806)
[ 1508.004185] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1508.992372] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:34944).
[ 1537.037416] Lustre: DEBUG MARKER: == sanity-quota test 14: check panic in qmt_site_recalc_cb ========================================================== 03:54:04 (1789890844)
[ 1547.432519] hrtimer: interrupt took 4022905 ns
[ 1573.397761] Lustre: Failing over lustre-OST0000
[ 1573.550721] Lustre: server umount lustre-OST0000 complete
[ 1576.422223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107
[ 1576.431966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1576.441659] LustreError: 8397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1576.451378] Lustre: Skipped 2 previous similar messages
[ 1576.491196] LustreError: 8397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages
[ 1581.547521] LustreError: 14866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1581.583807] LustreError: 14866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 1586.659418] LustreError: 84430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1586.673339] LustreError: 84430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 1588.883320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1589.132912] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 1589.161137] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect
[ 1590.952700] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
[ 1591.151770] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted.
[ 1591.158833] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo)
[ 1591.173796] Lustre: Skipped 2 previous similar messages
[ 1593.751383] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1621.701391] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ========= 03:55:29 (1789890929)
[ 1636.937296] Lustre: DEBUG MARKER: == sanity-quota test 16a: lfs quota should skip the inactive MDT/OST ========================================================== 03:55:45 (1789890945)
[ 1648.978897] Lustre: lustre-MDT0001: Client f7f0872b-bb3d-4c19-8a30-d4006c985d93 (at 192.168.203.53@tcp) reconnecting
[ 1666.560589] Lustre: DEBUG MARKER: == sanity-quota test 16b: lfs quota should skip the nonexistent MDT/OST ========================================================== 03:56:14 (1789890974)
[ 1667.666621] Lustre: DEBUG MARKER: SKIP: sanity-quota test_16b needs >= 3 MDTs
[ 1668.956854] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 03:56:17 (1789890977)
[ 1676.319259] Lustre: DEBUG MARKER: sanity-quota test_17: @@@@@@ FAIL: Used space(27776) for user quota_usr isn't 0.