[ 1545.752641] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1547.584650] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1548.136236] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:27776).
[ 1548.671328] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1551.163834] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1551.666805] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1552.296542] Lustre: Failing over lustre-MDT0000
[ 1552.352778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1552.356890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1552.645748] Lustre: server umount lustre-MDT0000 complete
[ 1553.117302] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 1553.119941] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1553.125094] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages
[ 1557.472522] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1557.478762] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 1566.558489] LDISKFS-fs (dm-0): 10 truncates cleaned up
[ 1566.559611] LDISKFS-fs (dm-0): recovery complete
[ 1566.564070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1566.600799] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1566.631771] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.44@tcp (not set up)
[ 1566.634190] Lustre: Skipped 2 previous similar messages
[ 1566.669243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1566.707012] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1567.942693] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1568.362608] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1571.810658] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1571.813096] Lustre: Skipped 1 previous similar message
[ 1571.823057] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 1571.838612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1801 to 0x280000401:1825)
[ 1571.838621] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1760 to 0x2c0000401:1793)
[ 1573.520331] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1574.018331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1583.579441] Lustre: DEBUG MARKER: (dd_pid=100971, time=7, timeout=600)
[ 1598.987406] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1600.721920] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1601.251321] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:95232).
[ 1601.805250] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1604.441582] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1604.980325] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1605.667139] Lustre: Failing over lustre-MDT0000
[ 1605.909337] Lustre: server umount lustre-MDT0000 complete
[ 1606.624449] LustreError: lustre-MDT0000-lwp-OST0001: operation quota_acquire to node 0@lo failed: rc = -107
[ 1606.624493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1606.627677] LustreError: Skipped 1 previous similar message
[ 1606.627956] LustreError: 7655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1606.632095] Lustre: Skipped 4 previous similar messages
[ 1606.640866] LustreError: 7655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages
[ 1619.902718] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 1619.904040] LDISKFS-fs (dm-0): recovery complete
[ 1619.906598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1619.942069] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1620.019510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1620.056059] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1621.352756] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1623.976487] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1625.059372] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1625.062063] Lustre: Skipped 3 previous similar messages
[ 1625.070288] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 1625.086593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1827 to 0x280000401:1857)
[ 1625.086682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1760 to 0x2c0000401:1825)
[ 1626.944838] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1627.482295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1629.817085] Lustre: DEBUG MARKER: (dd_pid=103867, time=0, timeout=600)
[ 1648.651885] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:42:55 (1789893775)
[ 1654.875856] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1655.423237] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400).
[ 1657.329541] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1657.876932] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400).
[ 1669.981772] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:43:16 (1789893796)
[ 1678.707861] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 1682.095254] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 1682.559546] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 1683.026181] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 1683.576562] Lustre: DEBUG MARKER: Set quota for 1 times
[ 1685.250878] Lustre: DEBUG MARKER: Set quota for 2 times
[ 1686.911967] Lustre: DEBUG MARKER: Set quota for 3 times
[ 1688.497065] Lustre: DEBUG MARKER: Set quota for 4 times
[ 1690.055903] Lustre: DEBUG MARKER: Set quota for 5 times
[ 1691.654590] Lustre: DEBUG MARKER: Set quota for 6 times
[ 1693.252035] Lustre: DEBUG MARKER: Set quota for 7 times
[ 1694.850770] Lustre: DEBUG MARKER: Set quota for 8 times
[ 1696.458628] Lustre: DEBUG MARKER: Set quota for 9 times
[ 1697.996742] Lustre: DEBUG MARKER: Set quota for 10 times
[ 1699.579546] Lustre: DEBUG MARKER: Set quota for 11 times
[ 1701.226329] Lustre: DEBUG MARKER: Set quota for 12 times
[ 1703.038093] Lustre: DEBUG MARKER: Set quota for 13 times
[ 1704.680695] Lustre: DEBUG MARKER: Set quota for 14 times
[ 1706.372369] Lustre: DEBUG MARKER: Set quota for 15 times
[ 1708.076371] Lustre: DEBUG MARKER: Set quota for 16 times
[ 1709.722806] Lustre: DEBUG MARKER: Set quota for 17 times
[ 1711.348189] Lustre: DEBUG MARKER: Set quota for 18 times
[ 1727.020214] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:44:13 (1789893853)
[ 1735.635963] Lustre: server umount lustre-MDT0000 complete
[ 1736.739629] LustreError: 74266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789893864 with bad export cookie 4697206803269881629
[ 1736.741494] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1736.743125] LustreError: 74266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 1736.844413] Lustre: server umount lustre-MDT0001 complete
[ 1753.312111] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893865/real 1789893865]  req@ffff91aa43763800 x1876837994754176/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789893881 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1753.320265] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1753.324531] Lustre: Skipped 2 previous similar messages
[ 1757.408133] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893869/real 1789893869]  req@ffff91a954a02300 x1876837994754816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789893885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1757.416145] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1757.698764] Lustre: server umount lustre-OST0000 complete
[ 1758.919302] Lustre: server umount lustre-OST0001 complete
[ 1763.649827] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local
[ 1766.854024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1766.970558] LustreError: 115619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1766.977178] LustreError: 115619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages
[ 1766.991469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1768.164983] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1770.740075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1770.847743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 1772.045185] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1772.938700] Lustre: 116774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 1774.888124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1774.978402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 1776.741840] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1777.058856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1859 to 0x280000401:1889)
[ 1779.339921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1780.453609] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 1780.453921] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1857)
[ 1781.299976] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1784.196167] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 1785.235703] Lustre: 118647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 1791.504470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 1791.506414] LustreError: Skipped 1 previous similar message
[ 1791.507517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1791.510306] Lustre: Skipped 1 previous similar message
[ 1791.511451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1796.065335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1796.067271] Lustre: Skipped 3 previous similar messages
[ 1797.717054] Lustre: server umount lustre-MDT0000 complete
[ 1798.912781] LustreError: 115601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789893926 with bad export cookie 4697206803269891737
[ 1798.914684] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1798.916339] LustreError: 115601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 1799.021970] Lustre: server umount lustre-MDT0001 complete
[ 1810.369753] Lustre: server umount lustre-OST0000 complete
[ 1821.780471] Lustre: server umount lustre-OST0001 complete
[ 1826.966583] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local
[ 1830.384492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1830.515209] LustreError: 121287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1830.520430] LustreError: 121287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 1830.541339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1830.543551] Lustre: Skipped 1 previous similar message
[ 1831.800444] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1834.399284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1835.783054] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1836.704033] Lustre: 122428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 1838.880538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1840.807656] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1841.058997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1859 to 0x280000401:1921)
[ 1843.465690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1844.579018] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 1844.579561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1889)
[ 1845.244969] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1848.065684] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 1849.097691] Lustre: 124299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 1852.099221] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:46:18 (1789893978)
[ 1852.568893] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 6144
[ 1852.994409] Lustre: DEBUG MARKER: run for 4MB test file
[ 1857.492462] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 1859.174853] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 1859.633790] Lustre: DEBUG MARKER: Write half of file
[ 1860.171132] Lustre: DEBUG MARKER: Write out of block quota ...
[ 1860.734775] Lustre: DEBUG MARKER: Step1: done
[ 1861.218578] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 1861.719309] Lustre: DEBUG MARKER: Step2: done
[ 1875.570540] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 61440
[ 1876.044900] Lustre: DEBUG MARKER: run for 40MB test file
[ 1879.395426] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 1881.120335] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1881.586907] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2048).
[ 1882.085414] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 1882.546854] Lustre: DEBUG MARKER: Write half of file
[ 1883.373519] Lustre: DEBUG MARKER: Write out of block quota ...
[ 1884.119617] Lustre: DEBUG MARKER: Step1: done
[ 1884.605795] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 1885.115597] Lustre: DEBUG MARKER: Step2: done
[ 1905.740567] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:47:12 (1789894032)
[ 1927.441642] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:47:33 (1789894053)
[ 1944.665137] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6144) for user quota_usr isn't 0.
[ 1970.414505] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:48:15 (1789894095)
[ 1975.558206] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:48:21 (1789894101)
[ 1982.191828] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:48:28 (1789894108)
[ 1987.850963] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:48:33 (1789894113)
[ 1993.051105] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:48:39 (1789894119)
[ 2025.828570] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2031.353138] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (8192) for user 60000 isn't 0.
[ 2040.697469] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2045.998391] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (8192) for user 60000 isn't 0.
[ 2053.330814] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:49:39 (1789894179)
[ 2072.829159] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used inodes for user 60000 is 0, expected 1
[ 2092.162981] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:50:18 (1789894218)
[ 2097.579512] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space (8192) for user 60000 isn't 0.
[ 2104.455830] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:50:30 (1789894230)
[ 2108.926583] Lustre: DEBUG MARKER: sanity-quota test_38: @@@@@@ FAIL: Used space (8192) for user 60000 isn't 0.
[ 2116.455227] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:50:42 (1789894242)
[ 2121.185166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2121.186531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2121.187338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2121.194835] Lustre: Skipped 5 previous similar messages
[ 2126.441671] Lustre: server umount lustre-MDT0000 complete
[ 2128.196718] LustreError: 122032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789894255 with bad export cookie 4697206803269894040
[ 2128.202373] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2128.204877] LustreError: 122032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2128.367396] Lustre: server umount lustre-MDT0001 complete
[ 2139.711819] Lustre: server umount lustre-OST0000 complete
[ 2147.297256] LustreError: 3617:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff91aa7f210a80 x1876837995070208/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 2147.314257] LustreError: 3617:0:(qsd_handler.c:318:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff91a95226ec78 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1580 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 2151.894331] Lustre: server umount lustre-OST0001 complete
[ 2159.699485] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local
[ 2165.616536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2165.856448] LustreError: 143207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2165.872030] LustreError: 143207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 2165.939732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2165.943589] Lustre: Skipped 3 previous similar messages
[ 2167.953616] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2172.199134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2174.914996] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2176.811264] Lustre: 144350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2180.716735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2182.956144] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 2184.115582] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2189.166712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2190.381896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1927 to 0x280000401:1953)
[ 2190.393345] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1891 to 0x2c0000401:1921)
[ 2192.616789] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2197.287908] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2199.301376] Lustre: 146219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2218.853298] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:52:24 (1789894344)
[ 2237.615315] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:52:43 (1789894363)
[ 2259.177842] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:53:05 (1789894385)
[ 2265.989978] Lustre: DEBUG MARKER: sanity-quota test_40c: @@@@@@ FAIL: file count expected 3 got 2
[ 2284.177865] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:53:30 (1789894410)
[ 2308.977604] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:53:54 (1789894434)
[ 2368.406666] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:54:54 (1789894494)
[ 2406.895849] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:55:31 (1789894531)
[ 2470.507335] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:56:36 (1789894596)
[ 2475.242014] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2475.754906] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2475.757455] Lustre: Skipped 43 previous similar messages
[ 2476.803569] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2476.805361] Lustre: Skipped 67 previous similar messages
[ 2478.813229] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2478.815128] Lustre: Skipped 107 previous similar messages
[ 2482.854886] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2482.864942] Lustre: Skipped 299 previous similar messages
[ 2490.875280] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2490.879908] Lustre: Skipped 535 previous similar messages
[ 2506.886916] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2506.889880] Lustre: Skipped 1101 previous similar messages
[ 2538.890378] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2538.892238] Lustre: Skipped 2033 previous similar messages
[ 2907.118222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2907.137041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2907.141265] Lustre: Skipped 3 previous similar messages
[ 2907.165134] Lustre: Skipped 10 previous similar messages
[ 2912.229070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2912.235915] Lustre: Skipped 3 previous similar messages
[ 2914.410927] Lustre: server umount lustre-MDT0000 complete
[ 2917.372819] LustreError: 145436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2917.385547] LustreError: 145436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages
[ 2918.015529] LustreError: 143187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789895045 with bad export cookie 4697206803269901614
[ 2918.016809] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2918.023391] LustreError: 143187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2922.467241] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 2922.476946] Lustre: Skipped 1 previous similar message
[ 2924.393331] Lustre: server umount lustre-MDT0001 complete
[ 2928.585427] Lustre: server umount lustre-OST0000 complete
[ 2932.136351] Lustre: server umount lustre-OST0001 complete
[ 2938.728967] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid
[ 2946.742872] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local
[ 2987.110309] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local
[ 2996.556462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2996.819322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 2996.852773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 2996.932382] Lustre: lustre-MDT0000: new disk, initializing
[ 2997.031493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2997.039394] Lustre: Skipped 3 previous similar messages
[ 2997.056795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 3000.920578] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3010.753149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3010.879440] Lustre: 163391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 3010.909830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 3010.912595] Lustre: Skipped 1 previous similar message
[ 3010.969933] Lustre: lustre-MDT0001: new disk, initializing
[ 3011.021226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 3011.045740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 3011.050536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 3015.040280] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3019.431079] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 3025.869138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3026.039302] Lustre: lustre-OST0000: new disk, initializing
[ 3026.041923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 3026.047115] Lustre: 165021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 3026.098462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3028.049600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 3028.069155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 3028.109242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 3031.812631] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3040.129764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3040.223797] Lustre: lustre-OST0001: new disk, initializing
[ 3040.227116] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 3040.232237] Lustre: 165891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 3041.867394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 3041.872906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 3041.928634] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 3044.496986] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3051.843916] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3054.252619] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 3073.337917] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 05:06:39 (1789895199)
[ 3078.994184] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 05:06:44 (1789895204)
[ 3106.321584] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 05:07:12 (1789895232)
[ 3142.036813] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 05:07:47 (1789895267)
[ 3160.752120] Lustre: DEBUG MARKER: rename directory return 255
[ 3193.561212] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 05:08:39 (1789895319)
[ 3211.808535] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 05:08:56 (1789895336)
[ 3236.701183] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 05:09:22 (1789895362)
[ 3311.197269] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 05:10:37 (1789895437)
[ 3327.880714] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 05:10:53 (1789895453)
[ 3345.079346] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 05:11:11 (1789895471)
[ 3428.153221] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 05:12:34 (1789895554)
[ 3430.883237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3430.887916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3430.895173] Lustre: Skipped 3 previous similar messages
[ 3430.898473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3430.901291] Lustre: Skipped 2 previous similar messages
[ 3436.371855] Lustre: server umount lustre-MDT0000 complete
[ 3436.520377] LustreError: 168128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3436.531571] LustreError: 168128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages
[ 3438.606446] LustreError: 163381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789895566 with bad export cookie 4697206803270273818
[ 3438.609675] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3438.616577] LustreError: 163381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 3438.834778] Lustre: server umount lustre-MDT0001 complete
[ 3450.915341] Lustre: server umount lustre-OST0000 complete
[ 3463.431171] Lustre: server umount lustre-OST0001 complete
[ 3474.084239] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local
[ 3479.924332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3480.163732] LustreError: 183325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3480.177730] LustreError: 183325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages
[ 3480.227597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3480.234703] Lustre: Skipped 1 previous similar message
[ 3480.239428] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3482.189309] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3486.296442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3486.447133] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3488.420943] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3489.961616] Lustre: 184468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3493.011598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3493.166876] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3496.190832] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3500.511082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3500.612705] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3503.566623] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3504.742181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[ 3504.744298] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[ 3512.600220] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3514.167298] Lustre: 186343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3517.495563] LustreError: 185300:0:(osd_handler.c:3426:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[ 3520.483468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3520.484323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3520.497269] Lustre: Skipped 6 previous similar messages
[ 3520.501728] Lustre: Skipped 7 previous similar messages
[ 3525.227130] Lustre: server umount lustre-MDT0000 complete
[ 3526.746731] LustreError: 183307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789895654 with bad export cookie 4697206803270322622
[ 3526.747076] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3526.751943] LustreError: 183307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 3526.867484] Lustre: server umount lustre-MDT0001 complete
[ 3538.955206] Lustre: server umount lustre-OST0000 complete
[ 3551.012776] Lustre: server umount lustre-OST0001 complete
[ 3559.785663] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local
[ 3564.519961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3564.706082] LustreError: 188735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3564.716293] LustreError: 188735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages
[ 3564.738524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3564.741387] Lustre: Skipped 3 previous similar messages
[ 3566.615302] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3570.902838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3573.010125] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3574.462911] Lustre: 189882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3577.938491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3581.080447] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3585.161910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3588.036528] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3589.351639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[ 3589.362637] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[ 3597.407947] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3599.262659] Lustre: 191754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3615.271975] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 05:15:41 (1789895741)
[ 3635.711959] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 3636.477368] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 05:16:02 (1789895762)
[ 3645.551330] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 3646.205781] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 05:16:12 (1789895772)
[ 3651.406048] Lustre: DEBUG MARKER: sanity-quota test_64: @@@@@@ FAIL: expected 0 got 6
[ 3665.540851] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 3666.132722] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 05:16:32 (1789895792)
[ 3686.166586] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 05:16:52 (1789895812)
[ 3690.333255] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 3692.520285] Lustre: 198742:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[ 3692.524595] Lustre: 198742:0:(qsd_reint.c:249:qsd_reint_index()) Skipped 1 previous similar message
[ 3693.984552] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3695.999773] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3697.397346] Lustre: DEBUG MARKER: Write...
[ 3707.054129] LustreError: 200084:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 3710.789813] Lustre: DEBUG MARKER: Write...
[ 3715.129527] Lustre: DEBUG MARKER: Write...
[ 3737.347165] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 3737.959899] Lustre: DEBUG MARKER: sanity-quota test_67: @@@@@@ FAIL: user quota isn't released after deletion