[ 1698.074241] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1702.137456] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1703.446960] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:27776).
[ 1704.946224] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1710.406818] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1711.571247] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1713.032350] Lustre: Failing over lustre-MDT0000
[ 1713.377891] Lustre: server umount lustre-MDT0000 complete
[ 1713.764419] LustreError: 6996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1713.785663] LustreError: 6996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 1713.812569] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 1713.819768] LustreError: Skipped 1 previous similar message
[ 1713.823338] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1722.337988] LustreError: 17164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1722.352599] LustreError: 17164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages
[ 1732.576178] Lustre: 3620:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891026/real 1789891026]  req@ffff903090d19500 x1876835046815104/t0(0) o400->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789891042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1732.617791] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1733.483420] LDISKFS-fs (dm-0): 10 truncates cleaned up
[ 1733.486223] LDISKFS-fs (dm-0): recovery complete
[ 1733.504197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1741.454354] LustreError: 11049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1741.474022] LustreError: 11049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages
[ 1743.100973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1743.134233] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1743.345639] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1747.093192] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1748.459968] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1748.468622] Lustre: Skipped 1 previous similar message
[ 1748.531630] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 1748.589257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1698 to 0x280000401:1729)
[ 1748.590538] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1670 to 0x2c0000401:1697)
[ 1755.932570] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1757.027812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1761.417243] Lustre: DEBUG MARKER: (dd_pid=101872, time=0, timeout=600)
[ 1785.321805] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1791.264179] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1792.933038] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400).
[ 1794.559763] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1800.929328] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1802.086462] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1803.452084] Lustre: Failing over lustre-MDT0000
[ 1803.726607] Lustre: server umount lustre-MDT0000 complete
[ 1804.769698] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1804.772774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 1804.787200] Lustre: Skipped 3 previous similar messages
[ 1804.796086] LustreError: Skipped 1 previous similar message
[ 1804.801607] LustreError: 17164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1804.837197] LustreError: 17164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages
[ 1821.153767] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891114/real 1789891114]  req@ffff903092ab6680 x1876835046873856/t0(0) o400->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789891130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1821.168477] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1821.850281] LDISKFS-fs (dm-0): 2 truncates cleaned up
[ 1821.852466] LDISKFS-fs (dm-0): recovery complete
[ 1821.859809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1830.370360] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd5b980e0d623e2da
[ 1830.374614] Lustre: MGC192.168.203.153@tcp: Connection restored to 0@lo (at 0@lo)
[ 1830.378268] Lustre: Skipped 3 previous similar messages
[ 1830.464810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1830.506436] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1831.564361] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 1832.272368] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1835.490294] LustreError: 3617:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff903085771b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 1835.508916] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 1835.527957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1731 to 0x280000401:1761)
[ 1835.527981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1670 to 0x2c0000401:1729)
[ 1837.437051] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1837.998349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1840.505205] Lustre: DEBUG MARKER: (dd_pid=104604, time=0, timeout=600)
[ 1860.266404] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 03:59:28 (1789891168)
[ 1867.970631] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1868.641915] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400).
[ 1870.724045] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1871.253694] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400).
[ 1885.985367] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 03:59:54 (1789891194)
[ 1894.830085] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 1898.495025] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 1898.979901] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 1899.474199] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 1900.086170] Lustre: DEBUG MARKER: Set quota for 1 times
[ 1901.719616] Lustre: DEBUG MARKER: Set quota for 2 times
[ 1903.526814] Lustre: DEBUG MARKER: Set quota for 3 times
[ 1905.199508] Lustre: DEBUG MARKER: Set quota for 4 times
[ 1906.848876] Lustre: DEBUG MARKER: Set quota for 5 times
[ 1908.598746] Lustre: DEBUG MARKER: Set quota for 6 times
[ 1910.264917] Lustre: DEBUG MARKER: Set quota for 7 times
[ 1911.913902] Lustre: DEBUG MARKER: Set quota for 8 times
[ 1913.684108] Lustre: DEBUG MARKER: Set quota for 9 times
[ 1915.347476] Lustre: DEBUG MARKER: Set quota for 10 times
[ 1917.008452] Lustre: DEBUG MARKER: Set quota for 11 times
[ 1918.646693] Lustre: DEBUG MARKER: Set quota for 12 times
[ 1920.335023] Lustre: DEBUG MARKER: Set quota for 13 times
[ 1921.961420] Lustre: DEBUG MARKER: Set quota for 14 times
[ 1923.573239] Lustre: DEBUG MARKER: Set quota for 15 times
[ 1925.206211] Lustre: DEBUG MARKER: Set quota for 16 times
[ 1926.841404] Lustre: DEBUG MARKER: Set quota for 17 times
[ 1928.478206] Lustre: DEBUG MARKER: Set quota for 18 times
[ 1947.100222] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:00:55 (1789891255)
[ 1953.249234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1953.249433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 1953.249489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1953.252973] Lustre: Skipped 4 previous similar messages
[ 1954.712152] Lustre: server umount lustre-MDT0000 complete
[ 1955.921773] LustreError: 28312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891265 with bad export cookie 15400482103923827418
[ 1955.923679] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1955.925425] LustreError: 28312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 1956.042639] Lustre: server umount lustre-MDT0001 complete
[ 1967.616948] Lustre: server umount lustre-OST0000 complete
[ 1979.189939] Lustre: server umount lustre-OST0001 complete
[ 1984.458775] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 1988.145196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1988.297674] LustreError: 116640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1988.304732] LustreError: 116640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 31 previous similar messages
[ 1988.323716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1989.766550] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1992.417297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 1992.542743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 1993.771466] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1994.724408] Lustre: 117793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 1996.906568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 1997.021141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 1999.011812] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1999.075931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1764 to 0x280000401:1793)
[ 2001.787735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2002.850788] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 2002.852490] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1731 to 0x2c0000401:1761)
[ 2003.664994] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2006.505761] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2007.577108] Lustre: 119666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2018.272922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2018.273112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2018.273220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2018.273225] Lustre: Skipped 3 previous similar messages
[ 2018.276056] Lustre: Skipped 3 previous similar messages
[ 2021.590412] Lustre: server umount lustre-MDT0000 complete
[ 2022.774529] LustreError: 119918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891332 with bad export cookie 15400482103923837141
[ 2022.775895] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2022.777952] LustreError: 119918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2022.868635] Lustre: server umount lustre-MDT0001 complete
[ 2034.239907] Lustre: server umount lustre-OST0000 complete
[ 2045.711701] Lustre: server umount lustre-OST0001 complete
[ 2050.829014] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 2054.249186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2054.407174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2054.409902] Lustre: Skipped 1 previous similar message
[ 2055.731637] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2058.535079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2060.115269] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2061.097379] Lustre: 123497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2063.338605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2064.483234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1764 to 0x280000401:1825)
[ 2065.414352] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2068.265673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2069.347415] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 2069.348027] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1731 to 0x2c0000401:1793)
[ 2070.233098] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2073.285928] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2074.426134] Lustre: 125367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2077.395794] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:03:06 (1789891386)
[ 2077.848660] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 6144
[ 2078.300213] Lustre: DEBUG MARKER: run for 4MB test file
[ 2083.051739] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 2084.737894] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2085.195801] Lustre: DEBUG MARKER: Write half of file
[ 2085.719193] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2086.214906] Lustre: DEBUG MARKER: Step1: done
[ 2086.651847] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2087.135844] Lustre: DEBUG MARKER: Step2: done
[ 2100.976291] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 61440
[ 2101.436258] Lustre: DEBUG MARKER: run for 40MB test file
[ 2104.683376] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 2106.349258] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 2106.849158] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2048).
[ 2107.367441] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2107.867920] Lustre: DEBUG MARKER: Write half of file
[ 2108.637958] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2109.374887] Lustre: DEBUG MARKER: Step1: done
[ 2109.836062] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2110.305907] Lustre: DEBUG MARKER: Step2: done
[ 2134.622145] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:04:03 (1789891443)
[ 2155.510164] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:04:24 (1789891464)
[ 2166.843621] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6144) for user quota_usr isn't 0.
[ 2179.753910] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:04:48 (1789891488)
[ 2181.621917] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:04:50 (1789891490)
[ 2185.674249] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:04:54 (1789891494)
[ 2188.782638] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:04:57 (1789891497)
[ 2191.822278] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:05:00 (1789891500)
[ 2217.027378] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2220.159084] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (9216) for user 60000 isn't 0.
[ 2225.964853] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2229.035383] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (9216) for user 60000 isn't 0.
[ 2233.837755] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:05:42 (1789891542)
[ 2249.628809] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used inodes for user 60000 is 0, expected 1
[ 2262.929777] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:06:11 (1789891571)
[ 2266.427219] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space (9216) for user 60000 isn't 0.
[ 2271.955693] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:06:20 (1789891580)
[ 2275.451633] Lustre: DEBUG MARKER: sanity-quota test_38: @@@@@@ FAIL: Used space (9216) for user 60000 isn't 0.
[ 2280.920571] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:06:29 (1789891589)
[ 2284.000801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2284.002925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2284.006477] Lustre: Skipped 1 previous similar message
[ 2284.007959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2284.010352] Lustre: Skipped 3 previous similar messages
[ 2289.634333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2289.636213] Lustre: Skipped 7 previous similar messages
[ 2290.010275] Lustre: server umount lustre-MDT0000 complete
[ 2291.173220] LustreError: 122339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891600 with bad export cookie 15400482103923839444
[ 2291.175836] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2291.179219] LustreError: 122339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 2291.300133] Lustre: server umount lustre-MDT0001 complete
[ 2302.976496] Lustre: server umount lustre-OST0000 complete
[ 2313.429572] Lustre: server umount lustre-OST0001 complete
[ 2318.335148] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 2322.117447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2322.270434] LustreError: 144404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2322.276576] LustreError: 144404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages
[ 2322.296778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2322.299060] Lustre: Skipped 3 previous similar messages
[ 2323.663546] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2326.545990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2328.065701] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2329.048691] Lustre: 145546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2331.250979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2332.388278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1831 to 0x280000401:1857)
[ 2333.292661] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2336.177877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2337.251800] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 2337.252594] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1795 to 0x2c0000401:1825)
[ 2338.129429] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2341.289276] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2342.449635] Lustre: 147418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2357.323150] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:07:45 (1789891665)
[ 2372.317906] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:08:00 (1789891680)
[ 2387.209179] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:08:15 (1789891695)
[ 2390.623091] Lustre: DEBUG MARKER: sanity-quota test_40c: @@@@@@ FAIL: file count expected 3 got 2
[ 2407.678983] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:08:36 (1789891716)
[ 2424.131059] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:08:52 (1789891732)
[ 2464.742671] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:09:33 (1789891773)
[ 2476.475850] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:09:45 (1789891785)
[ 2505.563129] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:10:14 (1789891814)
[ 2507.383763] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2507.883036] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2507.884311] Lustre: Skipped 99 previous similar messages
[ 2508.888815] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2508.890631] Lustre: Skipped 183 previous similar messages
[ 2510.890794] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2510.892060] Lustre: Skipped 381 previous similar messages
[ 2514.904747] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2514.906908] Lustre: Skipped 785 previous similar messages
[ 2522.905711] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2522.907559] Lustre: Skipped 1511 previous similar messages
[ 2538.905334] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2538.906622] Lustre: Skipped 3115 previous similar messages
[ 2659.810697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2659.812892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2659.813476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2659.815377] Lustre: Skipped 5 previous similar messages
[ 2664.138505] Lustre: server umount lustre-MDT0000 complete
[ 2664.929537] LustreError: 144405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2664.934742] LustreError: 144405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 2665.407087] LustreError: 144387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891975 with bad export cookie 15400482103923846885
[ 2665.410601] LustreError: 144387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2665.411173] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2670.050396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 2670.052267] Lustre: Skipped 4 previous similar messages
[ 2671.612017] Lustre: server umount lustre-MDT0001 complete
[ 2673.193466] Lustre: server umount lustre-OST0000 complete
[ 2674.603114] Lustre: server umount lustre-OST0001 complete
[ 2676.753685] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid
[ 2679.854574] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 2698.393801] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 2701.790680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2701.886336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 2701.896608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 2701.936424] Lustre: lustre-MDT0000: new disk, initializing
[ 2701.960060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2701.962535] Lustre: Skipped 3 previous similar messages
[ 2701.967926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 2703.318928] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2707.506241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2707.545596] Lustre: 165154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 2707.559377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 2707.562245] Lustre: Skipped 1 previous similar message
[ 2707.607027] Lustre: lustre-MDT0001: new disk, initializing
[ 2707.634086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 2707.636995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 2708.928722] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2711.166523] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 2713.363537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2713.457097] Lustre: lustre-OST0000: new disk, initializing
[ 2713.458632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 2713.461021] Lustre: 166787:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 2715.181170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 2715.183958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 2715.209601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 2715.487728] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2719.664091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2719.706996] Lustre: lustre-OST0001: new disk, initializing
[ 2719.708411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 2719.710466] Lustre: 167658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 2721.200331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 2721.202992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 2721.224479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 2721.693126] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2726.100643] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2727.269641] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 2736.659220] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:14:05 (1789892045)
[ 2738.657276] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:14:07 (1789892047)
[ 2751.269565] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:14:19 (1789892059)
[ 2771.145401] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:14:39 (1789892079)
[ 2778.282130] Lustre: DEBUG MARKER: rename directory return 255
[ 2797.788229] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:15:06 (1789892106)
[ 2804.850216] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:15:13 (1789892113)
[ 2813.922730] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:15:22 (1789892122)
[ 2846.680396] LustreError: 169744:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 2857.998601] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:16:06 (1789892166)
[ 2866.664618] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:16:15 (1789892175)
[ 2880.016488] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:16:28 (1789892188)
[ 2929.702212] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:17:18 (1789892238)
[ 2931.170289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2931.171025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2931.174746] Lustre: Skipped 5 previous similar messages
[ 2931.176734] Lustre: Skipped 1 previous similar message
[ 2937.051578] Lustre: server umount lustre-MDT0000 complete
[ 2938.272062] LustreError: 165147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892248 with bad export cookie 15400482103924219082
[ 2938.275074] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2938.276290] LustreError: 165147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2938.389216] Lustre: server umount lustre-MDT0001 complete
[ 2949.637951] Lustre: server umount lustre-OST0000 complete
[ 2961.152726] Lustre: server umount lustre-OST0001 complete
[ 2967.494515] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 2970.967687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2971.134657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2971.136566] Lustre: Skipped 3 previous similar messages
[ 2971.139250] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 2972.507458] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2975.411475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2975.508121] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 2976.834800] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2977.793691] Lustre: 186417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2979.952988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2980.063732] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 2981.090892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[ 2981.907817] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2984.629011] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 2984.680453] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 2985.701496] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[ 2986.771496] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2989.844104] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2993.203555] LustreError: 185268:0:(osd_handler.c:3426:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[ 2996.195848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2996.199354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2996.204872] Lustre: Skipped 9 previous similar messages
[ 3000.472291] Lustre: server umount lustre-MDT0000 complete
[ 3001.667142] LustreError: 185252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892311 with bad export cookie 15400482103924267620
[ 3001.670312] LustreError: 185252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 3001.670315] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3001.793315] Lustre: server umount lustre-MDT0001 complete
[ 3012.620818] Lustre: server umount lustre-OST0000 complete
[ 3024.275202] Lustre: server umount lustre-OST0001 complete
[ 3031.331876] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local
[ 3035.114212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3036.642485] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3039.506818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3040.919314] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3041.900886] Lustre: 191822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3041.903985] Lustre: 191822:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message
[ 3044.007895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3046.036528] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3046.178858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[ 3048.770372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3049.894152] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[ 3050.977151] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3054.478420] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3069.151604] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:19:37 (1789892377)
[ 3089.676279] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 3090.219267] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:19:58 (1789892398)
[ 3097.621458] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 3098.164112] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 04:20:06 (1789892406)
[ 3102.827124] Lustre: DEBUG MARKER: sanity-quota test_64: @@@@@@ FAIL: expected 0 got 6
[ 3116.125183] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 3116.669336] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:20:25 (1789892425)
[ 3135.557660] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:20:44 (1789892444)
[ 3139.252869] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 3141.434261] Lustre: 200821:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[ 3141.438997] Lustre: 200821:0:(qsd_reint.c:249:qsd_reint_index()) Skipped 2 previous similar messages
[ 3142.975050] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 3144.618899] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 3145.712601] Lustre: DEBUG MARKER: Write...
[ 3153.641531] LustreError: 202162:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 3157.196343] Lustre: DEBUG MARKER: Write...
[ 3161.131245] Lustre: DEBUG MARKER: Write...
[ 3179.899637] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 3180.432945] Lustre: DEBUG MARKER: sanity-quota test_67: @@@@@@ FAIL: user quota isn't released after deletion
[ 3203.034937] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:21:51 (1789892511)
[ 3212.871933] LustreError: 206554:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 3231.883727] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:22:20 (1789892540)
[ 3243.579075] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 3244.138102] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 3274.933522] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:23:03 (1789892583)
[ 3294.865283] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:23:23 (1789892603)
[ 3310.815900] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:23:39 (1789892619)
[ 3314.385175] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[ 3326.752092] Lustre: DEBUG MARKER: sanity-quota test_71a: @@@@@@ FAIL: Used space(10240) for user quota_usr isn't 0.
[ 3345.735297] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:24:14 (1789892654)
[ 3349.944332] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[ 3359.371317] Lustre: DEBUG MARKER: sanity-quota test_71b: @@@@@@ FAIL: Used space(10240) for user quota_usr isn't 0.
[ 3378.579393] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:24:47 (1789892687)
[ 3382.193721] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 3388.244858] Lustre: DEBUG MARKER: sanity-quota test_72: @@@@@@ FAIL: Used space(10240) for user quota_usr isn't 0.
[ 3399.135615] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:25:07 (1789892707)
[ 3410.066749] Lustre: DEBUG MARKER: set to use default quota
[ 3410.590120] Lustre: DEBUG MARKER: set default quota
[ 3411.096119] Lustre: DEBUG MARKER: get default quota
[ 3412.844588] Lustre: DEBUG MARKER: sanity-quota test_73a: @@@@@@ FAIL: Used space for quota_usr isn't 0.
[ 3426.213569] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 04:25:34 (1789892734)
[ 3434.690548] Lustre: DEBUG MARKER: set default quota for qpool1
[ 3435.233228] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 3458.875589] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 04:26:07 (1789892767)
[ 3490.852199] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 04:26:39 (1789892799)
[ 3515.306879] Lustre: DEBUG MARKER: Write...
[ 3516.339763] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3523.995377] Lustre: DEBUG MARKER: Files for user (60000), count=4:
[ 3524.512104] Lustre: DEBUG MARKER: sanity-quota test_75: @@@@@@ FAIL: user write success, but expect EDQUOT
[ 3545.790079] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 04:27:34 (1789892854)
[ 3561.120373] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 04:27:49 (1789892869)
[ 3563.390314] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 04:27:52 (1789892872)
[ 3567.354547] Lustre: DEBUG MARKER: sanity-quota test_78A: @@@@@@ FAIL: fallocate did not use quota. kbytes returned:30724*
[ 3581.915247] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 04:28:10 (1789892890)
[ 3602.494332] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 04:28:31 (1789892911)
[ 3611.867455] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 04:28:40 (1789892920)
[ 3616.596443] Lustre: DEBUG MARKER: sanity-quota test_80: @@@@@@ FAIL: Used space(30724) for user quota_usr is not 0.