[ 1698.074241] Lustre: DEBUG MARKER: User quota (limit: 200) [ 1702.137456] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0: [ 1703.446960] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:27776). [ 1704.946224] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 1710.406818] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1711.571247] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 1713.032350] Lustre: Failing over lustre-MDT0000 [ 1713.377891] Lustre: server umount lustre-MDT0000 complete [ 1713.764419] LustreError: 6996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.785663] LustreError: 6996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1713.812569] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107 [ 1713.819768] LustreError: Skipped 1 previous similar message [ 1713.823338] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.337988] LustreError: 17164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.352599] LustreError: 17164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 1732.576178] Lustre: 3620:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891026/real 1789891026] req@ffff903090d19500 x1876835046815104/t0(0) o400->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789891042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1732.617791] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.483420] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 1733.486223] LDISKFS-fs (dm-0): recovery complete [ 1733.504197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.454354] LustreError: 11049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.474022] LustreError: 11049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 1743.100973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.134233] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1743.345639] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1747.093192] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1748.459968] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1748.468622] Lustre: Skipped 1 previous similar message [ 1748.531630] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1748.589257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1698 to 0x280000401:1729) [ 1748.590538] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1670 to 0x2c0000401:1697) [ 1755.932570] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1757.027812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1761.417243] Lustre: DEBUG MARKER: (dd_pid=101872, time=0, timeout=600) [ 1785.321805] Lustre: DEBUG MARKER: User quota (limit: 200) [ 1791.264179] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0: [ 1792.933038] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400). [ 1794.559763] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 1800.929328] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1802.086462] Lustre: DEBUG MARKER: Fail mds for 40 seconds [ 1803.452084] Lustre: Failing over lustre-MDT0000 [ 1803.726607] Lustre: server umount lustre-MDT0000 complete [ 1804.769698] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.772774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1804.787200] Lustre: Skipped 3 previous similar messages [ 1804.796086] LustreError: Skipped 1 previous similar message [ 1804.801607] LustreError: 17164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1804.837197] LustreError: 17164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 1821.153767] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891114/real 1789891114] req@ffff903092ab6680 x1876835046873856/t0(0) o400->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789891130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1821.168477] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1821.850281] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1821.852466] LDISKFS-fs (dm-0): recovery complete [ 1821.859809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.370360] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd5b980e0d623e2da [ 1830.374614] Lustre: MGC192.168.203.153@tcp: Connection restored to 0@lo (at 0@lo) [ 1830.378268] Lustre: Skipped 3 previous similar messages [ 1830.464810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.506436] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1831.564361] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1832.272368] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1835.490294] LustreError: 3617:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff903085771b00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1835.508916] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1835.527957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1731 to 0x280000401:1761) [ 1835.527981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1670 to 0x2c0000401:1729) [ 1837.437051] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1837.998349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1840.505205] Lustre: DEBUG MARKER: (dd_pid=104604, time=0, timeout=600) [ 1860.266404] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 03:59:28 (1789891168) [ 1867.970631] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 1868.641915] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400). [ 1870.724045] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1: [ 1871.253694] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102400). [ 1885.985367] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 03:59:54 (1789891194) [ 1894.830085] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 1898.495025] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 1898.979901] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 1899.474199] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000 [ 1900.086170] Lustre: DEBUG MARKER: Set quota for 1 times [ 1901.719616] Lustre: DEBUG MARKER: Set quota for 2 times [ 1903.526814] Lustre: DEBUG MARKER: Set quota for 3 times [ 1905.199508] Lustre: DEBUG MARKER: Set quota for 4 times [ 1906.848876] Lustre: DEBUG MARKER: Set quota for 5 times [ 1908.598746] Lustre: DEBUG MARKER: Set quota for 6 times [ 1910.264917] Lustre: DEBUG MARKER: Set quota for 7 times [ 1911.913902] Lustre: DEBUG MARKER: Set quota for 8 times [ 1913.684108] Lustre: DEBUG MARKER: Set quota for 9 times [ 1915.347476] Lustre: DEBUG MARKER: Set quota for 10 times [ 1917.008452] Lustre: DEBUG MARKER: Set quota for 11 times [ 1918.646693] Lustre: DEBUG MARKER: Set quota for 12 times [ 1920.335023] Lustre: DEBUG MARKER: Set quota for 13 times [ 1921.961420] Lustre: DEBUG MARKER: Set quota for 14 times [ 1923.573239] Lustre: DEBUG MARKER: Set quota for 15 times [ 1925.206211] Lustre: DEBUG MARKER: Set quota for 16 times [ 1926.841404] Lustre: DEBUG MARKER: Set quota for 17 times [ 1928.478206] Lustre: DEBUG MARKER: Set quota for 18 times [ 1947.100222] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:00:55 (1789891255) [ 1953.249234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.249433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1953.249489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.252973] Lustre: Skipped 4 previous similar messages [ 1954.712152] Lustre: server umount lustre-MDT0000 complete [ 1955.921773] LustreError: 28312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891265 with bad export cookie 15400482103923827418 [ 1955.923679] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1955.925425] LustreError: 28312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1956.042639] Lustre: server umount lustre-MDT0001 complete [ 1967.616948] Lustre: server umount lustre-OST0000 complete [ 1979.189939] Lustre: server umount lustre-OST0001 complete [ 1984.458775] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1988.145196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.297674] LustreError: 116640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.304732] LustreError: 116640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 31 previous similar messages [ 1988.323716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.766550] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1992.417297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.542743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1993.771466] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1994.724408] Lustre: 117793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1996.906568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1997.021141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.011812] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1999.075931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1764 to 0x280000401:1793) [ 2001.787735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2002.850788] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 2002.852490] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1731 to 0x2c0000401:1761) [ 2003.664994] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2006.505761] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2007.577108] Lustre: 119666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2018.272922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.273112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2018.273220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.273225] Lustre: Skipped 3 previous similar messages [ 2018.276056] Lustre: Skipped 3 previous similar messages [ 2021.590412] Lustre: server umount lustre-MDT0000 complete [ 2022.774529] LustreError: 119918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891332 with bad export cookie 15400482103923837141 [ 2022.775895] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.777952] LustreError: 119918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2022.868635] Lustre: server umount lustre-MDT0001 complete [ 2034.239907] Lustre: server umount lustre-OST0000 complete [ 2045.711701] Lustre: server umount lustre-OST0001 complete [ 2050.829014] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2054.249186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.407174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.409902] Lustre: Skipped 1 previous similar message [ 2055.731637] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2058.535079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.115269] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2061.097379] Lustre: 123497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2063.338605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2064.483234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1764 to 0x280000401:1825) [ 2065.414352] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2068.265673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2069.347415] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129) [ 2069.348027] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1731 to 0x2c0000401:1793) [ 2070.233098] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2073.285928] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2074.426134] Lustre: 125367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2077.395794] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:03:06 (1789891386) [ 2077.848660] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 6144 [ 2078.300213] Lustre: DEBUG MARKER: run for 4MB test file [ 2083.051739] Lustre: DEBUG MARKER: User quota (limit: 4 MB) [ 2084.737894] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [ 2085.195801] Lustre: DEBUG MARKER: Write half of file [ 2085.719193] Lustre: DEBUG MARKER: Write out of block quota ... [ 2086.214906] Lustre: DEBUG MARKER: Step1: done [ 2086.651847] Lustre: DEBUG MARKER: Step2: rewrite should succeed [ 2087.135844] Lustre: DEBUG MARKER: Step2: done [ 2100.976291] Lustre: DEBUG MARKER: OST0_SIZE: 3605412 required: 61440 [ 2101.436258] Lustre: DEBUG MARKER: run for 40MB test file [ 2104.683376] Lustre: DEBUG MARKER: User quota (limit: 40 MB) [ 2106.349258] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0: [ 2106.849158] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2048). [ 2107.367441] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [ 2107.867920] Lustre: DEBUG MARKER: Write half of file [ 2108.637958] Lustre: DEBUG MARKER: Write out of block quota ... [ 2109.374887] Lustre: DEBUG MARKER: Step1: done [ 2109.836062] Lustre: DEBUG MARKER: Step2: rewrite should succeed [ 2110.305907] Lustre: DEBUG MARKER: Step2: done [ 2134.622145] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:04:03 (1789891443) [ 2155.510164] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:04:24 (1789891464) [ 2166.843621] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6144) for user quota_usr isn't 0. [ 2179.753910] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:04:48 (1789891488) [ 2181.621917] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:04:50 (1789891490) [ 2185.674249] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:04:54 (1789891494) [ 2188.782638] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:04:57 (1789891497) [ 2191.822278] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:05:00 (1789891500) [ 2217.027378] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [ 2220.159084] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (9216) for user 60000 isn't 0. [ 2225.964853] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [ 2229.035383] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (9216) for user 60000 isn't 0.