[ 1900.713854] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1902.690846] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1903.272829] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:30069).
[ 1903.890515] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1904.979649] LustreError: 86882:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1905.299069] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1905.896495] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1906.601997] Lustre: Failing over lustre-MDT0000
[ 1906.950246] Lustre: server umount lustre-MDT0000 complete
[ 1919.250427] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1919.367666] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1919.426806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1919.447562] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1920.834057] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1922.552207] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1922.569609] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1922.584909] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1024 to 0x280000400:1057)
[ 1922.584909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1038 to 0x240000400:1057)
[ 1924.579322] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1924.654808] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1925.145237] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1925.600258] Lustre: 3291:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891221/real 1789891221]  req@ffff9cd2c1ad0700 x1876835061165568/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891237 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1925.607385] Lustre: 3291:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1927.576459] Lustre: DEBUG MARKER: (dd_pid=88713, time=0, timeout=600)
[ 1930.720150] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891226/real 1789891226]  req@ffff9cd3ef43b100 x1876835061165824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1958.012885] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1959.913786] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1960.453117] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 1960.999242] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1962.091623] LustreError: 89976:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1962.418749] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1963.039860] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1963.751686] Lustre: Failing over lustre-MDT0000
[ 1964.152341] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 1964.155140] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1964.160104] Lustre: Skipped 1 previous similar message
[ 1964.192069] Lustre: server umount lustre-MDT0000 complete
[ 1976.710471] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1976.849289] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1976.906667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1976.930440] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1978.309362] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1978.871565] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1978.909890] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1978.926531] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1978.928537] Lustre: Skipped 1 previous similar message
[ 1978.938335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1059 to 0x240000400:1089)
[ 1978.939197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1024 to 0x280000400:1089)
[ 1981.961104] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891277/real 1789891277]  req@ffff9cd30863d500 x1876835061195264/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891293 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1981.970148] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1982.055137] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1982.559198] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1986.016133] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891281/real 1789891281]  req@ffff9cd30863d180 x1876835061195776/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1989.153930] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1990.112184] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891285/real 1789891285]  req@ffff9cd30863ce00 x1876835061196288/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2001.657987] Lustre: DEBUG MARKER: (dd_pid=91562, time=16, timeout=600)
[ 2041.763118] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:02:33 (1789891353)
[ 2053.164276] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2053.664803] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:101391).
[ 2055.362648] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2055.838951] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:101391).
[ 2077.796035] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:03:09 (1789891389)
[ 2086.664113] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2095.016043] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2095.566332] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2096.108804] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2096.634097] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2098.267862] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2099.989858] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2101.709399] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2103.365350] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2105.043043] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2106.679538] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2108.339636] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2110.066067] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2111.772255] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2113.526923] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2115.292538] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2117.046482] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2118.781249] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2120.501786] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2122.321317] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2124.136834] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2125.990532] Lustre: DEBUG MARKER: Set quota for 18 times
[ 2149.756874] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:04:21 (1789891461)
[ 2158.048625] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2158.048863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2158.052208] Lustre: Skipped 1 previous similar message
[ 2158.057666] Lustre: Skipped 1 previous similar message
[ 2161.016478] Lustre: server umount lustre-MDT0000 complete
[ 2162.376994] LustreError: 36665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891474 with bad export cookie 6611340678040632409
[ 2162.378335] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2162.382369] LustreError: 36665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2162.418189] Lustre: server umount lustre-OST0000 complete
[ 2163.759727] Lustre: server umount lustre-OST0001 complete
[ 2168.696546] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local
[ 2172.035066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2173.491495] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2174.508686] Lustre: 100498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2176.372087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 2178.416614] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2180.816334] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 2181.864762] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1091 to 0x280000400:1121)
[ 2181.864866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1092 to 0x240000400:1121)
[ 2182.905304] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2186.193976] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2187.527288] Lustre: 102306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2197.472679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2197.472972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2197.476562] Lustre: Skipped 1 previous similar message
[ 2201.326408] Lustre: server umount lustre-MDT0000 complete
[ 2202.703995] LustreError: 100046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891514 with bad export cookie 6611340678040655327
[ 2202.705548] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2202.708449] LustreError: 100046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2202.737546] Lustre: server umount lustre-OST0000 complete
[ 2204.126848] Lustre: server umount lustre-OST0001 complete
[ 2208.990476] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local
[ 2211.910852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2213.296703] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2214.260023] Lustre: 105002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2217.950321] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2221.421411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1091 to 0x280000400:1153)
[ 2221.421970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1092 to 0x240000400:1153)
[ 2222.436121] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2225.515969] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2226.682876] Lustre: 106806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2229.879295] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:05:41 (1789891541)
[ 2230.431883] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 2231.014445] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:05:42 (1789891542)
[ 2266.854119] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:06:18 (1789891578)
[ 2285.277424] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6150) for user quota_usr isn't 0.
[ 2304.989161] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:06:56 (1789891616)
[ 2307.039075] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:06:58 (1789891618)
[ 2311.279010] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:07:02 (1789891622)
[ 2314.571229] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:07:06 (1789891626)
[ 2317.595078] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:07:09 (1789891629)
[ 2359.129598] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2367.462380] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (15370) for user 60000 isn't 0.
[ 2380.334213] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2388.536027] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (15370) for user 60000 isn't 0.
[ 2401.208160] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:08:32 (1789891712)
[ 2433.045382] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used space for group 60000 is 0, expected 2M
[ 2458.783226] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:09:30 (1789891770)
[ 2467.380696] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space (15370) for user 60000 isn't 0.
[ 2480.137391] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:09:51 (1789891791)
[ 2488.822636] Lustre: DEBUG MARKER: sanity-quota test_38: @@@@@@ FAIL: Used space (15370) for user 60000 isn't 0.
[ 2501.489743] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:10:13 (1789891813)
[ 2513.376942] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2513.385196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2513.390501] Lustre: Skipped 2 previous similar messages
[ 2515.536356] Lustre: server umount lustre-MDT0000 complete
[ 2516.932401] LustreError: 109169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891828 with bad export cookie 6611340678040656860
[ 2516.934122] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2516.935686] LustreError: 109169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 2516.963938] Lustre: server umount lustre-OST0000 complete
[ 2518.322701] Lustre: server umount lustre-OST0001 complete
[ 2523.078779] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local
[ 2526.162997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2526.165228] Lustre: Skipped 2 previous similar messages
[ 2527.513602] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2528.473313] Lustre: 124110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2532.148622] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2535.598298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1156 to 0x240000400:1185)
[ 2535.604718] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1156 to 0x280000400:1185)
[ 2536.621653] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2539.834301] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2541.055043] Lustre: 125914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2566.424961] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:11:18 (1789891878)
[ 2596.910671] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:11:48 (1789891908)
[ 2627.571681] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:12:19 (1789891939)
[ 2628.076538] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 2628.659785] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:12:20 (1789891940)
[ 2629.178635] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 2629.746495] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:12:21 (1789891941)
[ 2684.392077] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:13:15 (1789891995)
[ 2711.294318] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:13:42 (1789892022)
[ 2760.538764] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:14:32 (1789892072)
[ 2767.540159] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2768.054215] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2768.055669] Lustre: Skipped 83 previous similar messages
[ 2769.061413] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2769.062520] Lustre: Skipped 189 previous similar messages
[ 2771.062397] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2771.063838] Lustre: Skipped 367 previous similar messages
[ 2775.069058] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2775.070523] Lustre: Skipped 735 previous similar messages
[ 2783.070533] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2783.072172] Lustre: Skipped 1359 previous similar messages
[ 2799.078264] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2799.079827] Lustre: Skipped 2865 previous similar messages
[ 2986.464843] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2986.465347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2986.468543] Lustre: Skipped 2 previous similar messages
[ 2986.471290] Lustre: Skipped 1 previous similar message
[ 2988.294205] Lustre: server umount lustre-MDT0000 complete
[ 2989.785333] LustreError: 129715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892301 with bad export cookie 6611340678040661592
[ 2989.789367] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2989.925874] Lustre: server umount lustre-OST0000 complete
[ 2991.465078] Lustre: server umount lustre-OST0001 complete
[ 2993.660042] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid
[ 2996.270681] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local
[ 2999.353887]  vdc: vdc1 vdc9
[ 3002.482621]  vde: vde1 vde9
[ 3006.036654]  vdf: vdf1 vdf9
[ 3011.983367] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local
[ 3015.207994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 3015.291297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 3015.331261] Lustre: lustre-MDT0000: new disk, initializing
[ 3015.422846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3015.424735] Lustre: Skipped 2 previous similar messages
[ 3015.440826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 3015.443006] Lustre: Skipped 1 previous similar message
[ 3016.863748] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3019.177600] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 3020.991789] Lustre: lustre-OST0000: new disk, initializing
[ 3020.994663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 3020.998347] Lustre: Skipped 1 previous similar message
[ 3022.152910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 3022.157192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 3022.162736] Lustre: Skipped 1 previous similar message
[ 3022.192330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 3023.054578] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3026.690960] Lustre: lustre-OST0001: new disk, initializing
[ 3026.693159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 3027.841252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 3027.844032] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 3027.872622] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 3028.744918] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3033.163897] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3034.296725] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 3054.312583] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:19:25 (1789892365)
[ 3056.566091] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:19:28 (1789892368)
[ 3088.885493] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:20:00 (1789892400)
[ 3124.353770] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:20:35 (1789892435)
[ 3138.718983] Lustre: DEBUG MARKER: rename directory return 255
[ 3165.309643] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:21:16 (1789892476)
[ 3183.183778] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:21:34 (1789892494)
[ 3203.120374] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:21:54 (1789892514)
[ 3240.516256] LustreError: 141233:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 3262.007320] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:22:53 (1789892573)
[ 3281.690641] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:23:13 (1789892593)
[ 3308.227240] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:23:39 (1789892619)
[ 3391.076525] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:25:02 (1789892702)
[ 3391.600405] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 3392.211418] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:25:03 (1789892703)
[ 3425.627855] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 3426.187822] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:25:37 (1789892737)
[ 3443.713516] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 3444.307113] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 04:25:55 (1789892755)
[ 3454.261866] Lustre: DEBUG MARKER: sanity-quota test_64: @@@@@@ FAIL: expected 0 got 6
[ 3480.639246] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 3481.193961] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:26:32 (1789892792)
[ 3513.045731] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:27:04 (1789892824)
[ 3513.538471] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 3514.134306] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:27:05 (1789892825)
[ 3529.244372] LustreError: 165671:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 3551.998374] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:27:43 (1789892863)
[ 3567.749431] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 3568.277817] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 3625.237207] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:28:56 (1789892936)
[ 3652.070429] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:29:23 (1789892963)
[ 3669.153574] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:29:40 (1789892980)
[ 3669.691434] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 3670.233348] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:29:41 (1789892981)
[ 3670.753371] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 3671.269503] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:29:42 (1789892982)
[ 3679.936427] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 3685.506902] Lustre: DEBUG MARKER: sanity-quota test_72: @@@@@@ FAIL: Used space(10244) for user quota_usr isn't 0.
[ 3703.588137] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:30:15 (1789893015)
[ 3717.298273] Lustre: DEBUG MARKER: set to use default quota
[ 3717.827930] Lustre: DEBUG MARKER: set default quota
[ 3718.360873] Lustre: DEBUG MARKER: get default quota
[ 3720.186280] Lustre: DEBUG MARKER: sanity-quota test_73a: @@@@@@ FAIL: Used space for quota_usr isn't 0.