[ 1900.713854] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1902.690846] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1903.272829] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:30069).
[ 1903.890515] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1904.979649] LustreError: 86882:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1905.299069] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1905.896495] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1906.601997] Lustre: Failing over lustre-MDT0000
[ 1906.950246] Lustre: server umount lustre-MDT0000 complete
[ 1919.250427] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1919.367666] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1919.426806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1919.447562] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1920.834057] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1922.552207] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1922.569609] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1922.584909] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1024 to 0x280000400:1057)
[ 1922.584909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1038 to 0x240000400:1057)
[ 1924.579322] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1924.654808] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1925.145237] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1925.600258] Lustre: 3291:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891221/real 1789891221]  req@ffff9cd2c1ad0700 x1876835061165568/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891237 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1925.607385] Lustre: 3291:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1927.576459] Lustre: DEBUG MARKER: (dd_pid=88713, time=0, timeout=600)
[ 1930.720150] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891226/real 1789891226]  req@ffff9cd3ef43b100 x1876835061165824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1958.012885] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1959.913786] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1960.453117] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 1960.999242] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1962.091623] LustreError: 89976:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1962.418749] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1963.039860] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1963.751686] Lustre: Failing over lustre-MDT0000
[ 1964.152341] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 1964.155140] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1964.160104] Lustre: Skipped 1 previous similar message
[ 1964.192069] Lustre: server umount lustre-MDT0000 complete
[ 1976.710471] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1976.849289] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1976.906667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1976.930440] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1978.309362] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1978.871565] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1978.909890] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1978.926531] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1978.928537] Lustre: Skipped 1 previous similar message
[ 1978.938335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1059 to 0x240000400:1089)
[ 1978.939197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1024 to 0x280000400:1089)
[ 1981.961104] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891277/real 1789891277]  req@ffff9cd30863d500 x1876835061195264/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891293 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1981.970148] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1982.055137] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1982.559198] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1986.016133] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891281/real 1789891281]  req@ffff9cd30863d180 x1876835061195776/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1989.153930] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1990.112184] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891285/real 1789891285]  req@ffff9cd30863ce00 x1876835061196288/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2001.657987] Lustre: DEBUG MARKER: (dd_pid=91562, time=16, timeout=600)
[ 2041.763118] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:02:33 (1789891353)
[ 2053.164276] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2053.664803] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:101391).
[ 2055.362648] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2055.838951] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:101391).
[ 2077.796035] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:03:09 (1789891389)
[ 2086.664113] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2095.016043] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2095.566332] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2096.108804] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2096.634097] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2098.267862] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2099.989858] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2101.709399] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2103.365350] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2105.043043] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2106.679538] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2108.339636] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2110.066067] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2111.772255] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2113.526923] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2115.292538] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2117.046482] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2118.781249] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2120.501786] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2122.321317] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2124.136834] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2125.990532] Lustre: DEBUG MARKER: Set quota for 18 times
[ 2149.756874] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:04:21 (1789891461)
[ 2158.048625] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2158.048863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2158.052208] Lustre: Skipped 1 previous similar message
[ 2158.057666] Lustre: Skipped 1 previous similar message
[ 2161.016478] Lustre: server umount lustre-MDT0000 complete
[ 2162.376994] LustreError: 36665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891474 with bad export cookie 6611340678040632409
[ 2162.378335] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2162.382369] LustreError: 36665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2162.418189] Lustre: server umount lustre-OST0000 complete
[ 2163.759727] Lustre: server umount lustre-OST0001 complete
[ 2168.696546] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local
[ 2172.035066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2173.491495] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2174.508686] Lustre: 100498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2176.372087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 2178.416614] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2180.816334] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 2181.864762] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1091 to 0x280000400:1121)
[ 2181.864866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1092 to 0x240000400:1121)
[ 2182.905304] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2186.193976] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2187.527288] Lustre: 102306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2197.472679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2197.472972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2197.476562] Lustre: Skipped 1 previous similar message
[ 2201.326408] Lustre: server umount lustre-MDT0000 complete
[ 2202.703995] LustreError: 100046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891514 with bad export cookie 6611340678040655327
[ 2202.705548] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2202.708449] LustreError: 100046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2202.737546] Lustre: server umount lustre-OST0000 complete
[ 2204.126848] Lustre: server umount lustre-OST0001 complete
[ 2208.990476] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local
[ 2211.910852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2213.296703] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2214.260023] Lustre: 105002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2217.950321] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2221.421411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1091 to 0x280000400:1153)
[ 2221.421970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1092 to 0x240000400:1153)
[ 2222.436121] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2225.515969] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2226.682876] Lustre: 106806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2229.879295] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:05:41 (1789891541)
[ 2230.431883] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 2231.014445] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:05:42 (1789891542)
[ 2266.854119] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:06:18 (1789891578)
[ 2285.277424] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6150) for user quota_usr isn't 0.
[ 2304.989161] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:06:56 (1789891616)
[ 2307.039075] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:06:58 (1789891618)
[ 2311.279010] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:07:02 (1789891622)
[ 2314.571229] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:07:06 (1789891626)
[ 2317.595078] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:07:09 (1789891629)
[ 2359.129598] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2367.462380] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (15370) for user 60000 isn't 0.
[ 2380.334213] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2388.536027] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (15370) for user 60000 isn't 0.
[ 2401.208160] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:08:32 (1789891712)
[ 2433.045382] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used space for group 60000 is 0, expected 2M
[ 2458.783226] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:09:30 (1789891770)
[ 2467.380696] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space (15370) for user 60000 isn't 0.