[ 1800.937197] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1802.641152] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1803.107786] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:13698).
[ 1803.614360] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1804.700161] LustreError: 86267:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1805.038681] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1805.592686] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1806.254482] Lustre: Failing over lustre-MDT0000
[ 1806.702667] Lustre: server umount lustre-MDT0000 complete
[ 1819.056412] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1819.163247] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1819.171286] Lustre: Skipped 1 previous similar message
[ 1819.219224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1819.245425] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1820.582451] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1820.601991] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1820.604967] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1820.622317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1932 to 0x240000400:1953)
[ 1820.622925] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1917 to 0x280000400:1953)
[ 1824.225649] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1824.406804] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1824.944692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1825.247183] Lustre: 3277:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893937/real 1789893937]  req@ffff90f087215880 x1876837988509952/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789893953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1827.371528] Lustre: DEBUG MARKER: (dd_pid=88447, time=0, timeout=600)
[ 1830.367198] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893942/real 1789893942]  req@ffff90f1486c8700 x1876837988510336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789893958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1830.375913] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1857.421613] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1859.361494] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1859.904038] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 1860.464917] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1861.617199] LustreError: 89334:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1861.995077] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1862.606848] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1863.359339] Lustre: Failing over lustre-MDT0000
[ 1863.558868] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 1863.563203] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1863.591281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1863.775129] Lustre: server umount lustre-MDT0000 complete
[ 1876.160431] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1876.262526] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1876.311478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1876.335120] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1877.698058] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1879.676832] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1879.709138] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1879.725422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1917 to 0x280000400:1985)
[ 1879.725433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1955 to 0x240000400:1985)
[ 1880.416883] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1880.418627] Lustre: Skipped 1 previous similar message
[ 1881.376070] Lustre: 3276:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893993/real 1789893993]  req@ffff90f087214a80 x1876837988539264/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789894009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1881.386054] Lustre: 3276:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1881.515152] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1882.055163] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1885.663140] Lustre: 3276:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893998/real 1789893998]  req@ffff90f087217480 x1876837988540160/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789894014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1885.675168] Lustre: 3276:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1890.783101] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789894003/real 1789894003]  req@ffff90f042471500 x1876837988540672/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789894019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1892.641255] Lustre: DEBUG MARKER: (dd_pid=91268, time=8, timeout=600)
[ 1944.050438] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:47:52 (1789894072)
[ 1955.172063] hrtimer: interrupt took 4144786 ns
[ 1967.619921] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1969.442767] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:101390).
[ 1974.799638] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1976.116673] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:101390).
[ 2000.715827] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:48:49 (1789894129)
[ 2013.869814] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2024.953944] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2025.822241] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2026.735855] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2027.841639] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2030.085462] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2032.405789] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2034.743071] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2036.854215] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2039.048807] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2041.347613] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2043.693855] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2046.000596] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2048.338349] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2050.624106] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2052.808880] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2055.099895] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2057.197441] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2081.212547] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:50:09 (1789894209)
[ 2089.127870] Lustre: server umount lustre-MDT0000 complete
[ 2091.465079] LustreError: 7450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789894219 with bad export cookie 14360426835585196730
[ 2091.470715] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2100.767666] Lustre: server umount lustre-OST0000 complete
[ 2113.293114] Lustre: server umount lustre-OST0001 complete
[ 2121.179348] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local
[ 2126.615372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2129.439308] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2131.329163] Lustre: 99403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2134.971172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 2135.978442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1987 to 0x240000400:2017)
[ 2136.994869] LustreError: 99730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2137.005250] LustreError: 99730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 2138.671519] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2142.179255] LustreError: 99729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2143.098415] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 2147.247745] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2148.328224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1988 to 0x280000400:2017)
[ 2152.396506] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2154.403688] Lustre: 101233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2168.800241] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2168.800657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2168.809235] Lustre: Skipped 1 previous similar message
[ 2168.818645] Lustre: Skipped 1 previous similar message
[ 2171.993638] Lustre: server umount lustre-MDT0000 complete
[ 2174.385187] LustreError: 98952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789894302 with bad export cookie 14360426835585214027
[ 2174.385702] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2174.396462] LustreError: 98952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2174.457318] Lustre: server umount lustre-OST0000 complete
[ 2176.894046] Lustre: server umount lustre-OST0001 complete
[ 2184.921970] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local
[ 2189.900423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2192.283322] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2193.979405] Lustre: 103928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2200.096860] LustreError: 104254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2200.106979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1987 to 0x240000400:2049)
[ 2200.569727] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2207.906604] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2209.773269] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1988 to 0x280000400:2049)
[ 2212.505860] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2214.294063] Lustre: 105744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2219.158310] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:52:27 (1789894347)
[ 2219.994383] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 2220.996947] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:52:29 (1789894349)
[ 2262.974708] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:53:11 (1789894391)
[ 2292.220295] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6150) for user quota_usr isn't 0.
[ 2320.016960] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:54:08 (1789894448)
[ 2323.734466] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:54:12 (1789894452)
[ 2330.160597] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:54:18 (1789894458)
[ 2335.367289] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:54:24 (1789894464)
[ 2339.967961] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:54:28 (1789894468)
[ 2412.368906] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2428.480782] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2454.658600] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2467.872901] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2489.933942] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:56:58 (1789894618)
[ 2532.935349] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used space for group 60000 is 0, expected 2M