[ 1800.937197] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1802.641152] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1803.107786] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:13698).
[ 1803.614360] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1804.700161] LustreError: 86267:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1805.038681] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1805.592686] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1806.254482] Lustre: Failing over lustre-MDT0000
[ 1806.702667] Lustre: server umount lustre-MDT0000 complete
[ 1819.056412] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1819.163247] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1819.171286] Lustre: Skipped 1 previous similar message
[ 1819.219224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1819.245425] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1820.582451] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1820.601991] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1820.604967] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1820.622317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1932 to 0x240000400:1953)
[ 1820.622925] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1917 to 0x280000400:1953)
[ 1824.225649] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1824.406804] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1824.944692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1825.247183] Lustre: 3277:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893937/real 1789893937]  req@ffff90f087215880 x1876837988509952/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789893953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1827.371528] Lustre: DEBUG MARKER: (dd_pid=88447, time=0, timeout=600)
[ 1830.367198] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893942/real 1789893942]  req@ffff90f1486c8700 x1876837988510336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789893958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1830.375913] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1857.421613] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1859.361494] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1859.904038] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 1860.464917] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1861.617199] LustreError: 89334:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1861.995077] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1862.606848] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1863.359339] Lustre: Failing over lustre-MDT0000
[ 1863.558868] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 1863.563203] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1863.591281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 1863.775129] Lustre: server umount lustre-MDT0000 complete
[ 1876.160431] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1876.262526] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1876.311478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1876.335120] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1877.698058] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1879.676832] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1879.709138] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1879.725422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1917 to 0x280000400:1985)
[ 1879.725433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1955 to 0x240000400:1985)
[ 1880.416883] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 1880.418627] Lustre: Skipped 1 previous similar message
[ 1881.376070] Lustre: 3276:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893993/real 1789893993]  req@ffff90f087214a80 x1876837988539264/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789894009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1881.386054] Lustre: 3276:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1881.515152] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1882.055163] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1885.663140] Lustre: 3276:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789893998/real 1789893998]  req@ffff90f087217480 x1876837988540160/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789894014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1885.675168] Lustre: 3276:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1890.783101] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789894003/real 1789894003]  req@ffff90f042471500 x1876837988540672/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789894019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1892.641255] Lustre: DEBUG MARKER: (dd_pid=91268, time=8, timeout=600)
[ 1944.050438] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:47:52 (1789894072)
[ 1955.172063] hrtimer: interrupt took 4144786 ns
[ 1967.619921] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1969.442767] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:101390).
[ 1974.799638] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 1976.116673] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:101390).
[ 2000.715827] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:48:49 (1789894129)
[ 2013.869814] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2024.953944] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2025.822241] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2026.735855] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2027.841639] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2030.085462] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2032.405789] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2034.743071] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2036.854215] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2039.048807] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2041.347613] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2043.693855] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2046.000596] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2048.338349] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2050.624106] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2052.808880] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2055.099895] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2057.197441] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2081.212547] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:50:09 (1789894209)
[ 2089.127870] Lustre: server umount lustre-MDT0000 complete
[ 2091.465079] LustreError: 7450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789894219 with bad export cookie 14360426835585196730
[ 2091.470715] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2100.767666] Lustre: server umount lustre-OST0000 complete
[ 2113.293114] Lustre: server umount lustre-OST0001 complete
[ 2121.179348] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local
[ 2126.615372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2129.439308] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2131.329163] Lustre: 99403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2134.971172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 2135.978442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1987 to 0x240000400:2017)
[ 2136.994869] LustreError: 99730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2137.005250] LustreError: 99730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages
[ 2138.671519] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2142.179255] LustreError: 99729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2143.098415] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 2147.247745] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2148.328224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1988 to 0x280000400:2017)
[ 2152.396506] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2154.403688] Lustre: 101233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2168.800241] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2168.800657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2168.809235] Lustre: Skipped 1 previous similar message
[ 2168.818645] Lustre: Skipped 1 previous similar message
[ 2171.993638] Lustre: server umount lustre-MDT0000 complete
[ 2174.385187] LustreError: 98952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789894302 with bad export cookie 14360426835585214027
[ 2174.385702] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2174.396462] LustreError: 98952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2174.457318] Lustre: server umount lustre-OST0000 complete
[ 2176.894046] Lustre: server umount lustre-OST0001 complete
[ 2184.921970] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local
[ 2189.900423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2192.283322] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2193.979405] Lustre: 103928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2200.096860] LustreError: 104254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2200.106979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1987 to 0x240000400:2049)
[ 2200.569727] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2207.906604] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2209.773269] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1988 to 0x280000400:2049)
[ 2212.505860] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2214.294063] Lustre: 105744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2219.158310] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:52:27 (1789894347)
[ 2219.994383] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 2220.996947] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:52:29 (1789894349)
[ 2262.974708] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:53:11 (1789894391)
[ 2292.220295] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6150) for user quota_usr isn't 0.
[ 2320.016960] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:54:08 (1789894448)
[ 2323.734466] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:54:12 (1789894452)
[ 2330.160597] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:54:18 (1789894458)
[ 2335.367289] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:54:24 (1789894464)
[ 2339.967961] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:54:28 (1789894468)
[ 2412.368906] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2428.480782] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2454.658600] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2467.872901] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2489.933942] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:56:58 (1789894618)
[ 2532.935349] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used space for group 60000 is 0, expected 2M
[ 2561.476178] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:58:09 (1789894689)
[ 2574.409338] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2596.283730] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:58:44 (1789894724)
[ 2608.867348] Lustre: DEBUG MARKER: sanity-quota test_38: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2629.711761] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:59:18 (1789894758)
[ 2650.994694] Lustre: server umount lustre-MDT0000 complete
[ 2658.628872] LustreError: 3276:0:(qsd_handler.c:318:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff90f15139fab8 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2359 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 2658.650888] LustreError: 108008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789894787 with bad export cookie 14360426835585215560
[ 2658.683991] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2669.159644] Lustre: server umount lustre-OST0000 complete
[ 2670.562967] Lustre: 3278:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789894783/real 1789894783]  req@ffff90f080919f80 x1876837988856192/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789894799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2670.601919] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2674.733891] Lustre: server umount lustre-OST0001 complete
[ 2690.628353] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local
[ 2698.432482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2698.437174] Lustre: Skipped 2 previous similar messages
[ 2701.881532] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2704.208600] Lustre: 122618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2713.469852] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2722.057731] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2723.376324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2053 to 0x240000400:2081)
[ 2723.377895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2051 to 0x280000400:2081)
[ 2727.103927] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2729.209678] Lustre: 124436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2755.855551] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 05:01:24 (1789894884)
[ 2791.477244] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 05:02:00 (1789894920)
[ 2821.993585] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 05:02:30 (1789894950)
[ 2822.892995] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 2823.880978] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 05:02:32 (1789894952)
[ 2824.789299] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 2825.752224] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 05:02:34 (1789894954)
[ 2912.934817] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 05:04:01 (1789895041)
[ 2954.771613] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 05:04:43 (1789895083)
[ 3015.756291] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 05:05:44 (1789895144)
[ 3024.099165] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3024.616048] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3024.620159] Lustre: Skipped 47 previous similar messages
[ 3025.635801] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3025.638899] Lustre: Skipped 87 previous similar messages
[ 3027.656217] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3027.659040] Lustre: Skipped 171 previous similar messages
[ 3031.656182] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3031.660099] Lustre: Skipped 352 previous similar messages
[ 3039.656909] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3039.661736] Lustre: Skipped 737 previous similar messages
[ 3055.674495] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3055.676470] Lustre: Skipped 1432 previous similar messages
[ 3087.707475] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3087.710695] Lustre: Skipped 2503 previous similar messages
[ 3368.416442] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3368.424946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3368.428513] Lustre: Skipped 1 previous similar message
[ 3371.614966] Lustre: server umount lustre-MDT0000 complete
[ 3373.444200] LustreError: 122170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789895501 with bad export cookie 14360426835585220369
[ 3373.450599] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3373.621359] Lustre: server umount lustre-OST0000 complete
[ 3375.480627] Lustre: server umount lustre-OST0001 complete
[ 3378.313883] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid
[ 3381.448254] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local
[ 3385.935234]  vdc: vdc1 vdc9
[ 3390.532692]  vde: vde1 vde9
[ 3395.104812]  vdf: vdf1 vdf9
[ 3402.309815] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local
[ 3406.169384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 3406.297642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 3406.339581] Lustre: lustre-MDT0000: new disk, initializing
[ 3406.507427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3406.512080] Lustre: Skipped 2 previous similar messages
[ 3406.546357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 3408.351795] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3410.969843] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 3413.160967] Lustre: lustre-OST0000: new disk, initializing
[ 3413.163306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 3413.165415] Lustre: Skipped 1 previous similar message
[ 3413.204507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3414.505462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 3414.509288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 3414.548765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 3415.693198] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3420.516914] Lustre: lustre-OST0001: new disk, initializing
[ 3420.519569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 3422.307282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 3422.311296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 3422.352036] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 3423.165787] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3428.155824] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3429.707955] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 3451.819080] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 05:13:00 (1789895580)
[ 3454.601173] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 05:13:03 (1789895583)
[ 3483.243881] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 05:13:32 (1789895612)
[ 3519.093352] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 05:14:08 (1789895648)
[ 3534.567342] Lustre: DEBUG MARKER: rename directory return 255
[ 3565.427073] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 05:14:54 (1789895694)
[ 3584.492278] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 05:15:13 (1789895713)
[ 3608.504461] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 05:15:37 (1789895737)
[ 3651.922629] LustreError: 149225:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 3675.390802] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 05:16:44 (1789895804)
[ 3697.513844] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 05:17:06 (1789895826)
[ 3729.724847] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 05:17:38 (1789895858)
[ 3816.512086] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 05:19:05 (1789895945)
[ 3817.120810] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 3817.751289] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 05:19:06 (1789895946)
[ 3853.066277] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 3853.789163] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 05:19:42 (1789895982)
[ 3872.774411] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 3873.388344] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 05:20:02 (1789896002)
[ 3884.355312] Lustre: DEBUG MARKER: sanity-quota test_64: @@@@@@ FAIL: expected 0 got 6