[ 1928.361292] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1930.681933] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1931.307403] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:54629).
[ 1931.913922] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 1933.060489] LustreError: 85992:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1933.412820] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1934.098225] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1934.763970] Lustre: Failing over lustre-MDT0000
[ 1935.188889] Lustre: server umount lustre-MDT0000 complete
[ 1947.513472] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 1947.617382] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 1947.624359] Lustre: Skipped 1 previous similar message
[ 1947.663697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1947.691059] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1948.335603] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 1948.347931] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 1948.363995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2430 to 0x280000400:2465)
[ 1948.364040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2446 to 0x240000400:2465)
[ 1948.903207] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 1952.531930] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1952.737953] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 1953.022053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1954.784106] Lustre: 3278:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891111/real 1789891111]  req@ffff9a3185c14a80 x1876834893087104/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1955.314295] Lustre: DEBUG MARKER: (dd_pid=88215, time=0, timeout=600)
[ 1959.904170] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891116/real 1789891116]  req@ffff9a3185c15880 x1876834893087360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 1959.913631] Lustre: 3279:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 1985.308166] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 1987.067739] Lustre: DEBUG MARKER: Files for user (quota_usr), count=0:
[ 1987.545270] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 1988.029170] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 1988.969885] LustreError: 89068:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 1989.256444] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1989.725580] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 1990.356099] Lustre: Failing over lustre-MDT0000
[ 1990.447126] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping)
[ 1990.545385] Lustre: server umount lustre-MDT0000 complete
[ 2002.780132] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2002.903954] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2002.911965] Lustre: Skipped 1 previous similar message
[ 2002.953689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2002.979421] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2004.320988] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2008.032199] Lustre: 3280:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891164/real 1789891164]  req@ffff9a32b5e06a00 x1876834893115008/t0(0) o601->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789891180 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 2008.035060] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 2008.047165] Lustre: 3280:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 2008.050034] Lustre: Skipped 1 previous similar message
[ 2008.746732] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2008.759378] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2008.775171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2467 to 0x240000400:2497)
[ 2008.775203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2430 to 0x280000400:2497)
[ 2010.559335] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2011.070676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2013.461360] Lustre: DEBUG MARKER: (dd_pid=91050, time=0, timeout=600)
[ 2014.176138] Lustre: 3277:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891171/real 1789891171]  req@ffff9a31935dea00 x1876834893115776/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891187 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2014.190613] Lustre: 3277:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 2053.617905] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:00:26 (1789891226)
[ 2065.400873] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2065.896288] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 2067.649677] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 2068.115279] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:102414).
[ 2088.809392] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:01:01 (1789891261)
[ 2089.883702] LustreError: 3279:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 2097.889124] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2106.518546] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2107.039352] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2107.550309] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2108.179409] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2109.964461] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2111.802086] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2113.501308] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2115.346927] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2117.198792] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2119.003037] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2120.903863] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2122.659684] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2124.379949] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2126.147402] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2127.918104] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2129.757962] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2131.609643] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2133.461305] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2135.302991] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2137.164266] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2160.784775] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:02:13 (1789891333)
[ 2166.752662] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2166.752912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2166.756086] Lustre: Skipped 1 previous similar message
[ 2166.760102] Lustre: Skipped 1 previous similar message
[ 2167.996429] Lustre: server umount lustre-MDT0000 complete
[ 2169.368980] LustreError: 5748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891342 with bad export cookie 14388611505878732397
[ 2169.370364] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2169.373194] LustreError: 5748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2169.419255] Lustre: server umount lustre-OST0000 complete
[ 2170.788787] Lustre: server umount lustre-OST0001 complete
[ 2175.446453] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local
[ 2178.953299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2180.362864] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2181.353805] Lustre: 99464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2183.086201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 2185.026178] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2187.290570] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 2188.329333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2499 to 0x240000400:2529)
[ 2188.331296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2500 to 0x280000400:2529)
[ 2189.250757] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2192.237324] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2193.332848] Lustre: 101273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2198.496668] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2198.497137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2198.501322] Lustre: Skipped 1 previous similar message
[ 2203.616476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2203.619082] Lustre: Skipped 2 previous similar messages
[ 2204.137231] Lustre: server umount lustre-MDT0000 complete
[ 2205.436665] LustreError: 101523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891378 with bad export cookie 14388611505878752704
[ 2205.437895] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2205.440712] LustreError: 101523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2205.470709] Lustre: server umount lustre-OST0000 complete
[ 2206.835564] Lustre: server umount lustre-OST0001 complete
[ 2211.459721] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local
[ 2214.282765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2215.556950] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2216.501315] Lustre: 103871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2220.131883] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2224.493767] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2224.616738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2499 to 0x240000400:2561)
[ 2224.617890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2500 to 0x280000400:2561)
[ 2227.511468] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2228.648139] Lustre: 105669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2231.943852] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:03:24 (1789891404)
[ 2232.428692] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 2232.963681] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:03:25 (1789891405)
[ 2265.020708] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:03:57 (1789891437)
[ 2283.337967] Lustre: DEBUG MARKER: sanity-quota test_25: @@@@@@ FAIL: Used space(6150) for user quota_usr isn't 0.
[ 2303.258906] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:04:36 (1789891476)
[ 2305.384579] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:04:38 (1789891478)
[ 2309.789400] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:04:42 (1789891482)
[ 2313.146442] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:04:45 (1789891485)
[ 2316.436831] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:04:49 (1789891489)
[ 2362.932222] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2371.408613] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2385.278569] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2394.058207] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2406.855161] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:06:19 (1789891579)
[ 2443.260714] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used space for group 60000 is 0, expected 2M
[ 2467.475536] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:07:20 (1789891640)
[ 2476.002188] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2488.684690] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:07:41 (1789891661)
[ 2497.240956] Lustre: DEBUG MARKER: sanity-quota test_38: @@@@@@ FAIL: Used space (9222) for user 60000 isn't 0.
[ 2510.799269] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:08:03 (1789891683)
[ 2521.569041] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2521.569410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2521.573692] Lustre: Skipped 1 previous similar message
[ 2521.577009] Lustre: Skipped 1 previous similar message
[ 2525.044120] Lustre: server umount lustre-MDT0000 complete
[ 2526.338067] LustreError: 103428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891699 with bad export cookie 14388611505878754223
[ 2526.340286] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2526.341233] LustreError: 103428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 2526.382280] Lustre: server umount lustre-OST0000 complete
[ 2527.642690] Lustre: server umount lustre-OST0001 complete
[ 2532.814425] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local
[ 2536.110220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2536.112655] Lustre: Skipped 2 previous similar messages
[ 2537.485646] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2538.556495] Lustre: 122994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 2542.252058] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2546.862698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2563 to 0x280000400:2593)
[ 2546.862784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2565 to 0x240000400:2593)
[ 2546.942222] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2550.652842] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2552.056217] Lustre: 124796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 2577.339116] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:09:10 (1789891750)
[ 2608.863817] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:09:41 (1789891781)
[ 2638.047262] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:10:10 (1789891810)
[ 2638.609675] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 2639.360609] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:10:11 (1789891811)
[ 2639.852377] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 2640.376779] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:10:13 (1789891813)
[ 2689.935877] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:11:02 (1789891862)
[ 2717.035274] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:11:29 (1789891889)
[ 2763.131124] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:12:15 (1789891935)
[ 2770.330900] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2770.835670] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2770.837027] Lustre: Skipped 103 previous similar messages
[ 2771.843394] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2771.845548] Lustre: Skipped 167 previous similar messages
[ 2773.845759] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2773.847104] Lustre: Skipped 353 previous similar messages
[ 2777.851549] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2777.854075] Lustre: Skipped 747 previous similar messages
[ 2785.861058] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2785.862225] Lustre: Skipped 1513 previous similar messages
[ 2801.862739] Lustre: *** cfs_fail_loc=a09, val=0***
[ 2801.863955] Lustre: Skipped 2789 previous similar messages
[ 2967.009237] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2967.017230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2973.576554] Lustre: server umount lustre-MDT0000 complete
[ 2975.536991] LustreError: 122549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892148 with bad export cookie 14388611505878759039
[ 2975.545599] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2975.737505] Lustre: server umount lustre-OST0000 complete
[ 2977.814346] Lustre: server umount lustre-OST0001 complete
[ 2981.216855] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid
[ 2984.416584] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local
[ 2987.915741]  vdc: vdc1 vdc9
[ 2991.126321]  vde: vde1 vde9
[ 2994.656719]  vdf: vdf1 vdf9
[ 3000.560668] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local
[ 3003.568152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 3003.661818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 3003.695632] Lustre: lustre-MDT0000: new disk, initializing
[ 3003.798638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3003.801956] Lustre: Skipped 2 previous similar messages
[ 3003.818895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 3005.288913] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3007.554504] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 3009.231464] Lustre: lustre-OST0000: new disk, initializing
[ 3009.232774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 3009.234169] Lustre: Skipped 1 previous similar message
[ 3010.378079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 3010.381325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 3010.405514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 3011.198199] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3014.767958] Lustre: lustre-OST0001: new disk, initializing
[ 3014.769589] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 3016.063420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 3016.065553] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 3016.092158] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 3016.644456] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3021.023548] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3022.187667] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 3041.880779] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:16:54 (1789892214)
[ 3043.928279] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:16:56 (1789892216)
[ 3071.972966] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:17:24 (1789892244)
[ 3107.199325] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:18:00 (1789892280)
[ 3120.516065] Lustre: DEBUG MARKER: rename directory return 255
[ 3147.836439] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:18:40 (1789892320)
[ 3165.325283] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:18:58 (1789892338)
[ 3184.532468] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:19:17 (1789892357)
[ 3220.285115] LustreError: 139741:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 3246.111084] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:20:18 (1789892418)
[ 3266.153117] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:20:38 (1789892438)
[ 3298.523377] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:21:11 (1789892471)
[ 3384.297447] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:22:36 (1789892556)
[ 3384.966863] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 3385.743946] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:22:38 (1789892558)
[ 3420.456126] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 3421.014935] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:23:13 (1789892593)
[ 3439.215225] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 3439.741123] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 04:23:32 (1789892612)
[ 3449.837126] Lustre: DEBUG MARKER: sanity-quota test_64: @@@@@@ FAIL: expected 0 got 6
[ 3474.117586] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 3474.692306] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:24:07 (1789892647)
[ 3507.211269] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:24:40 (1789892680)
[ 3507.702943] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 3508.225361] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:24:41 (1789892681)
[ 3522.974731] LustreError: 163931:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 3545.252468] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:25:18 (1789892718)
[ 3559.927585] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 3560.528894] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 3625.119651] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:26:37 (1789892797)
[ 3655.635381] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:27:08 (1789892828)
[ 3670.130304] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:27:22 (1789892842)
[ 3670.864212] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 3671.525685] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:27:24 (1789892844)
[ 3672.018313] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 3672.600405] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:27:25 (1789892845)
[ 3682.062881] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 3689.677410] Lustre: DEBUG MARKER: sanity-quota test_72: @@@@@@ FAIL: Used space(10245) for user quota_usr isn't 0.
[ 3710.832385] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:28:03 (1789892883)
[ 3726.580091] Lustre: DEBUG MARKER: set to use default quota
[ 3727.247849] Lustre: DEBUG MARKER: set default quota
[ 3727.975912] Lustre: DEBUG MARKER: get default quota
[ 3730.704030] Lustre: DEBUG MARKER: sanity-quota test_73a: @@@@@@ FAIL: Used space for quota_usr isn't 0.