[ 3255.145982] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3257.539395] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 3258.625889] LustreError: 122103:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3258.941385] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3259.495768] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3260.196849] Lustre: Failing over lustre-MDT0000
[ 3260.615756] Lustre: server umount lustre-MDT0000 complete
[ 3272.789443] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3272.891456] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3272.942744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3272.963419] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3274.149234] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3274.694188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3274.727374] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3274.742844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1743 to 0x240000400:1761)
[ 3274.742957] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1740 to 0x280000400:1761)
[ 3277.024100] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891729/real 1789891729]  req@ffff8d6c060f0000 x1876834162512256/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789891745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3277.621787] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3278.063472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3278.306755] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 3279.328184] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891732/real 1789891732]  req@ffff8d6c060f0380 x1876834162512640/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3284.448100] Lustre: 3287:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891737/real 1789891737]  req@ffff8d6c33dc2d80 x1876834162513024/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3284.462339] Lustre: 3287:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3285.802924] Lustre: DEBUG MARKER: (dd_pid=113167, time=5, timeout=600)
[ 3316.624289] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3318.735887] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 3319.802151] LustreError: 125040:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3320.135282] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3320.696638] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3321.405435] Lustre: Failing over lustre-MDT0000
[ 3321.716500] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 3321.719163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3321.843870] Lustre: server umount lustre-MDT0000 complete
[ 3334.529228] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3334.624872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up)
[ 3334.706387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3334.731618] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3336.134143] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3336.170147] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3336.172944] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3336.191861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1763 to 0x240000400:1793)
[ 3336.192353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1740 to 0x280000400:1793)
[ 3340.237563] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3340.774771] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3341.792200] Lustre: 3286:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891793/real 1789891793]  req@ffff8d6c34b33100 x1876834162544128/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3351.865236] Lustre: DEBUG MARKER: (dd_pid=115684, time=8, timeout=600)
[ 3394.307823] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:11:03 (1789891863)
[ 3406.522075] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3407.024880] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3408.889895] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3409.406587] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3432.174394] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:11:41 (1789891901)
[ 3433.265616] LustreError: 3285:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3441.562547] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 3450.124128] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 3450.585689] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 3451.053855] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 3451.594455] Lustre: DEBUG MARKER: Set quota for 1 times
[ 3453.197326] Lustre: DEBUG MARKER: Set quota for 2 times
[ 3454.900334] Lustre: DEBUG MARKER: Set quota for 3 times
[ 3456.565115] Lustre: DEBUG MARKER: Set quota for 4 times
[ 3458.188387] Lustre: DEBUG MARKER: Set quota for 5 times
[ 3459.827802] Lustre: DEBUG MARKER: Set quota for 6 times
[ 3461.449801] Lustre: DEBUG MARKER: Set quota for 7 times
[ 3463.077510] Lustre: DEBUG MARKER: Set quota for 8 times
[ 3464.754354] Lustre: DEBUG MARKER: Set quota for 9 times
[ 3466.476511] Lustre: DEBUG MARKER: Set quota for 10 times
[ 3468.123732] Lustre: DEBUG MARKER: Set quota for 11 times
[ 3469.785566] Lustre: DEBUG MARKER: Set quota for 12 times
[ 3471.415089] Lustre: DEBUG MARKER: Set quota for 13 times
[ 3473.052244] Lustre: DEBUG MARKER: Set quota for 14 times
[ 3474.693163] Lustre: DEBUG MARKER: Set quota for 15 times
[ 3476.340268] Lustre: DEBUG MARKER: Set quota for 16 times
[ 3477.970198] Lustre: DEBUG MARKER: Set quota for 17 times
[ 3479.628964] Lustre: DEBUG MARKER: Set quota for 18 times
[ 3503.420849] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:12:52 (1789891972)
[ 3509.728830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3509.731105] Lustre: Skipped 1 previous similar message
[ 3511.835233] Lustre: server umount lustre-MDT0000 complete
[ 3513.593797] LustreError: 107718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891982 with bad export cookie 5864709643194209458
[ 3513.595617] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3513.598995] LustreError: 107718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3513.635967] Lustre: server umount lustre-OST0000 complete
[ 3515.588548] Lustre: server umount lustre-OST0001 complete
[ 3520.942430] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 3524.464835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3525.942202] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3527.040912] Lustre: 135544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3528.874973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3530.883262] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3533.399597] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 3534.447631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1795 to 0x240000400:1825)
[ 3534.448172] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1796 to 0x280000400:1825)
[ 3535.453255] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3538.470851] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3539.641516] Lustre: 137347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3548.544606] Lustre: server umount lustre-MDT0000 complete
[ 3549.949056] LustreError: 137601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892018 with bad export cookie 5864709643194219188
[ 3549.956088] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3559.420576] Lustre: server umount lustre-OST0000 complete
[ 3570.955389] Lustre: server umount lustre-OST0001 complete
[ 3576.208110] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 3579.332261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3580.845083] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3581.982127] Lustre: 139993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3585.730500] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3589.420099] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1796 to 0x280000400:1857)
[ 3589.421930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1795 to 0x240000400:1857)
[ 3590.659940] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3594.204169] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3595.597812] Lustre: 141799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3598.852664] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:14:27 (1789892067)
[ 3599.331904] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 3599.857299] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:14:28 (1789892068)
[ 3635.241276] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:15:04 (1789892104)
[ 3651.743563] Lustre: DEBUG MARKER: Write...
[ 3652.600397] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3690.327967] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:15:58 (1789892158)
[ 3693.461764] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:16:02 (1789892162)
[ 3698.184857] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:16:07 (1789892167)
[ 3701.686933] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:16:10 (1789892170)
[ 3705.286183] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:16:14 (1789892174)
[ 3747.465095] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3779.481974] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space for user 60000:0, expected:20480
[ 3806.945435] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3843.922901] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space for user 60000 is 0, expected 2048
[ 3867.630982] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:18:56 (1789892336)
[ 3896.291904] Lustre: DEBUG MARKER: sanity-quota test_35: @@@@@@ FAIL: Used space for user 60000 is 0, expected 2M
[ 3919.040131] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:19:47 (1789892387)
[ 3950.642211] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3951.123417] Lustre: DEBUG MARKER: sanity-quota test_37: @@@@@@ FAIL: Used space is 0
[ 3975.065815] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:20:43 (1789892443)
[ 4408.489348] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:27:57 (1789892877)
[ 4418.629837] Lustre: server umount lustre-MDT0000 complete
[ 4419.956971] LustreError: 141103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892888 with bad export cookie 5864709643194220721
[ 4419.960425] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4429.314660] Lustre: server umount lustre-OST0000 complete
[ 4430.627609] LustreError: 3284:0:(qsd_handler.c:318:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff8d6b02485398 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2375 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 4440.860443] Lustre: server umount lustre-OST0001 complete
[ 4445.355681] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 4448.387301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4448.390357] Lustre: Skipped 2 previous similar messages
[ 4449.716581] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4450.697985] Lustre: 163411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4452.412127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 4454.218634] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4456.538750] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 4458.460985] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4458.600886] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6866 to 0x280000400:6881)
[ 4458.601437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6868 to 0x240000400:6913)
[ 4461.557587] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4462.673729] Lustre: 165216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4489.062941] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:29:17 (1789892957)
[ 4519.439143] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:29:48 (1789892988)
[ 4550.734977] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:30:19 (1789893019)
[ 4551.258050] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 4551.853171] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:30:20 (1789893020)
[ 4552.367179] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 4552.908888] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:30:21 (1789893021)
[ 4607.144693] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:31:16 (1789893076)
[ 4634.071866] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:31:42 (1789893102)
[ 4678.338890] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:32:27 (1789893147)
[ 4685.337752] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4685.838471] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4685.839696] Lustre: Skipped 105 previous similar messages
[ 4686.839315] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4686.840500] Lustre: Skipped 216 previous similar messages
[ 4688.849624] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4688.851270] Lustre: Skipped 412 previous similar messages
[ 4692.852748] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4692.854196] Lustre: Skipped 765 previous similar messages
[ 4700.855511] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4700.857555] Lustre: Skipped 1603 previous similar messages
[ 4716.857603] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4716.859372] Lustre: Skipped 2833 previous similar messages
[ 4894.176877] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4894.177398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4894.181402] Lustre: Skipped 6 previous similar messages
[ 4894.185625] Lustre: Skipped 1 previous similar message
[ 4896.637276] Lustre: server umount lustre-MDT0000 complete
[ 4898.141949] LustreError: 162964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789893367 with bad export cookie 5864709643195964148
[ 4898.145806] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4898.293096] Lustre: server umount lustre-OST0000 complete
[ 4899.814165] Lustre: server umount lustre-OST0001 complete
[ 4902.087793] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid
[ 4904.843384] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 4908.475203]  vdc: vdc1 vdc9
[ 4911.820747]  vde: vde1 vde9
[ 4915.423647]  vdf: vdf1 vdf9
[ 4921.076412] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 4923.829207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 4923.911964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 4923.946714] Lustre: lustre-MDT0000: new disk, initializing
[ 4924.026930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4924.046749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 4925.497493] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4927.742944] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 4929.381216] Lustre: lustre-OST0000: new disk, initializing
[ 4929.382965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 4929.385250] Lustre: Skipped 1 previous similar message
[ 4930.830544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 4930.832889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 4930.864489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 4931.303854] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4934.779578] Lustre: lustre-OST0001: new disk, initializing
[ 4934.781237] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 4936.696808] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4936.827312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 4936.830657] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 4936.859124] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 4941.208367] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4942.397433] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 4961.831339] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:37:10 (1789893430)
[ 4963.815855] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:37:12 (1789893432)
[ 4992.486979] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:37:41 (1789893461)
[ 5029.308408] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:38:18 (1789893498)
[ 5042.415610] Lustre: DEBUG MARKER: rename directory return 255
[ 5068.894130] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:38:57 (1789893537)
[ 5086.272477] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:39:15 (1789893555)
[ 5105.132947] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:39:34 (1789893574)
[ 5141.371582] LustreError: 180479:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 5167.237770] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:40:36 (1789893636)
[ 5186.332373] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:40:55 (1789893655)
[ 5217.833930] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:41:26 (1789893686)
[ 5298.846590] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:42:47 (1789893767)
[ 5299.331846] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 5299.856329] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:42:48 (1789893768)
[ 5335.371245] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 5335.896723] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:43:24 (1789893804)
[ 5352.888948] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 5353.381250] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 04:43:42 (1789893822)
[ 5362.951947] Lustre: DEBUG MARKER: sanity-quota test_64: @@@@@@ FAIL: expected 0 got 6
[ 5388.360593] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 5388.907306] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:44:17 (1789893857)
[ 5417.147642] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:44:46 (1789893886)
[ 5417.640877] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 5418.206632] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:44:47 (1789893887)
[ 5433.093074] LustreError: 205314:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 5455.161043] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:45:24 (1789893924)
[ 5470.815432] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 5471.352516] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 5523.443793] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:46:32 (1789893992)
[ 5551.496346] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:47:00 (1789894020)
[ 5567.201226] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:47:16 (1789894036)
[ 5567.703948] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 5568.256234] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:47:17 (1789894037)
[ 5568.735577] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 5569.294203] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:47:18 (1789894038)
[ 5577.967737] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 5588.360204] Lustre: DEBUG MARKER: Write...
[ 5589.706770] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5637.771937] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:48:26 (1789894106)
[ 5658.473334] Lustre: DEBUG MARKER: set to use default quota
[ 5659.605568] Lustre: DEBUG MARKER: set default quota
[ 5660.520460] Lustre: DEBUG MARKER: get default quota
[ 5663.017585] Lustre: DEBUG MARKER: Test not out of quota
[ 5664.654689] Lustre: DEBUG MARKER: Test out of quota
[ 5670.087863] Lustre: DEBUG MARKER: Increase default quota
[ 5696.461814] Lustre: DEBUG MARKER: Set quota to override default quota
[ 5696.486093] LustreError: 180506:0:(qmt_entry.c:559:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1790498965 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 5701.889456] Lustre: DEBUG MARKER: Set to use default quota again
[ 5714.865603] Lustre: DEBUG MARKER: Cleanup
[ 5767.747514] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 04:50:36 (1789894236)
[ 5783.532873] Lustre: DEBUG MARKER: set default quota for qpool1
[ 5784.373058] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 5823.388440] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 04:51:32 (1789894292)
[ 5882.270707] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 04:52:30 (1789894350)
[ 5936.036907] Lustre: DEBUG MARKER: Write...
[ 5937.909467] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5976.324765] Lustre: DEBUG MARKER: Files for user (60000), count=3:
[ 5977.421439] Lustre: DEBUG MARKER: sanity-quota test_75: @@@@@@ FAIL: user quota not released after deletion