[ 3255.145982] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3257.539395] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 3258.625889] LustreError: 122103:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3258.941385] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3259.495768] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3260.196849] Lustre: Failing over lustre-MDT0000
[ 3260.615756] Lustre: server umount lustre-MDT0000 complete
[ 3272.789443] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3272.891456] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3272.942744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3272.963419] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3274.149234] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3274.694188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3274.727374] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3274.742844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1743 to 0x240000400:1761)
[ 3274.742957] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1740 to 0x280000400:1761)
[ 3277.024100] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891729/real 1789891729]  req@ffff8d6c060f0000 x1876834162512256/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1789891745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3277.621787] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3278.063472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3278.306755] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 3279.328184] Lustre: 3285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891732/real 1789891732]  req@ffff8d6c060f0380 x1876834162512640/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3284.448100] Lustre: 3287:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891737/real 1789891737]  req@ffff8d6c33dc2d80 x1876834162513024/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3284.462339] Lustre: 3287:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3285.802924] Lustre: DEBUG MARKER: (dd_pid=113167, time=5, timeout=600)
[ 3316.624289] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3318.735887] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 3319.802151] LustreError: 125040:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3320.135282] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3320.696638] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3321.405435] Lustre: Failing over lustre-MDT0000
[ 3321.716500] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 3321.719163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3321.843870] Lustre: server umount lustre-MDT0000 complete
[ 3334.529228] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3334.624872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up)
[ 3334.706387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3334.731618] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3336.134143] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3336.170147] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3336.172944] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3336.191861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1763 to 0x240000400:1793)
[ 3336.192353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1740 to 0x280000400:1793)
[ 3340.237563] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3340.774771] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3341.792200] Lustre: 3286:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789891793/real 1789891793]  req@ffff8d6c34b33100 x1876834162544128/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789891809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3351.865236] Lustre: DEBUG MARKER: (dd_pid=115684, time=8, timeout=600)
[ 3394.307823] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:11:03 (1789891863)
[ 3406.522075] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3407.024880] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3408.889895] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3409.406587] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3432.174394] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:11:41 (1789891901)
[ 3433.265616] LustreError: 3285:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3441.562547] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 3450.124128] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 3450.585689] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 3451.053855] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 3451.594455] Lustre: DEBUG MARKER: Set quota for 1 times
[ 3453.197326] Lustre: DEBUG MARKER: Set quota for 2 times
[ 3454.900334] Lustre: DEBUG MARKER: Set quota for 3 times
[ 3456.565115] Lustre: DEBUG MARKER: Set quota for 4 times
[ 3458.188387] Lustre: DEBUG MARKER: Set quota for 5 times
[ 3459.827802] Lustre: DEBUG MARKER: Set quota for 6 times
[ 3461.449801] Lustre: DEBUG MARKER: Set quota for 7 times
[ 3463.077510] Lustre: DEBUG MARKER: Set quota for 8 times
[ 3464.754354] Lustre: DEBUG MARKER: Set quota for 9 times
[ 3466.476511] Lustre: DEBUG MARKER: Set quota for 10 times
[ 3468.123732] Lustre: DEBUG MARKER: Set quota for 11 times
[ 3469.785566] Lustre: DEBUG MARKER: Set quota for 12 times
[ 3471.415089] Lustre: DEBUG MARKER: Set quota for 13 times
[ 3473.052244] Lustre: DEBUG MARKER: Set quota for 14 times
[ 3474.693163] Lustre: DEBUG MARKER: Set quota for 15 times
[ 3476.340268] Lustre: DEBUG MARKER: Set quota for 16 times
[ 3477.970198] Lustre: DEBUG MARKER: Set quota for 17 times
[ 3479.628964] Lustre: DEBUG MARKER: Set quota for 18 times
[ 3503.420849] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:12:52 (1789891972)
[ 3509.728830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3509.731105] Lustre: Skipped 1 previous similar message
[ 3511.835233] Lustre: server umount lustre-MDT0000 complete
[ 3513.593797] LustreError: 107718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789891982 with bad export cookie 5864709643194209458
[ 3513.595617] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3513.598995] LustreError: 107718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3513.635967] Lustre: server umount lustre-OST0000 complete
[ 3515.588548] Lustre: server umount lustre-OST0001 complete
[ 3520.942430] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 3524.464835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3525.942202] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3527.040912] Lustre: 135544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3528.874973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3530.883262] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3533.399597] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 3534.447631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1795 to 0x240000400:1825)
[ 3534.448172] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1796 to 0x280000400:1825)
[ 3535.453255] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3538.470851] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3539.641516] Lustre: 137347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3548.544606] Lustre: server umount lustre-MDT0000 complete
[ 3549.949056] LustreError: 137601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789892018 with bad export cookie 5864709643194219188
[ 3549.956088] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3559.420576] Lustre: server umount lustre-OST0000 complete
[ 3570.955389] Lustre: server umount lustre-OST0001 complete
[ 3576.208110] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 3579.332261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3580.845083] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3581.982127] Lustre: 139993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3585.730500] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3589.420099] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1796 to 0x280000400:1857)
[ 3589.421930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1795 to 0x240000400:1857)
[ 3590.659940] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3594.204169] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3595.597812] Lustre: 141799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3598.852664] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:14:27 (1789892067)
[ 3599.331904] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 3599.857299] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:14:28 (1789892068)
[ 3635.241276] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:15:04 (1789892104)
[ 3651.743563] Lustre: DEBUG MARKER: Write...
[ 3652.600397] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3690.327967] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:15:58 (1789892158)
[ 3693.461764] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:16:02 (1789892162)
[ 3698.184857] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:16:07 (1789892167)
[ 3701.686933] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:16:10 (1789892170)
[ 3705.286183] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:16:14 (1789892174)
[ 3747.465095] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3779.481974] Lustre: DEBUG MARKER: sanity-quota test_33: @@@@@@ FAIL: Used space for user 60000:0, expected:20480
[ 3806.945435] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3843.922901] Lustre: DEBUG MARKER: sanity-quota test_34: @@@@@@ FAIL: Used space for user 60000 is 0, expected 2048