[ 3268.852558] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3270.697976] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 3271.674170] LustreError: 129264:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3271.956945] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3272.468786] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3273.111939] Lustre: Failing over lustre-MDT0000
[ 3273.405547] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 3273.408497] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3273.411732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3273.506142] Lustre: server umount lustre-MDT0000 complete
[ 3285.686313] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3285.875130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3285.895170] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3287.073441] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3288.934374] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3288.962331] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3288.977106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1600 to 0x240000400:1633)
[ 3288.977123] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1595 to 0x280000400:1633)
[ 3290.570065] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3290.979236] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 3291.072980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3292.000148] Lustre: 3292:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782633757/real 1782633757]  req@ffff976188422680 x1869223613400448/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782633773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3292.006777] Lustre: 3292:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3297.248083] Lustre: 3292:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782633762/real 1782633762]  req@ffff9761cfa42d80 x1869223613401344/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782633778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3305.758359] Lustre: DEBUG MARKER: (dd_pid=119772, time=12, timeout=600)
[ 3333.579742] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 3335.404843] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 3336.427547] LustreError: 132171:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3336.731129] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3337.273899] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 3337.963643] Lustre: Failing over lustre-MDT0000
[ 3338.263241] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 3338.265753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3338.369676] Lustre: server umount lustre-MDT0000 complete
[ 3350.643979] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3350.843163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3350.865398] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3351.957835] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3351.990554] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3352.008607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1595 to 0x280000400:1665)
[ 3352.008620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1635 to 0x240000400:1665)
[ 3352.061807] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3355.543846] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3356.086498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3357.920087] Lustre: 3292:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782633823/real 1782633823]  req@ffff9762bcdaed80 x1869223613433600/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782633839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3363.296136] Lustre: 3292:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782633828/real 1782633828]  req@ffff97619040df80 x1869223613434240/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782633844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3367.656908] Lustre: DEBUG MARKER: (dd_pid=122255, time=9, timeout=600)
[ 3405.538378] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:04:46 (1782633886)
[ 3418.101798] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3418.647646] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3420.556733] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 3421.045729] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 3441.697730] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:05:22 (1782633922)
[ 3442.779286] LustreError: 3291:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3450.449078] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 3458.993349] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 3459.489875] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 3459.973304] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 3460.532870] Lustre: DEBUG MARKER: Set quota for 1 times
[ 3462.219797] Lustre: DEBUG MARKER: Set quota for 2 times
[ 3463.869513] Lustre: DEBUG MARKER: Set quota for 3 times
[ 3465.581721] Lustre: DEBUG MARKER: Set quota for 4 times
[ 3467.322564] Lustre: DEBUG MARKER: Set quota for 5 times
[ 3469.055468] Lustre: DEBUG MARKER: Set quota for 6 times
[ 3470.723898] Lustre: DEBUG MARKER: Set quota for 7 times
[ 3472.332976] Lustre: DEBUG MARKER: Set quota for 8 times
[ 3473.984686] Lustre: DEBUG MARKER: Set quota for 9 times
[ 3475.597052] Lustre: DEBUG MARKER: Set quota for 10 times
[ 3477.219697] Lustre: DEBUG MARKER: Set quota for 11 times
[ 3478.794715] Lustre: DEBUG MARKER: Set quota for 12 times
[ 3480.422404] Lustre: DEBUG MARKER: Set quota for 13 times
[ 3482.033427] Lustre: DEBUG MARKER: Set quota for 14 times
[ 3483.663726] Lustre: DEBUG MARKER: Set quota for 15 times
[ 3485.258139] Lustre: DEBUG MARKER: Set quota for 16 times
[ 3486.858367] Lustre: DEBUG MARKER: Set quota for 17 times
[ 3488.480856] Lustre: DEBUG MARKER: Set quota for 18 times
[ 3490.025794] Lustre: DEBUG MARKER: Set quota for 19 times
[ 3513.435583] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:06:34 (1782633994)
[ 3520.992869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3520.995290] Lustre: Skipped 1 previous similar message
[ 3521.353840] Lustre: server umount lustre-MDT0000 complete
[ 3522.573613] LustreError: 5761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634003 with bad export cookie 7591067977031985741
[ 3522.574630] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3522.576695] LustreError: 5761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3522.604592] Lustre: server umount lustre-OST0000 complete
[ 3523.791448] Lustre: server umount lustre-OST0001 complete
[ 3527.881505] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 3530.587988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3531.784161] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3532.664123] Lustre: 142706:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3534.307624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3536.123779] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3538.339329] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 3540.178696] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3540.395677] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1667 to 0x280000400:1697)
[ 3540.396527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1668 to 0x240000400:1697)
[ 3543.216796] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3544.330861] Lustre: 144518:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3550.688525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3550.688761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3550.691759] Lustre: Skipped 6 previous similar messages
[ 3552.201612] Lustre: server umount lustre-MDT0000 complete
[ 3553.592331] LustreError: 142255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634034 with bad export cookie 7591067977031995541
[ 3553.594330] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3553.600804] LustreError: 142255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3555.637614] Lustre: server umount lustre-OST0000 complete
[ 3557.010477] Lustre: server umount lustre-OST0001 complete
[ 3563.555069] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 3567.414315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3569.066363] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3570.550883] Lustre: 147162:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 3575.365984] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3577.057910] LustreError: 147486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3577.066033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1668 to 0x240000400:1729)
[ 3581.276533] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3583.992190] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1667 to 0x280000400:1729)
[ 3584.687115] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3586.233568] Lustre: 148970:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 3589.994279] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:07:50 (1782634070)
[ 3590.624358] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 3591.342049] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:07:51 (1782634071)
[ 3630.028526] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:08:30 (1782634110)
[ 3649.452312] Lustre: DEBUG MARKER: Write...
[ 3650.185566] Lustre: DEBUG MARKER: Write out of block quota ...
[ 3683.787906] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:09:24 (1782634164)
[ 3685.727758] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:09:26 (1782634166)
[ 3690.447882] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:09:31 (1782634171)
[ 3693.779795] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:09:34 (1782634174)
[ 3697.175238] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:09:37 (1782634177)
[ 3740.484027] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 3821.253876] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 3940.957731] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:13:41 (1782634421)
[ 3981.075663] Lustre: DEBUG MARKER: Restart...
[ 3983.330297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3983.335723] Lustre: Skipped 2 previous similar messages
[ 3986.400413] LustreError: 3291:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff97629884ca80 x1869223613745536/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 3986.416336] LustreError: 3291:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff9762b0c52600 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2660 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 3988.514384] Lustre: server umount lustre-MDT0000 complete
[ 3990.194412] LustreError: 149720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634471 with bad export cookie 7591067977031997074
[ 3990.196845] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3990.198621] LustreError: 149720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3990.229126] Lustre: server umount lustre-OST0000 complete
[ 3991.724043] Lustre: server umount lustre-OST0001 complete
[ 3996.266088] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 3999.877676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3999.879459] Lustre: Skipped 2 previous similar messages
[ 4001.265137] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4002.353255] Lustre: 168791:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4006.969251] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4010.535800] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1738 to 0x280000400:1761)
[ 4010.538996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1738 to 0x240000400:1761)
[ 4011.492388] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4014.488600] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4015.596358] Lustre: 170606:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 4045.580280] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:15:26 (1782634526)
[ 4096.470711] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:16:17 (1782634577)
[ 4501.483269] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:23:02 (1782634982)
[ 4511.262282] Lustre: server umount lustre-MDT0000 complete
[ 4512.443808] LustreError: 168344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634993 with bad export cookie 7591067977032003465
[ 4512.447225] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4521.980556] Lustre: server umount lustre-OST0000 complete
[ 4533.378331] Lustre: server umount lustre-OST0001 complete
[ 4537.616970] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 4540.445637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4540.447436] Lustre: Skipped 2 previous similar messages
[ 4541.598207] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4542.456832] Lustre: 178760:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 4545.729995] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4549.679475] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4549.927819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6763 to 0x240000400:6785)
[ 4549.928495] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6763 to 0x280000400:6785)
[ 4552.587486] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4580.442064] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:24:21 (1782635061)
[ 4610.009719] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:24:50 (1782635090)
[ 4640.854085] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:25:21 (1782635121)
[ 4641.306037] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 4641.811467] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:25:22 (1782635122)
[ 4642.265196] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 4642.755502] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:25:23 (1782635123)
[ 4692.580096] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:26:13 (1782635173)
[ 4718.630705] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:26:39 (1782635199)
[ 4763.880375] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:27:24 (1782635244)
[ 4770.689184] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4770.690283] Lustre: Skipped 1 previous similar message
[ 4771.697472] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4771.698605] Lustre: Skipped 231 previous similar messages
[ 4773.697213] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4773.698457] Lustre: Skipped 407 previous similar messages
[ 4777.697670] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4777.698745] Lustre: Skipped 812 previous similar messages
[ 4785.700578] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4785.701772] Lustre: Skipped 1728 previous similar messages
[ 4801.710819] Lustre: *** cfs_fail_loc=a09, val=0***
[ 4801.712821] Lustre: Skipped 3589 previous similar messages
[ 4939.233099] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4939.233796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 4939.238409] Lustre: Skipped 3 previous similar messages
[ 4942.177855] Lustre: server umount lustre-MDT0000 complete
[ 4943.514347] LustreError: 184715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635424 with bad export cookie 7591067977033739766
[ 4943.519177] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4943.651173] Lustre: server umount lustre-OST0000 complete
[ 4945.059731] Lustre: server umount lustre-OST0001 complete
[ 4947.179301] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid
[ 4949.456781] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 4952.432692]  vdc: vdc1 vdc9
[ 4955.329878]  vde: vde1 vde9
[ 4958.441096]  vdf: vdf1 vdf9
[ 4963.141342] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local
[ 4965.883479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 4965.952051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 4965.993983] Lustre: lustre-MDT0000: new disk, initializing
[ 4966.070508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4966.072271] Lustre: Skipped 2 previous similar messages
[ 4966.086789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 4967.287854] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4969.466759] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 4971.118773] Lustre: lustre-OST0000: new disk, initializing
[ 4971.120911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 4971.123454] Lustre: Skipped 1 previous similar message
[ 4972.227553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[ 4972.231903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[ 4972.258966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[ 4972.988427] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4976.557122] Lustre: lustre-OST0001: new disk, initializing
[ 4976.558476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 4978.450238] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 4978.559841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[ 4978.563563] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[ 4978.589021] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[ 4982.675747] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 4983.834168] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 5003.025766] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:31:23 (1782635483)
[ 5004.905057] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:31:25 (1782635485)
[ 5033.507480] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:31:54 (1782635514)
[ 5069.419753] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:32:30 (1782635550)
[ 5082.169829] Lustre: DEBUG MARKER: rename directory return 255
[ 5105.957870] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 04:33:06 (1782635586)
[ 5123.004986] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 04:33:23 (1782635603)
[ 5141.623176] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 04:33:42 (1782635622)
[ 5174.741364] LustreError: 206109:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 5197.486929] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 04:34:38 (1782635678)
[ 5217.000562] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 04:34:57 (1782635697)
[ 5245.587315] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 04:35:26 (1782635726)
[ 5336.256322] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 04:36:56 (1782635816)
[ 5336.768354] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[ 5337.291566] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 04:36:58 (1782635818)
[ 5370.901476] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 5371.429310] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 04:37:32 (1782635852)
[ 5389.243056] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 5389.805096] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 04:37:50 (1782635870)
[ 5423.203964] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 5423.747117] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 04:38:24 (1782635904)
[ 5453.116813] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 04:38:53 (1782635933)
[ 5453.583838] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[ 5454.133574] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 04:38:54 (1782635934)
[ 5467.191420] LustreError: 219571:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 5491.215577] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 04:39:31 (1782635971)
[ 5504.609244] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 5505.149675] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 5561.147277] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:40:41 (1782636041)
[ 5590.798178] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:41:11 (1782636071)
[ 5606.645612] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:41:27 (1782636087)
[ 5607.133803] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[ 5607.674376] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:41:28 (1782636088)
[ 5608.173600] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[ 5608.676038] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:41:29 (1782636089)
[ 5617.334075] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[ 5625.020781] Lustre: DEBUG MARKER: Write...
[ 5625.853886] Lustre: DEBUG MARKER: Write out of block quota ...
[ 5663.953908] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:42:24 (1782636144)
[ 5678.745472] Lustre: DEBUG MARKER: set to use default quota
[ 5679.242139] Lustre: DEBUG MARKER: set default quota
[ 5679.712711] Lustre: DEBUG MARKER: get default quota
[ 5681.499941] Lustre: DEBUG MARKER: Test not out of quota
[ 5682.672960] Lustre: DEBUG MARKER: Test out of quota
[ 5686.118953] Lustre: DEBUG MARKER: Increase default quota
[ 5709.387297] Lustre: DEBUG MARKER: Set quota to override default quota
[ 5709.413448] LustreError: 195765:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1783240990 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 5712.893427] Lustre: DEBUG MARKER: Set to use default quota again
[ 5722.555466] Lustre: DEBUG MARKER: Cleanup
[ 5769.981636] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 04:44:10 (1782636250)
[ 5783.644264] Lustre: DEBUG MARKER: set default quota for qpool1
[ 5784.139509] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[ 5812.288460] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 04:44:53 (1782636293)
[ 5856.554566] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 04:45:36 (1782636336)
[ 5877.168124] hrtimer: interrupt took 11111890 ns
[ 5921.687979] Lustre: DEBUG MARKER: Write...
[ 5923.488597] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6021.590586] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 04:48:20 (1782636500)
[ 6071.937206] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 04:49:11 (1782636551)
[ 6079.570750] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 04:49:19 (1782636559)
[ 6081.257653] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[ 6083.009580] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 04:49:22 (1782636562)
[ 6084.623464] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[ 6086.235720] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 04:49:26 (1782636566)
[ 6104.281341] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 04:49:44 (1782636584)
[ 6105.583406] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[ 6107.114811] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 04:49:47 (1782636587)
[ 6122.148293] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 6127.028468] LustreError: 245347:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[ 6130.658392] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6130.659295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6130.659300] Lustre: Skipped 1 previous similar message
[ 6130.682703] Lustre: Skipped 1 previous similar message
[ 6134.164064] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping)
[ 6134.171658] Lustre: Skipped 1 previous similar message
[ 6135.784855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6135.789329] Lustre: Skipped 1 previous similar message
[ 6137.064128] LustreError: 245347:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[ 6137.684783] Lustre: server umount lustre-MDT0000 complete
[ 6144.791234] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6145.383413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6145.390588] Lustre: Skipped 2 previous similar messages
[ 6145.552795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[ 6145.576164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[ 6150.193316] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6160.896190] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 6160.904979] Lustre: Skipped 3 previous similar messages
[ 6182.724720] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 04:51:02 (1782636662)
[ 6214.242995] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 04:51:33 (1782636693)
[ 6244.840980] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 04:52:04 (1782636724)
[ 6295.276617] Lustre: *** cfs_fail_loc=a08, val=0***
[ 6295.284384] Lustre: Skipped 1227 previous similar messages
[ 6295.302942] Lustre: *** cfs_fail_loc=a08, val=0***
[ 6412.969474] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 04:54:52 (1782636892)
[ 6495.063343] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 04:56:14 (1782636974)
[ 6574.560871] LustreError: 248395:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1783241855 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 6732.081332] LustreError: 245878:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1783242013 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 6740.183819] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed