[ 5571.183324] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5578.965502] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5582.534391] LustreError: 120977:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5583.608489] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5585.749891] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5588.284510] Lustre: Failing over lustre-MDT0000
[ 5588.741015] Lustre: server umount lustre-MDT0000 complete
[ 5606.373718] Lustre: 3316:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787556906/real 1787556906]  req@ffff96368c38e680 x1874383516985472/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787556922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5606.421044] Lustre: 3316:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5606.427626] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5606.436971] Lustre: Skipped 1 previous similar message
[ 5607.395187] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5612.511374] Lustre: 3313:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787556912/real 1787556912]  req@ffff9636c87afb80 x1874383516986240/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787556928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5612.548455] Lustre: 3313:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5617.759116] Lustre: 3316:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787556917/real 1787556917]  req@ffff9636cd13b800 x1874383516986496/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787556933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5617.777780] Lustre: 3316:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5617.943683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5618.023857] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5618.487518] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5618.727817] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5618.782813] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:624 to 0x280000400:641)
[ 5618.783209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:628 to 0x240000400:673)
[ 5623.371883] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5632.162896] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5632.178743] Lustre: Skipped 1 previous similar message
[ 5632.963393] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5634.546735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5650.242492] Lustre: DEBUG MARKER: (dd_pid=114363, time=9, timeout=600)
[ 5698.728762] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5705.223614] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5708.388702] LustreError: 123868:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5709.219388] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5710.746234] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5712.941444] Lustre: Failing over lustre-MDT0000
[ 5713.249276] Lustre: server umount lustre-MDT0000 complete
[ 5729.759101] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787557029/real 1787557029]  req@ffff9637b786c000 x1874383517027072/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787557045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5729.796624] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5729.805201] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5729.826187] Lustre: Skipped 1 previous similar message
[ 5731.873462] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5742.972088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5743.039876] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5746.127637] Lustre: 3314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787557045/real 1787557045]  req@ffff9636cd139180 x1874383517028480/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787557061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5746.186885] Lustre: 3314:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5748.467269] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5755.041687] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5755.051314] Lustre: Skipped 1 previous similar message
[ 5756.711972] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5756.795485] LustreError: 124521:0:(tgt_handler.c:534:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffff9636d07d3800 x1874383517042432/t0(0) o601->lustre-MDT0000-lwp-OST0000_UUID@0@lo:208/0 lens 336/0 e 0 to 0 dl 1787557083 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 5756.808630] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -11
[ 5756.880593] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5756.930805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5756.940720] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:624 to 0x280000400:673)
[ 5762.192266] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5763.769924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5773.348890] Lustre: DEBUG MARKER: (dd_pid=116779, time=4, timeout=600)
[ 5842.537696] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 03:39:16 (1787557156)
[ 5866.502310] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5868.107272] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5874.565352] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5876.309785] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5915.310166] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 03:40:29 (1787557229)
[ 5942.713360] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5959.341996] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5961.446160] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5963.335127] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5966.118812] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5970.527581] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5975.256250] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5979.681953] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5983.721941] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5987.719731] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5991.549470] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6062.634875] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 03:42:57 (1787557377)
[ 6074.849601] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6074.856665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6074.866212] Lustre: Skipped 2 previous similar messages
[ 6074.902309] Lustre: Skipped 1 previous similar message
[ 6076.278728] Lustre: server umount lustre-MDT0000 complete
[ 6080.601239] LustreError: 31603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557396 with bad export cookie 995929096743035012
[ 6080.605178] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6080.608575] LustreError: 31603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 6080.686851] Lustre: server umount lustre-OST0000 complete
[ 6085.000378] Lustre: server umount lustre-OST0001 complete
[ 6100.929942] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local
[ 6111.846230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6117.174404] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6120.515657] Lustre: 133665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6127.933369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6135.414777] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6137.062734] LustreError: 134051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6137.092299] LustreError: 134051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 6137.139497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737)
[ 6142.440639] LustreError: 134500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6145.465822] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6150.663187] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:705)
[ 6152.087234] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6159.706205] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6164.202991] Lustre: 135606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6184.911252] Lustre: server umount lustre-MDT0000 complete
[ 6190.034457] LustreError: 133214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557505 with bad export cookie 995929096743041774
[ 6190.046849] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6200.390541] Lustre: server umount lustre-OST0000 complete
[ 6202.655253] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787557502/real 1787557502]  req@ffff963785d1b100 x1874383517191424/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787557518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6204.709479] Lustre: server umount lustre-OST0001 complete
[ 6220.983529] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local
[ 6232.360543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6237.431545] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6240.713609] Lustre: 138259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6255.481066] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6257.128558] LustreError: 138635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6257.146094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769)
[ 6262.240152] LustreError: 138634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6264.893440] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6264.897892] Lustre: Skipped 1 previous similar message
[ 6269.932610] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:737)
[ 6272.017886] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6281.271241] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6286.239973] Lustre: 140180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6301.156291] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 03:46:55 (1787557615)
[ 6302.645199] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6304.405650] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 03:46:58 (1787557618)
[ 6363.007208] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 03:47:57 (1787557677)
[ 6420.080967] Lustre: DEBUG MARKER: Write...
[ 6423.300250] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6483.711581] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 03:49:58 (1787557798)
[ 6490.511067] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 03:50:04 (1787557804)
[ 6500.308225] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 03:50:14 (1787557814)
[ 6510.612502] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 03:50:24 (1787557824)
[ 6518.633548] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 03:50:33 (1787557833)
[ 6587.007743] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6738.758375] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6927.995628] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 03:57:22 (1787558242)
[ 6990.646742] Lustre: DEBUG MARKER: Restart...
[ 6999.005420] Lustre: server umount lustre-MDT0000 complete
[ 7005.401128] LustreError: 145197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558321 with bad export cookie 995929096743043307
[ 7005.418038] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7016.138645] Lustre: server umount lustre-OST0000 complete
[ 7017.440248] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787558317/real 1787558317]  req@ffff9637bb7f1880 x1874383517422080/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787558333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7017.505091] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7017.529813] Lustre: Skipped 1 previous similar message
[ 7021.321287] LustreError: 3314:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff963790099500 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2377 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7021.495477] Lustre: server umount lustre-OST0001 complete
[ 7035.518737] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local
[ 7045.649411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7050.528819] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7053.281536] Lustre: 158698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7059.724361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7065.942793] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7070.945984] LustreError: 159078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7070.971831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:778 to 0x240000400:801)
[ 7074.557637] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7079.918979] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:746 to 0x280000400:769)
[ 7080.375896] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7088.325841] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7092.104802] Lustre: 160621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7159.791422] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:01:14 (1787558474)
[ 7236.956143] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:02:30 (1787558550)
[ 8996.759360] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 04:31:50 (1787560310)
[ 9015.095151] Lustre: server umount lustre-MDT0000 complete
[ 9019.227992] LustreError: 158251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560334 with bad export cookie 995929096743049803
[ 9019.248156] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9028.692530] Lustre: server umount lustre-OST0000 complete
[ 9030.496371] Lustre: 3315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787560330/real 1787560330]  req@ffff9636c5c8fb80 x1874383520384384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787560346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9030.519981] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9033.484713] Lustre: server umount lustre-OST0001 complete
[ 9051.802272] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local
[ 9060.945509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9065.118547] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9067.965824] Lustre: 169102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9074.298388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9079.928165] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9085.551318] LustreError: 169462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9085.590525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5804 to 0x240000400:5825)
[ 9089.402586] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9094.663960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5770 to 0x280000400:5793)
[ 9097.501947] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9106.248636] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9110.367196] Lustre: 171018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9146.852696] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 04:34:21 (1787560461)
[ 9194.032444] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 04:35:08 (1787560508)
[ 9238.666172] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 04:35:53 (1787560553)
[ 9240.419344] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9241.983850] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 04:35:56 (1787560556)
[ 9243.592534] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9245.734690] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 04:35:59 (1787560559)
[ 9338.281472] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 04:37:32 (1787560652)
[ 9392.235106] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 04:38:25 (1787560705)
[ 9473.702873] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 04:39:47 (1787560787)
[ 9486.216581] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9486.219444] Lustre: Skipped 4 previous similar messages
[ 9488.241412] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9488.243566] Lustre: Skipped 95 previous similar messages
[ 9492.279535] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9492.283884] Lustre: Skipped 183 previous similar messages
[ 9500.361425] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9500.367624] Lustre: Skipped 307 previous similar messages
[ 9516.412122] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9516.416876] Lustre: Skipped 681 previous similar messages
[ 9548.432052] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9548.435719] Lustre: Skipped 1465 previous similar messages
[ 9612.467111] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9612.472963] Lustre: Skipped 2955 previous similar messages
[10027.363652] Lustre: *** cfs_fail_loc=a08, val=0***
[10027.366795] Lustre: *** cfs_fail_loc=a08, val=0***
[10027.371452] Lustre: Skipped 2315 previous similar messages
[10027.383317] Lustre: Skipped 3 previous similar messages
[10287.584641] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10287.585389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10287.603584] Lustre: Skipped 1 previous similar message
[10287.616190] Lustre: Skipped 1 previous similar message
[10288.347843] Lustre: server umount lustre-MDT0000 complete
[10292.619277] LustreError: 182575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561608 with bad export cookie 995929096744827425
[10292.633989] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10293.075733] Lustre: server umount lustre-OST0000 complete
[10298.288829] Lustre: server umount lustre-OST0001 complete
[10306.769385] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid
[10315.908471] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local
[10329.813419]  vdc: vdc1 vdc9
[10342.580741]  vde: vde1 vde9
[10354.368792]  vdf: vdf1 vdf9
[10375.201211] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local
[10387.119567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10387.501740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10387.609478] Lustre: lustre-MDT0000: new disk, initializing
[10388.292055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10388.503367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10393.643385] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10398.521935] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10404.544383] Lustre: lustre-OST0000: new disk, initializing
[10404.547171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10404.551651] Lustre: Skipped 1 previous similar message
[10404.647975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10406.601882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10406.620743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10406.864546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10411.635076] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10422.361197] Lustre: lustre-OST0001: new disk, initializing
[10422.364273] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10422.442747] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10423.951601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10423.974712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10424.195916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10430.346896] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10441.083158] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10445.037211] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10485.068276] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 04:56:39 (1787561799)
[10491.761338] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 04:56:45 (1787561805)
[10544.314892] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 04:57:38 (1787561858)
[10611.686853] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 04:58:46 (1787561926)
[10643.203943] Lustre: DEBUG MARKER: rename directory return 255
[10694.609938] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 05:00:08 (1787562008)
[10726.813490] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 05:00:41 (1787562041)
[10773.911641] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 05:01:28 (1787562088)
[10921.849757] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 05:03:56 (1787562236)
[10960.159746] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 05:04:34 (1787562274)
[11011.487297] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 05:05:25 (1787562325)
[11204.159062] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 05:08:38 (1787562518)
[11205.671948] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11207.598188] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 05:08:41 (1787562521)
[11273.735718] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11275.773943] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 05:09:49 (1787562589)
[11308.107555] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11310.077077] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 05:10:24 (1787562624)
[11367.552376] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11369.761486] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 05:11:23 (1787562683)
[11418.489987] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 05:12:12 (1787562732)
[11419.985339] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11422.390750] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 05:12:16 (1787562736)
[11453.883671] LustreError: 209444:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11500.355484] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 05:13:34 (1787562814)
[11527.032177] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11528.759768] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11727.372830] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 05:17:21 (1787563041)
[11779.805637] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 05:18:14 (1787563094)
[11812.435990] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 05:18:46 (1787563126)
[11813.954773] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11815.446247] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 05:18:49 (1787563129)
[11817.138798] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11818.842532] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 05:18:53 (1787563133)
[11835.277322] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11853.268372] Lustre: DEBUG MARKER: Write...
[11856.183801] Lustre: DEBUG MARKER: Write out of block quota ...
[11913.760234] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 05:20:28 (1787563228)
[11942.760285] Lustre: DEBUG MARKER: set to use default quota
[11944.749489] Lustre: DEBUG MARKER: set default quota
[11946.528931] Lustre: DEBUG MARKER: get default quota
[11952.595284] Lustre: DEBUG MARKER: Test not out of quota
[11957.448055] Lustre: DEBUG MARKER: Test out of quota
[11972.146165] Lustre: DEBUG MARKER: Increase default quota
[12005.702902] Lustre: DEBUG MARKER: Set quota to override default quota
[12005.773329] LustreError: 186087:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45064 time:1788168121 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12018.101369] Lustre: DEBUG MARKER: Set to use default quota again
[12038.606631] Lustre: DEBUG MARKER: Cleanup
[12123.283144] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 05:23:57 (1787563437)
[12148.807083] Lustre: DEBUG MARKER: set default quota for qpool1
[12150.309506] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12201.871497] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 05:25:15 (1787563515)
[12289.236328] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 05:26:43 (1787563603)
[12371.112832] Lustre: DEBUG MARKER: Write...
[12374.706045] Lustre: DEBUG MARKER: Write out of block quota ...
[12498.395771] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 05:30:12 (1787563812)
[12545.186638] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 05:30:59 (1787563859)
[12553.165524] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 05:31:07 (1787563867)
[12554.601980] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12556.098199] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 05:31:10 (1787563870)
[12557.690763] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12559.365275] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 05:31:13 (1787563873)
[12578.123575] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 05:31:32 (1787563892)
[12579.710823] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12581.467693] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 05:31:35 (1787563895)
[12599.151451] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12606.268125] LustreError: 234429:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12610.018078] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12610.040080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12610.053262] Lustre: Skipped 1 previous similar message
[12612.423407] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.33@tcp (stopping)
[12615.143201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12615.154318] Lustre: Skipped 1 previous similar message
[12616.271152] LustreError: 234429:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12616.550842] Lustre: server umount lustre-MDT0000 complete
[12625.500802] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12626.175233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12626.283106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12626.294798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12631.425327] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12639.726829] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12639.736651] Lustre: Skipped 1 previous similar message
[12668.556157] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 05:33:02 (1787563982)
[12703.645907] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 05:33:37 (1787564017)
[12741.294213] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 05:34:15 (1787564055)
[12799.758443] Lustre: *** cfs_fail_loc=a08, val=0***
[12799.774113] Lustre: Skipped 7 previous similar messages
[12799.802622] Lustre: *** cfs_fail_loc=a08, val=0***
[12799.806420] Lustre: Skipped 8 previous similar messages
[12916.227751] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 05:37:10 (1787564230)
[13002.339461] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 05:38:36 (1787564316)
[13086.901068] LustreError: 235012:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788169202 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13252.359181] LustreError: 243454:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788169368 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13414.707964] LustreError: 243454:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1788169530 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13423.838768] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed