[ 5815.433850] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5822.103550] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5825.366893] LustreError: 125538:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5826.518358] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5828.281763] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5830.502535] Lustre: Failing over lustre-MDT0000
[ 5830.692046] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5830.708582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5830.724307] Lustre: Skipped 1 previous similar message
[ 5831.035992] Lustre: server umount lustre-MDT0000 complete
[ 5850.273448] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5851.024203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5851.170651] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5851.943315] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5852.924333] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5852.998237] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5853.034987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:628 to 0x240000400:673)
[ 5853.037325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:625 to 0x280000400:641)
[ 5855.623543] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5863.704829] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5865.083448] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5869.584120] Lustre: DEBUG MARKER: (dd_pid=117955, time=0, timeout=600)
[ 5927.905938] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5933.495751] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5936.330178] LustreError: 128371:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5937.094246] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5938.546417] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5940.471510] Lustre: Failing over lustre-MDT0000
[ 5940.956814] Lustre: server umount lustre-MDT0000 complete
[ 5957.536117] Lustre: 3317:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641877/real 1788641877]  req@ffff979187486d80 x1875520845527168/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1788641893 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5957.563669] Lustre: 3317:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5957.577389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5957.592702] Lustre: Skipped 1 previous similar message
[ 5958.203214] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5958.560109] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (not set up)
[ 5958.721460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5958.774613] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5959.736852] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5959.779659] Lustre: 3316:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641879/real 1788641879]  req@ffff9791cc064e00 x1875520845527424/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788641895 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5959.909828] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5959.973349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5959.980583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:625 to 0x280000400:673)
[ 5962.740615] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5962.747976] Lustre: Skipped 1 previous similar message
[ 5963.087930] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5965.792839] Lustre: 3316:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641885/real 1788641885]  req@ffff979288830380 x1875520845527936/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788641901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5965.825714] Lustre: 3316:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5970.912177] Lustre: 3318:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641890/real 1788641890]  req@ffff9791cc067800 x1875520845528192/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788641906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5970.960359] Lustre: 3318:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5971.966710] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5973.600768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5983.681226] Lustre: DEBUG MARKER: (dd_pid=120312, time=4, timeout=600)
[ 6044.224537] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 16:59:38 (1788641978)
[ 6064.652684] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6066.464346] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6072.684422] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6074.156874] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6110.863229] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 17:00:44 (1788642044)
[ 6137.630894] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6153.475629] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6155.222274] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6157.124861] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6158.872905] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6162.466797] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6165.792044] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6168.877157] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6172.070372] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6175.195932] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6178.412495] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6181.857517] Lustre: DEBUG MARKER: Set quota for 8 times
[ 6185.384647] Lustre: DEBUG MARKER: Set quota for 9 times
[ 6228.933750] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 17:02:43 (1788642163)
[ 6242.961948] Lustre: server umount lustre-MDT0000 complete
[ 6247.428322] LustreError: 6714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642183 with bad export cookie 5785403843361465281
[ 6247.443530] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6257.714077] Lustre: server umount lustre-OST0000 complete
[ 6259.681563] Lustre: 3317:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788642179/real 1788642179]  req@ffff97928593d880 x1875520845647616/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788642195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6259.730511] Lustre: 3317:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6259.741433] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6259.755340] Lustre: Skipped 1 previous similar message
[ 6262.590194] Lustre: server umount lustre-OST0001 complete
[ 6277.872218] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local
[ 6287.869927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6292.300288] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6295.270994] Lustre: 138188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6301.621544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6307.953081] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6312.865178] LustreError: 138557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6312.915919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737)
[ 6316.734658] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6322.201562] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:705)
[ 6323.456236] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6330.469528] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6333.658797] Lustre: 140104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6351.798665] Lustre: server umount lustre-MDT0000 complete
[ 6356.771836] LustreError: 140106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642292 with bad export cookie 5785403843361476453
[ 6356.788715] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6367.294558] Lustre: server umount lustre-OST0000 complete
[ 6369.184419] Lustre: 3318:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788642288/real 1788642288]  req@ffff9791cb7b7480 x1875520845689344/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788642304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6372.153788] Lustre: server umount lustre-OST0001 complete
[ 6388.092317] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local
[ 6397.513835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6402.298451] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6405.111399] Lustre: 142760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6417.988197] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6421.730392] LustreError: 143139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6421.747990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769)
[ 6425.902375] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6425.906504] Lustre: Skipped 1 previous similar message
[ 6431.267126] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:737)
[ 6432.472634] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6441.001483] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6445.278567] Lustre: 144679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6456.072343] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 17:06:29 (1788642389)
[ 6457.436336] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6459.837494] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 17:06:33 (1788642393)
[ 6515.441154] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 17:07:29 (1788642449)
[ 6565.206602] Lustre: DEBUG MARKER: Write...
[ 6567.724761] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6619.832802] LustreError: 3316:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 6619.853412] LustreError: 3316:0:(qsd_reint.c:633:qqi_reint_delayed()) Skipped 2 previous similar messages
[ 6623.431650] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 17:09:17 (1788642557)
[ 6630.654930] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 17:09:24 (1788642564)
[ 6641.024410] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 17:09:34 (1788642574)
[ 6650.341410] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 17:09:44 (1788642584)
[ 6658.711103] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 17:09:52 (1788642592)
[ 6729.137431] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6880.090721] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7073.391681] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 17:16:47 (1788643007)
[ 7140.825454] Lustre: DEBUG MARKER: Restart...
[ 7148.058962] Lustre: server umount lustre-MDT0000 complete
[ 7152.434273] LustreError: 144681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643088 with bad export cookie 5785403843361477993
[ 7152.446771] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7163.038614] Lustre: server umount lustre-OST0000 complete
[ 7164.335088] Lustre: 3315:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788643083/real 1788643083]  req@ffff9792bbaa6300 x1875520845916032/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788643099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7164.364087] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7164.378733] Lustre: Skipped 1 previous similar message
[ 7167.995649] LustreError: 3316:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff979284cefc80 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2377 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7168.307340] Lustre: server umount lustre-OST0001 complete
[ 7187.166078] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local
[ 7198.676381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7203.436482] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7206.888049] Lustre: 163253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7214.097887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7220.823934] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7224.299549] LustreError: 163620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7224.341954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:779 to 0x240000400:801)
[ 7229.240179] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7234.560126] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:745 to 0x280000400:769)
[ 7235.883969] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7244.220428] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7247.914676] Lustre: 165159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7313.769948] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 17:20:47 (1788643247)
[ 7389.167549] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 17:22:03 (1788643323)
[ 9250.966082] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 17:53:05 (1788645185)
[ 9266.351650] Lustre: server umount lustre-MDT0000 complete
[ 9270.707525] LustreError: 162808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645206 with bad export cookie 5785403843361484489
[ 9270.718878] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9281.166066] Lustre: server umount lustre-OST0000 complete
[ 9283.044339] Lustre: 3315:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788645202/real 1788645202]  req@ffff9791c5716300 x1875520848914048/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788645218 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 9283.083145] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9286.583042] Lustre: server umount lustre-OST0001 complete
[ 9302.232566] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local
[ 9313.837476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9319.270282] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9322.197219] Lustre: 173603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9329.839228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9336.947986] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9338.989893] LustreError: 173970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9339.052135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5804 to 0x240000400:5825)
[ 9344.494165] LustreError: 174436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9348.117891] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9353.234961] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5770 to 0x280000400:5793)
[ 9355.782660] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9364.076463] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9367.457724] Lustre: 175524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9400.297268] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 17:55:34 (1788645334)
[ 9449.063502] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 17:56:22 (1788645382)
[ 9494.668605] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 17:57:08 (1788645428)
[ 9496.738969] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9499.006593] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 17:57:12 (1788645432)
[ 9500.777361] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9502.609559] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 17:57:16 (1788645436)
[ 9600.819504] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 17:58:54 (1788645534)
[ 9655.245892] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 17:59:49 (1788645589)
[ 9743.265888] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 18:01:17 (1788645677)
[ 9755.874791] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9755.878807] Lustre: Skipped 1 previous similar message
[ 9757.878632] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9757.884612] Lustre: Skipped 60 previous similar messages
[ 9761.965498] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9761.973850] Lustre: Skipped 128 previous similar messages
[ 9769.988605] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9770.003621] Lustre: Skipped 277 previous similar messages
[ 9786.043558] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9786.045359] Lustre: Skipped 715 previous similar messages
[ 9818.068064] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9818.071534] Lustre: Skipped 1387 previous similar messages
[ 9882.095657] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9882.100154] Lustre: Skipped 2769 previous similar messages
[10285.735969] Lustre: *** cfs_fail_loc=a08, val=0***
[10285.738605] Lustre: Skipped 2657 previous similar messages
[10285.773144] Lustre: *** cfs_fail_loc=a08, val=0***
[10576.611148] Lustre: *** cfs_fail_loc=a08, val=0***
[10576.629727] Lustre: Skipped 14 previous similar messages
[10576.639810] Lustre: *** cfs_fail_loc=a08, val=0***
[10576.648886] Lustre: Skipped 15 previous similar messages
[10581.738253] Lustre: *** cfs_fail_loc=a08, val=0***
[10581.746870] Lustre: Skipped 47 previous similar messages
[10586.887313] Lustre: *** cfs_fail_loc=a08, val=0***
[10586.898070] Lustre: Skipped 242 previous similar messages
[10591.995846] Lustre: *** cfs_fail_loc=a08, val=0***
[10591.998499] Lustre: Skipped 278 previous similar messages
[10610.658447] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10610.674253] Lustre: Skipped 1 previous similar message
[10610.678366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10610.685105] Lustre: Skipped 1 previous similar message
[10615.494491] Lustre: server umount lustre-MDT0000 complete
[10619.713764] LustreError: 173159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646555 with bad export cookie 5785403843363266024
[10619.732482] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10620.162090] Lustre: server umount lustre-OST0000 complete
[10625.405624] Lustre: server umount lustre-OST0001 complete
[10634.691970] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid
[10644.549425] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local
[10653.861317]  vdc: vdc1 vdc9
[10663.648465]  vde: vde1 vde9
[10673.642213]  vdf: vdf1 vdf9
[10689.306684] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local
[10699.645672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10699.923802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10700.033575] Lustre: lustre-MDT0000: new disk, initializing
[10700.561673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10700.614397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10705.523388] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10710.119483] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10716.805935] Lustre: lustre-OST0000: new disk, initializing
[10716.808949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10716.816618] Lustre: Skipped 1 previous similar message
[10716.903928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10718.154666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10718.163941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10718.386472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10724.423173] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10734.978853] Lustre: lustre-OST0001: new disk, initializing
[10734.982925] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10735.174820] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10736.107059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10736.115319] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10736.248268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10741.562100] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10750.608390] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10753.944508] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10788.361492] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 18:18:42 (1788646722)
[10794.759437] Lustre: DEBUG MARKER: == sanity-quota test 49c: lfs quota long options don't consume an extra argument ========================================================== 18:18:48 (1788646728)
[10801.425824] Lustre: DEBUG MARKER: == sanity-quota test 49d: lfs quota -d and --delimiter both work ========================================================== 18:18:55 (1788646735)
[10807.886482] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 18:19:02 (1788646742)
[10853.002547] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 18:19:47 (1788646787)
[10910.001184] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 18:20:44 (1788646844)
[10939.318996] Lustre: DEBUG MARKER: rename directory return 255
[10985.538209] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 18:21:59 (1788646919)
[11014.225225] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 18:22:28 (1788646948)
[11053.331664] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 18:23:07 (1788646987)
[11144.703453] LustreError: 190440:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11181.899469] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 18:25:15 (1788647115)
[11216.949369] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 18:25:51 (1788647151)
[11263.754288] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 18:26:37 (1788647197)
[11450.133625] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 18:29:44 (1788647384)
[11451.590898] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11453.065837] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 18:29:47 (1788647387)
[11517.226183] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11519.209395] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 18:30:52 (1788647452)
[11552.240912] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11553.857640] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 18:31:27 (1788647487)
[11610.841324] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11612.448856] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 18:32:26 (1788647546)
[11661.555391] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 18:33:15 (1788647595)
[11663.117423] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11664.605555] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 18:33:18 (1788647598)
[11695.092757] LustreError: 214080:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11738.675375] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 18:34:32 (1788647672)
[11767.196543] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11768.906464] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11977.246320] LustreError: 3317:0:(qsd_reint.c:633:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[11977.275854] LustreError: 3317:0:(qsd_reint.c:633:qqi_reint_delayed()) Skipped 1 previous similar message
[11980.680310] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 18:38:34 (1788647914)
[12036.903634] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 18:39:30 (1788647970)
[12074.778580] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 18:40:08 (1788648008)
[12076.457952] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12078.803334] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 18:40:12 (1788648012)
[12080.265818] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12082.343736] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 18:40:16 (1788648016)
[12097.672654] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12114.245585] Lustre: DEBUG MARKER: Write...
[12118.118036] Lustre: DEBUG MARKER: Write out of block quota ...
[12181.420950] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 18:41:55 (1788648115)
[12210.452586] Lustre: DEBUG MARKER: set to use default quota
[12212.459420] Lustre: DEBUG MARKER: set default quota
[12214.377500] Lustre: DEBUG MARKER: get default quota
[12222.236695] Lustre: DEBUG MARKER: Test not out of quota
[12228.109449] Lustre: DEBUG MARKER: Test out of quota
[12241.008390] Lustre: DEBUG MARKER: Increase default quota
[12270.213703] Lustre: DEBUG MARKER: Set quota to override default quota
[12270.259226] LustreError: 190439:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1789253005 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12282.679909] Lustre: DEBUG MARKER: Set to use default quota again
[12309.037372] Lustre: DEBUG MARKER: Cleanup
[12381.248916] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 18:45:15 (1788648315)
[12404.783803] Lustre: DEBUG MARKER: set default quota for qpool1
[12406.210635] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12457.922828] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 18:46:31 (1788648391)
[12532.382649] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 18:47:46 (1788648466)
[12602.172616] Lustre: DEBUG MARKER: Write...
[12604.830964] Lustre: DEBUG MARKER: Write out of block quota ...
[12728.650044] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 18:51:02 (1788648662)
[12776.236904] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 18:51:50 (1788648710)
[12783.748513] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 18:51:58 (1788648718)
[12786.488604] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A fallocate not supported
[12788.768574] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 18:52:02 (1788648722)
[12791.701593] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a fallocate not supported
[12793.885957] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 18:52:07 (1788648727)
[12814.456292] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 18:52:28 (1788648748)
[12815.893672] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12817.654730] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 18:52:31 (1788648751)
[12835.539942] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12841.581465] LustreError: 239392:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12845.567650] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12845.623938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12845.633849] Lustre: Skipped 1 previous similar message
[12849.020521] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping)
[12850.661394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12850.669475] Lustre: Skipped 1 previous similar message
[12851.640578] LustreError: 239392:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12852.079934] Lustre: server umount lustre-MDT0000 complete
[12864.657591] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12865.462782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12865.713170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12865.718902] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12871.049329] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12875.749510] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12875.755441] Lustre: Skipped 1 previous similar message
[12904.354978] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 18:53:58 (1788648838)
[12941.680814] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 18:54:35 (1788648875)
[12971.467356] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 18:55:05 (1788648905)
[13020.964820] Lustre: *** cfs_fail_loc=a08, val=0***
[13020.971312] Lustre: Skipped 883 previous similar messages
[13020.977128] Lustre: *** cfs_fail_loc=a08, val=0***
[13020.979140] Lustre: Skipped 313 previous similar messages
[13143.421651] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 18:57:56 (1788649076)
[13229.907869] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 18:59:24 (1788649164)
[13313.297462] LustreError: 244890:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1789254048 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13321.792812] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed