How to properly configure a Service Account in Google AppEngine Flex w/ Python?
How to properly configure a Service Account in Google AppEngine Flex w/ Python?
I am using Google AppEngine (GAE) Flexible environment with a custom runtime (python2.7). I am using custom so I can just manage a Dockerfile and have easy parity between my laptop and when deployed in GAE.
I have a python 2.7 Flask app that is doing a query against DataStore. I get the following (when starting python main.py in my local docker):
python main.py
root@24dcf9b8712d:/home/vmagent/app# curl localhost:5000/things
'/home/vmagent/app/clientid-searchapp.json'
[2018-09-04 14:54:19,969] ERROR in app: Exception on /things [GET]
Traceback (most recent call last):
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "main.py", line 55, in products
for rec in query.fetch(limit=100):
File "/env/lib/python2.7/site-packages/google/api_core/page_iterator.py", line 199, in _items_iter
for page in self._page_iter(increment=False):
File "/env/lib/python2.7/site-packages/google/api_core/page_iterator.py", line 230, in _page_iter
page = self._next_page()
File "/env/local/lib/python2.7/site-packages/google/cloud/datastore/query.py", line 517, in _next_page
query=query_pb,
File "/env/local/lib/python2.7/site-packages/google/cloud/datastore_v1/gapic/datastore_client.py", line 294, in run_query
request, retry=retry, timeout=timeout, metadata=metadata)
File "/env/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
return wrapped_func(*args, **kwargs)
File "/env/lib/python2.7/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
on_error=on_error,
File "/env/lib/python2.7/site-packages/google/api_core/retry.py", line 177, in retry_target
return target()
File "/env/lib/python2.7/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
return func(*args, **kwargs)
File "/env/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "/env/local/lib/python2.7/site-packages/six.py", line 737, in raise_from
raise value
PermissionDenied: 403 Missing or insufficient permissions.
127.0.0.1 - - [04/Sep/2018 14:54:19] "GET /things HTTP/1.1" 500 -
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
Snippets from main.py
main.py
app = Flask(__name__)
ds = datastore.Client()
<snip>
@app.route("/things")
def things():
global ds
pprint(os.environ['GOOGLE_APPLICATION_CREDENTIALS'])
query = ds.query(kind='Things', order=('thing_name',))
results = list()
for rec in query.fetch(limit=100):
results.append(rec)
return jsonify(results)
Output of gcloud auth list:
gcloud auth list
root@24dcf9b8712d:/home/vmagent/app# gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* <myapp>@<project-name>.iam.gserviceaccount.com
This is the configured service account. I have over-privileged it with excessive Role membership; I'm pretty certain the service account isn't lacking in permissions. In fact, it has 'DataStore Owner'
In main.py I am outputting:
pprint(os.environ['GOOGLE_APPLICATION_CREDENTIALS'])
main.py
And this outputs the path to my client-<myapp>.json file for the service account. It is the same file supplied in the Dockerfile as:
client-<myapp>.json
Dockerfile
RUN gcloud auth activate-service-account --key-file=clientid-<myapp>.json --project <project-name> --quiet
I get the same results when running the Flask app in my local Docker environment and when I deploy to GAE Flexible. The Flask app has other endpoints that work as they don't call any other services. Only /things uses an API to call DataStore and it receives the above error.
/things
I've been through all the docs. I'm sure it's something basic and obvious but afaict, everything looks right.
UPDATE:
I've tried creating the DataStore client with an explicit pass of project name with no change in result. Also, I've run this command from within the Docker container and this seems like unusual output to me but I don't know that it's incorrect for a Service Account.
root@e02b74cd269d:/home/vmagent/app# gcloud projects list
API [cloudresourcemanager.googleapis.com] not enabled on project
[<project id>]. Would you like to enable and retry (this will take a
few minutes)? (y/N)? y
When I respond w/ Y,:
ERROR: (gcloud.projects.list) PERMISSION_DENIED: Not allowed to get project settings for project <project id>
Output of gcloud config list:
gcloud config list
root@d18c83cae166:/home/vmagent/app# gcloud config list
[core]
account = <myapp>@<myproject>.iam.gserviceaccount.com
disable_usage_reporting = False
project = <myproject>
Your active configuration is: [default]
print(open(os.environ['GOOGLE_APPLICATION_CREDENTIALS']).read())
Hi @DustinI.Yes, I just tried that and it did print the contents of the json file to STDOUT. Good idea, though--it was a better way to confirm.
– Erik Nord
Sep 4 '18 at 15:34
Have you tried passing the
project parameter to datastore.Client()?– Dustin Ingram
Sep 4 '18 at 16:05
project
datastore.Client()
Another great idea, @Dustin. I've updated the question above with some new information, but this change had no effect.
– Erik Nord
Sep 4 '18 at 17:16
What's the output of
gcloud config list inside the container?– Dustin Ingram
Sep 4 '18 at 17:52
gcloud config list
0
Thanks for contributing an answer to Stack Overflow!
But avoid …
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
But avoid …
To learn more, see our tips on writing great answers.
Required, but never shown
Required, but never shown
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
Is the path to the default credentials correct? Can you do something like
print(open(os.environ['GOOGLE_APPLICATION_CREDENTIALS']).read())and see the contents of the file?– Dustin Ingram
Sep 4 '18 at 15:30