I'm trying to run a python flask based application on google app engine using cloud datastore. I'm following the bookshelf example -
https://cloud.google.com/python/getting-started/using-cloud-datastore
Everything seems to work in local machine but on app engine, I'm getting
ImportError: No module named pwd, which is being imported by oauth2client library.
Is there anything I'm missing. Any help will be appreciated.
The question referenced as duplicate is different. The application there seems to fail only locally, whereas for me everything works locally but fails to work in GAE.
Here's the stack trace (partial) -
...
1384/v1.394932573930853146/application/home/model.py", line 16, in save_user
ds = get_client()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/application/home/model.py", line 7, in get_client
return datastore.Client('bookshelf-1384')#current_app.config['PROJECT_ID'])
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/gcloud/datastore/client.py", line 173, in __init__
super(Client, self).__init__(credentials, http)
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/gcloud/client.py", line 122, in __init__
credentials = get_credentials()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/gcloud/credentials.py", line 82, in get_credentials
return client.GoogleCredentials.get_application_default()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/oauth2client/client.py", line 1288, in get_application_default
return GoogleCredentials._get_implicit_credentials()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/oauth2client/client.py", line 1273, in _get_implicit_credentials
credentials = checker()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/oauth2client/client.py", line 1226, in _implicit_credentials_from_files
credentials_filename = _get_well_known_file()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/oauth2client/client.py", line 1392, in _get_well_known_file
default_config_dir = os.path.join(os.path.expanduser('~'),
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/posixpath.py", line 268, in expanduser
import pwd
ImportError: No module named pwd
The error is caused by http://github.com/google/oauth2client/issues/578.
It will be fixed in a couple of weeks with a new AppEngine SDK. In the meantime you can downgrade the version of the oauth2client library in your app.yaml file.
I had this same issue which is a known issue with the SDK. I tried the other solutions suggested on SO (see Google App Engine 'No module named pwd') but it didn't work. I found a work around by installing an older version of oauth2client-2.0.0 overwriting the oauth2client-4.0.0 that came with the pubsub pip install. Try running
pip install -t ./lib/ --upgrade --force-reinstall oauth2client==2.0.0
in the project directory and rerunning.
Related
I am working in pycharm(in windows) environment.
I am building a website to send query to datastore and display values. I also have data under kind="SpecialTest" that is automatically saved through Node.js. Now I wanted to Query the data from Python.
I installed google-cloud-datastore from command line.
But I can't establish connection to datastore.
My code:
from google.cloud import datastore
cli=datastore.Client(project="My_project_I")
query4= cli.query(kind="SpecialTest").order('Published_at')
print query4
My result:
Internal Server Error
The server has either erred or is incapable of performing the requested operation.
Traceback (most recent call last):
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1272, in default_dispatcher
self.handlers[handler] = handler = import_string(handler)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1850, in import_string
return getattr(__import__(module, None, None, [obj]), obj)
File "C:\Users\Harry\workspace\www\app\account.py", line 11, in <module>
from google.cloud import datastore
File "C:\Python27\lib\site-packages\google\cloud\datastore\__init__.py", line 60, in <module>
from google.cloud.datastore.batch import Batch
File "C:\Python27\lib\site-packages\google\cloud\datastore\batch.py", line 24, in <module>
from google.cloud.datastore import helpers
File "C:\Python27\lib\site-packages\google\cloud\datastore\helpers.py", line 23, in <module>
from google.protobuf import struct_pb2
ImportStringError: import_string() failed for 'app.account.UserAccount'. Possible reasons are:
- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;
Original exception:
ImportError: No module named protobuf
Debugged import:
- 'app' found in 'C:\\Users\\Harry\\workspace\\www\\app\\__init__.pyc'.
- 'app.account' not found.
The datastore client library is not supplied by GAE, you need to install/vendor it into your application (different method than for a standalone script).
Alternatively you can use the GAE-specific datastore library - it's actually recommended as it offers some advantages over the general purpose one. From Google App Engine Standard Environment Client Libraries:
Integrate Cloud Datastore with your App Engine Standard Environment
applications by using the App Engine client libraries.
...
Python Google Datastore NDB Client Library
Note: For App Engine applications that are written in Python, the Google Datastore DB Client Library is no longer recommended; use the
Google Datastore NDB Client Library instead.
All gcloud commands suddenly give this error:
ERROR: gcloud failed to load: No module named google.appengine.ext
This usually indicates corruption in your gcloud installation or
problems with your Python interpreter.
Please verify that the following is the path to a working Python 2.7
executable:
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
If it is not, please set the CLOUDSDK_PYTHON environment variable to
point to a working Python 2.7 executable.
If you are still experiencing problems, please reinstall the Cloud SDK
using the instructions here:
https://cloud.google.com/sdk/
Here are some relevant details and things that I've tried:
Mac 10.11.6
Python 2.7.8 at /Library/Frameworks/Python.framework/Versions/2.7/bin/python
Python otherwise works fine
I completely removed google-cloud-sdk and redownloaded from https://cloud.google.com/sdk/
Tried rebooting
Tried "export CLOUDSDK_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python" but it didn't help. Also added this line to my .bash_profile
Anyone else having this problem or can suggest a fix?
I have a fix below, but I'll happily award the answer to anyone who can explain what is going on.
I had the same issue . I solved it by using the default python2.7 in Ubuntu
Type this in termimal
export CLOUDSDK_PYTHON=/usr/bin/python
Not sure if this will work the same on Windows
I followed the answer here, worked for me
https://stackoverflow.com/a/59816812/2210667
brew switch openssl 1.0.2q (or whatever latest version you have from ls /usr/local/Cellar/openssl)
I figured out what was causing the problem, though I don't understand what is going on... When reinstalling gcloud I skipped the install script (./google-cloud-sdk/install.sh) since it is an optional step and I already had the needed lines in my bash profile.
When running this script, however, I got this error:
$ ./google-cloud-sdk/install.sh
Welcome to the Google Cloud SDK!
Traceback (most recent call last):
File "/Users/.../google-cloud-sdk/bin/bootstrapping/install.py", line 8, in <module>
import bootstrapping
File "/Users/.../google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 17, in <module>
import oauth2client.contrib.gce as gce
File "/Users/.../google-cloud-sdk/lib/third_party/oauth2client/contrib/gce.py", line 23, in <module>
import httplib2
File "/Users/.../google-cloud-sdk/lib/third_party/httplib2/__init__.py", line 29, in <module>
import email
File "/Users/.../GIT/GAE_Project/email.py", line 1, in <module>
ImportError: No module named google.appengine.ext
A file I recently created in my GAE project named email.py broke gcloud!?! Renaming this file fixed gcloud.
This is my PYTHONPATH:
/Users/.../google-cloud-sdk/platform/google_appengine:/Users/.../GIT/GAE_Project
So even though the Google cloud libraries come before my GAE project in my PYTHONPATH, a file in my GAE project broke gcloud.
I have already got boto installed to my python virtual environment, as shown in the screenshot of my pycharm project.
However I got an ImportError of boto.cloudfront when I run my script from pycharm
ERROR 2015-04-20 00:17:39,590 wsgi.py:263]
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/Users/antkong/dev/test/testmain.py", line 7, in <module>
from boto.cloudfront import CloudFrontConnection
ImportError: No module named boto.cloudfront
In the python console, I can import the library just fine:
Any suggestion what can go wrong here?
My project is a Google App Engine project. So even though boto is installed locally, it is not visible to my code within my Google App Engine project
My solution is to copy the boto and all dependency into a lib subdirectory and then add it to the sys.path
This is a follow-up question for Google App Engine and Django support:
The tutorial works great for an empty project, however when I try to deploy an existing Django app to Google App Engine, it starts throwing errors:
Traceback (most recent call last):
File "/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 223, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/python27_runtime/python27_lib/versions/third_party/django-1.4/django/core/handlers/wsgi.py", line 219, in __call__
self.load_middleware()
File "/python27_runtime/python27_lib/versions/third_party/django-1.4/django/core/handlers/base.py", line 39, in load_middleware
for middleware_path in settings.MIDDLEWARE_CLASSES:
File "/python27_runtime/python27_lib/versions/third_party/django-1.4/django/utils/functional.py", line 184, in inner
self._setup()
File "/python27_runtime/python27_lib/versions/third_party/django-1.4/django/conf/__init__.py", line 40, in _setup
raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
And a second, which might be related:
Traceback (most recent call last):
File "/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 223, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/python27_runtime/python27_lib/versions/third_party/django-1.4/django/core/handlers/wsgi.py", line 219, in __call__
self.load_middleware()
File "/python27_runtime/python27_lib/versions/third_party/django-1.4/django/core/handlers/base.py", line 39, in load_middleware
for middleware_path in settings.MIDDLEWARE_CLASSES:
File "/python27_runtime/python27_lib/versions/third_party/django-1.4/django/utils/functional.py", line 184, in inner
self._setup()
File "/python27_runtime/python27_lib/versions/third_party/django-1.4/django/conf/__init__.py", line 42, in _setup
self._wrapped = Settings(settings_module)
File "/python27_runtime/python27_lib/versions/third_party/django-1.4/django/conf/__init__.py", line 95, in __init__
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'settings' (Is it on sys.path?): No module named settings
I have modified my settings.py as per the tutorial. The app deploys without problems and the syncdb works as well; the database and all required tables are in place.
I found this reference to the error message as shown in the log but its suggested fix did not help.
Any ideas what might be causing this?
EDIT:
To shorten this already lengthy question I removed the previously posted wsgi.py file. I had nothing to do with this issue.
EDIT 2:
I think I made a small improvement and GAE is now complaining:
ImportError: No module named properties
I presume there might be an entry missing in my app.yaml file but I have no clue as to which file that may be. I found some references regarding missing modules but none that reflect this error message...
After much trial and error I managed to solve this, albeit partially:
I did not manage to get my existing PyCharm project deployed (yet). As I was keen to have a version running on GAE I went for a different approach and created a new empty Django project (project 2) on my local machine.
I first deployed the blank project using this tutorial which ran fine. I then copied my models.py file from my existing PyCharm project to project 2 and added an app called properties which uses the models.py file. Initially this went wrong as GAE kept complaining it could not find my properties app. After fiddling about with it, it suddenly appeared in my admin. I'm not sure what caused the initial problem but it's working now.
Things crucial to success for me:
I added the following to the very top of my settings.py:
import os
ROOT_PATH = os.path.dirname(__file__)
... and then replaced the default STATIC_ROOT with:
STATIC_ROOT = ROOT_PATH + os.sep + 'static'
Then I ran:
python manage.py collectstatic
from the dir of project 2. This collects the required static files which Django admin uses (image files, css files etc)
So, I currently have a stripped down version of my original project running on GAE. Things that are missing:
The original project used grappelli for a more styled admin interface. I'm not sure whether this will run on GAE. EDIT: Deploying Grappelli on GAE is reasonbly easy. Just copy the Grappelli package to your project dir and run manage.py collectstatic. Also add grappelli urls as per documention. Redeploy. Done.
The original project uses a module called templated_email to send out templated emails from my properties app. Same as grappelli: I'm not sure whether this is supported by GAE. Hopefully it will run on GAE as it is crucial for my specific app. EDIT: I bumped into this little gem which makes templated_email obsolete and lets me plug into GAE mail system.
To be honest, this approach partially defies the benefits of PyCharm which should fully support Django and GAE out of the box. It will probably require some additional research and tweaking to get that up and running.
Any comments or insights re the above are obviously appreciated!
I am trying to deploy my django app to amazon's ec2. I ran into some trouble getting my site-packages and my middleware working on the ec2 server. i am using bitnami djangostack for my AMI but am confused where on the server to put my site-packages (like registration south etc) and where to put my middleware (like pagination).
right now i am running the server and getting the following error:
Traceback (most recent call last):
File "/opt/bitnami/apps/django/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 283, in run
self.result = application(self.environ, self.start_response)
File "/opt/bitnami/apps/django/lib/python2.6/site-packages/django/contrib/staticfiles/handlers.py", line 68, in __call__
return self.application(environ, start_response)
File "/opt/bitnami/apps/django/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 250, in __call__
self.load_middleware()
File "/opt/bitnami/apps/django/lib/python2.6/site-packages/django/core/handlers/base.py", line 47, in load_middleware
raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
ImproperlyConfigured: Error importing middleware pagination.middleware: "No module named pagination.middleware"
you can see the error at the following link
site-packages is not somewhere you manually add items. Let Python's installation tools - pip and easy_install do the job for you.
As for middleware, it can go anywhere that is in your PYTHONPATH. The best place to put it is in a file that is in one of your app's directory (same place you see models.py); typically this file is called middleware.py
If the middleware is part of a third-party application; first install that application. Typically this is done with sudo easy_install package-name. Use sudo to install it in the global python site-packages directory.
A better approach is to use a virtual environment.