importing google.cloud.datastore not working - python-2.7

I am working in pycharm(in windows) environment.
I am building a website to send query to datastore and display values. I also have data under kind="SpecialTest" that is automatically saved through Node.js. Now I wanted to Query the data from Python.
I installed google-cloud-datastore from command line.
But I can't establish connection to datastore.
My code:
from google.cloud import datastore
cli=datastore.Client(project="My_project_I")
query4= cli.query(kind="SpecialTest").order('Published_at')
print query4
My result:
Internal Server Error
The server has either erred or is incapable of performing the requested operation.
Traceback (most recent call last):
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1272, in default_dispatcher
self.handlers[handler] = handler = import_string(handler)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1850, in import_string
return getattr(__import__(module, None, None, [obj]), obj)
File "C:\Users\Harry\workspace\www\app\account.py", line 11, in <module>
from google.cloud import datastore
File "C:\Python27\lib\site-packages\google\cloud\datastore\__init__.py", line 60, in <module>
from google.cloud.datastore.batch import Batch
File "C:\Python27\lib\site-packages\google\cloud\datastore\batch.py", line 24, in <module>
from google.cloud.datastore import helpers
File "C:\Python27\lib\site-packages\google\cloud\datastore\helpers.py", line 23, in <module>
from google.protobuf import struct_pb2
ImportStringError: import_string() failed for 'app.account.UserAccount'. Possible reasons are:
- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;
Original exception:
ImportError: No module named protobuf
Debugged import:
- 'app' found in 'C:\\Users\\Harry\\workspace\\www\\app\\__init__.pyc'.
- 'app.account' not found.

The datastore client library is not supplied by GAE, you need to install/vendor it into your application (different method than for a standalone script).
Alternatively you can use the GAE-specific datastore library - it's actually recommended as it offers some advantages over the general purpose one. From Google App Engine Standard Environment Client Libraries:
Integrate Cloud Datastore with your App Engine Standard Environment
applications by using the App Engine client libraries.
...
Python Google Datastore NDB Client Library
Note: For App Engine applications that are written in Python, the Google Datastore DB Client Library is no longer recommended; use the
Google Datastore NDB Client Library instead.

Related

ImportError: cannot import name cygrpc

I am trying to use Firebase Native mode on Google App Engine - Standard. My python language is Python 2.7. When I try to from google.cloud import firestore, I get an error ImportError: cannot import name cygrpc
I have deployed virtualenv described in the documentation here.
pip install virtualenv
virtualenv env
source env/bin/activate
My appengine_config.py is
from google.appengine.ext import vendor
import os.path
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
vendor.add(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib'))
my_app.py includes
from google.appengine.ext.webapp import template
from google.appengine.ext import ndb
from google.appengine.api import mail
import os
from google.cloud import firestore
(/base/alloc/tmpfs/dynamic_runtimes/python27g/43d5822312de17fd/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py:269)
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/43d5822312de17fd/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/43d5822312de17fd/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 311, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/43d5822312de17fd/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/main.py", line 4, in <module>
from controllers import server, common, header
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/controllers/server.py", line 10, in <module>
from google.cloud import firestore
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/lib/google/cloud/firestore.py", line 18, in <module>
from google.cloud.firestore_v1 import __version__
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/lib/google/cloud/firestore_v1/__init__.py", line 22, in <module>
from google.cloud.firestore_v1._helpers import GeoPoint
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/lib/google/cloud/firestore_v1/_helpers.py", line 21, in <module>
import grpc
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/lib/grpc/__init__.py", line 23, in <module>
from grpc._cython import cygrpc as _cygrpc
ImportError: cannot import name cygrpc
The question I have - can you use Firestore Native mode on Google App Engine Standard using Python 2.7? I need GAE- Standard because we are using Google Endpoint that is not support on GAE-Flex.
The doc here says App Engine Client library integration is not supported on Python2.7 GAE Standard environment. But I am not trying App Engine Client library, I am trying App Engine Server library in GAE Standard Environment.
How do I solve for import error for cygrpc? The solution here, says -
python -m pip install grpcio --ignore-installed
Is this recommended?
It's 2020, and you can now use grpcio (without explicitly adding it yourself as it's a built-in library) w/Cloud Firestore natively on App Engine with Python 2.7. Three things to make this work:
Add google-cloud-firestore to your requirements.txt file and install with pip install -U -t lib -r requirements.txt.
Add these lines to the libraries section of your app.yaml file:
libraries:
- name: grpcio
version: 1.0.0
- name: setuptools
version: 36.6.0
Ensure these lines are in your appengine_config.py file:
import pkg_resources
from google.appengine.ext import vendor
# Set path to your libraries folder.
path = 'lib'
# Add libraries installed in the path folder.
vendor.add(path)
# Add libraries to pkg_resources working set to find the distribution.
pkg_resources.working_set.add_entry(path)
Of course, we do recommend you eventually migrate to Python 3 to take advantage of the greater flexibility the next generations (of Python and App Engine) provide, esp. the ability to tap into other GCP services. However, caveat such a port isn't without effort if your app is complex & deeply-dependent on App Engine's 1st gen built-in services. Most of the suggestions above are derived from this section of the migration docs.
Once you're on Python 3, everything related to 3P libraries on App Engine becomes much easier, as I demonstrate in this other SO answer.
A while ago GRPC wasn't supported on GAE standard, see GRPC and types import error in App Engine Datastore. I didn't try since, but I don't see newer activity on issue 149.
Also the cython reference in the traceback suggests that it may include compiled code, which would violate the pure python standard environment sandbox restrictions that applies to the code you deploy.
Make sure that your installed python version bit(32/64) and your system bit(32/64) are matched.
If they are not matched please follow the solution mentioned at https://azurelessons.com/cannot-import-name-cygrpc/

How do I fix packaging issue when generating OpenAPI configuration file?

I am working through the Python Quickstart for Cloud Endpoints Frameworks on App Engine. I try to generate the OpenAPI configuration file by invoking the Endpoints Tool and get this error:
python lib/endpoints/endpointscfg.py get_swagger_spec main.EchoApi --hostname echo.endpoints.[YOUR-PROJECT-ID].cloud.goog
Traceback (most recent call last):
File "lib/endpoints/endpointscfg.py", line 59, in <module>
import _endpointscfg_setup # pylint: disable=unused-import
File "/Users/myName/lab/python-docs-samples/appengine/standard/endpoints-frameworks-v2/echo/lib/endpoints/_endpointscfg_setup.py", line 98, in <module>
_SetupPaths()
File "/Users/myName/lab/python-docs-samples/appengine/standard/endpoints-frameworks-v2/echo/lib/endpoints/_endpointscfg_setup.py", line 94, in _SetupPaths
from google.appengine.ext import vendor
ImportError: No module named appengine.ext
My understanding is that this is some kind of packaging issue? This is issue is discussed here but i am still stuck: Error running endpointscfg.py get_swagger_spec. Any help would be much appreciated.
It turned out to be an issue with my system variable path. Specifically I added:
ENDPOINTS_GAE_SDK="path to google_appengine"

ImportError: no module named pwd in app engine

I'm trying to run a python flask based application on google app engine using cloud datastore. I'm following the bookshelf example -
https://cloud.google.com/python/getting-started/using-cloud-datastore
Everything seems to work in local machine but on app engine, I'm getting
ImportError: No module named pwd, which is being imported by oauth2client library.
Is there anything I'm missing. Any help will be appreciated.
The question referenced as duplicate is different. The application there seems to fail only locally, whereas for me everything works locally but fails to work in GAE.
Here's the stack trace (partial) -
...
1384/v1.394932573930853146/application/home/model.py", line 16, in save_user
ds = get_client()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/application/home/model.py", line 7, in get_client
return datastore.Client('bookshelf-1384')#current_app.config['PROJECT_ID'])
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/gcloud/datastore/client.py", line 173, in __init__
super(Client, self).__init__(credentials, http)
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/gcloud/client.py", line 122, in __init__
credentials = get_credentials()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/gcloud/credentials.py", line 82, in get_credentials
return client.GoogleCredentials.get_application_default()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/oauth2client/client.py", line 1288, in get_application_default
return GoogleCredentials._get_implicit_credentials()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/oauth2client/client.py", line 1273, in _get_implicit_credentials
credentials = checker()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/oauth2client/client.py", line 1226, in _implicit_credentials_from_files
credentials_filename = _get_well_known_file()
File "/base/data/home/apps/s~bookshelf-1384/v1.394932573930853146/lib/oauth2client/client.py", line 1392, in _get_well_known_file
default_config_dir = os.path.join(os.path.expanduser('~'),
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/posixpath.py", line 268, in expanduser
import pwd
ImportError: No module named pwd
The error is caused by http://github.com/google/oauth2client/issues/578.
It will be fixed in a couple of weeks with a new AppEngine SDK. In the meantime you can downgrade the version of the oauth2client library in your app.yaml file.
I had this same issue which is a known issue with the SDK. I tried the other solutions suggested on SO (see Google App Engine 'No module named pwd') but it didn't work. I found a work around by installing an older version of oauth2client-2.0.0 overwriting the oauth2client-4.0.0 that came with the pubsub pip install. Try running
pip install -t ./lib/ --upgrade --force-reinstall oauth2client==2.0.0
in the project directory and rerunning.

PyCharm IDE: boto is already installed but got a "No module named boto.cloudfront" importError

I have already got boto installed to my python virtual environment, as shown in the screenshot of my pycharm project.
However I got an ImportError of boto.cloudfront when I run my script from pycharm
ERROR 2015-04-20 00:17:39,590 wsgi.py:263]
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/Users/antkong/dev/test/testmain.py", line 7, in <module>
from boto.cloudfront import CloudFrontConnection
ImportError: No module named boto.cloudfront
In the python console, I can import the library just fine:
Any suggestion what can go wrong here?
My project is a Google App Engine project. So even though boto is installed locally, it is not visible to my code within my Google App Engine project
My solution is to copy the boto and all dependency into a lib subdirectory and then add it to the sys.path

datastore error after migrating to python2.7 - works fine on localhost

I understand that google.appengine.dist was removed from python2.7. What should I use instead?
application works fine on localhost, deployment seems to be successful, but I am getting this message online:
Error: Server Error
The server encountered an error and could not complete your request.
If the problem persists, please report your problem and mention this error message and the query that caused it.
Here is what my log says:
Traceback (most recent call last):
File
"/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 196, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 255, in _LoadHandler
handler = __import__(path[0])
File "/base/data/home/apps/s~quick-test/1.368856085074570769/django_bootstrap.py", line 54, in <module>
from google.appengine.dist import use_library
ImportError: No module named dist
Here's the line 54 from django_bootstrap.py:
from google.appengine.dist import use_library
use_library('django', '1.2')
GAE LAuncher version 1.8.2, had this problem with 1.8 too.
Python 2.7 configuration requires third-party libraries specified in app.yaml, so it might be sufficient to have this in your app.yaml file:
libraries:
- name: django
version: "1.2"
You can also use webapp2 that includes Django’s templating engine. Version 1.2 included with the SDK is part of App Engine, and you do not need to bundle Django yourself to use it.
import os
from google.appengine.ext.webapp import template
Also, with this code you don't need to call use_library() to explicitly select a Django version:
webapp_django_version = "1.2"