I am trying to run a BigQuery query from Google AppEngine (deployed) using Python 2.7, but I am seeing this error in StackDriver's Error Reporting:
ImportError: No module named cloud
This is my code (main.py):
from __future__ import absolute_import
import webapp2
from google.cloud import bigquery
class MainPage(webapp2.RequestHandler):
def get(self):
# Instantiates a client
bigquery_client = bigquery.Client()
# The name for the new dataset
dataset_name = 'my_new_set'
# Prepares the new dataset
dataset = bigquery_client.dataset(dataset_name)
# Creates the new dataset
dataset.create()
# Remove unwanted chars
#self.response.write(str(container))
app = webapp2.WSGIApplication([
('/', MainPage),
], debug=True)
This is my (app.yaml):
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: main.app
The error message would make me assume that the BigQuery's library is not being imported. However, if this code is being deployed in AppEngine, shouldn't the library already be installed in AppEngine by default?
Trying to solve the problem
Attempt # 1
I found this post that refers to a similar issue. The suggestion was to add this line to the top of the file. I added the line to my file, but the problem still exists:
from __future__ import absolute_import
Source:
No module named cloud while using google.cloud import bigquery
Attempt # 2
I installed BigQuery's client locally in my laptop:
pip install google-cloud-bigquery==0.22.1
I also installed the same client in the "lib" folder to have it uploaded to AppEngine once it is deployed:
pip install --target='lib' google-cloud-bigquery==0.22.1
This last, also requires a file named "appengine_config.py" to be created with this content:
# appengine_config.py
from google.appengine.ext import vendor
# Add any libraries install in the "lib" folder.
vendor.add('lib')
Source: https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27
However, this attempt did not work either. The error message changed to the following:
*File "/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/httplib2/__init__.py", line 352: print('%s:' % h, end=' ', file=self._fp) ^ SyntaxError: invalid syntax
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google_auth_httplib2.py:23)
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/_helpers.py:31)
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/bigquery/_helpers.py:21)
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/bigquery/__init__.py:26)
at get (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/main.py:75)
at dispatch (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:545)
at dispatch (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:547)
at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1077)
at default_dispatcher (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1253)
at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1505)
at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1511)*
How can I import the BigQuery library correctly in AppEngine (deployed)?
Thanks for your help.
The following solution worked for me without having to use from __future__ import absolute_import. There are three main steps to follow.
1. Copy google-api-python-client and google-cloud into project folder
Even though it sounds counterintuitive, according to the documentation
[...] Python client libraries are not installed in the App Engine Python runtime environment, [so] they must be vendored into your application just like third-party libraries.
So in order to use google.cloud one must copy the library code into the project's source directory. The library code, along with the application code, is uploaded to App Engine.
To copy a library code into your project:
Create a directory (e.g. lib/) to store third-party libraries in your project's root folder
mkdir lib
Copy the google-api-python-client and google-cloud libraries into the folder you just created. I use pip in the following example.
pip install -t lib/ --upgrade google-api-python-client
pip install -t lib/ --upgrade google-cloud
2. Link installed libraries to app
Create a file named appengine_config.pyin the same folder as your app.yaml file
Edit appengine_config.py and include the following code
# appengine_config.py
from google.appengine.ext import vendor
# Add any libraries install in the "lib" folder.
vendor.add('lib')
3. Include added libraries to requirements.txt
Edit your requirements.txt file and include the names of your added libraries
# other requirements
google-api-python-client
google-cloud
You should now be able to use from google.cloud import bigquery with no problem after deploying your app.
For more information see using third-party libraries
Related
I am trying to use Firebase Native mode on Google App Engine - Standard. My python language is Python 2.7. When I try to from google.cloud import firestore, I get an error ImportError: cannot import name cygrpc
I have deployed virtualenv described in the documentation here.
pip install virtualenv
virtualenv env
source env/bin/activate
My appengine_config.py is
from google.appengine.ext import vendor
import os.path
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
vendor.add(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib'))
my_app.py includes
from google.appengine.ext.webapp import template
from google.appengine.ext import ndb
from google.appengine.api import mail
import os
from google.cloud import firestore
(/base/alloc/tmpfs/dynamic_runtimes/python27g/43d5822312de17fd/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py:269)
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/43d5822312de17fd/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/43d5822312de17fd/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 311, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/43d5822312de17fd/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/main.py", line 4, in <module>
from controllers import server, common, header
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/controllers/server.py", line 10, in <module>
from google.cloud import firestore
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/lib/google/cloud/firestore.py", line 18, in <module>
from google.cloud.firestore_v1 import __version__
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/lib/google/cloud/firestore_v1/__init__.py", line 22, in <module>
from google.cloud.firestore_v1._helpers import GeoPoint
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/lib/google/cloud/firestore_v1/_helpers.py", line 21, in <module>
import grpc
File "/base/data/home/apps/s~openbarn-prod/20190602t102855.418624175446659791/lib/grpc/__init__.py", line 23, in <module>
from grpc._cython import cygrpc as _cygrpc
ImportError: cannot import name cygrpc
The question I have - can you use Firestore Native mode on Google App Engine Standard using Python 2.7? I need GAE- Standard because we are using Google Endpoint that is not support on GAE-Flex.
The doc here says App Engine Client library integration is not supported on Python2.7 GAE Standard environment. But I am not trying App Engine Client library, I am trying App Engine Server library in GAE Standard Environment.
How do I solve for import error for cygrpc? The solution here, says -
python -m pip install grpcio --ignore-installed
Is this recommended?
It's 2020, and you can now use grpcio (without explicitly adding it yourself as it's a built-in library) w/Cloud Firestore natively on App Engine with Python 2.7. Three things to make this work:
Add google-cloud-firestore to your requirements.txt file and install with pip install -U -t lib -r requirements.txt.
Add these lines to the libraries section of your app.yaml file:
libraries:
- name: grpcio
version: 1.0.0
- name: setuptools
version: 36.6.0
Ensure these lines are in your appengine_config.py file:
import pkg_resources
from google.appengine.ext import vendor
# Set path to your libraries folder.
path = 'lib'
# Add libraries installed in the path folder.
vendor.add(path)
# Add libraries to pkg_resources working set to find the distribution.
pkg_resources.working_set.add_entry(path)
Of course, we do recommend you eventually migrate to Python 3 to take advantage of the greater flexibility the next generations (of Python and App Engine) provide, esp. the ability to tap into other GCP services. However, caveat such a port isn't without effort if your app is complex & deeply-dependent on App Engine's 1st gen built-in services. Most of the suggestions above are derived from this section of the migration docs.
Once you're on Python 3, everything related to 3P libraries on App Engine becomes much easier, as I demonstrate in this other SO answer.
A while ago GRPC wasn't supported on GAE standard, see GRPC and types import error in App Engine Datastore. I didn't try since, but I don't see newer activity on issue 149.
Also the cython reference in the traceback suggests that it may include compiled code, which would violate the pure python standard environment sandbox restrictions that applies to the code you deploy.
Make sure that your installed python version bit(32/64) and your system bit(32/64) are matched.
If they are not matched please follow the solution mentioned at https://azurelessons.com/cannot-import-name-cygrpc/
I created an Odoo Module in Python using the Python library ujson.
I installed this library on my development server manually with pip install ujson.
Now I want to install the Module on my live server. Can I somehow tell the Odoo Module to install the ujson library when it is installed? So I just have to add the Module to my addons path and install it via the Odoo Web Interface?
Another reason to have this automated would be if I like to share my custom module, so others don't have to install the library manually on their server.
Any suggestions how to configure my Module that way? Or should I just include the library's directory in my module?
You should try-except the import to handle problems on odoo server start:
try:
from external_dependency import ClassA
except ImportError:
pass
And for other users of your module, extend the external_dependencies in your module manifest (v9 and less: __openerp__.py; v10+: __manifest__.py), which will prompt a warning on installation:
"external_dependencies": {
'python': ['external_dependency']
},
Big thanks goes to Ivan and his Blog
Thank you for your help, #Walid Mashal and #CZoellner, you both pointed me to the right direction.
I solved this task now with the following code added to the __init__.py of my module:
import pip
try:
import ujson
except ImportError:
print('\n There was no such module named -ujson- installed')
print('xxxxxxxxxxxxxxxx installing ujson xxxxxxxxxxxxxx')
pip.main(['install', 'ujson'])
In python file using the following command, you can install it (it works for odoo only). Eg: Here I am going to install xlsxwriter
try:
import xlsxwriter
except:
os.system("pip install xlsxwriter")
import xlsxwriter
The following is the code that is used in odoo base module report in base addons inside report.py (odoo_root_folder/addons/report/models/report.py) to install wkhtmltopdf.
from openerp.tools.misc import find_in_path
import subprocess
def _get_wkhtmltopdf_bin():
return find_in_path('wkhtmltopdf')
try:
process = subprocess.Popen([_get_wkhtmltopdf_bin(), '--version'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except (OSError, IOError):
_logger.info('You need Wkhtmltopdf to print a pdf version of the reports.')
basically you need to find some python code that will run the library and install it and include that code in one of you .py files, and that should do it.
Goal: Use GAE with Python and Google Cloud Storage to store and serve an image more efficiently to ultimately use the image API.
Problem: Cannot find correct modules (httplib2 and six) despite successful install.
Run time example
Python Code Sample A:
from google.cloud import storage
from google.appengine.api import app_identity
from google.appengine.ext import blobstore
from google.appengine.ext.webapp import blobstore_handlers
import webapp2
Returns Sample A:
ImportError: No module named google_auth_httplib2
Note: Also had a similar error for six. "no module named six"
Installed Details
python installed at:
C:\Python27
six installed at:
C:\python27\lib\site-packages\six-1.10.0-py2.7.egg
httplib2 installed at:
C:\Python27\Lib\site-packages\httplib2-0.9.2-py2.7.egg-info
Running “C:>pip install httplib2” in the command line returns:
“Requirement already satisfied: httplib2 in c:\python27\lib\site-packages”
Running “C:>pip install six” in the command line returns:
Requirement already satisfied: six in c:\python27\lib\site-packages\six-1.10.0-py2.7.egg
GAE Cloud Storage Client installed at:
C:\Python27\Lib\site-packages\GoogleAppEngineCloudStorageClient-1.9.22.1-py2.7.egg-info
GAE SDK Server Hosting using "dev_appserver.py ." at:
C:\Users\sebastian\Documents\Web Projects\Cookbook
This location also contains the app.yaml file.
Copied modules to app.yaml location
Copied the httplib2 and six-1.10.0-py2.7.egg folders to my app.yaml directory.
Appendix 1:
App.yaml
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /
script: homegcs.app
- url: /static
static_dir: static
- url: /.*
script: home.app
- url: /index\.html
script: home.app
- url: /stylesheets
static_dir: stylesheets
- url: /(.*\.(gif|png|jpg))$
static_files: static/\1
upload: static/.*\.(gif|png|jpg)$
- url: /admin/.*
script: admin.app
login: admin
- url: /.*
script: not_found.app
Folder structure containing App.yaml
Your packages either need to be uploaded with the project, or added in app.yaml, if they are available in App Engine. six is an available library, so, in app.yaml, add:
libraries:
- name: six
version: "1.9.0"
If you put the httplib2 package at the same level as app.yaml, it should upload with the project, and be available in production.
Another user added google_auth_httplib2 as a package as well, and uploaded it with the project. Though I think that should be available directly:
Module google_auth_httplib2 not found after pip installing google-cloud How can I fix it?
** You also have an issue in your url handlers in app.yaml. This is a wildcard for all urls:
- url: /.*
script: home.app
So, every handler below that will not ever be hit.
You need to install google-cloud in your project, like this:
pip install google-cloud -t [my_project]/lib/google-cloud
Make sure you create that google-cloud folder first, inside your lib folder. Once you do that, change or create appengine_config.py (in /my_project) and include this:
from google.appengine.ext import vendor
vendor.add('lib/google-cloud')
Everything shoul work now.
i have built an app engine application to load data into bigquery table using google app engine launcher but when I run it on local host or on the cloud i get the No module named cloud while using google.cloud import bigquery error message in log file. I have installed the google cloud client library but it is still giving me the same error. please see below the code I am using
---main.py file contains
import argparse
import time
import uuid
from google.cloud import bigquery
def load_data_from_gcs(dataset_name, table_name, source):
bigquery_client = bigquery.Client()
dataset = bigquery_client.dataset(dataset_name)
table = dataset.table(table_name)
job_name = str(uuid.uuid4())
job = bigquery_client.load_table_from_storage(
job_name, table, source)
job.begin()
wait_for_job(job)
print('Loaded {} rows into {}:{}.'.format(
job.output_rows, dataset_name, table_name))
def wait_for_job(job):
while True:
job.reload()
if job.state == 'DONE':
if job.error_result:
raise RuntimeError(job.error_result)
return
time.sleep(1)
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('Test')
parser.add_argument('mytable')
parser.add_argument('gs://week/geninfo.csv')
args = parser.parse_args()
load_data_from_gcs(
args.dataset_name,
args.table_name,
args.source)
--app.yaml file contains the following code
application: mycloudproject
version: 1
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: /favicon\.ico
static_files: favicon.ico
upload: favicon\.ico
- url: .*
script: main.app
Please let me know what is missing or if I am doing something wrong here?
This can be a bit tricky. Google Cloud uses the new Python namespace format (if you look at the source you'll notice that there's no __init__.py in the directory structure).
This was changed in Python 3.3 with PEP-420
Fortunately in Python 2.7 you can fix this easily by avoiding implicit imports. Just add this to the top of your file:
from __future__ import absolute_import
Hope that helps.
Find the directory containing google/cloud/..., and add that directory to the PYTHONPATH so that python can find it. See this post for details on how to add to PYTHONPATH. It outlines two common ways to do it:
Here's how to do it with a bash command:
export PYTHONPATH=$PYTHONPATH:/<path_to_modules>
Or you could append it to the path in your script:
# if the google/ directory is in the directory /path/to/directory/
path_to_look_for_module = '/path/to/directory/'
import sys
if not path_to_look_for_module in sys.path:
sys.path.append(path_to_look_for_module)
If that doesn't work, here is some code I found in one of my projects for importing Google Appengine modules:
def fixup_paths(path):
"""Adds GAE SDK path to system path and appends it to the google path
if that already exists."""
# Not all Google packages are inside namespace packages, which means
# there might be another non-namespace package named `google` already on
# the path and simply appending the App Engine SDK to the path will not
# work since the other package will get discovered and used first.
# This emulates namespace packages by first searching if a `google` package
# exists by importing it, and if so appending to its module search path.
try:
import google
google.__path__.append("{0}/google".format(path))
except ImportError:
pass
sys.path.insert(0, path)
# and then call later in your code:
fixup_paths(path_to_google_sdk)
from google.cloud import bigquery
It looks like you are trying to use the Cloud Datastore client library in a Google App Engine's standard environment. As documented in Google's documentation you should not be doing this. Instead, either use the NDB Client Library or do not use the standard environment.
Are you sure you've updated to the latest version of the library? The version installed by pip may be out of date. Previously, the module was imported as:
from gcloud import bigquery
If that works, you're running an older version. To install the latest, I'd recommend pulling from the master in the github project.
I am running a project at a remote cluster but I do not have root access. Also pip or any other python manager is not installed so I try to include all the files of the beautifulsoup to my project and import them naively. Currently I installed the package to my root folder of the project, I run python setup.py build and include the lien to my source code
sys.path.append("beautifulsoup/bs4")
from bs4 import BeautifulSoup
However it does not work and the program I wrote cannot see the package.
(My code and the beautifulsoup folder at the same level of folder hierarchy)
How could I make it work ?
Relative paths cannot be used that way to my knowledge. To append a relative path to sys.path you can do:
import os, sys
bspath = os.path.join(os.path.dirname(os.path.abspath(__file__)),"beautifulsoup")
sys.path.append(bspath)
In your program/script you could then import the library like this:
import bs4