No module named google_auth_httplib2 - python-2.7

Goal: Use GAE with Python and Google Cloud Storage to store and serve an image more efficiently to ultimately use the image API.
Problem: Cannot find correct modules (httplib2 and six) despite successful install.
Run time example
Python Code Sample A:
from google.cloud import storage
from google.appengine.api import app_identity
from google.appengine.ext import blobstore
from google.appengine.ext.webapp import blobstore_handlers
import webapp2
Returns Sample A:
ImportError: No module named google_auth_httplib2
Note: Also had a similar error for six. "no module named six"
Installed Details
python installed at:
C:\Python27
six installed at:
C:\python27\lib\site-packages\six-1.10.0-py2.7.egg
httplib2 installed at:
C:\Python27\Lib\site-packages\httplib2-0.9.2-py2.7.egg-info
Running “C:>pip install httplib2” in the command line returns:
“Requirement already satisfied: httplib2 in c:\python27\lib\site-packages”
Running “C:>pip install six” in the command line returns:
Requirement already satisfied: six in c:\python27\lib\site-packages\six-1.10.0-py2.7.egg
GAE Cloud Storage Client installed at:
C:\Python27\Lib\site-packages\GoogleAppEngineCloudStorageClient-1.9.22.1-py2.7.egg-info
GAE SDK Server Hosting using "dev_appserver.py ." at:
C:\Users\sebastian\Documents\Web Projects\Cookbook
This location also contains the app.yaml file.
Copied modules to app.yaml location
Copied the httplib2 and six-1.10.0-py2.7.egg folders to my app.yaml directory.
Appendix 1:
App.yaml
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /
script: homegcs.app
- url: /static
static_dir: static
- url: /.*
script: home.app
- url: /index\.html
script: home.app
- url: /stylesheets
static_dir: stylesheets
- url: /(.*\.(gif|png|jpg))$
static_files: static/\1
upload: static/.*\.(gif|png|jpg)$
- url: /admin/.*
script: admin.app
login: admin
- url: /.*
script: not_found.app
Folder structure containing App.yaml

Your packages either need to be uploaded with the project, or added in app.yaml, if they are available in App Engine. six is an available library, so, in app.yaml, add:
libraries:
- name: six
version: "1.9.0"
If you put the httplib2 package at the same level as app.yaml, it should upload with the project, and be available in production.
Another user added google_auth_httplib2 as a package as well, and uploaded it with the project. Though I think that should be available directly:
Module google_auth_httplib2 not found after pip installing google-cloud How can I fix it?
** You also have an issue in your url handlers in app.yaml. This is a wildcard for all urls:
- url: /.*
script: home.app
So, every handler below that will not ever be hit.

You need to install google-cloud in your project, like this:
pip install google-cloud -t [my_project]/lib/google-cloud
Make sure you create that google-cloud folder first, inside your lib folder. Once you do that, change or create appengine_config.py (in /my_project) and include this:
from google.appengine.ext import vendor
vendor.add('lib/google-cloud')
Everything shoul work now.

Related

Receiving error 502 BAD CONNECTION nginx when deploying django app on Google app engine

Ive been trying to my django website but I've been having the same 502 BAD CONNECTION nginx error every time and I'm not sure how to troubleshoot it.
When I run the program using py.manager runserver, it works perfectly fine, but but I only receive an error once I run gcloud app deploy, looking at the logs I haven't received any errors, just one warning stating that Container called exit(1).
I've tried one solution from the Google's website: https://cloud.google.com/endpoints/docs/openapi/troubleshoot-response-errors and I added resources: memory_gb: 4 to my app.yaml file and I still end up with an error.
I am not sure this is of much help my app.yaml currently looks like this:
runtime: python39
handlers:
# This configures Google App Engine to serve the files in the app's static
# directory.
- url: /static
static_dir: static/
# This handler routes all requests not caught above to your main app. It is
# required when static routes are defined, but can be omitted (along with
# the entire handlers section) when there are no static files defined.
- url: /.*
script: auto
resources:
memory_gb: 4
And my gcloudignore looks like this:
# This file specifies files that are *not* uploaded to Google Cloud Platform
# using gcloud. It follows the same syntax as .gitignore, with the addition of
# "#!include" directives (which insert the entries of the given .gitignore-style
# file at that point).
#
# For more information, run:
# $ gcloud topic gcloudignore
#
.gcloudignore
# If you would like to upload your .git directory, .gitignore file or files
# from your .gitignore file, remove the corresponding line
# below:
.git
.gitignore
secret_variables.py
.venv
Pipfile
Pipfile.lock
# Python pycache:
__pycache__/
# Ignored by the build system
/setup.cfg
Thank you in advance for the help
Cloned and tried running the code locally after running pip install -r requirements.txt. This is the error I got:
ModuleNotFoundError: No module named 'django_filters'
This means that this requirement needs to be added before the code is deployed to the GAE.
add django-filter to the requirements.txt file to fix it.

ImportError: no module named [directory] Google App Engine

I keep getting an error that says no module named backend, this is the directory where my webapp2 application is.
My folder structure:
/project
/backend
/env #python virtual env libraries
main.py #my main entry point where webapp2 app instance is
requirements.txt
app.yaml
My app.yaml:
service: default
handlers:
- url: /dist
static_dir: dist
- url: /.*
script: backend.main.app
libraries:
- name: webapp2
version: latest
- name: jinja2
version: latest
Before my app.yaml was in backend, but I decided to move to root. Now when I run dev_appserver.py in root, I keep getting ImportError: No module named backend
I created the virualenv and installed the requirements.txt packages inside the backend directory.
EDIT: I am unsure if this makes a difference, but I have already deployed my application when the app.yaml was inside the backend folder. I am guessing this should not matter since I am trying to test locally by moving the app.yaml in my project root and running dev_appserver.py app.yaml, but it seems to not work when I do this.
The directory containing the app.yaml file for a GAE service is the service's top-level directory. The content of this directory is what will be uploaded to GAE when you deploy the service. All paths referenced in the service's code or configurations are relative to this top level dir. So moving the app.yaml file around without updating the related code and configurations accordingly will break the app's functionality.
You don't seem to grasp the meaning of the script: statement very well. From Handlers element:
A script: directive must be a python import path, for example,
package.module.app that points to a WSGI application. The last
component of a script: directive using a Python module path is the
name of a global variable in the module: that variable must be a WSGI
app, and is usually called app by convention.
Note: just like for a Python import statement, each subdirectory that
is a package must contain a file named __init__.py
So, assuming your app.yaml file is located in your project dir, the
script: backend.pythonAttack.app
would mean:
having an __init__.py file inside the backend dir, to make backend a package
having a pythonAttack.py file in the backend dir, with an app variable pointing to your webapp2 application
According to your description you don't meet any of these conditions.
My recommendation:
keep the app.yaml inside the backend dir (which doesn't need to be a python package dir)
update its script line to match your code. Assuming the app variable for your webapp2 app is actually in the main.py file the line would be:
script: main.app
run the app locally by explicitly passing the app.yaml file as argument (in general a good habit and also the only way to run apps with multiple services and/or a dispatch.yaml file):
dev_appserver.py backend/app.yaml
store your service python dependencies inside a backend/lib directory (to follow the naming convention), separated from your virtualenv packages
store the env virtualenv package dir outside the backend directory to prevent unnecessarily uploading them to GAE when deploying the service (and potential interference with the app's operation). The goal of the virtualenv is to properly emulate the GAE sandbox locally so that you can run the development server correctly.
Potentially of interest for structuring a multi-service app: Can a default service/module in a Google App Engine app be a sibling of a non-default one in terms of folder structure?

Django app works locally but deployment on GCP say 500 Server Error, "ImportError: Could not import settings"

On GCP dashboard I get this error but on my machine all works fine:
ImportError: Could not import settings 'site.settings' (Is it on sys.path?): No module named settings
at __init__ (/base/alloc/tmpfs/dynamic_runtimes/python27g/3b44e98ed7fbb86b/python27/python27_lib/versions/third_party/django-1.4/django/conf/__init__.py:95)
at _setup (/base/alloc/tmpfs/dynamic_runtimes/python27g/3b44e98ed7fbb86b/python27/python27_lib/versions/third_party/django-1.4/django/conf/__init__.py:42)
at __setattr__ (/base/alloc/tmpfs/dynamic_runtimes/python27g/3b44e98ed7fbb86b/python27/python27_lib/versions/third_party/django-1.4/django/utils/functional.py:207)
at <module> (/base/data/home/apps/g~unitutor-221411/1.413727868439486493/main.py:9)
at LoadObject (/base/alloc/tmpfs/dynamic_runtimes/python27g/3b44e98ed7fbb86b/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py:85)
at _LoadHandler (/base/alloc/tmpfs/dynamic_runtimes/python27g/3b44e98ed7fbb86b/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py:299)
at Handle (/base/alloc/tmpfs/dynamic_runtimes/python27g/3b44e98ed7fbb86b/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py:240)
This is my app.yaml:
application: *********
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /static
static_dir: static
- url: /.*
script: main.app
libraries:
- name: django
version: "1.4"
- name: jinja2
version: "latest"
Obviously I have, locally and on GCP, a directory "site" containing "settings.py". On my machine the app works well. I tried in many ways and I read many posts but I can't find a solution to solve the "500 Server Error" on GCP.
Local and production are two different things. You are getting this error because your paths are incorrectly handled in GCP. Try to check where it is looking for settings.py file and then reconfigure it.

Error with Stripe calls on Google App Engine even after upgrading to latest

I receive runtime errors like this when making Stripe calls such as stripe.Customer.create(email = email)
APIError: Stripe no longer supports API requests made with TLS 1.0.
Please initiate HTTPS connections with TLS 1.2 or later.
You can learn more about this at https://stripe.com/blog/upgrading-tls.
This error shows up only on the local development server, but not on production. I have of course upgraded my stripe library like this:
pip install -t lib --upgrade stripe==1.19.1
To eliminate the possibility of this being an issue with pip upgrade, I have also tried removing the old lib/stripe directory and it's dependencies, followed by a fresh install. But that does not make a difference.
Does your appengine_config.py have this in it yet?
import os
from google.appengine.ext import vendor
from google.appengine.ext.appstats import recording
appstats_CALC_RPC_COSTS = True
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
def webapp_add_wsgi_middleware(app):
app = recording.appstats_wsgi_middleware(app)
return app
# if on localhost
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
import imp
import os.path
import inspect
from google.appengine.tools.devappserver2.python import sandbox
sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
# Use the system socket.
real_os_src_path = os.path.realpath(inspect.getsourcefile(os))
psocket = os.path.join(os.path.dirname(real_os_src_path), 'socket.py')
imp.load_source('socket', psocket)
else:
# Doing this on dev_appserver/localhost seems to cause outbound https requests to fail
from lib import requests
from lib.requests_toolbelt.adapters import appengine as requests_toolbelt_appengine
# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt_appengine.monkeypatch()
I copied my whole file because I'm not sure which parts you already have, but the key part is that final if-else. I went through a whole mess of trouble getting TLS 1.2 working on prod, which primarily came down to specifying version 2.7.11 for App Engine's special ssl library & requests_toolbelt_appengine.monkeypatch().
Like you this broke ssl on localhost for me, so now I only do requests_toolbelt_appengine.monkeypatch() on prod, and on localhost I do that 'white-listing the native sockets library' trick you've probably seen. Part of this comes down what combination of versions of things you're using. Hopefully this helps.
Notable items from my app.yaml:
env_variables:
theme: 'default'
GAE_USE_SOCKETS_HTTPLIB : 'true' # TLS 1.2
libraries:
- name: jinja2
version: "2.6"
- name: webapp2
version: "2.5.2"
- name: markupsafe
version: "0.15"
- name: ssl
version: "2.7.11" # TLS 1.2
- name: pycrypto
version: "2.6"
- name: lxml
version: latest
Also I'm using python-requests 2.18.2
EDIT:
In my ~/.bash_profile, google cloud sdk was added to my path:
export PATH="/Users/alex/google-cloud-sdk/platform/google_appengine/:$PATH".
If I go to that folder, I can follow the imports for from google.appengine.tools.devappserver2.python import sandbox all the way through. (screenshot included below)

How to import BigQuery in AppEngine for Python

I am trying to run a BigQuery query from Google AppEngine (deployed) using Python 2.7, but I am seeing this error in StackDriver's Error Reporting:
ImportError: No module named cloud
This is my code (main.py):
from __future__ import absolute_import
import webapp2
from google.cloud import bigquery
class MainPage(webapp2.RequestHandler):
def get(self):
# Instantiates a client
bigquery_client = bigquery.Client()
# The name for the new dataset
dataset_name = 'my_new_set'
# Prepares the new dataset
dataset = bigquery_client.dataset(dataset_name)
# Creates the new dataset
dataset.create()
# Remove unwanted chars
#self.response.write(str(container))
app = webapp2.WSGIApplication([
('/', MainPage),
], debug=True)
This is my (app.yaml):
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: main.app
The error message would make me assume that the BigQuery's library is not being imported. However, if this code is being deployed in AppEngine, shouldn't the library already be installed in AppEngine by default?
Trying to solve the problem
Attempt # 1
I found this post that refers to a similar issue. The suggestion was to add this line to the top of the file. I added the line to my file, but the problem still exists:
from __future__ import absolute_import
Source:
No module named cloud while using google.cloud import bigquery
Attempt # 2
I installed BigQuery's client locally in my laptop:
pip install google-cloud-bigquery==0.22.1
I also installed the same client in the "lib" folder to have it uploaded to AppEngine once it is deployed:
pip install --target='lib' google-cloud-bigquery==0.22.1
This last, also requires a file named "appengine_config.py" to be created with this content:
# appengine_config.py
from google.appengine.ext import vendor
# Add any libraries install in the "lib" folder.
vendor.add('lib')
Source: https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27
However, this attempt did not work either. The error message changed to the following:
*File "/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/httplib2/__init__.py", line 352: print('%s:' % h, end=' ', file=self._fp) ^ SyntaxError: invalid syntax
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google_auth_httplib2.py:23)
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/_helpers.py:31)
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/bigquery/_helpers.py:21)
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/bigquery/__init__.py:26)
at get (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/main.py:75)
at dispatch (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:545)
at dispatch (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:547)
at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1077)
at default_dispatcher (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1253)
at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1505)
at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1511)*
How can I import the BigQuery library correctly in AppEngine (deployed)?
Thanks for your help.
The following solution worked for me without having to use from __future__ import absolute_import. There are three main steps to follow.
1. Copy google-api-python-client and google-cloud into project folder
Even though it sounds counterintuitive, according to the documentation
[...] Python client libraries are not installed in the App Engine Python runtime environment, [so] they must be vendored into your application just like third-party libraries.
So in order to use google.cloud one must copy the library code into the project's source directory. The library code, along with the application code, is uploaded to App Engine.
To copy a library code into your project:
Create a directory (e.g. lib/) to store third-party libraries in your project's root folder
mkdir lib
Copy the google-api-python-client and google-cloud libraries into the folder you just created. I use pip in the following example.
pip install -t lib/ --upgrade google-api-python-client
pip install -t lib/ --upgrade google-cloud
2. Link installed libraries to app
Create a file named appengine_config.pyin the same folder as your app.yaml file
Edit appengine_config.py and include the following code
# appengine_config.py
from google.appengine.ext import vendor
# Add any libraries install in the "lib" folder.
vendor.add('lib')
3. Include added libraries to requirements.txt
Edit your requirements.txt file and include the names of your added libraries
# other requirements
google-api-python-client
google-cloud
You should now be able to use from google.cloud import bigquery with no problem after deploying your app.
For more information see using third-party libraries