test function with Google App Engine `files` api - unit-testing

I have a function that uses the Google Blobstore API, and here's a degenerate case:
#!/usr/bin/python
from google.appengine.ext import testbed
def foo():
from google.appengine.api import files
blob_filename = files.blobstore.create(mime_type='text/plain')
with files.open(blob_filename, 'a') as googfile:
googfile.write("Test data")
files.finalize(blob_filename)
tb = testbed.Testbed()
tb.activate()
tb.init_blobstore_stub()
foo() # in reality, I'm a function called from a 'faux client'
# in a unittest testcase.
The error this generates is:
Traceback (most recent call last):
File "e.py", line 18, in
foo() # in reality, I'm a function called from a 'faux client'
File "e.py", line 8, in foo
blob_filename = files.blobstore.create(mime_type='text/plain')
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/blobstore.py", line 68, in create
return files._create(_BLOBSTORE_FILESYSTEM, params=params)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/file.py", line 491, in _create
_make_call('Create', request, response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/file.py", line 230, in _make_call
rpc = _create_rpc(deadline=deadline)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/file.py", line 213, in _create_rpc
return apiproxy_stub_map.UserRPC('file', deadline)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 393, in __init__
self.__rpc = CreateRPC(service, stubmap)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 67, in CreateRPC
assert stub, 'No api proxy found for service "%s"' % service
AssertionError: No api proxy found for service "file"
I don't want to have to modify foo in order to be able to test it. Is there a way to make foo work as expected (i.e. create the given file) in Google App Engine's unit tests?
I would expect to be able to do this with Google's API Proxy, but I don't understand it well enough to figure it out on my own.
I'd be grateful for your thoughts and suggestions.
Thanks for reading.

It seems like testbed.init_blobstore_stub() is outdated, because dev_appserver inits blobstore stubs differently. Here is my implementation of init_blobstore_stub that allows you to write to and read from blobstore in your tests.
from google.appengine.ext import testbed
from google.appengine.api.blobstore import blobstore_stub, file_blob_storage
from google.appengine.api.files import file_service_stub
class TestbedWithFiles(testbed.Testbed):
def init_blobstore_stub(self):
blob_storage = file_blob_storage.FileBlobStorage('/tmp/testbed.blobstore',
testbed.DEFAULT_APP_ID)
blob_stub = blobstore_stub.BlobstoreServiceStub(blob_storage)
file_stub = file_service_stub.FileServiceStub(blob_storage)
self._register_stub('blobstore', blob_stub)
self._register_stub('file', file_stub)
# Your code...
def foo():
from google.appengine.api import files
blob_filename = files.blobstore.create(mime_type='text/plain')
with files.open(blob_filename, 'a') as googfile:
googfile.write("Test data")
files.finalize(blob_filename)
tb = TestbedWithFiles()
tb.activate()
tb.init_blobstore_stub()
foo()

I don't know if it was added later to the SDK, but using Testbed.init_files_stub should fix it:
tb = testbed.Testbed()
tb.activate()
tb.init_blobstore_stub()
tb.init_files_stub()

Any chance that you are trying to do this using the gaeunit.py test runner? I see the same error while using that, since it does it's own code to replace the api proxy.
The error disappears when I added 'file' to the "as-is" list of proxies in the _run_test_suite function of gaeunit.py.
Honestly, I'm not sure that the gaeunit.py proxy replacement code is needed at all since I'm also using the more recently recommended testbed code in the test cases as per http://code.google.com/appengine/docs/python/tools/localunittesting.html. So, at this point I've commented it all out of gaeunit.py, which also seems to be working.
Note that I'm doing all this on a dev server only, in highly experimental mode on python27 in GAE with Python 2.7.
Hope this helps.

Related

How to extend Django's test database with custom SQL view?

I'm using Django 2.1 with MySQL.
I have one custom SQL view, which is bound with a model with Meta managed = False. Django's TestCase has no idea how the view is created, so I'd like to provide SQL command to create this view. The best option would be to do this on database create, but I have no idea how to do that.
What I've done so far was to override TestCase's setUp method. It looks like that:
class TaskDoneViewTest(TestCase):
def setUp(self):
"""
Create custom SQL view
"""
cursor = connection.cursor()
file_handle = open('app/tests/create_sql_view.sql', 'r+')
sql_file = File(file_handle)
sql = sql_file.read()
cursor.execute(sql)
cursor.close()
def test_repeatable_task_done(self):
# ...
def test_one_time_task_done(self):
# ...
I've got this solution from similar SO post: How to use database view in test cases. It would be a nice temporary solution, but the problem is with all those 2 test cases active I'm getting following error:
$ python manage.py test app.tests
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...E..
======================================================================
ERROR: test_repeatable_task_done (app.tests.test_views.TaskDoneViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/asmox/AppDev/Python/bubblechecklist/project_root/app/tests/test_views.py", line 80, in setUp
cursor.execute(sql)
File "/home/asmox/AppDev/Python/bubblechecklist/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/asmox/AppDev/Python/bubblechecklist/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/asmox/AppDev/Python/bubblechecklist/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 80, in _execute
self.db.validate_no_broken_transaction()
File "/home/asmox/AppDev/Python/bubblechecklist/env/lib/python3.6/site-packages/django/db/backends/base/base.py", line 437, in validate_no_broken_transaction
"An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
For some reason this error doesn't happen when I have only one test case active (why?).This error remains until I change my test's base class from TestCase to TransactionTestCase.
Well, I would ask why this happen and if there is any solution to get it okay with simple TestCase class, because my test for now has nothing to do with transactions and I find this working solutions a bit too dirty, but...
I would more likely stick to the main issue, that is, to globally (for all my test cases) do the following thing:
When testing database is created, do one more custom SQL from provided file. It is going to create required view
Can you please help me how to do that?
If you read the documentation for TestCase, you'll see that it wraps each test in a double transaction, one at the class level and one at the test level. The setUp() method runs for each test and is thus inside this double wrapping.
As shown in the above mentioned docs, it is suggested you use setUpTestData() to set up your db at the class level. This is also where you'd add initial data to your db for all your tests to use.

httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com

When i try to fetch the details of gmail using google-api's (googleapiclient,oauth2client), i am getting below error:
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/google_api_python_client-1.7.8-py2.7.egg/googleapiclient/_helpers.py", line 130, in positional_wrapper
File "/usr/local/lib/python2.7/site-packages/google_api_python_client-1.7.8-py2.7.egg/googleapiclient/discovery.py", line 224, in build
File "/usr/local/lib/python2.7/site-packages/google_api_python_client-1.7.8-py2.7.egg/googleapiclient/discovery.py", line 274, in _retrieve_discovery_doc
File "/usr/local/lib/python2.7/site-packages/httplib2-0.12.1-py2.7.egg/httplib2/__init__.py", line 2135, in request
cachekey,
File "/usr/local/lib/python2.7/site-packages/httplib2-0.12.1-py2.7.egg/httplib2/__init__.py", line 1796, in _request
conn, request_uri, method, body, headers
File "/usr/local/lib/python2.7/site-packages/httplib2-0.12.1-py2.7.egg/httplib2/__init__.py", line 1707, in _conn_request
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com
but it is working fine in my pc but not from remote location.
code:
from googleapiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
credentials = ServiceAccountCredentials.from_json_keyfile_name(
"quickstart-1551349397232-e8bcb3368ae1.json", scopes=
['https://www.googleapis.com/auth/admin.directory.group', 'https://www.googleapis.com/auth/admin.directory.user', 'https://www.googleapis.com/auth/admin.directory.domain', 'https://www.googleapis.com/auth/gmail.readonly'])
delegated_credentials = credentials.create_delegated('jango#carbonitedepot.com')
DIRECOTORY = build('admin', 'directory_v1', credentials=delegated_credentials)
try:
results = DIRECOTORY.users().list(customer='my_customer').execute()
users = results.get('users', [])
res = []
for info in users:
print(info)
res.append(info.get("primaryEmail"))
print(res)
except Exception as e:
print(e)
Any help would be much appreciated.
Thanks in advance.
I had the same issue and I searched a lot to get it fixed, turns out I was at fault here.
I don't know from where there was podman installed instead of docker and that caused the problem.
I suggest you check your docker version and make sure the latest is running at the server, otherwise, the library works fine! Please let me know if you still face this issue, would like to dig deep!
The httplib2 is likely scanning through network interface DHCP DNS nameservers (in your registry or visible to docker) and then trying to connect through a stale DNS or possible an IPv6. Patching your socket in the main routine may solve it:
# Monkey patch to force IPv4, since FB seems to hang on IPv6
import socket
old_getaddrinfo = socket.getaddrinfo
def new_getaddrinfo(*args, **kwargs):
responses = old_getaddrinfo(*args, **kwargs)
return [response
for response in responses
if response[0] == socket.AF_INET]
socket.getaddrinfo = new_getaddrinfo

Encountering remote error using grpc with protobuf2.6 in python

I am using grpc with protobuf 2.6.1 in python 2.7, and when I run my client side code, I have the following errors:
Traceback (most recent call last):
File "debate_client.py", line 31, in <module>
run_client()
File "debate_client.py", line 17, in run_client
reply = stub.Answer(debate_pb2.AnswerRequest(question=question, timeout=timeout), 30)
File "/Users/elaine/Desktop/gitHub/grpc/python2.7_virtual_environment/lib/python2.7/site-packages/grpc/framework/crust/implementations.py", line 73, in __call__
protocol_options, metadata, request)
File "/Users/elaine/Desktop/gitHub/grpc/python2.7_virtual_environment/lib/python2.7/site-packages/grpc/framework/crust/_calls.py", line 109, in blocking_unary_unary
return next(rendezvous)
File "/Users/elaine/Desktop/gitHub/grpc/python2.7_virtual_environment/lib/python2.7/site-packages/grpc/framework/crust/_control.py", line 412, in next
raise self._termination.abortion_error
grpc.framework.interfaces.face.face.RemoteError: RemoteError(code=StatusCode.UNKNOWN, details="")
Here is my client side code:
from grpc.beta import implementations
import debate_pb2
import sys
def run_client():
params = sys.argv
print params
how = params[1]
question = params[2]
channel = implementations.insecure_channel('localhost', 29999)
stub = debate_pb2.beta_create_Candidate_stub(channel)
if how.lower() == "answer":
timeout = int(params[3])
reply = stub.Answer(debate_pb2.AnswerRequest(question=question, timeout=timeout), 30)
elif how.lower() == "elaborate":
blah = params[3:len(sys.argv)]
for i in range(0, len(blah)):
blah[i] = int(blah[i])
reply = stub.Elaborate(debate_pb2.ElaborateRequest(topic=question, blah_run=blah), 30)
if reply is None:
print "No comment"
else:
print reply.answer
if __name__ == "__main__":
run_client()
And here is my server side code:
import debate_pb2
import consultation_pb2
import re
import random
from grpc.beta import implementations
class Debate(debate_pb2.BetaCandidateServicer):
def Answer(self, request, context=None):
#Answer implementation
def Elaborate(self, request, context=None):
#Elaborate implementation
def run_server():
server = debate_pb2.beta_create_Candidate_server(Debate())
server.add_insecure_port('localhost:29999')
server.start()
if __name__ == "__main__":
run_server()
Any idea where the remote error comes from? Thank you so much!
Hello Elaine and thank you for trying out gRPC Python.
Nothing leaps out at me as an obvious smoking gun, but a couple of things I see are:
gRPC Python isn't known to work with protobuf 2.6.1. Have you tried working with the very latest protobuf release (3.0.0a3 at this time)?
context isn't an optional keyword parameter in servicer methods; it's a required, positional parameter. Does dropping =None from your servicer method implementations effect any change?
The same happened to me just now, and I figured out why.
Make sure the messages in your proto definition and the message in your implementations match the format.
For example, my Response message had a message= param in my python server, but not in my proto definition.
I think your function implementations should be outside class Debate or might be your functions are not correctly implemented to give the desired result.
I faced a similar error because my functions were inside the class but moving it outside the class fixed it.

Google Python Admin SDK using Oauth2 for a Service Account (Education Edition)-"oauth2client.client.AccessTokenRefreshError: access_denied" exception

I have been trying to get the Service Account authentication working for the Google Admin SDK for a few days now to no avail. I am using the google-api-python-client-1.2 library freshly installed from Google.
I have been following Google's documentation on the topic. Links are here:
htps://developers.google.com/accounts/docs/OAuth2ServiceAccount
htps://developers.google.com/api-client-library/python/guide/aaa_oauth
htp://google-api-python-client.googlecode.com/hg/docs/epy/oauth2client.client.SignedJwtAssertionCredentials-class.html
And have the tasks.py Service Account example working which you can be found here:
htp://code.google.com/p/google-api-python-client/source/browse/samples/service_account/tasks.py?r=c21573904a2df1334d13b4380f63463c94c8d0e8
And have been closely studying these two Stack Overflow threads on a related topic here:
google admin sdk directory api 403 python
Google Admin API using Oauth2 for a Service Account (Education Edition) - 403 Error
And have studied the relevant code in gam.py (Dito GAM).
Yet I'm still missing something as I am getting an 'oauth2client.client.AccessTokenRefreshError: access_denied' exception in nearly every test case I write.
Here is a concise example of a test authentication:
import httplib2
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
f = file('myKey.p12', 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials(
'myServiceAdmin#developer.gserviceaccount.com',
key,
sub='myAdminUser#my.googleDomain.edu',
scope = ['https://www.googleapis.com/auth/admin.directory.user',])
http = httplib2.Http()
http = credentials.authorize(http)
service = build('admin', 'directory_v1', http=http)
When I run the above code I get this stack dump and exception:
Traceback (most recent call last):
File "./test.py", line 17, in <module>
service = build('admin', 'directory_v1', http=http)
File "/usr/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/apiclient/discovery.py", line 192, in build resp, content = http.request(requested_url)
File "/usr/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/oauth2client/client.py", line 475, in new_request
self._refresh(request_orig)
File "/usr/lib/python2.7/dist-packages/oauth2client/client.py", line 653, in _refresh
self._do_refresh_request(http_request)
File "/usr/lib/python2.7/dist-packages/oauth2client/client.py", line 710, in _do_refresh_request
raise AccessTokenRefreshError(error_msg)
oauth2client.client.AccessTokenRefreshError: access_denied
I've tried multiple super user accounts, service accounts, and keys and always end up with the same exception. If I add sub to the tasks.py example I end up with the same error. Replacing sub with prn also generates this exception and adding private_key_password='notasecret' does nothing (it is the default). The Admin SDK is activated in the Google Developers Console and the target accounts have super user privileges. This makes me think something is missing on the Google domain side but I cannot think of anything else to check.
Any one have an idea what I am doing wrong?
Have you granted the third party client access in your Admin Console for your service account?
My to go instruction when it comes to setting up Service Account is the instruction Google has for Drive Api.
https://developers.google.com/drive/web/delegation
Take a look at the "Delegate domain-wide authority to your service account" part and see if you have completed those steps.
Maybe not OP's problem, but I had this same error and my issue was that I was setting the sub field in the credentials object
credentials = SignedJwtAssertionCredentials(SERVICE_ACCOUNT_EMAIL, key,
scope=SCOPES, sub=**<DON'T SET ME>**)
If you're using domain-wide delegation, you need to not set a sub (because your "user" is the domain administrator.) The docs are a bit confusing on this point. I just removed the sub field and it worked for me.

Unit testing in Web2py

I'm following the instructions from this post but cannot get my methods recognized globally.
The error message:
ERROR: test_suggest_performer (__builtin__.TestSearch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "applications/myapp/tests/test_search.py", line 24, in test_suggest_performer
suggs = suggest_flavors("straw")
NameError: global name 'suggest_flavors' is not defined
My test file:
import unittest
from gluon.globals import Request
db = test_db
execfile("applications/myapp/controllers/search.py", globals())
class TestSearch(unittest.TestCase):
def setUp(self):
request = Request()
def test_suggest_flavors(self):
suggs = suggest_flavors("straw")
self.assertEqual(len(suggs), 1)
self.assertEqual(suggs[0][1], 'Strawberry')
My controller:
def suggest_flavors(term):
return []
Has anyone successfully completed unit testing like this in web2py?
Please see: http://web2py.com/AlterEgo/default/show/260
Note that in your example the function 'suggest_flavors' should be defined at 'applications/myapp/controllers/search.py'.
I don't have any experience with web2py, but used other frameworks a lot. And looking at your code I'm confused a bit. Is there an objective reason why execfile should be used? Isn't it better to use regular import statement. So instead of execfile you may write:
from applications.myapp.controllers.search import suggest_flavors
It's more clear code for pythoners.
Note, that you should place __init__.py in each directory along the path in this case, so that dirs will form package/module hierarchy.