I am using grpc with protobuf 2.6.1 in python 2.7, and when I run my client side code, I have the following errors:
Traceback (most recent call last):
File "debate_client.py", line 31, in <module>
run_client()
File "debate_client.py", line 17, in run_client
reply = stub.Answer(debate_pb2.AnswerRequest(question=question, timeout=timeout), 30)
File "/Users/elaine/Desktop/gitHub/grpc/python2.7_virtual_environment/lib/python2.7/site-packages/grpc/framework/crust/implementations.py", line 73, in __call__
protocol_options, metadata, request)
File "/Users/elaine/Desktop/gitHub/grpc/python2.7_virtual_environment/lib/python2.7/site-packages/grpc/framework/crust/_calls.py", line 109, in blocking_unary_unary
return next(rendezvous)
File "/Users/elaine/Desktop/gitHub/grpc/python2.7_virtual_environment/lib/python2.7/site-packages/grpc/framework/crust/_control.py", line 412, in next
raise self._termination.abortion_error
grpc.framework.interfaces.face.face.RemoteError: RemoteError(code=StatusCode.UNKNOWN, details="")
Here is my client side code:
from grpc.beta import implementations
import debate_pb2
import sys
def run_client():
params = sys.argv
print params
how = params[1]
question = params[2]
channel = implementations.insecure_channel('localhost', 29999)
stub = debate_pb2.beta_create_Candidate_stub(channel)
if how.lower() == "answer":
timeout = int(params[3])
reply = stub.Answer(debate_pb2.AnswerRequest(question=question, timeout=timeout), 30)
elif how.lower() == "elaborate":
blah = params[3:len(sys.argv)]
for i in range(0, len(blah)):
blah[i] = int(blah[i])
reply = stub.Elaborate(debate_pb2.ElaborateRequest(topic=question, blah_run=blah), 30)
if reply is None:
print "No comment"
else:
print reply.answer
if __name__ == "__main__":
run_client()
And here is my server side code:
import debate_pb2
import consultation_pb2
import re
import random
from grpc.beta import implementations
class Debate(debate_pb2.BetaCandidateServicer):
def Answer(self, request, context=None):
#Answer implementation
def Elaborate(self, request, context=None):
#Elaborate implementation
def run_server():
server = debate_pb2.beta_create_Candidate_server(Debate())
server.add_insecure_port('localhost:29999')
server.start()
if __name__ == "__main__":
run_server()
Any idea where the remote error comes from? Thank you so much!
Hello Elaine and thank you for trying out gRPC Python.
Nothing leaps out at me as an obvious smoking gun, but a couple of things I see are:
gRPC Python isn't known to work with protobuf 2.6.1. Have you tried working with the very latest protobuf release (3.0.0a3 at this time)?
context isn't an optional keyword parameter in servicer methods; it's a required, positional parameter. Does dropping =None from your servicer method implementations effect any change?
The same happened to me just now, and I figured out why.
Make sure the messages in your proto definition and the message in your implementations match the format.
For example, my Response message had a message= param in my python server, but not in my proto definition.
I think your function implementations should be outside class Debate or might be your functions are not correctly implemented to give the desired result.
I faced a similar error because my functions were inside the class but moving it outside the class fixed it.
Related
I have to monitor an XML file being written by a tool running all the day. But the XML file is properly completed and closed only at the end of the day.
Same constraints as XML stream processing:
Parse an incomplete XML file on-the-fly and trigger actions
Keep track of the last position within the file to avoid processing it again from the beginning
On answer of Need to read XML files as a stream using BeautifulSoup in Python, slezica suggests xml.sax, xml.etree.ElementTree and cElementTree. But no success with my attempts to use xml.etree.ElementTree and cElementTree. There are also xml.dom, xml.parsers.expat and lxml but I do not see support for "on-the-fly parsing".
I need more obvious examples...
I am currently using Python 2.7 on Linux, but I will migrate to Python 3.x => please also provide tips on new Python 3.x features. I also use watchdog to detect XML file modifications => Optionally, reuse the watchdog mechanism. Optionally support also Windows.
Please provide easy to understand/maintain solutions. If it is too complex, I may just use tell()/seek() to move within the file, use stupid text search in the raw XML and finally extract the values using basic regex.
XML sample:
<dfxml xmloutputversion='1.0'>
<creator version='1.0'>
<program>TCPFLOW</program>
<version>1.4.6</version>
</creator>
<configuration>
<fileobject>
<filename>file1</filename>
<filesize>288</filesize>
<tcpflow packets='12' srcport='1111' dstport='2222' family='2' />
</fileobject>
<fileobject>
<filename>file2</filename>
<filesize>352</filesize>
<tcpflow packets='12' srcport='3333' dstport='4444' family='2' />
</fileobject>
<fileobject>
<filename>file3</filename>
<filesize>456</filesize>
...
...
First test using SAX failed:
import xml.sax
class StreamHandler(xml.sax.handler.ContentHandler):
def startElement(self, name, attrs):
print 'start: name=', name
def endElement(self, name):
print 'end: name=', name
if name == 'root':
raise StopIteration
if __name__ == '__main__':
parser = xml.sax.make_parser()
parser.setContentHandler(StreamHandler())
with open('f.xml') as f:
parser.parse(f)
Shell:
$ while read line; do echo $line; sleep 1; done <i.xml >f.xml &
...
$ ./test-using-sax.py
start: name= dfxml
start: name= creator
start: name= program
end: name= program
start: name= version
end: name= version
Traceback (most recent call last):
File "./test-using-sax.py", line 17, in <module>
parser.parse(f)
File "/usr/lib64/python2.7/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib64/python2.7/xml/sax/xmlreader.py", line 125, in parse
self.close()
File "/usr/lib64/python2.7/xml/sax/expatreader.py", line 220, in close
self.feed("", isFinal = 1)
File "/usr/lib64/python2.7/xml/sax/expatreader.py", line 214, in feed
self._err_handler.fatalError(exc)
File "/usr/lib64/python2.7/xml/sax/handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: report.xml:15:0: no element found
Since yesterday I found the Peter Gibson's answer about the undocumented xml.etree.ElementTree.XMLTreeBuilder._parser.EndElementHandler.
This example is similar to the other one but uses xml.etree.ElementTree (and watchdog).
It does not work when ElementTree is replaced by cElementTree :-/
import time
import watchdog.events
import watchdog.observers
import xml.etree.ElementTree
class XmlFileEventHandler(watchdog.events.PatternMatchingEventHandler):
def __init__(self):
watchdog.events.PatternMatchingEventHandler.__init__(self, patterns=['*.xml'])
self.xml_file = None
self.parser = xml.etree.ElementTree.XMLTreeBuilder()
def end_tag_event(tag):
node = self.parser._end(tag)
print 'tag=', tag, 'node=', node
self.parser._parser.EndElementHandler = end_tag_event
def on_modified(self, event):
if not self.xml_file:
self.xml_file = open(event.src_path)
buffer = self.xml_file.read()
if buffer:
self.parser.feed(buffer)
if __name__ == '__main__':
observer = watchdog.observers.Observer()
event_handler = XmlFileEventHandler()
observer.schedule(event_handler, path='.')
try:
observer.start()
while True:
time.sleep(10)
finally:
observer.stop()
observer.join()
While the script is running, do not forget to touch one XML file, or simulate the on-the-fly writing using this one line script:
while read line; do echo $line; sleep 1; done <in.xml >out.xml &
For information, the xml.etree.ElementTree.iterparse does not seem to support a file being written. My test code:
from __future__ import print_function, division
import xml.etree.ElementTree
if __name__ == '__main__':
context = xml.etree.ElementTree.iterparse('f.xml', events=('end',))
for action, elem in context:
print(action, elem.tag)
My output:
end program
end version
end creator
end filename
end filesize
end tcpflow
end fileobject
end filename
end filesize
end tcpflow
end fileobject
end filename
end filesize
Traceback (most recent call last):
File "./iter.py", line 9, in <module>
for action, elem in context:
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1281, in next
self._root = self._parser.close()
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1654, in close
self._raiseerror(v)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
raise err
xml.etree.ElementTree.ParseError: no element found: line 20, column 0
Three hours after posting my question, no answer received. But I have finally implemented the simple example I was looking for.
My inspiration is from saaj's answer and is based on xml.sax and watchdog.
from __future__ import print_function, division
import time
import watchdog.events
import watchdog.observers
import xml.sax
class XmlStreamHandler(xml.sax.handler.ContentHandler):
def startElement(self, tag, attributes):
print(tag, 'attributes=', attributes.items())
self.tag = tag
def characters(self, content):
print(self.tag, 'content=', content)
class XmlFileEventHandler(watchdog.events.PatternMatchingEventHandler):
def __init__(self):
watchdog.events.PatternMatchingEventHandler.__init__(self, patterns=['*.xml'])
self.file = None
self.parser = xml.sax.make_parser()
self.parser.setContentHandler(XmlStreamHandler())
def on_modified(self, event):
if not self.file:
self.file = open(event.src_path)
self.parser.feed(self.file.read())
if __name__ == '__main__':
observer = watchdog.observers.Observer()
event_handler = XmlFileEventHandler()
observer.schedule(event_handler, path='.')
try:
observer.start()
while True:
time.sleep(10)
finally:
observer.stop()
observer.join()
While the script is running, do not forget to touch one XML file, or simulate the on-the-fly writing using the following command:
while read line; do echo $line; sleep 1; done <in.xml >out.xml &
I'm running unittests in the callbacks for motor database calls, and I'm successfully catching AssertionErrors and having them surface when running nosetests, but the AssertionErrors are being caught in the wrong test. The tracebacks are to different files.
My unittests look generally like this:
def test_create(self):
#self.callback
def create_callback(result, error):
self.assertIs(error, None)
self.assertIsNot(result, None)
question_db.create(QUESTION, create_callback)
self.wait()
And the unittest.TestCase class I'm using looks like this:
class MotorTest(unittest.TestCase):
bucket = Queue.Queue()
# Ensure IOLoop stops to prevent blocking tests
def callback(self, func):
def wrapper(*args, **kwargs):
try:
func(*args, **kwargs)
except Exception as e:
self.bucket.put(traceback.format_exc())
IOLoop.current().stop()
return wrapper
def wait(self):
IOLoop.current().start()
try:
raise AssertionError(self.bucket.get(block = False))
except Queue.Empty:
pass
The errors I'm seeing:
======================================================================
FAIL: test_sync_user (app.tests.db.test_user_db.UserDBTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/----/Documents/app/app-Server/app/tests/db/test_user_db.py", line 39, in test_sync_user
self.wait()
File "/Users/----/Documents/app/app-Server/app/tests/testutils/mongo.py", line 25, in wait
raise AssertionError(self.bucket.get(block = False))
AssertionError: Traceback (most recent call last):
File "/Users/----/Documents/app/app-Server/app/tests/testutils/mongo.py", line 16, in wrapper
func(*args, **kwargs)
File "/Users/----/Documents/app/app-Server/app/tests/db/test_question_db.py", line 32, in update_callback
self.assertEqual(result["question"], "updated question?")
TypeError: 'NoneType' object has no attribute '__getitem__'
Where the error is reported to be in UsersDbTest but is clearly in test_questions_db.py (which is QuestionsDbTest)
I'm having issues with nosetests and asynchronous tests in general, so if anyone has any advice on that, it'd be greatly appreciated as well.
I can't fully understand your code without an SSCCE, but I'd say you're taking an unwise approach to async testing in general.
The particular problem you face is that you don't wait for your test to complete (asynchronously) before leaving the test function, so there's work still pending in the IOLoop when you resume the loop in your next test. Use Tornado's own "testing" module -- it provides convenient methods for starting and stopping the loop, and it recreates the loop between tests so you don't experience interference like what you're reporting. Finally, it has extremely convenient means of testing coroutines.
For example:
import unittest
from tornado.testing import AsyncTestCase, gen_test
import motor
# AsyncTestCase creates a new loop for each test, avoiding interference
# between tests.
class Test(AsyncTestCase):
def callback(self, result, error):
# Translate from Motor callbacks' (result, error) convention to the
# single arg expected by "stop".
self.stop((result, error))
def test_with_a_callback(self):
client = motor.MotorClient()
collection = client.test.collection
collection.remove(callback=self.callback)
# AsyncTestCase starts the loop, runs until "remove" calls "stop".
self.wait()
collection.insert({'_id': 123}, callback=self.callback)
# Arguments passed to self.stop appear as return value of "self.wait".
_id, error = self.wait()
self.assertIsNone(error)
self.assertEqual(123, _id)
collection.count(callback=self.callback)
cnt, error = self.wait()
self.assertIsNone(error)
self.assertEqual(1, cnt)
#gen_test
def test_with_a_coroutine(self):
client = motor.MotorClient()
collection = client.test.collection
yield collection.remove()
_id = yield collection.insert({'_id': 123})
self.assertEqual(123, _id)
cnt = yield collection.count()
self.assertEqual(1, cnt)
if __name__ == '__main__':
unittest.main()
(In this example I create a new MotorClient for each test, which is a good idea when testing applications that use Motor. Your actual application must not create a new MotorClient for each operation. For decent performance you must create one MotorClient when your application begins, and use that same one client throughout the process's lifetime.)
Take a look at the testing module, and particularly the gen_test decorator:
http://tornado.readthedocs.org/en/latest/testing.html
These test conveniences take care of many details related to unittesting Tornado applications.
I gave a talk and wrote an article about testing in Tornado, there's more info here:
http://emptysqua.re/blog/eventually-correct-links/
I'm trying to use the Python package ComplexNetworkSim, which inherits from networkx and SimPy, to simulate an agent-based model of how messages propagate within networks.
Here is my code:
from ComplexNetworkSim import NetworkSimulation, NetworkAgent, Sim
import networkx as nx
#define constants for our example of states
NO_MESSAGE = 0
MESSAGE = 1
class Message(object):
def __init__(self,topic_pref):
self.relevance = topic_pref
class myAgent(NetworkAgent):
def __init__(self, state, initialiser):
NetworkAgent.__init__(self, state, initialiser)
self.state = MESSAGE
self.topic_pref = 0.5
def Run(self):
while True:
if self.state == MESSAGE:
self.message = self.Message(topic_pref, self, TIMESTEP)
yield Sim.hold, self, NetworkAgent.TIMESTEP_DEFAULT
elif self.state == NO_MESSAGE:
yield Sim.hold, self, NetworkAgent.TIMESTEP_DEFAULT
# Network and initial states of agents
nodes = 30
G = nx.scale_free_graph(nodes)
states = [MESSAGE for n in G.nodes()]
# Simulation constants
MAX_SIMULATION_TIME = 25.0
TRIALS = 2
def main():
directory = 'test' #output directory
# run simulation with parameters
# - complex network structure
# - initial state list
# - agent behaviour class
# - output directory
# - maximum simulation time
# - number of trials
simulation = NetworkSimulation(G,
states,
myAgent,
directory,
MAX_SIMULATION_TIME,
TRIALS)
simulation.runSimulation()
if __name__ == '__main__':
main()
(There may be other problems downstream with this code and it is not fully tested.)
My problem is that the myAgent object is not properly calling the method Run as an attribute. Specifically, this is the error message that I get when I try to run the above code:
Starting simulations...
---Trial 0 ---
set up agents...
Traceback (most recent call last):
File "simmessage.py", line 55, in <module>
main()
File "simmessage.py", line 52, in main
simulation.runSimulation()
File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/ComplexNetworkSim-0.1.2-py2.7.egg/ComplexNetworkSim/simulation.py", line 71, in runSimulation
self.runTrial(i)
File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/ComplexNetworkSim-0.1.2-py2.7.egg/ComplexNetworkSim/simulation.py", line 88, in runTrial
self.activate(agent, agent.Run())
AttributeError: 'myAgent' object has no attribute 'Run'
Does anybody know why this is? I can't figure how my code differs substantially from the example in ComplexNetworkSim.
I've run your code on my machine and there the Run method gets called.
My best guess is what Paulo Scardine wrote, but since i can't reproduce the problem i can't actually debug it.
I have a function that uses the Google Blobstore API, and here's a degenerate case:
#!/usr/bin/python
from google.appengine.ext import testbed
def foo():
from google.appengine.api import files
blob_filename = files.blobstore.create(mime_type='text/plain')
with files.open(blob_filename, 'a') as googfile:
googfile.write("Test data")
files.finalize(blob_filename)
tb = testbed.Testbed()
tb.activate()
tb.init_blobstore_stub()
foo() # in reality, I'm a function called from a 'faux client'
# in a unittest testcase.
The error this generates is:
Traceback (most recent call last):
File "e.py", line 18, in
foo() # in reality, I'm a function called from a 'faux client'
File "e.py", line 8, in foo
blob_filename = files.blobstore.create(mime_type='text/plain')
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/blobstore.py", line 68, in create
return files._create(_BLOBSTORE_FILESYSTEM, params=params)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/file.py", line 491, in _create
_make_call('Create', request, response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/file.py", line 230, in _make_call
rpc = _create_rpc(deadline=deadline)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/files/file.py", line 213, in _create_rpc
return apiproxy_stub_map.UserRPC('file', deadline)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 393, in __init__
self.__rpc = CreateRPC(service, stubmap)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 67, in CreateRPC
assert stub, 'No api proxy found for service "%s"' % service
AssertionError: No api proxy found for service "file"
I don't want to have to modify foo in order to be able to test it. Is there a way to make foo work as expected (i.e. create the given file) in Google App Engine's unit tests?
I would expect to be able to do this with Google's API Proxy, but I don't understand it well enough to figure it out on my own.
I'd be grateful for your thoughts and suggestions.
Thanks for reading.
It seems like testbed.init_blobstore_stub() is outdated, because dev_appserver inits blobstore stubs differently. Here is my implementation of init_blobstore_stub that allows you to write to and read from blobstore in your tests.
from google.appengine.ext import testbed
from google.appengine.api.blobstore import blobstore_stub, file_blob_storage
from google.appengine.api.files import file_service_stub
class TestbedWithFiles(testbed.Testbed):
def init_blobstore_stub(self):
blob_storage = file_blob_storage.FileBlobStorage('/tmp/testbed.blobstore',
testbed.DEFAULT_APP_ID)
blob_stub = blobstore_stub.BlobstoreServiceStub(blob_storage)
file_stub = file_service_stub.FileServiceStub(blob_storage)
self._register_stub('blobstore', blob_stub)
self._register_stub('file', file_stub)
# Your code...
def foo():
from google.appengine.api import files
blob_filename = files.blobstore.create(mime_type='text/plain')
with files.open(blob_filename, 'a') as googfile:
googfile.write("Test data")
files.finalize(blob_filename)
tb = TestbedWithFiles()
tb.activate()
tb.init_blobstore_stub()
foo()
I don't know if it was added later to the SDK, but using Testbed.init_files_stub should fix it:
tb = testbed.Testbed()
tb.activate()
tb.init_blobstore_stub()
tb.init_files_stub()
Any chance that you are trying to do this using the gaeunit.py test runner? I see the same error while using that, since it does it's own code to replace the api proxy.
The error disappears when I added 'file' to the "as-is" list of proxies in the _run_test_suite function of gaeunit.py.
Honestly, I'm not sure that the gaeunit.py proxy replacement code is needed at all since I'm also using the more recently recommended testbed code in the test cases as per http://code.google.com/appengine/docs/python/tools/localunittesting.html. So, at this point I've commented it all out of gaeunit.py, which also seems to be working.
Note that I'm doing all this on a dev server only, in highly experimental mode on python27 in GAE with Python 2.7.
Hope this helps.
I'm following the instructions from this post but cannot get my methods recognized globally.
The error message:
ERROR: test_suggest_performer (__builtin__.TestSearch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "applications/myapp/tests/test_search.py", line 24, in test_suggest_performer
suggs = suggest_flavors("straw")
NameError: global name 'suggest_flavors' is not defined
My test file:
import unittest
from gluon.globals import Request
db = test_db
execfile("applications/myapp/controllers/search.py", globals())
class TestSearch(unittest.TestCase):
def setUp(self):
request = Request()
def test_suggest_flavors(self):
suggs = suggest_flavors("straw")
self.assertEqual(len(suggs), 1)
self.assertEqual(suggs[0][1], 'Strawberry')
My controller:
def suggest_flavors(term):
return []
Has anyone successfully completed unit testing like this in web2py?
Please see: http://web2py.com/AlterEgo/default/show/260
Note that in your example the function 'suggest_flavors' should be defined at 'applications/myapp/controllers/search.py'.
I don't have any experience with web2py, but used other frameworks a lot. And looking at your code I'm confused a bit. Is there an objective reason why execfile should be used? Isn't it better to use regular import statement. So instead of execfile you may write:
from applications.myapp.controllers.search import suggest_flavors
It's more clear code for pythoners.
Note, that you should place __init__.py in each directory along the path in this case, so that dirs will form package/module hierarchy.