As per last question about invoking remote RPCs, an additional example to the one included in the documentation would be this one:
$ cat nameko.sh
#!/bin/bash
/usr/local/bin/nameko run service1:Microservice1 &
nameko_id=$!
echo 'Microservice 1 PID: ' $nameko_id
/usr/local/bin/nameko run service2:Microservice2 &
nameko_id=$!
echo 'Microservice 2 PID: ' $nameko_id
wait 2> /dev/null
$ cat service1.py
# -*- coding: utf-8 -*-
from nameko.rpc import rpc, RpcProxy
class Microservice1(object):
name = "microservice1"
#rpc
def hello(self):
print 'Microservice1 hello method invoked'
return True
$ cat service2.py
# -*- coding: utf-8 -*-
from nameko.rpc import rpc, RpcProxy
class Microservice2(object):
name = "microservice2"
microservice1 = RpcProxy('microservice1')
#rpc
def remote_hello(self):
print 'Microservice2 invokes hello method from Microservice1'
self.microservice1.hello()
return True
I am trying to build an architecture where microservices register themselves against a central microservice on startup (basically this central microservice is in charge of displaying a REST API, where each microservice deal with their part of the REST API -this is done to avoid the usage of a reverse proxy and deal with the port numbers-).
In Nameko, how could you launch a remote procedure just when the microservice is registered? As mentioned in the above post's answers, remote RPC invoke can not be done outside a #rpc method.
Actually stand alone proxy as answered here is the way to achive this:
Nameko - invoking RPC method from another service
use the once entrypoint
from nameko.testing.service import once
class Microservice2(object):
name = "microservice2"
microservice1 = RpcProxy('microservice1')
#rpc
def remote_hello(self):
print 'Microservice2 invokes hello method from Microservice1'
self.microservice1.hello()
return True
Related
I am trying to understand how nameko works for basic RPC.
I am looking to define microservices in separate files and being able to run them from command shell. With this structure service2 is not being able to invoke service1's RPC method. What is missing to get this working?
I have the following file structure:
-rwxrwxr-x 1 user user 240 Dec 15 01:49 nameko.sh*
-rw-rw-r-- 1 user user 251 Dec 15 01:46 service1.py
-rw-rw-r-- 1 user user 305 Dec 15 01:47 service2.py
Content of files are:
$ cat nameko.sh
#!/bin/bash
/usr/local/bin/nameko run service1:Microservice1 &
nameko_id=$!
echo 'Microservice 1 PID: ' $nameko_id
/usr/local/bin/nameko run service2:Microservice2 &
nameko_id=$!
echo 'Microservice 2 PID: ' $nameko_id
wait 2> /dev/null
$ cat service1.py
# -*- coding: utf-8 -*-
from nameko.rpc import rpc, RpcProxy
class Microservice1(object):
name = "microservice1"
#rpc
def hello(self):
print 'Microservice1 hello method invoked'
return True
$ cat service2.py
# -*- coding: utf-8 -*-
from nameko.rpc import rpc, RpcProxy
class Microservice2(object):
name = "microservice2"
microservice1 = RpcProxy('microservice1')
microservice1.hello()
#rpc
def hello(self):
print 'Microservice2 hello method invoked'
return True
And I am not being able to understand how to invoke hello method in Microservice 1 from Microservice 2:
$ ./nameko.sh
Microservice 1 PID: 14782
Microservice 2 PID: 14783
Traceback (most recent call last):
File "/usr/local/bin/nameko", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/nameko/cli/main.py", line 66, in main
args.main(args)
File "/usr/local/lib/python2.7/dist-packages/nameko/cli/commands.py", line 85, in main
starting services: microservice1
main(args)
File "/usr/local/lib/python2.7/dist-packages/nameko/cli/run.py", line 179, in main
import_service(path)
File "/usr/local/lib/python2.7/dist-packages/nameko/cli/run.py", line 46, in import_service
__import__(module_name)
File "./service2.py", line 5, in <module>
class Microservice2(object):
File "./service2.py", line 11, in Microservice2
microservice1.hello()
AttributeError: 'RpcProxy' object has no attribute 'hello'
Connected to amqp://guest:**#127.0.0.1:5672//
But invoking nameko shell and running rpc method from microservice1 works:
Broker: pyamqp://guest:guest#localhost
>>> n.rpc.microservice1.hello()
True
Added information
Following Matt's answer I have edited service2.py as follows:
# -*- coding: utf-8 -*-
from nameko.rpc import rpc, RpcProxy
class Microservice2(object):
name = "microservice2"
#rpc
def toctoc(self):
print 'Microservice2 called hello method from Microservice1'
m1 = RpcProxy('microservice1')
m1.hello()
return True
Now both services run, But still does not work, when I run a namejo shell and invoke toctoc method from Microservice2:
$ nameko shell
Nameko Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] shell on linux2
Broker: pyamqp://guest:guest#localhost
>>> n.rpc.microservice2.toctoc()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/nameko/rpc.py", line 374, in
__call__
return reply.result()
File "/usr/local/lib/python2.7/dist-packages/nameko/rpc.py", line 332, in
result
raise deserialize(error)
RemoteError: AttributeError 'RpcProxy' object has no attribute 'hello'
>>>
Last working code
With the help provided in the answers, a working version of this would be (in case it clarifies/helps others):
$ cat nameko.sh
#!/bin/bash
/usr/local/bin/nameko run service1:Microservice1 &
nameko_id=$!
echo 'Microservice 1 PID: ' $nameko_id
/usr/local/bin/nameko run service2:Microservice2 &
nameko_id=$!
echo 'Microservice 2 PID: ' $nameko_id
wait 2> /dev/null
$ cat service1.py
# -*- coding: utf-8 -*-
from nameko.rpc import rpc, RpcProxy
class Microservice1(object):
name = "microservice1"
#rpc
def hello(self):
print 'Microservice1 hello method invoked'
return True
$ cat service2.py
# -*- coding: utf-8 -*-
from nameko.rpc import rpc, RpcProxy
class Microservice2(object):
name = "microservice2"
microservice1 = RpcProxy('microservice1')
#rpc
def remote_hello(self):
print 'Microservice2 invokes hello method from Microservice1'
self.microservice1.hello()
return True
there's an example of service-to-service rpc in the docs
class ServiceX:
name = "service_x"
# this _declares_ the dependency (and triggers nameko to set things up)
y = RpcProxy("service_y")
# note that the proxy isn't usable until a service worker is
# running, i.e. until you're inside an executing method
# y.foo() <- this doesn't work.
#rpc
def remote_method(self, value):
res = u"{}-x".format(value)
# this _invokes_ the dependency
return self.y.append_identifier(res)
The problem is that service2.py isn't valid. You can't invoke the RPC proxy outside of a service method.
If you want to call a running service from an external script, use the standalone proxy.
I want to build a python client to talk to my python Google Cloud Endpoints API. My simple HelloWorld example is suffering from an HTTPException in the python client and I can't figure out why.
I've setup simple examples as suggested in this extremely helpful thread. The GAE Endpoints API is running on localhost:8080 with no problems - I can successfully access it in the API Explorer. Before I added the offending service = build() line, my simple client ran fine on localhost:8080.
When trying to get the client to talk to the endpoints API, I get the following error:
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/dist27/gae_override/httplib.py", line 526, in getresponse
raise HTTPException(str(e))
HTTPException: Deadline exceeded while waiting for HTTP response from URL: http://localhost:8080/_ah/api/discovery/v1/apis/helloworldendpoints/v1/rest?userIp=%3A%3A1
I've tried extending the http deadline. Not only did that not help, but such a simple first call on localhost should not be exceeding a default 5s deadline. I've also tried accessing the discovery URL directly within a browser and that works fine, too.
Here is my simple code. First the client, main.py:
import webapp2
import os
import httplib2
from apiclient.discovery import build
http = httplib2.Http()
# HTTPException happens on the following line:
# Note that I am using http, not https
service = build("helloworldendpoints", "v1", http=http,
discoveryServiceUrl=("http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest"))
# result = service.resource().method([parameters]).execute()
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-type'] = 'text/plain'
self.response.out.write("Hey, this is working!")
app = webapp2.WSGIApplication(
[('/', MainPage)],
debug=True)
Here's the Hello World endpoint, helloworld.py:
"""Hello World API implemented using Google Cloud Endpoints.
Contains declarations of endpoint, endpoint methods,
as well as the ProtoRPC message class and container required
for endpoint method definition.
"""
import endpoints
from protorpc import messages
from protorpc import message_types
from protorpc import remote
# If the request contains path or querystring arguments,
# you cannot use a simple Message class.
# Instead, you must use a ResourceContainer class
REQUEST_CONTAINER = endpoints.ResourceContainer(
message_types.VoidMessage,
name=messages.StringField(1),
)
package = 'Hello'
class Hello(messages.Message):
"""String that stores a message."""
greeting = messages.StringField(1)
#endpoints.api(name='helloworldendpoints', version='v1')
class HelloWorldApi(remote.Service):
"""Helloworld API v1."""
#endpoints.method(message_types.VoidMessage, Hello,
path = "sayHello", http_method='GET', name = "sayHello")
def say_hello(self, request):
return Hello(greeting="Hello World")
#endpoints.method(REQUEST_CONTAINER, Hello,
path = "sayHelloByName", http_method='GET', name = "sayHelloByName")
def say_hello_by_name(self, request):
greet = "Hello {}".format(request.name)
return Hello(greeting=greet)
api = endpoints.api_server([HelloWorldApi])
Finally, here is my app.yaml file:
application: <<my web client id removed for stack overflow>>
version: 1
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: /_ah/spi/.*
script: helloworld.api
secure: always
# catchall - must come last!
- url: /.*
script: main.app
secure: always
libraries:
- name: endpoints
version: latest
- name: webapp2
version: latest
Why am I getting an HTTP Deadline Exceeded and how to I fix it?
On your main.py you forgot to add some variables to your discovery service url string, or you just copied the code here without it. By the looks of it you were probably suppose to use the format string method.
"http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest".format(api='helloworldendpoints', apiVersion="v1")
By looking at the logs you'll probably see something like this:
INFO 2015-11-19 18:44:51,562 module.py:794] default: "GET /HTTP/1.1" 500 -
INFO 2015-11-19 18:44:51,595 module.py:794] default: "POST /_ah/spi/BackendService.getApiConfigs HTTP/1.1" 200 3109
INFO 2015-11-19 18:44:52,110 module.py:794] default: "GET /_ah/api/discovery/v1/apis/helloworldendpoints/v1/rest?userIp=127.0.0.1 HTTP/1.1" 200 3719
It's timing out first and then "working".
Move the service discovery request inside the request handler:
class MainPage(webapp2.RequestHandler):
def get(self):
service = build("helloworldendpoints", "v1",
http=http,
discoveryServiceUrl=("http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest")
.format(api='helloworldendpoints', apiVersion='v1'))
# -*- coding: utf-8 -*-
# coding: utf-8
import sys
import os
import time
b = 'sudo tshark -i eth0 -R “tcp contains “attack”” -T fields -e ip.src -a duration:60>output.txt'
a = os.popen(b)
time.sleep(32)
f = open('output.txt','r')
text = 'IP address of attacker is'
print (text), f.read()
f.close
I am trying to execute this code to capture packets using tshark but i am getting this error:
tshark: "�" was unexpected in this context.
Please help me why that error is caused, thank you
The error message is because tshark tries to disable some dangerous functions in Lua like dofile.
If you don't need Lua in tshark, you can disable Lua support: edit init.lua, change disable_lua = false to disable_lua = true.
If you need Lua support, read Platform-Specific information about capture privileges, see how to capture packets without root rivilege.
I would like to create env variables once to use elsewhere in my fabric file. For example:
from fabric.api import *
# environments
def dtconfig():
env.path = 'David'
# tasks
def hello():
require('path', provided_by=[dtconfig])
print (env.path)
print ('Hello $(path)')
print ('Hello ' + env.path)
The output from running 'fab dtconfig hello' is:
David
Hello $(path)
Hello David
Why doesn't the $(path) get replaced with 'David'? thx
Looks like bash variables were removed in later versions of fabric. just used plain old %s string substitutions instead.
I want to write a unit test for a Django manage.py command that does a backend operation on a database table. How would I invoke the management command directly from code?
I don't want to execute the command on the Operating System's shell from tests.py because I can't use the test environment set up using manage.py test (test database, test dummy email outbox, etc...)
The best way to test such things - extract needed functionality from command itself to standalone function or class. It helps to abstract from "command execution stuff" and write test without additional requirements.
But if you by some reason cannot decouple logic form command you can call it from any code using call_command method like this:
from django.core.management import call_command
call_command('my_command', 'foo', bar='baz')
Rather than do the call_command trick, you can run your task by doing:
from myapp.management.commands import my_management_task
cmd = my_management_task.Command()
opts = {} # kwargs for your command -- lets you override stuff for testing...
cmd.handle_noargs(**opts)
the following code:
from django.core.management import call_command
call_command('collectstatic', verbosity=3, interactive=False)
call_command('migrate', 'myapp', verbosity=3, interactive=False)
...is equal to the following commands typed in terminal:
$ ./manage.py collectstatic --noinput -v 3
$ ./manage.py migrate myapp --noinput -v 3
See running management commands from django docs.
The Django documentation on the call_command fails to mention that out must be redirected to sys.stdout. The example code should read:
from django.core.management import call_command
from django.test import TestCase
from django.utils.six import StringIO
import sys
class ClosepollTest(TestCase):
def test_command_output(self):
out = StringIO()
sys.stdout = out
call_command('closepoll', stdout=out)
self.assertIn('Expected output', out.getvalue())
Building on Nate's answer I have this:
def make_test_wrapper_for(command_module):
def _run_cmd_with(*args):
"""Run the possibly_add_alert command with the supplied arguments"""
cmd = command_module.Command()
(opts, args) = OptionParser(option_list=cmd.option_list).parse_args(list(args))
cmd.handle(*args, **vars(opts))
return _run_cmd_with
Usage:
from myapp.management import mycommand
cmd_runner = make_test_wrapper_for(mycommand)
cmd_runner("foo", "bar")
The advantage here being that if you've used additional options and OptParse, this will sort the out for you. It isn't quite perfect - and it doesn't pipe outputs yet - but it will use the test database. You can then test for database effects.
I am sure use of Micheal Foords mock module and also rewiring stdout for the duration of a test would mean you could get some more out of this technique too - test the output, exit conditions etc.
The advanced way to run manage command with a flexible arguments and captured output
argv = self.build_argv(short_dict=kwargs)
cmd = self.run_manage_command_raw(YourManageCommandClass, argv=argv)
# Output is saved cmd.stdout.getvalue() / cmd.stderr.getvalue()
Add code to your base Test class
#classmethod
def build_argv(cls, *positional, short_names=None, long_names=None, short_dict=None, **long_dict):
"""
Build argv list which can be provided for manage command "run_from_argv"
1) positional will be passed first as is
2) short_names with be passed after with one dash (-) prefix
3) long_names with be passed after with one tow dashes (--) prefix
4) short_dict with be passed after with one dash (-) prefix key and next item as value
5) long_dict with be passed after with two dashes (--) prefix key and next item as value
"""
argv = [__file__, None] + list(positional)[:]
for name in short_names or []:
argv.append(f'-{name}')
for name in long_names or []:
argv.append(f'--{name}')
for name, value in (short_dict or {}).items():
argv.append(f'-{name}')
argv.append(str(value))
for name, value in long_dict.items():
argv.append(f'--{name}')
argv.append(str(value))
return argv
#classmethod
def run_manage_command_raw(cls, cmd_class, argv):
"""run any manage.py command as python object"""
command = cmd_class(stdout=io.StringIO(), stderr=io.StringIO())
with mock.patch('django.core.management.base.connections.close_all'):
# patch to prevent closing db connecction
command.run_from_argv(argv)
return command