I would like to create env variables once to use elsewhere in my fabric file. For example:
from fabric.api import *
# environments
def dtconfig():
env.path = 'David'
# tasks
def hello():
require('path', provided_by=[dtconfig])
print (env.path)
print ('Hello $(path)')
print ('Hello ' + env.path)
The output from running 'fab dtconfig hello' is:
David
Hello $(path)
Hello David
Why doesn't the $(path) get replaced with 'David'? thx
Looks like bash variables were removed in later versions of fabric. just used plain old %s string substitutions instead.
Related
Want to verify if site domain contains "com". Assume I have shell varibale as
export FIRST_URL="http://www.11111.com"
export SECOND_URL="http://www.22222.org"
User calls Python script with parameter (partial shell varibale) as
python2.7 FIRST # OR
python2.7 SECOND
Python script is,
import sys, os, subprocess
PART_URL = sys.argv[1]
print( "PART_URL=",PART_URL)
COMPLETE_URL = PART_URL+'_URL' # Formed a full shell varibale
cmd_str='echo {} | grep \"com\".format(COMPLETE_URL)' # equivalent to echo $FIRST_URL | grep com
my_site=subprocess.check_output(cmd_str, shell=True) # Note we cant use subprocess.run() in Python Python 2.7
print("The validated Site is ", my_site)
The output should be "The validated Site is http://www.11111.com"
Refer How to access environment variable values?
$ export FIRST_VAR="http://www.11111.com"
$ python
>>> foo = 'FIRST'
>>> print os.environ[foo + '_VAR']
http://www.11111.com
>>>
Figured it out,
COMPLETE_URL = PART_URL+'_URL'
cmd_str='echo ${} | grep \'com\'.format(COMPLETE_URL)'
my_site=subprocess.check_output(cmd_str, shell=True)
Alteratively, we can use,
my_site=subprocess.call(shlex.split(cmd_str))
I've been trying to make a script that can print the Ubuntu SSH key located in ~/.ssh/authorised_keys/
Basically I want the script to print out exactly what cat ~/.ssh/authorised_keys/ would output.
I have tried using subprocess.check_output but it always returns an error.
Thanks
What about this ?
import os
os.system('cat ~/.ssh/authorised_keys')
If you want to capture the output to a variable, use subprocess. If not, you can use os.system as user803422 has mentioned
import os, subprocess
path = '~/.ssh/authorized_keys'
cmd = 'cat ' + os.path.expanduser(path)
output = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
response = output.communicate()
print (response)
You can read the file directly in Python, there is not really a need to use subprocess:
import os
print(open(os.path.expanduser('~/.ssh/authorized_keys')).read())
As per last question about invoking remote RPCs, an additional example to the one included in the documentation would be this one:
$ cat nameko.sh
#!/bin/bash
/usr/local/bin/nameko run service1:Microservice1 &
nameko_id=$!
echo 'Microservice 1 PID: ' $nameko_id
/usr/local/bin/nameko run service2:Microservice2 &
nameko_id=$!
echo 'Microservice 2 PID: ' $nameko_id
wait 2> /dev/null
$ cat service1.py
# -*- coding: utf-8 -*-
from nameko.rpc import rpc, RpcProxy
class Microservice1(object):
name = "microservice1"
#rpc
def hello(self):
print 'Microservice1 hello method invoked'
return True
$ cat service2.py
# -*- coding: utf-8 -*-
from nameko.rpc import rpc, RpcProxy
class Microservice2(object):
name = "microservice2"
microservice1 = RpcProxy('microservice1')
#rpc
def remote_hello(self):
print 'Microservice2 invokes hello method from Microservice1'
self.microservice1.hello()
return True
I am trying to build an architecture where microservices register themselves against a central microservice on startup (basically this central microservice is in charge of displaying a REST API, where each microservice deal with their part of the REST API -this is done to avoid the usage of a reverse proxy and deal with the port numbers-).
In Nameko, how could you launch a remote procedure just when the microservice is registered? As mentioned in the above post's answers, remote RPC invoke can not be done outside a #rpc method.
Actually stand alone proxy as answered here is the way to achive this:
Nameko - invoking RPC method from another service
use the once entrypoint
from nameko.testing.service import once
class Microservice2(object):
name = "microservice2"
microservice1 = RpcProxy('microservice1')
#rpc
def remote_hello(self):
print 'Microservice2 invokes hello method from Microservice1'
self.microservice1.hello()
return True
I'm attempting to use the re module to look through some terminal output. When I ping a server through terminal using ping -n 1 host (I'm using Windows), it gives me much more information than I want. I want just the amount of time that it takes to get a reply from the server, which in this case is always denoted by an integer and then the letters 'ms'. The error I get explains that the output from the terminal is not a string, so I cannot use regular expressions on it.
from os import system as system_call
import re
def ping(host):
return system_call("ping -n 1 " + host) == 0
host = input("Select a host to ping: ")
regex = re.compile(r"\w\wms")
final_ping = regex.search(ping(host))
print(final_ping)
system returns 0, not anything too useful. However, if we were to do subprocess, we can get teh output, and store it to a variable, out, then we can regex search that.
import subprocess
import re
def ping(host):
ping = subprocess.Popen(["ping", "-n", "1", host], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, error = ping.communicate()
return str(out)
host = input("Select a host to ping: ")
final_ping = re.findall("\d+ms",ping(host))[0]
print(final_ping)
Output:
22ms
There are two problems with your code:
Your ping function doesn't return the terminal output. It only returns a bool that reports if the ping succeeded. The ping output is directly forwarded to the terminal that runs the Python script.
Python 3 differentiates between strings (for text, consisting of Unicode codepoints) and bytes (for any data, consisting of bytes). As Python cannot know that ping only outputs ASCII text, you will get a bytes object if you don't specify which text encoding is in use.
It would be the best to use the subprocess module instead of os.system. This is also suggested by the Python documentation.
One possible way is to use subprocess.check_output with the encoding parameter to get a string instead of bytes:
from subprocess import check_output
import sys
def ping(host):
return check_output(
"ping -n 1 " + host,
shell=True,
encoding=sys.getdefaultencoding()
)
...
EDIT: The encoding parameter is only supported since Python 3.6. If you are using an older version, try this:
from subprocess import check_output
import sys
def ping(host):
return check_output(
"ping -n 1 " + host,
shell=True
).decode()
...
I have multiple hadoop commands to be run and these are going to be invoked from a python script. Currently, I tried the following way.
import os
import xml.etree.ElementTree as etree
import subprocess
filename = "sample.xml"
__currentlocation__ = os.getcwd()
__fullpath__ = os.path.join(__currentlocation__,filename)
tree = etree.parse(__fullpath__)
root = tree.getroot()
hivetable = root.find("hivetable").text
dburl = root.find("dburl").text
username = root.find("username").text
password = root.find("password").text
tablename = root.find("tablename").text
mappers = root.find("mappers").text
targetdir = root.find("targetdir").text
print hivetable
print dburl
print username
print password
print tablename
print mappers
print targetdir
p = subprocess.call(['hadoop','fs','-rmr',targetdir],stdout = subprocess.PIPE, stderr = subprocess.PIPE)
But, the code is not working.It is neither throwing an error not deleting the directory.
I suggest you slightly change your approach, or this is how I'm doing it. I make use of python library import commands which then depends how you will use it (https://docs.python.org/2/library/commands.html).
Here is a lil demo:
import commands as com
print com.getoutput('hadoop fs -ls /')
This gives you output like (depending on what you have in the HDFS dir )
/usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/hadoop-env.sh: line 25: /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home: Is a directory
Found 2 items
drwxr-xr-x - someone supergroup 0 2017-03-29 13:48 /hdfs_dir_1
drwxr-xr-x - someone supergroup 0 2017-03-24 13:42 /hdfs_dir_2
Note: the lib commands doesn't work with python 3 (to my knowledge), I'm using python 2.7.
Note: Be aware of the limitation of commands
If you will use subprocess which is the equivalent to commands for python 3 then you might consider to find a proper way to deal with your 'pipelines'. I find this discussion useful in that sense: (subprocess popen to run commands (HDFS/hadoop))
I hope this suggestion helps you!
Best