i need to convert os.system() to subprocess.call()
I have 2 complex commands executing inside os.system as (cmd1 +"&&"+cmd2), I need to convert it to subprocess.call() with shell=False(for security reasons)
example of my code :-
cmd1 ="echo paras"
cmd2 ="rm -rf paras1.txt"
os.system as (cmd1 +"&&"+cmd2)
I Tried:-
subprocess.call([cmd3,"&&",cmd4],shell=False)
subprocess.call(shlex.split(cmd3+ "&&" +cmd4),shell=False)
but failed in both cases, i need to know how can i run 2 commands inside subprocess.call() separated by "&&" with shell=False.
You need to execute the commands within a shell (for example bash):
import subprocess
shell = '/bin/bash' # you may need to adjust the path!
cmds = ['echo "foo"', '&&', 'echo "bar"']
cmd = [shell, '-c', ' '.join(cmds)]
print cmd
print subprocess.call(cmd, shell=False)
Output:
['/bin/bash', '-c', 'echo "foo" && echo "bar"']
foo
bar
0
Related
I am trying to automate the process of sending my temporary Amazon AWS keys as environment variables to a Docker image using Windows. I have a file, credentials.txt that contains my AWS credentials (the 3 ids are always the same, but the string values change regularly). I am using Windows command prompt.
Input:
(includes 2 empty lines at end) credentials.txt:
[default]
aws_access_key_id = STR/+ing1
aws_secret_access_key = STR/+ing2
aws_session_token = STR/+ing3
Desired output:
I need to issue the following command in order to run a Docker image (substituting the strings with the actual strings):
docker run -e AWS_ACCESS_KEY_ID=STR/+ing1 -e AWS_SECRET_ACCESS_KEY=STR/+ing2 -e AWS_SESSION_TOKEN=STR/+ing3 my-aws-container
My idea is to try to use regex on credentials.txt to convert it to:
SET aws_access_key_id=STR/+ing1
SET aws_secret_access_key=STR/+ing2
SET aws_session_token=STR/+ing3
And then run:
docker run -e AWS_ACCESS_KEY_ID=%aws_access_key_id% -e AWS_SECRET_ACCESS_KEY=%aws_secret_access_key% -e AWS_SESSION_TOKEN=%aws_session_token% my-aws-container
Does anyone have any advice on how to achieve this?
You can parse your credentials.txt with a for /f loop to set the variables (effectively removing the spaces):
for /f "tokens=1,3" %%a in ('type credentials.txt ^| find "="') do set "%%a=%%b"
and then run the last code line from your question:
docker run -e AWS_ACCESS_KEY_ID=%aws_access_key_id% -e AWS_SECRET_ACCESS_KEY=%aws_secret_access_key% -e AWS_SESSION_TOKEN=%aws_session_token% my-aws-container
Note: the values should not contain spaces or commas.
I've had a go in python that seems to work. Someone else may have a better answer.
I create the python file:
docker_run.py
import re
import os
myfile = 'C:/fullpath/credentials'
with open(myfile,'r') as f:
mystr = f.read()
vals = re.findall('=[\s]*([^\n]+)',mystr)
keys = ['AWS_ACCESS_KEY_ID','AWS_SECRET_ACCESS_KEY','AWS_SESSION_TOKEN']
environment_vars = ''.join([' -e ' + k + '=' + v for k,v in zip(keys,vals)])
cmd = 'docker run'+environment_vars+' my-aws-container'
os.system(cmd)
Then from command prompt I run:
python docker_run.py
This succeeds in running docker
(note: I tried using exec() in the final line rather than os.system(), but got the error "SyntaxError: invalid syntax")
NOTE: not interested in any modules like Pramiko
I'm trying to save some binary data on remote server without creating local file.
As a test I read from file but later I'm replacing it with data feed:
ps = subprocess.Popen(['cat', "/delta/ftp/GSM.PRICINT_TBL.dmp"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Next step I want to ssh data to remote server
ssh = subprocess.Popen(["ssh", '-XC', '-c', 'blowfish-cbc,arcfour', 'deltadmin#archiveserver', 'echo - >/tmp/test.log'],
shell=False,
stdin = ps.stdout,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
result = ssh.stdout.readlines()
if result == []:
error = ssh.stderr.readlines()
print >>sys.stderr, "ERROR: %s" % error
else:
print result
I use '-' so cat can accept standard input.
Expected result is data in /tmp/test.log but i see only
'-\n'
Any idea how to make it work?
I figured it:
echo 'test'|ssh -XC -c blowfish-cbc,arcfour bicadmin#nitarchive -T 'gzip - >/tmp/test.gz'
then on remote server:
zcat /tmp/test.gz
test
For cat we need space after redirect:
cat - > /tmp/test.txt
I have a function read_command defined as:
function read_command {
local __newline __lines __input __newstr __tmp i;
exec 3< "$*";
__newline=$'\n';
__lines=();
while IFS= read <&3 -r __line && [ "$__line" != '####' ]; do
echo "$__line";
__lines+=("$__line");
done
while IFS= read <&3 -r __line && [ "$__line" != '####' ]; do
read -e -p "${__line#*:}$PS2" __input;
local ${__line%%:*}="$__input";
done
__command="";
for i in "${__lines[#]}"; do
__tmp=$(echo "${i}");
__command="${__command} ${__newline} ${__tmp}";
done
echo -e "$__command";
}
In the current directory there is a file named "test", with the following
content:
greet ${a:-"Bob"}
greet ${b:-"Jim"}
####
a: name of friend a
b: name of friend b
####
In the terminal, the command executed is
read_command test
With no input, I am expecting the output of the last statement to be:
greet Bob
greet Jim
But what I get is:
greet ${a:-"Bob"}
greet ${b:-"Jim"}
What is wrong here?
Edit: As suggested by David, adding eval works in some cases except the following one.
j=1;i="export DISPLAY=:0 && Xephyr :${j}&";k=$(eval echo "$i");
echo $k
export DISPLAY=:0
I am expecting k to be "export DISPLAY=:0 && Xephyr :1&", what's wrong here?
Edit: I tried with the following
k=$(eval "echo \"$i\"")
This is the link to the script I am working on.
https://gist.github.com/QiangF/565102ba3b6123942b9bf6b897c05f87
During the first while loop, in echo "$__line", you have __line='greet ${a:-"Bob"}'. When you try to print that, Bash won't be expanding ${a:-"Bob"} into Bob. (Even if you remove the quotes around $__line this won't happen.) To get that effect, you need to add eval, as in, e.g., eval echo "$__line". Unfortunately eval comes with its can of worms, you have to start worrying about interactions between quoting levels and such.
I'm trying to write a program that will allow easier management of Arduino projects. So I wrote bash script that creates all the necessary folders and files for me and when I execute it I runs like champ. Because I want to change directory in the working terminal inside the script I run script like this
. ./initialize.sh
This is also working great, but because I am writing C++ program, sourcing this script from program is giving me headache.
So inside a program I run this script like this:
system(". /usr/lib/avrduino/script/initialize.sh");
and then when I run the program I get this error:
sh: 25: /usr/lib/avrduino/script/initialize.sh: Syntax error: "(" unexpected (expecting "}")
Running the script from the program like this:
system("/usr/lib/avrduino/script/initialize.sh");
works without error but it runs in subshell.
Syntax error points to this line in script
options=("uno" "mega" "mega2560" "atmega8" "atmega168" "atmega328" "pro" "pro5v" "pro328" "pro5v328")
How come that when I run this script outside of the program it's working like champ, but run this script from program and you have a problem ?
EDIT:
Script code
#!/bin/bash
BLACK='\033[0;30m'
RED='\033[0;31m'
GREEN='\033[0;32m'
BROWN='\033[0;33m'
BLUE='\033[0;34m'
PURPLE='\033[0;35m'
CYAN='\033[0;36m'
LGRAY='\033[0;37m'
DGRAY='\033[1;30m'
LRED='\033[1;31m'
LGREEN='\033[1;32m'
YELLOW='\033[1;33m'
LBLUE='\033[1;34m'
LPURPLE='\033[1;35m'
LCYAN='\033[1;36m'
WHITE='\033[1;37m'
NC='\033[0m'
makefile()
{
BOARD="default"
PS3='Chose your board: '
options=("uno" "mega" "mega2560" "atmega8" "atmega168" "atmega328" "pro" "pro5v" "pro328" "pro5v328")
select opt in "${options[#]}"
do case $opt in
"uno") BOARD="uno"
cp -r /usr/lib/avrduino/data/boards_info/uno/board-info.h .
MCU="atmega328P"
F_CPU="16000000UL"
;;
"mega") BOARD="mega"
cp -r /usr/lib/avrduino/data/boards_info/mega/board-info.h .
F_CPU="16000000UL"
;;
"mega2560") BOARD="mega2560"
cp -r /usr/lib/avrduino/data/boards_info/mega2560/board-info.h .
F_CPU="16000000UL"
;;
"atmega8") BOARD="atmega8"
cp -r /usr/lib/avrduino/data/boards_info/atmega8/board_-nfo.h .
MCU="atmega8"
F_CPU="16000000UL"
;;
"atmega168") BOARD="atmega168"
cp -r /usr/lib/avrduino/data/boards_info/atmega168/board-info.h .
MCU="atmega168"
F_CPU="16000000UL"
;;
"atmega328") BOARD="atmega328"
cp -r /usr/lib/avrduino/data/boards_info/atmega328/board-info.h .
MCU="atmega328P"
F_CPU="16000000UL"
;;
"pro") BOARD="pro"
cp -r /usr/lib/avrduino/data/boards_info/pro/board-info.h .
MCU="unknow"
F_CPU="16000000UL"
;;
"pro5v") BOARD="pro5v"
cp -r /usr/lib/avrduino/data/boards_info/pro5v/board-info.h .
MCU="unknown"
F_CPU="16000000UL"
;;
"pro328") BOARD="pro328"
cp -r /usr/lib/avrduino/data/boards_info/pro328/board-info.h .
MCU="atmega328P"
F_CPU="16000000UL"
;;
"pro5v328") BOARD= "pro5v328"
cp -r /usr/lib/avrduino/data/boards_info/pro5v328/board-info.h .
MCU="atmega328P"
F_CPU="16000000UL"
;;
*)
echo "Error : Input is not valid"
echo "Exiting..."
return 1
;;
esac
break
done
[ -e Makefile ] && rm Makefile
read -p "Do you want to configure your Makefile settings [Y/n]: " CONFIGURE
if { [ "$CONFIGURE" == "Y" ] || [ "$CONFIGURE" == "y" ]; }; then
read -p "Enter your MCU: " MCU
read -p "Enter F_CPU: " F_CPU
fi
read -p "Enter ARDUINO_PORT: " ARDUINO_PORT
echo "ARDUINO_DIR = /usr/share/arduino">>Makefile
echo "BOARD_TAG = $BOARD">>Makefile
echo "ARDUINO_PORT = $ARDUINO_PORT">>Makefile
echo "NO_CORE = 1">>Makefile
echo "AVRDUDE_ARD_PROGRAMMER = arduino">>Makefile
echo "HEX_MAXIMUM_SIZE = 30720">>Makefile
echo "AVRDUDE_ARD_BAUDRATE = 115200">>Makefile
echo "#ISP_LOW_FUSE = 0xFF">>Makefile
echo "#ISP_HIGH_FUSE = 0xDA">>Makefile
echo "#ISP_EXT_FUSE = 0x05">>Makefile
echo "#ISP_LOCK_FUSE_PRE = 0x3F">>Makefile
echo "#ISP_LOCK_FUSE_POST = 0x0F">>Makefile
echo "MCU = $MCU">>Makefile
echo "F_CPU = $F_CPU">>Makefile
echo "VARIANT = standard">>Makefile
echo "ARDUINO_LIBS =">>Makefile
echo "include /usr/share/arduino/Arduino.mk">>Makefile
echo "$BOARD|$MCU|" >> .avrduino.txt
clear
echo -e "${LGREEN}Makefile settings${NC}"
echo -e "${LBLUE}ARDUINO_DIR = ${LRED}/usr/share/arduino ${NC}"
echo -e "${LBLUE}BOARD_TAG = ${LRED}$BOARD${NC}"
echo -e "${LBLUE}ARDUINO_PORT = ${LRED}$ARDUINO_PORT${NC}"
echo -e "${LBLUE}NO_CORE = ${LRED}1${NC}"
echo -e "${LBLUE}AVRDUDE_ARD_PROGRAMMER = ${LRED}arduino${NC}"
echo -e "${LBLUE}HEX_MAXIMUM_SIZE = ${LRED}30720${NC}"
echo -e "${LBLUE}AVRDUDE_ARD_BAUDRATE = ${LRED}115200${NC}"
echo -e "${DGRAY}#ISP_LOW_FUSE = ${RED}0xFF${NC}"
echo -e "${DGRAY}#ISP_HIGH_FUSE = ${RED}0xDA${NC}"
echo -e "${DGRAY}#ISP_EXT_FUSE = ${RED}0x05${NC}"
echo -e "${DGRAY}#ISP_LOCK_FUSE_PRE = ${RED}0x3F${NC}"
echo -e "${DGRAY}#ISP_LOCK_FUSE_POST = ${RED}0x0F${NC}"
echo -e "${LBLUE}MCU = ${LRED}$MCU${NC}"
echo -e "${LBLUE}F_CPU = ${LRED}$F_CPU${NC}"
echo -e "${LBLUE}VARIANT = ${LRED}standard${NC}"
echo -e "${LBLUE}ARDUINO_LIBS =${NC}"
}
initializeProject()
{
read -p "Project name: " PROJECT_NAME
if [ ! -e PROJECT_NAME ]; then
mkdir $PROJECT_NAME
cd $PROJECT_NAME
makefile #Call function that makes makefile
cp -r /usr/lib/avrduino/data/include/ .
echo -e "${LGREEN}Project created successfully ${NC}"
else
echo "AVRduino: Project with name [ $PROJECT_NAME ] already exists. "
echo "AVRduino: Stop project wizard and exit."
fi
}
clear
initializeProject
. doesn't execute the script as a process, it only loads it into your current shell process.
In that context, your "shebang" line, #!/bin/bash, is just a comment.
(You can put #! doodle poodle noodle there and it will run just as well.)
When you use system, it executes in /bin/sh, and thus your bash script has syntax errors.
One way to execute scripts is to make them executable:
chmod +x /usr/lib/avrduino/script/initialize.sh
and then you can just pass it directly to system:
system("/usr/lib/avrduino/script/initialize.sh");
Or, you could explictly execute it in bash:
system("/bin/bash /usr/lib/avrduino/script/initialize.sh");
OK, here's a way one might solve your "changing directory" problem:
Rewrite initialize.sh so it takes the project name as an argument instead of asking for it interactively (that's how normal Unix tools work, so stick with it).
Then add the following to your .bashrc:
make_project()
{
/usr/lib/avrduino/script/initialize.sh "$1" && cd "$1"
}
Then you can say make_project foo and get transported to the directory "foo".
Most likely it is the misplaced shebang causing a default shell to be run - make sure the shebang is at the beginning of the first line
#!/bin/bash
# rest of script
If that does not work change your system call to
system("/bin/bash /usr/lib/avrduino/script/initialize.sh");
Here is a test file:
gunzip -c file_1.gz
Line 1
Line 2
Line 3
I am executing bash commands this way:
cmd = "gunzip -c file_1.gz | grep 3"
subprocess.call(cmd, shell=True))
Line 3
I need to run this command on several files in parallel, then join the processes. SO it seems I have to use subprocess.Popen().communicate(). However Popen won't recognize the pipe correctly and will feed it to the first command, gunzip in my case:
subprocess.Popen(cmd.split()).communicate())
gunzip: can't stat: | (|.gz): No such file or directory
gunzip: can't stat: grep (grep.gz): No such file or directory
gunzip: can't stat: 8 (8.gz): No such file or directory
I would like to keep the whole command and to avoid separating it this way:
gunzip = subprocess.Popen('gunzip -c file_1.gz'.split(), stdout=subprocess.PIPE)
grep = subprocess.Popen('grep 3'.split(), stdin=gunzip.stdout, stdout=subprocess.PIPE)
gunzip.stdout.close()
output = grep.communicate()[0]
gunzip.wait()
Is there a way to not separate the commands and process the pipe correctly?
To run the grep 3 command you need the output from the previous command, so there is no way to run this successfully in a single command with subprocess.Popen.
If you always want to run grep 3 for all the files, you could just join the results of all the gunzip -c file_x.gz and then run the grep command only once on the entire list.
subprocess.Popen('gunzip -c file_1.gz'.split(), stdout=subprocess.PIPE)
subprocess.Popen('gunzip -c file_2.gz'.split(), stdout=subprocess.PIPE)
...
grep = subprocess.Popen('grep 3'.split(), stdin=all_gunzip_stdout, stdout=subprocess.PIPE)