I have a set of linux commands in a file, I am trying to execute each of them one by one in python script
for line in file:
p = subprocess.Popen(line,shell=True,stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
the above line does not execute any command as I cannot see any output.
If only the command is provided explicitly then it gets executed
cmd = "date"
p = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
You can use os.system or subprocess.call.
Complete code:
import os
with open("/path/to/file") as file:
command = file.readlines()
for line in command:
p = str(os.system(str(line)))
The syntax is
import os
os.system("path/to/executable option parameter")
or
os.system("executable option paramter")
For example,
os.system("ls -al /home")
or part of code(with subprocess):
for line in file:
subprocess.call(line, shell=True)
I got this info at https://docs.python.org/2/library/subprocess.html
NOTE: os.system is deprecated but it still works
Try after removing shell=True, My command was also facing same problem when I was executing this:
subprocess.Popen(
["python3", os.path.join(script_dir, script_name)] + list(args),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
cwd=script_dir,
shell=True
)
but once I remove shell it worked properly.
Related
I am trying to run a simple mapreduce code just to read using mapper.py, taking the output of mapper.py and reading by reducer.py. This code works on local computer but when I tried on aws-emr, it gives the following error -
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
Here is the input.txt, mapper.py and reducer.py
input.txt
scott,haris
jenifer,smith
ted,brandy
amanda,woods
bob,wilton
damn,halloween
mapper.py
#!/usr/bin/env python
import sys
for line in sys.stdin:
x = line.strip()
first,last = x.split(",")
print '%s\t%s' % (first, last)
reducer.py
#!/usr/bin/env python
import sys
for line in sys.stdin:
x = line.strip()
key, value = x.split('\t')
print '%s\t%s' % (key, value)
I am using the following command:
hadoop jar /usr/lib/hadoop/hadoop-streaming.jar -files s3://test/mapper.py,s3://test/reducer.py -mapper "python mapper.py" -reducer "python reducer.py" -input s3://test/input.txt -output s3://test/output
It seems like you have problem with python reducer / mapper script, can you check two things below
1.Are your Mapper and Reducer scripts executable(make sure you are pointing to right env with , like try #!/usr/bin/python ) and have right permissions?
2.Your Python program is right, for example if server is running python3, you need to have brackets for print() or any other issue with the script.
Try to execute python normally in emr with bash and see if it works
I'm using Python Version 2.7.10 with macOS Sierra 10.12.16 and Xcode 8.3.3.
I want to call latex from Python. os.system('latex myFile.tex') does not work because the complete path is needed. So I tried to get the path with the which program command.
import os
import subprocess
batcmd = '/usr/bin/which latex'
thePath = os.system(batcmd)
print("The path: "+str(thePath))
p = subprocess.Popen([batcmd], stdout=subprocess.PIPE, shell=True)
print(p.stdout.read())
The results I get from both variations are:
he path: 256
Program end
with exit code: 0
The system command delivers the number 256. The subprocess command derives an empty line. Other system calls like ls or dir work fine.
How can I get the path of a program with Python?
The path var is included in my custom bin dirs, including Latex.
import os
os.environ["PATH"] += os.pathsep + '/usr/local/.......'
print(os.getenv("PATH"))
I would like to submit jobs to a computer cluster via the scheduler SGE using a pipe:
$ echo -e 'date; sleep 2; date' | qsub -cwd -j y -V -q all.q -N test
(The queue might be different depending on the particular cluster.)
Running this command-line in a bash terminal works for me on the cluster I have access to, with GNU bash version 3.2.25, GE version 6.2u5 and Linux 2.6 x86_64.
In Python 2.7.2, here are my commands (the whole script is available as a gist):
import subprocess
queue = "all.q"
jobName = "test"
cmd = "date; sleep 2; date"
echoArgs = ["echo", "-e", "'%s'" % cmd]
qsubArgs = ["qsub", "-cwd", "-j", "y", "-V", "-q", queue, "-N", jobName]
Case 1: using shell=True makes it work:
wholeCmd = " ".join(echoArgs) + " | " + " ".join(qsubArgs)
out = subprocess.Popen(wholeCmd, shell=True, stdout=subprocess.PIPE)
out = out.communicate()[0]
jobId = out.split()[2]
But I would like to avoid that for security reasons explained in the official documentation.
Case 2: using the same code as above but with shell=False results in the following error message, so that the job is not even submitted:
Traceback (most recent call last):
File "./test.py", line 22, in <module>
out = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE)
File "/share/apps/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/share/apps/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Case 3: therefore, following the official documentation as well as this on SO, here is one proper way to do it:
echoProc = subprocess.Popen(echoArgs, stdout=subprocess.PIPE)
out = subprocess.check_output(qsubArgs, stdin=echoProc.stdout)
echoProc.wait()
The job is successfully submitted, but it returns the following error message:
/opt/gridengine/default/spool/compute-2-27/job_scripts/3873705: line 1: echo 3; date; sleep 2; date: command not found
This is something I don't understand.
Case 4: another proper way to do it following this is:
echoProc = subprocess.Popen(echoArgs, stdout=subprocess.PIPE)
qsubProc = subprocess.Popen(qsubArgs, stdin=echoProc.stdout, stdout=subprocess.PIPE)
echoProc.stdout.close()
out = qsubProc.communicate()[0]
echoProc.wait()
Here again the job is successfully submitted, but returns the following error message:
/opt/gridengine/default/spool/compute-2-32/job_scripts/3873706: line 1: echo 4; date; sleep 2; date: command not found
Did I make mistakes in my Python code? Could the problem come from the way Python or SGE were compiled and installed?
You're getting "command not found" because 'echo 3; date; sleep 2; date' is being interpreted as a single command.
Just change this line:
echoArgs = ["echo", "-e", "'%s'" % cmd]
to:
echoArgs = ["echo", "-e", "%s" % cmd]
(I.e., remove the single quotes.) That should make both Case 3 and Case 4 work (though it will break 1 and 2).
Your specific case could be implemented in Python 3 as:
#!/usr/bin/env python3
from subprocess import check_output
queue_name = "all.q"
job_name = "test"
cmd = b"date; sleep 2; date"
job_id = check_output('qsub -cwd -j y -V'.split() +
['-q', queue_name, '-N', job_name],
input=cmd).split()[2]
You could adapt it for Python 2, using Popen.communicate().
As I understand, whoever controls the input cmd may run arbitrary commands already and therefore there is no much point to avoid shell=True here:
#!/usr/bin/env python
from pipes import quote as shell_quote
from subprocess import check_output
pipeline = 'echo -e {cmd} | qsub -cwd -j y -V -q {queue_name} -N {job_name}'
job_id = check_output(pipeline.format(
cmd=shell_quote(cmd),
queue_name=shell_quote(queue_name),
job_name=shell_quote(job_name)),
shell=True).split()[2]
Implementing the pipeline by hand is error-prone. If you don't want to run the shell; you could use plumbum module that supports a similar pipeline syntax embedded in pure Python:
#!/usr/bin/env python
from plumbum.cmd import echo, qsub # $ pip install plumbum
qsub_args = '-cwd -j y -V -q'.split() + [queue_name, '-N', job_name]
job_id = (echo['-e', cmd] | qsub[qsub_args])().split()[2]
# or (qsub[qsub_args] << cmd)()
See How do I use subprocess.Popen to connect multiple processes by pipes?
I use this command on the Linux terminal to perform a function on a file ./pythonfile.py --count ../textfile.txt
now I use that command on windows on command prompt and it doesn't work.
I use python pythonfile.py --count ./textfile.txt to read and count the text after every white space .
When I press enter it gives the error.
error is : IOError: [Errno 2] NO such file or directory: '--count'
import sys
def cat(filename):
f=open(filename,'rU')
text=f.read()
print text
f.close()
def main():
cat(sys.argv[1])
if __name__=="__main__":
main()
You should get the same error on Linux. You can inspect sys.argv with:
print(sys.argv)
It prints:
['pythonfile.py', '--count', './textfile.txt']
Use sys.argv[2] to get to your file name:
def main():
print(sys.argv)
cat(sys.argv[2])
This should work on Windows too.
From a windows application written on C++ or python, how can I execute arbitrary shell commands?
My installation of Cygwin is normally launched from the following bat file:
#echo off
C:
chdir C:\cygwin\bin
bash --login -i
From Python, run bash with os.system, os.popen or subprocess and pass the appropriate command-line arguments.
os.system(r'C:\cygwin\bin\bash --login -c "some bash commands"')
The following function will run Cygwin's Bash program while making sure the bin directory is in the system path, so you have access to non-built-in commands. This is an alternative to using the login (-l) option, which may redirect you to your home directory.
def cygwin(command):
"""
Run a Bash command with Cygwin and return output.
"""
# Find Cygwin binary directory
for cygwin_bin in [r'C:\cygwin\bin', r'C:\cygwin64\bin']:
if os.path.isdir(cygwin_bin):
break
else:
raise RuntimeError('Cygwin not found!')
# Make sure Cygwin binary directory in path
if cygwin_bin not in os.environ['PATH']:
os.environ['PATH'] += ';' + cygwin_bin
# Launch Bash
p = subprocess.Popen(
args=['bash', '-c', command],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.wait()
# Raise exception if return code indicates error
if p.returncode != 0:
raise RuntimeError(p.stderr.read().rstrip())
# Remove trailing newline from output
return (p.stdout.read() + p.stderr.read()).rstrip()
Example use:
print cygwin('pwd')
print cygwin('ls -l')
print cygwin(r'dos2unix $(cygpath -u "C:\some\file.txt")')
print cygwin(r'md5sum $(cygpath -u "C:\another\file")').split(' ')[0]
Bash should accept a command from args when using the -c flag:
C:\cygwin\bin\bash.exe -c "somecommand"
Combine that with C++'s exec or python's os.system to run the command.