python2 how to handle file input as argument with subprocess.call - python-2.7

I need to use a file as argument to the command but subprocess.call returns following error
>>> with open('test.txt') as F:
... subprocess.call(['cat', F])
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
TypeError: execv() arg 2 must contain only strings
However when file is not stored as variable it works fine
>>> subprocess.call(['cat', 'test.txt'])
fruit=banana
fruit=mango
fruit=apple
0

Code with open('test.txt') as F opens the file and assigns its pointor to variable F.
If I understand correctly you wanted file's content. Since subprocess.call required a command like you execute from terminal/console(In your case cat test.txt). Here in the subprocess.call you are passing File Descriptor sometime called File Pointer instead of file itself.
Therefore from the code if you really want a file content, you can simply use F.read() or F.readlines(). For Example
with open('test.txt') as F:
print F.readlines()
Also note that subprocess.call expect arguments as string. In the first case its the file object i.e <open file 'test.txt', mode 'r' at 0x7fa64a6f5a50> that leads to a traceback whereas in the second case its string i.e test.txt that's why works without any issue.

Related

How to print the help (usage) section for the main parser of Argparse and not the usage of a subparser?

I have been writing a command line programs with Argparse for some time now, and I am trying to write it in such a way that when the user supplies the following to the command line:
$python my_script.py -h
A help section (usage) will be printed out that prints out help section of the main parser, as well as brief overviews of the subparsers.
But right now, anytime I type in the previous line into my terminal, I receive no usage and instead get a massive traceback and the following error:
TypeError: expected string or buffer
This error has never occurred to me before with argparse-based command line programs. Furthermore, if I supply the name of one of the subparsers,
$python my_script.py subparserA -h
I get a print-out of the subparser's usage. The same holds true for other subparsers.
So why is it not possible for me to get the usage for the main parser? This worked for me before so I don't know why it's not working now. I really would like for the user to be able to look at an overview of the different subparsers available.
My basic code is currently set up in the following way:
import argparse
import sys
if __name__ == "__main__":
Parser = argparse.ArgumentParser(prog= "My_program")
Parser.description= "This program does A and B things."
subparsers= Parser.add_subparsers(help= "SubparserA does A things and SubparserB does B things", dest='mode')
subparserA= subparsers.add_parser("subparserA", help= "Additional explanation of what A things entail")
subparserA.add_arguments("-foo", required=True, help= "foo is needed for SubparserA to work")
subparserB= subparsers.add_parser("subparserB", help="Additional explanation of what B things entail")
subparserB.add_argument("-bar", required=True, help= "bar is needed for SubparserB to work")
args= Parser.parse_args()
if args.mode == "subparserA":
###do things pertinent to subparserA
elif args.mode== "subparserB":
###do things pertinent to subparserB
else:
argparse.print_help()
argparse.ArgumentError("too few arguments")
UPDATE
Here is the full traceback of the error:
Traceback (most recent call last):
File "my_program.py", line 164, in <module>
args= Parser.parse_args()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1701, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1733, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1939, in _parse_known_args
start_index = consume_optional(start_index)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1879, in consume_optional
take_action(action, args, option_string)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1807, in take_action
action(self, namespace, argument_values, option_string)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 996, in __call__
parser.print_help()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 2340, in print_help
self._print_message(self.format_help(), file)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 2314, in format_help
return formatter.format_help()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 281, in format_help
help = self._root_section.format_help()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 211, in format_help
func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 485, in _format_text
return self._fill_text(text, text_width, indent) + '\n\n'
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 621, in _fill_text
text = self._whitespace_matcher.sub(' ', text).strip()
TypeError: expected string or buffer
You should be using
Parser.print_help()
Parser.error('too few arguments')
That is use methods of the existing Parser object.
When I run your script
1019:~/mypy$ python stack46754855.py
Traceback (most recent call last):
File "stack46754855.py", line 10, in <module>
subparserA= subparsers.add_parser("subparserA", help= "Additional explanation of what A things entail", dest= 'mode')
File "/usr/lib/python2.7/argparse.py", line 1066, in add_parser
parser = self._parser_class(**kwargs)
TypeError: __init__() got an unexpected keyword argument 'dest'
dest is a not a valid parameter for the add_parser method. It is a valid, and useful, parameter for add_subparsers.
subparsers= Parser.add_subparsers(dest='mode')
It also objects to the add_arguments method.
After correction those I get:
1022:~/mypy$ python stack46754855.py
usage: My_program [-h] {subparserA,subparserB} ...
My_program: error: too few arguments
In Py2, subparsers is a required argument. It is optional in Py3 (a bug), allowing the script to run to the invalid argparse.print_help call:
1022:~/mypy$ python3 stack46754855.py
Traceback (most recent call last):
File "stack46754855.py", line 27, in <module>
argparse.print_help()
AttributeError: module 'argparse' has no attribute 'print_help'
With the change I suggested above:
1025:~/mypy$ python3 stack46754855.py
usage: My_program [-h] {subparserA,subparserB} ...
This program does A and B things.
positional arguments:
{subparserA,subparserB}
SubparserA does A things and SubparserB does B things
subparserA Additional explanation of what A things entail
subparserB Additional explanation of what B things entail
optional arguments:
-h, --help show this help message and exit
usage: My_program [-h] {subparserA,subparserB} ...
My_program: error: too few arguments
The second usage comes from the Parser.error call.
I can't reproduce your
massive traceback and the following error:
TypeError: expected string or buffer
I need to see that traceback (or part of it) to see what exactly is raising the error. That's not a normal argparse error; certainly it isn't one that argparse traps and reroutes.
More on the required/not required subparser behavior at How to Set a Default Subparser using Argparse Module with Python 2.7
Use + instead of , for multi line help string in parser.add_argument. If you have split you argument help in multiple lines using ',' then, you will see this issue
parser.add_argument("xml",help=("long help here",
" long help second line"))
This will result in above exception
instead
parser.add_argument("xml",help=("long help here" +
" long help second line"))

Use Python subprocess.check_output() with LC_ALL as first argument

from a Python script I want to call a few external programs and use their output inside the script. Now I found that this script is running on my local system, but gets problems on another one that uses another language. So instead of checking the output in different languages, I want to call these programs with a specified "LC_ALL=en_US.utf8" as first argument. But it seems that subprocess does not like this:
lc = "LC_ALL=en_US.utf8"
uptimedata = subprocess.check_output([lc, "/bin/uptime"])
When I run the script, I get:
Traceback (most recent call last):
File "./serverwatch.py", line 22, in <module>
uptimedata = subprocess.check_output([lc, "/bin/uptime"])
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I understand that subprocess.check_output() wants the first argument to be the command to be called. So how can I run the command with the changed LC?
This is Python 2.7.3 on Debian Wheezy. The other system is running the same versions.
Execute your command through bash:
>>> lc = "LC_ALL=en_US.utf8"
>>> command = ["/usr/bin/bash", "-c", "%s /bin/uptime" % lc]
>>> command
['/usr/bin/bash', '-c', 'LC_ALL=en_US.utf8 /bin/uptime']
>>> uptimedata = subprocess.check_output(command)
>>> uptimedata
b' 13:26:01 up 17:43, 1 user, load average: 0.36, 0.41, 0.44\n'
>>>

Accessing a csv file in python

I keep getting an "invalid keyword" error when i try to read from a csv file in python. Any ideas for a work around for this?
C:\Python27\python.exe C:/Users/User_Name/PycharmProjects/untitled/car.py
Traceback (most recent call last): File
"C:/Users/User_Name/PycharmProjects/untitled/car.py", line 122, in <module> d = handle_refugee_data.DataTable(csvformat="generic", data_directory="car2014", start_date="2013-12-01") File
"C:\Users\User_Name\PycharmProjects\untitled\handle_refugee_data.py", line 78, in __init__ with open("%s/%s" % (data_directory, data_layout), newline='') as csvfile:
TypeError: 'newline' is an invalid keyword argument for this function
Process finished with exit code 1
========================================================================
newline is a valid keyword argument to open() in Python 3, but not in Python 2, which appears to be what you are using.
One solution, if possible, would be to execute the script with Python 3 instead. Alternatively, as pointed out by #dhke in the comments, you could use io.open() instead, which does accept a newline keyword argument.
Of course, you could probably use the csv module instead, depending on your use case (which is not clear from the original question).

issue with subprocess.call

Python 2.7.3 with ubuntu:
Trying to run a program, (youtube-dl in this example) with subprocess.call with some arguments, I encounter the following issue. consider the following script:
try.py:
#!/usr/bin/python
from subprocess import call
url = "https://www.youtube.com/watch?v=8SbUC-UaAxE"
myArray = ['./youtube-dl {}'.format(url),'-x','--audio-format mp3']
#print the array before executing:
for item in myArray:
print item,
#execute:
call(myArray)
This script prints outputs:
oris#oris:~/Desktop/YouTube/backend$ ./try.py
./youtube-dl https://www.youtube.com/watch?v=8SbUC-UaAxE -x --audio-format mp3
Traceback (most recent call last):
File "./try.py", line 16, in <module>
call(myArray)
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Strangly, On the first line of the output I see the script does output ./youtube-dl https://www.youtube.com/watch?v=8SbUC-UaAxE -x --audio-format mp3 This command runs perfectly from bash directly, but produces an OSError from python. I also thought maybe to try and supply the url as an argument like this:
myArray = ['./youtube-dl', url,'-x','--audio-format mp3']
but than youtube-dl has an error of incorrect usage:
oris#oris:~/Desktop/YouTube/backend$ ./try.py
Usage: youtube-dl [options] url [url...]
youtube-dl: error: no such option: --audio-format mp3
./youtube-dl https://www.youtube.com/watch?v=8SbUC-UaAxE -x --audio-format mp3
I've read youtube-dl source to see how it handles the supplied arguments with optparse. I don't see the url as an argument there, so I'm guessing i'm passing the arguments to subprocess.call incorrectly.
Side note: another thing I find odd is the fact the print here takes effect after the subprocess call, as opposed to their order on my script. Is something happening here asynchronously?
What am I missing here? Many thanks
--audio-format and mp3 should be passed as separated arguments:
myArray = ['./youtube-dl', url, '-x', '--audio-format', 'mp3']

Python2 setting time error

I write a function which sets system time in linux. I write next code:
import time
import subprocess
def SetSystemTime(val):
try:
val = float(val)
except ValueError:
return
command = 'date -s"' + time.ctime(val) + '"'
subprocess.call(command)
On calling of that i get:
File "crc.py", line 96, in
SetSystemTime(0)
File "crc.py", line 12, in SetSystemTime
subprocess.call(command)
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1239, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
subprocess.call() normally takes a list of strings as its first argument. If you hand it a string X (as you do), that one will be converted to list with that whole string as its first argument.
The first element of that list is executed with the rest of that list as arguments.
So the underlying OS tries to execute the executable 'date -s"XXYYXXZ"' and cannot find it.
This is different from os.system() that hands the parameter to a shell that most often splits the string it gets at the spaces and then executes the first element split off.
Try:
command = ['date', '-s"' + time.ctime(val) + '"']
subprocess.call(command)
as the last two lines.