I intend to run a c++ program on spark using rdd.pipe() operator in order to see the possible benefits of the program running in parallel.
In terminal i run it like this:
./program program_mode -t input -i output
In spark driver i've attempted:
mapDataset.pipe(s"/path/to/program program_mode -t $mapDataset -i /path/to/output"
where mapDataset is the input rdd (type .fasta file) that i have successfully loaded in spark driver,but this doesnt work.
The general problem is that the program expects to have its input through the flags but in spark the input is in the rdd that i've created on which pipe is called.
Any idea on how i can implement this communication correctly?
If your program uses streams, then change the way it behaves.
Instead of opening an ifstream for a file when it's on the command line, pass in stdin to your functions. Same for the output stream.
Related
I would like to run LIWC (installed in my Mac) within a python 2.7 script.
I have been reading about subprocess (popen and check_output seem the way to go), but I do not get the syntax for:
opening the program;
getting a text file to be analysed;
running the program;
getting the output (analysis) and storing it in a text file.
This is my first approach to subprocess, is this possible?
I appreciate the suggestions.
EDIT
This is the closest to implementing a solution (still does not work):
I can open the application.
subprocess.call(['open', '/file.app'])
But cannot make it process the input file and get an output one.
subprocess.Popen(['/file.app', '-input', 'input.txt', '-output', 'output.txt'])
Nothing comes out of this code.
EDIT 2
After reading dozens of posts, I am still very confused about the syntax for the solution.
Following How do I pipe a subprocess call to a text file?
I came out with this code:
g = open('in_file.txt', 'rb', 0)
f = open('out_file.txt', 'wb')
subprocess.call(['open', "file.app"] stdin=g, stdout=f)
The output file comes out empty.
EDIT 3
Following http://www.cplusplus.com/forum/unices/40680/
When I run the following shell script on the Terminal:
cat input.txt | /Path/LIWC > output.txt
The output txt file is empty.
EDIT 4
When I run:
subprocess.check_call(['/PATH/LIWC', 'PATH/input.txt', 'PATH/output.txt'])
It opens LIWC, does not create an output file and freezes.
EDIT 5
When I run:
subprocess.call(['/PATH/LIWC', 'PATH/input.txt', 'PATH/output.txt'])
It runs LIWC, creates an empty output.txt file and freezes (the process does not end).
The problem with using 'open' in subprocess.call(['open', "file.app"] stdin=g, stdout=f) is that it requests that a file be opened through a service, and doesn't directly attach it to your python process. You'll need to instead use the path to LIWC. I'm not sure that it supports reading from stdin, though, so you might need to even pass in the path to the file you'd like it to open.
I have a c++ program that run a command and pass some arguments to it. The code is as follow:
int RunApplication(fs::path applicationPathName,std::string arguments)
{
std::string applicationShortPath=GetShortFileName(applicationPathName);
std::string cmd="\""+applicationShortPath +"\" "+ arguments+" >>log.txt 2>&1 \"";
std::cout<<cmd<<std::endl;
int result=std::system(cmd.c_str());
return result;
}
When I run system command, the cmd window appears shortly and then closes, but the result is 1 and the cmd was not run (the command should generate output which is not generated).
To check that the cmd is correct, I stopped the application just before system line and copy/ paste cmd content to a cmd window and it worked.
I am wondering how can I find why application is not run in system()?
the cmd has this value just before running it:
"D:/DEVELO~3/x64/Debug/enfuse.exe" -w --hard-mask --exposure-weight=1 --saturation-weight=0.328 --contrast-weight=0.164 -o "C:/Users/m/AppData/Local/Temp/1.tif" "C:/Users/m/AppData/Local/Temp/1.jpg" "C:/Users/m/AppData/Local/Temp/2.jpg" >>log.txt 2>&1 "
How can I find why it is not working?
Is there any way that I set the system so it doesn't close cmd window so I can inspect it?
is there any better way to run a command on OS?
Does Boost has any solution for this?
Edit
After running it with cmd /k, I get this error message:
The input line is too long.
How can I fix it other than reducing cmd line?
There are two different things here: if you have to start a suprocess, "system" is not the best way of doing it (better to use the proper API, like CreateProcess, or a multiplatform wrapper, but avoid to go through the command interpreter, to avoid to open to potential malware injection).
But in this case system() is probably the right way to go since you in fact need the command interpreter (you cannot manage things like >>log.txt 2>&1 with only a process creation.)
The problem looks like a failure in the called program: may be the path is not correct or some of the files it has to work with are not existent or accessible with appropriate-permission and so on.
One of the firt thing to do: open a command prompt and paste the string you posted, in there. Does it run? Does it say something about any error?
Another thing to check is how escape sequence are used in C++ literals: to get a '\', you need '\\' since the first is the escape for the second (like \n, or \t etc.). Although it seems not the case, here, it is one of the most common mistakes.
Use cmd /k to keep the terminal: http://ss64.com/nt/cmd.html
Or just spawn cmd.exe instead and inspect the environment, permissions, etc. You can manually paste that command to see whether it would work from that shell. If it does, you know that paths, permssions and environment are ok, so you have some other issue on your hands (argument escaping, character encoding issues)
Check here How to execute a command and get output of command within C++ using POSIX?
Boost.Process is not official yet http://www.highscore.de/boost/process/
Question: Using C++ or a bash/shell script, how can I evaluate output of a long running linux process?
Example:
root#example.com~# iw event
(This command will run until manually killed.)
(It will output data that I will want to read and parse line by line.)
What is the most efficient way to evaluate the std output of this command when a new line is added to its buffer?
For example: iw event will output a line that says:
new station: 0e:0e:20:2d:20
I want to detect "new station" and run another command with the mac address. IE:
./myProgram -mac 0e:0e:20:2d:20
Thanks!
If you run the command as shown, all output will go to stdout and display on the terminal. To capture the output you have a few options:
Pipe the output to your monitor program, as in iw events | yourmonitorprogram which then reads stdin. iw should probably be modified to use unbuffered output.
Write the output of iw to a file and then use the same technique as the tail -f command to poll the file periodically
Have iw write to a named pipe or socket and have your monitor program read from that pipe or socket. This option requires modification to iw.
The simplest option is the first one
i have an interface where i use to execute the mml command in my solaris unix like below:
> eaw 0004
<RLTYP;
BSC SYSTEM TYPE DATA
GSYSTYPE
GSM1800
END
<
As soon as i do eaw <name> on the command line.It will start an interface where in i can execute mml commands and i can see the output of those commands executed.
My idea here is to parse the command output in c++.
I can do away with some logic for parsing.But to start with How can get the command to be executed inside c++ ? Is there any predefined way to do this.
This should be similar to executing sql queries inside c++.But we use other libraries to execute sql queries.I also donot want to run a shell script or create temporary files in between.
what i want is to execute the command inside c++ and get the output and even that in c++.
could anybody give me the right directions?
You have several options. From easiest and simplest to hardest and most complex to use:
Use the system() call to spawn a shell to run a command
Use the popen() call to spawn a subprocess and either write to its standard input stream or read from its standard output stream (but not both)
Use a combination of pipe(), fork(), dup()/dup2(), and exec*() to spawn a child process and set up pipes for a child process's standard input and output.
The below code is done with the sh command. This redirects stdout to a file named "out" which can be read later to process the output. Each command to the process can be written through the pipe.
#include <stdio.h>
int main()
{
FILE *fp;
fp = popen("sh > out", "w");
if (fp) {
fprintf(fp, "date\n");
fprintf(fp, "exit\n");
fclose(fp);
}
return 0;
}
How to keep writing the results of system("ping 10.50.132.10 -t"); in a text file using C++?
A way is to do directly with shell command:
system("ping 10.50.132.10 -t >> file.txt");
After your operations, you can read from "file.txt"!
There are a couple solutions to this. The first and simplest would be to simple add a redirect in the system call:
system("ping 10.50.132.10 -t > some_file.txt");
Another and more advanced way would be to read the output into your program, and write it out to file yourself. For this look either at _popen or CreateProcess.