I have a project that I thought was going to be relatively easy, but is turning out to be more of a pain that I had hoped. First, most of the code I'm interacting with is legacy code that I don't have control over, so I can't do big paradigm changes.
Here's a simplified explanation of what I need to do: Say I have a large number of simple programs that read from stdin and write to stdout. (These I can't touch). Basically, input to stdin is a command like "Set temperature to 100" or something like that. And the output is an event "Temperature has been set to 100" or "Temperature has fallen below setpoint".
What I'd like to do is write an application that can start a bunch of these simple programs, watch for events and then send commands to them as necessary. My initial plan was to something like popen, but I need a bidrectional popen to get both read and write pipes. I hacked something together that I call popen2 where I pass it the command to run and two FILE* that get filled with the read and write stream. Then all I need to do is write a simple loop that reads from each of the stdouts from each of the processes, does the logic that it needs and then writes commands back to the proper process.
Here's some pseudocode
FILE *p1read, *p1write;
FILE *p2read, *p2write;
FILE *p3read, *p3write;
//start each command, attach to stdin and stdout
popen2("process1",&p1read,&p1write);
popen2("process2",&p2read,&p2write);
popen2("process3",&p3read,&p3write);
while (1)
{
//read status from each process
char status1[1024];
char status2[1024];
char status3[1024];
fread(status1,1024,p1read);
fread(status2,1024,p2read);
fread(status3,1024,p3read);
char command1[1024];
char command2[1024];
char command3[1024];
//do some logic here
//write command back to each process
fwrite(command1,p1write);
fwrite(command2,p2write);
fwrite(command3,p3write);
}
The real program is more complicated where it peeks in the stream to see if anything is waiting, if not, it will skip that process, likewise if it doesn't need to send a command to a certain process it doesn't. But this code gives the basic idea.
Now this works great on my UNIX box and even pretty good on a Windows XP box with cygwin. However, now I need to get it to work on Win32 natively.
The hard part is that my popen2 uses fork() and execl() to start the process and assign the streams to stdin and stdout of the child processes. Is there a clean way I can do this in windows? Basically, I'd like to create a popen2 that works in windows the same way as my unix version. This way the only windows specific code would be in that function and I could get away with everything else working the same way.
Any Ideas?
Thanks!
On Windows, you invoke CreatePipe first (similar to pipe(2)), then CreateProcess. The trick here is that CreateProcess has a parameter where you can pass stdin, stdout, stderr of the newly-created process.
Notice that when you use stdio, you need to do fdopen to create the file object afterwards, which expects file numbers. In the Microsoft CRT, file numbers are different from OS file handles. So to return the other end of CreatePipe to the caller, you first need _open_osfhandle to get a CRT file number, and then fdopen on that.
If you want to see working code, check out _PyPopen in
http://svn.python.org/view/python/trunk/Modules/posixmodule.c?view=markup
I think you've made a very good start to your problem by using the popen2() function to abstract away the cross-platform issues. I was expecting to come and suggest 'sockets', but I'm sure that's not relevant after reading the question. You could use sockets instead of pipes - it would be hidden in the popen2() function.
I am 99% sure you can implement the required functionality on Windows, using Windows APIs. What I cannot do is point you to the right functions reliably. However, you should be aware the Microsoft has most of the POSIX-like API calls available, but the name is prefixed with '_'. There are also native API calls that achieve the effects of fork and exec.
Your comments suggest that you are aware of issues with availability of data and possible deadlocks - be cautious.
Related
I have two loggers in my program. One that I made inside a gui, and one that is super complicated but very well designed and prints to the console. I am trying to get the output from the nice console logger to the rendered one. I have tried everything under the sun to get this to work but I can't seem to figure it out (due to my lack of understanding of the code from the other logger(spdlog).) My conclusion is that taking the logs directly from what is printed is the best way to do this but I can't find online anyone asking how to do this. I have seen a few questions but they just post code as an answer and don't really explain what is going on. My question: Is there a way to grab printed statements from the console and what are the performance issues/complications that come with doing something like this.
For example, if i do std::cout << "hello!" << std::endl; or some printf statement, I want to be able to further down in the code be able to grab "hello!"
My conclusion is that taking the logs directly from what is printed is the best way to do this but I can't find online anyone asking how to do this.
Consoles nowadays are terminal emulators. The original terminals' output went to printers and couldn't be (easily) read back.
Application's stdout and stderr (console) streams are write-only. Moreover, in Windows and Unix/Linux you can pipe your program's (console) output (either or both stderr and stdout) into another application with | (pipe) that creates a pipe IPC between stdout of your application and stdin of another one. That IPC pipe is write-only, your application cannot possibly read back from it.
You may be able to get access to the contents of the frame buffer of Windows cmd.exe that controls its Windows console window, but that won't be the verbatim byte-exact copy of data you wrote into stdout because of the escape sequences interpreted by Windows console.
If stdout is redirected into a file you can re-open that file for reading, but there is no portable way to re-open that file.
In other words, there is no portable way to read console output back.
I have tried everything under the sun to get this to work but I can't seem to figure it out (due to my lack of understanding of the code from the other logger(spdlog).
I bet you haven't tried reading spdlog documentation, in particular logger with multi sinks. A sink is an output abstraction, which implementation can write into a file, memory or both. What you need is attach your own sink to spdlog that prints into your UI.
Derive your sink from base_sink and implement abstract member functions:
virtual void sink_it_(const details::log_msg &msg) = 0; to print into the UI, and,
virtual void flush_() = 0; to do nothing.
Then attach one object of your sink class to that spdlog.
I'm developing an interactive program that takes inputs from the user (through stdin) and outputs data accordingly (using stdout). While the program is running I need to open a background application (and leave it running while the main program is running). I have successfully done this using popen (using the "r" mode), however at times I get stdin conflicts. For example when the user enters an input intended for the main program, sometimes, the program treats it as an input to the background program. The user should never have to interact directly with the background program. Is there a way to completely decouple both the input and the output of the background program from stdin and stdout, while still being able to read and write using a file descriptor to the background process?
Just code exactly what you need. The popen function is a convenience function that you can use what it happens to do exactly what you happen to need. Otherwise, use pipe, fork, dup2, close, and whatever exec-family function you want.
You may find it helpful to look at a few implementations of popen/pclose to see how they work. Adjust as needed.
Simply don't use popen. Instead, use fork + one of the exec family of functions, which doesn't connect the io streams.
I have a string command I'd like to execute asynchronously while writing to its input and reading its output. Sounds easy, right, the devil is in the cross-platform. I'm targeting both MSVC/Win32 and gcc/Linux and obviously want to write the minimum amount of platform-specific code. My google-fu has failed me, I get too much noise for my queries, so I started with what I know.
popen - nice and easy, returns FILE* that is easy to consume everywhere. But here's what MSDN have to say about _popen:
If used in a Windows program, the
_popen function returns an invalid file pointer that causes the program
to stop responding indefinitely.
_popen works properly in a console application. To create a Windows
application that redirects input and
output, see Creating a Child Process
with Redirected Input and Output in
the Platform SDK.
and so popen is out of the question (edit: because I'd like my code to work in GUI application). The Windows way to do it is in my opinion rather ugly and verbose. I could live with platform specific spawn code but I'd want at least the I/O code to be the same. Here, however, I hit a wall between the WinAPI HANDLEs and C FILE*, and int file descriptor. Is there a way to "convert" a HANDLE to FILE* or int fd or vice-versa? (Google failed me once more on this one, all the keywords I tried are way overused)
Is there better way to do the whole thing with little platform-specific code?
External libraries are not out of the question, however dependency maintenance is a pain, especially so on multiple platforms so I'd like to reduce dependencies. I didn't find such library in boost also.
Just for the record, what worked for me in the end. On Windows/MSVC, CreatePipe() + CreateProcess() as outlined here, using _open_osfhandle() followed by _fdopen() to get FILE* to the process input and output. On Linux/GCC, nothing new here, creating pipe()s; fork() then dup2() the pipes; exec(); fdopen() on the relevant file descriptors. That way, only the process spawning code is platform dependent (which is ok, as on Windows I'd like to control additional STARTUPINFO parameters), writing input and reading output is done through standard FILE* and related functions.
Give libexecstream a whirl. It's cross platform, and allows you to trap the input, output and error streams of a process asynchronously as C++ style streams.
I've used it on Linux, Darwin and Windows and it seems to work. It's also pretty lightweight so integrates into projects with little pain, and has a pretty open BSD licence. I don't think there's any way around using a library (other than writing your own variations for each platform).
for converting windows HANDLEs to C file descriptors use _open_osfhandle http://msdn.microsoft.com/en-us/library/bdts1c9x%28VS.71%29.aspx
EDIT: this example once helped me aswell with a similar problem:
http://www.halcyon.com/~ast/dload/guicon.htm
I have a c++ program (very complicated, and lengthy both in code and execution time).
Once in a while this program stops and calls a user-specified shell script.
Before calling the script, my program creates a .out file with current data. I call the script via system() command. The script then reads the .out file, and creates its own script.out file and exits.
Then the system() function call ends, and my program reads and parses the script.out file.
Question: is there a better way to execute communication between my c++ program and a random shell script?
My intent is to have full communication between the two. Script could virtually "ask" the program "What data do you have right now?" and the program would reply with some strict convention. Then the script could say "Add this data...", or "delete all your previous data" etc.etc.
The reason I need this is because the shell script tells the program to modify its data. The exact data that was put in the original .out file. So after the modification is done -- the actual data held by the program does not correspond to the data written in the .out file.
Thanks!
P.S.
I swear I've searched around, but everyone suggests an intermediate file.
There are certainly ways to do that without intermediate files. The most common approach is to use command line arguments for input, and pipes for standard output; others also use pipes for input. The most straight-forward alternative to system then is to use popen.
On a unix-like system? Perhaps pipe (2) will work for you?
From the man page (Mac OS X 10.5 version):
SYNOPSIS
#include <unistd.h>
int pipe(int fildes[2]);
DESCRIPTION
The pipe() function creates a pipe (an object that allows unidirectional
data flow) and allocates a pair of file descriptors. The first descrip-
tor connects to the read end of the pipe; the second connects to the
write end.
You will, of course, have to follow the creation of the pipes with a fork and exec pair. Probably this has already been answered in detail, and now you know what to search on...
It's been a while since I did this, but:
In the main process, before forking the sub-process you call pipe twice. Now you have two pipes and control both ends of both of them.
You fork.
The main process will read from one pipe and write from the other. It doesn't matter which is which, but you need to be clear about this.
The child process will call one of the exec family of function to replace it's image with that of the shell you want to run but first you will use dup2 to replace it's standard input and output with the ends of the two pipes (again, this is where you need to be clear about which pipe is which).
At his point you have two processes, the main process can send things into one pipe ad they will be received on the standard input of the script, and anything the script writes to it's standard output will be sent up the other pipe to the controlling process. So they take turns, just like interacting with the shell.
You can use pipes or (maybe more convenient) sockets - for example frontends to gdb, or expect do that. It would require changes to your shell scripts, and switching from system() to more low-level fork() and exec().
It's rather complicated so please, be more specific about your environment and what you need to clarify.
You are asking the question on Interprocess Communication (IPC).
There are a lot of ways to do that. You can do a simply search and Internet will return you most answers.
If I am not wrong, Google chrome uses a technique called Named Pipe.
Anyway, I think the most "portable way" is probably a file. But if you know you are working on which operating system, you can definitely use most of the IPC techniques.
I have a string command I'd like to execute asynchronously while writing to its input and reading its output. Sounds easy, right, the devil is in the cross-platform. I'm targeting both MSVC/Win32 and gcc/Linux and obviously want to write the minimum amount of platform-specific code. My google-fu has failed me, I get too much noise for my queries, so I started with what I know.
popen - nice and easy, returns FILE* that is easy to consume everywhere. But here's what MSDN have to say about _popen:
If used in a Windows program, the
_popen function returns an invalid file pointer that causes the program
to stop responding indefinitely.
_popen works properly in a console application. To create a Windows
application that redirects input and
output, see Creating a Child Process
with Redirected Input and Output in
the Platform SDK.
and so popen is out of the question (edit: because I'd like my code to work in GUI application). The Windows way to do it is in my opinion rather ugly and verbose. I could live with platform specific spawn code but I'd want at least the I/O code to be the same. Here, however, I hit a wall between the WinAPI HANDLEs and C FILE*, and int file descriptor. Is there a way to "convert" a HANDLE to FILE* or int fd or vice-versa? (Google failed me once more on this one, all the keywords I tried are way overused)
Is there better way to do the whole thing with little platform-specific code?
External libraries are not out of the question, however dependency maintenance is a pain, especially so on multiple platforms so I'd like to reduce dependencies. I didn't find such library in boost also.
Just for the record, what worked for me in the end. On Windows/MSVC, CreatePipe() + CreateProcess() as outlined here, using _open_osfhandle() followed by _fdopen() to get FILE* to the process input and output. On Linux/GCC, nothing new here, creating pipe()s; fork() then dup2() the pipes; exec(); fdopen() on the relevant file descriptors. That way, only the process spawning code is platform dependent (which is ok, as on Windows I'd like to control additional STARTUPINFO parameters), writing input and reading output is done through standard FILE* and related functions.
Give libexecstream a whirl. It's cross platform, and allows you to trap the input, output and error streams of a process asynchronously as C++ style streams.
I've used it on Linux, Darwin and Windows and it seems to work. It's also pretty lightweight so integrates into projects with little pain, and has a pretty open BSD licence. I don't think there's any way around using a library (other than writing your own variations for each platform).
for converting windows HANDLEs to C file descriptors use _open_osfhandle http://msdn.microsoft.com/en-us/library/bdts1c9x%28VS.71%29.aspx
EDIT: this example once helped me aswell with a similar problem:
http://www.halcyon.com/~ast/dload/guicon.htm