I have two executables written in C++ on Windows. I generate some data in one, and want to call the other executable to process this data. I could write the data out to a file then read it in the other executable, but that seems rather expensive in terms of disk I/O. What is a better way of doing this? It seems like a simple enough question but google just isn't helping!
Let's say the data is around 100MB, and is generated in its entirety before needing to be sent (i.e. no streaming is needed).
Answers that work when mixing 32 bit and 64 bit processes gain bonus points.
If your processes can easily write to and read from file, just go ahead. Create the file with CreateFile and mark it as temporary & shareable. Windows uses this hint to delay physical writes, but all file semantics are still obeyed. Since your file is only 100 MB and actively in use, Windows is almost certainly able to keep its contents fully in RAM.
You can use Boost.MPI. It is from Boost, which has high quality standard, and the code sample seems pretty explicit:
http://www.boost.org/doc/libs/1_53_0/doc/html/mpi/tutorial.html#mpi.point_to_point
// The following program uses two MPI processes to write "Hello, world!"
// to the screen (hello_world.cpp):
int main(int argc, char* argv[])
{
mpi::environment env(argc, argv);
mpi::communicator world;
if (world.rank() == 0) {
world.send(1, 0, std::string("Hello"));
std::string msg;
world.recv(1, 1, msg);
std::cout << msg << "!" << std::endl;
} else {
std::string msg;
world.recv(0, 0, msg);
std::cout << msg << ", ";
std::cout.flush();
world.send(0, 1, std::string("world"));
}
return 0;
}
Assuming you only want to go "one direction" (that is, you don't need to get data BACK from the child process), you could use _popen(). You write your data to the pipe and the child process reads the data from stdin.
If you need bidirectional flow of data, then you will need to use two pipes, one as input and one as output, and you will need to set up a scheme for how the child process connects to those pipes [you can still set up the stdin/stdout to be the data path, but you could also use a pair of named pipes].
A third option is a shared memory region. I've never done this in Windows, but the principle is pretty much the same as what I've used in Linux [and many years back in OS/2]:
1. Create a memory region with a given name in your parent process.
2. The child process opens the same memory region.
3. Data is stored by parent process and read by child process.
4. If necessary, semaphores or similar can be used to signal completion/results ready/etc.
Related
I'm trying to learn how to use the pipe() and fork() system calls. I'm using pipe and fork to create parent and child processes where the child will read a character from the text file, and then send it through the pipe to the parent that will then output the character to the console, with the desired result that it will print out the entire text to the console. Later I'm going to be doing some text processing on the file with the child process reading and processing then sending the updated text to the parent but for now I just want to make sure I'm getting the basics of pipe() correct.
example file:
This is a test file; it is 1 of many.
Others will follow.
Relevant code:
pid = fork();
ifstream fin;
fin.open(inputFilename);
fin.get(inputChar);
if (pid == -1)
{
perror("Trouble");
exit(2);
}
else if (pid == 0) //child process that reads text file and writes to parent
{
close(pipefds[0]);
while(!fin.eof())
{
write(pipefds[1], &inputChar, sizeof(inputChar));
fin.get(inputChar);
}
close(pipefds[1]);
exit(0);
}
else
{
close(pipefds[1]);
read(pipefds[0], readbuffer, sizeof(readbuffer));
cout << readbuffer << endl;
close(pipefds[0]);
exit(0);
}
fin.close();
However, when I compile and run, the output is always of a varying length. Sometimes it will print the whole file, others it will just print out a few letters, or half of a line. Such as.
This i
I've tried going through the man pages and researching more but I haven't been able to find any answers. What exactly is going on with my program that it will sometimes read everything from the file but other times won't. Any help is greatly appreciated!
It looks as though you're trying to read all the data from the pipe with one call to read(2). But, as with any I/O operation, this may always return fewer bytes than you requested. You should always check the return value of read(2) and write(2) system calls (and others), to make sure that they acted as expected.
In this case, you should loop until you get some independent notification from the child process that they're done sending data. This can be signaled in this case by read(2) returning 0, meaning that the child closed their end of the pipe.
You are assuming that the parent can read everything written to the pipe by the child via one read() call. That might be a safe assumption for a pipe if the child were writing everything via a single write() call, as long as the overall data size did not exceed the size of the pipe's internal buffer. It is not at all safe when, as in this case, the child is sending data via many little writes.
How much data the parent actually gets will depend in part on how its one read() call is ordered relative to the child's writes. Inasmuch as the two are separate processes and you're employing no IPC other than the pipe itself, it's basically unpredictable how much data the parent will successfully read.
In the general case, one must assume that the reader will need to perform multiple read() calls to read all data that are sent. It must keep calling read() and processing the resulting data appropriately until read's return value indicates that an I/O error has occurred or that the end of the file has been reached. Note well that end of file does not mean just that no more bytes are available now, but that no more bytes will ever be available. That happens after all processes have closed all copies of the write end of the pipe.
I'm developing a cross platform project currently. On windows i had a class that ran a process/script (using a commandline), waited for it to end, and read everything from it's stdout/stderr to a buffer. I then printed the output to a custom 'console'. Note: This was not a redirection of child stdout to parent stdout, just a pipe from child stdout to parent.
I'm new to OSX/unix-like api's but i can understand the canonical way of doing something like this is forking and piping stdouts together. However, i dont want to redirect it to stdout and i would like to capture the output.. It should work pretty much like this (pseudocode, resemblance with unix functions purely coincidental):
class program
{
string name, cmdline;
string output;
program(char * name, char * cmdline)
: name(name), cmdline(cmdline) {};
int run()
{
// run program - spawn it as a new process
int pid = exec(name, cmdline);
// wait for it to finish
wait(pid);
char buf[size];
int n;
// read output of program's stdout
// keep appending data until there's nothing left to read
while (read(pid, buf, size, &n))
output.append(buf, n);
// return exit code of process
return getexitcode(pid);
}
const string & getOutput() { return output; }
};
How would i go about doing this on OSX?
E:
Okay so i studied the relevant api's and it seems that some kind of fork/exec combo is unavoidable. Problem at hand is that my process is very large and forking it really seems like a bad idea (i see that some unix implementations can't do it if the parent process takes up 50%+ of the system ram).
Can't i avoid this scheme in any way? I see that vfork() might be a possible contender, so maybe i could try to mimic the popen() function using vfork. But then again, most man pages state that vfork might very well just be fork()
You have a library call to do just that: popen. It will provide you with a return value of a file descriptor, and you can read that descriptor till eof. It's part of stdio, so you can do that on OSX, but other systems as well. Just remember to pclose() the descriptor.
#include <stdio.h>
FILE * popen(const char *command, const char *mode);
int pclose(FILE *stream);
if you want to keep output with absolutely no redirection, the only thing we can think of is using something like "tee" - a command which splits the output to a file but maintains its own stdout. It's fairly easy to implement that in code as well, but it might not be necessary in this case.
My Code:
std::ofstream m_myfile,
m_myfile.open ("zLog.txt");
m_myfile << "Writing this to a file " << " and this " << endl;
when this C++ Program runs, I have another program that needs to read this file. The problem is that the file is locked by C++ and I cannot read it from the other program. I know there is something I have to do where I write the code someway in the C++ Program where it allows sharing. Can someone write exactly what I need. I have googled this to death and still cannot get this to work.
Some people say close the file, before the other program reads it. I cannot do this, the file needs to be open.
You need to open the file with sharing enabled. Use the following overload of the open method:
void open(const char *szName, int nMode = ios::out, int nProt = filebuf::openprot);
and pass the appropriate share mode as nProt:
filebuf::sh_compat: Compatibility share mode
filebuf::sh_none: Exclusive mode; no sharing
filebuf::sh_read: Read sharing allowed
filebuf::sh_write: Write sharing allowed
There is also an overload of the ofstream constructor that takes the same arguments.
The sharing is going to be controlled at the OS level. So you need to look at the API for your OS and figure out how to turn read-write sharing on.
Note: you still probably won't get the results you want because there will be caching and buffering issues and what you think was written to the file may not actually be there.
If you want to share information between two processes, use named pipes or sockets. Both are available on just about every OS.
Use filebuf::sh_write while opening the file.
Other option is to use sockets. Check out this stackoverflow question: Is there a way for multiple processes to share a listening socket?
My native C++ Win32 program spawns a worker process and needs to pass a huge configuration string to it. Currently it just passes the string as a command line to CreateProcess(). The problem is the string is getting longer and now it doesn't fit into the 32K characters limitation imposed by Windows.
Of course I could do something like complicating the worker process start - I use the RPC server in it anyway and I could introduce an RPC request for passing the configuration string, but this will require a lot of changes and make the solution not so reliable. Saving the data into a file for passing is also not very elegant - the file could be left on the filesystem and become garbage.
What other simple ways are there for passing long strings to a worker process started by my program on Windows?
One possible strategy is to create a named Pipe and pass the handle ( or pipe name) to the other process. Then use normal Read\Write operations on Pipe to extract the data.
There are several good answers already, but the easiest way is to save it in a file, and pass the filename in the command line.
As well as being simple, an advantage of this approach is that the apps will be very loosely coupled (you'll potentially be able to use the child application stand-alone in other ways, rather than always having to launch it from a program that knows how to pipe data into it via a specialised interface)
If you want to be sure that the file is cleaned up after processing, mark it for deletion on the next reboot. THen if anybody forgets to clean it up, the OS will deal with it for you on the next reboot.
I would prefer Boost's message queue. It's extremely simple yet sophisticated. Here's example:
#include <boost/interprocess/ipc/message_queue.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/shared_ptr.hpp>
using namespace boost::interprocess;
// ------------------------------------------------------------------------------
// Your worker:
// ------------------------------------------------------------------------------
try {
message_queue::remove("NAME_OF_YOUR_QUEUE");
boost::shared_ptr<message_queue> mq(new message_queue(create_only, "NAME_OF_YOUR_QUEUE", 65535, 32));
char message[1024];
std::size_t size_received;
unsigned int priority;
if (mq->timed_receive(&message, sizeof(message), size_received, priority, boost::posix_time::ptime(boost::posix_time::second_clock::universal_time()) + boost::posix_time::seconds(1))) {
std::string s(message); // s now contains the message.
}
} catch (std::exception &) {
// ...
}
// ------------------------------------------------------------------------------
// And the sender:
// ------------------------------------------------------------------------------
try {
boost::shared_ptr<message_queue> mq(new message_queue(create_only, "NAME_OF_YOUR_QUEUE", 1024, 1024));
std::stringstream message;
message << "the very very very long message you wish to send over";
while (!mq.try_send(message.str().c_str(), message.str().length(), 0))
::Sleep(33);
} catch (std::exception &) {
// ...
}
Use shared memory. Pass to a worker process name of shared memory object. Another solution is to use WM_COPYDATA message.
How about reading it from stdin :) It seems to work for the Unix folks.
Guaranteed a lot easier than passing pipe names/handles around!
Here is some official code from MSDN for creating child processes with I/O pipes.
Is it a possibility to set up a named shared memory segment?
http://msdn.microsoft.com/en-us/library/aa366551(VS.85).aspx
You could use an inheritable handle to a section object. In your parent process create a section object (CreateFileMapping) and specify that its handle is to be inherited by the child process; then pass the handle value to the child process on the command line. The child process can then open the section object (OpenFileMapping). Though I would prefer a named section object as the semantics of using it are easier to understand.
I want to pass a value of an input variable in my program lets say#1 to another program #2 and i want #2 to print the data it got to screen, both are needed to be written in c++. The this will be on Linux.
Depending on the platform there are a number of options available. What you are trying to do is typically called inter-process communication (IPC).
Some options include:
Sockets
Pipes
Queues
Shared Memory
What is easiest is probably dependent on the platform youa are using.
As always, there is a Boost library for that (God, I like Boost).
Nic has covered all the 4 that I wanted to mention (on the same machine):
Sockets
Pipes
Queues
Shared Memory
If writing system calls is troublesome for you, you may want to use the following libraries:
Boost http://www.boost.org/
Poco http://pocoproject.org/blog/
Nokia Qt http://qt.nokia.com/
Something you can read from Qt portable IPC: only QSharedMemory?
If effeciency is not prime concern then use normal file i/o.
else go for IPC to do so.
As far as Windows is concern you have following options :
Clipboard ,
COM ,
Data Copy ,
DDE ,
File Mapping ,
Mailslots ,
Pipes ,
RPC ,
Windows Sockets
For Linux , use can use Name Pipes(efficient) or sockets.
If you're on Windows, you can use Microsoft Message Queueing. This is an example of queue mentioned previously.
If the data to be passed is just a variable, then one of the option is to set it as Environment Variable [ Var1 ] by program #1 and access it, in Program #2 [ if both are running on same env/machine ]. Guess this will be the easiest one, instead of making it complex, by using IPC/socket etc.
I think most of the answers have address the common IPC mechanisms. I'd just like to add that I would probably go for sockets because it's fairly most standard across several platforms. I decided to go for that when I needed to implement IPC that worked both on Symbian Series 60 and Windows Mobile.
The paradigm is straightforward and apart from a few platform glitches, the model worked the same for both platforms. I would also suggest using Protocol Buffers to format the data you send through. Google uses this a lot in its infrastructure. http://code.google.com/p/protobuf/
DBUS
QtDbus
DBus-mm
In response to your comment to Roopesh Majeti's answer, here's a very simple example using environment variables:
First program:
// p1.cpp - set the variable
#include <cstdlib>
using namespace std;;
int main() {
_putenv( "MYVAR=foobar" );
system( "p2.exe" );
}
Second program:
// p2.cpp - read the variable
#include <cstdlib>
#include <iostream>
using namespace std;;
int main() {
char * p = getenv( "MYVAR" );
if ( p == 0 ) {
cout << "Not set" << endl;
}
else {
cout << "Value: " << p << endl;
}
}
Note:
there is no standard way of setting an environment variable
you will need to construct the name=value string from the variable contents
For a very dirt and completely nonprofessional solution you can do it like me.
Save the variable in to a file and then read it (in an infinite loop every x time) with the other program.
fsexample.open("F:/etc etc ...");
fsexample >> data1 >> data2; // etc etc
and on the other side
fsexample.open("F:/etc etc ...");
fsexample << data1 << data2; // etc etc
The trick is that F is a virtual drive created with ramdisk so it is fast
and heavy-duty proof.
You could have problem of simultaneous access but you can check it with
if (!fsexample.is_open()) {
fsexample_error = 1;
}
and retry on failure.