I'm writing a custom shell where I try to add support for input, output redirections and pipes just like standard shell. I stuck at point where I cannot do input redirection, but output redirection is perfectly working. My implementation is something like this (only related part), you can assume that (string) input is non-empty
void execute() {
... // stuff before execution and initialization of variables
int *fds;
std::string content;
std::string input = readFromAFile(in_file); // for input redirection
for (int i = 0; i < commands.size(); i++) {
fds = subprocess(commands[i]);
dprintf(fds[1], "%s", input.data()); // write to write-end of pipe
close(fds[1]);
content += readFromFD(fds[0]); // read from read-end of pipe
close(fds[0]);
}
... // stuff after execution
}
int *subprocess(std::string &cmd) {
std::string s;
int *fds = new int[2];
pipe(fds);
pid_t pid = fork();
if (pid == -1) {
std::cerr << "Fork failed.";
}
if (pid == 0) {
dup2(fds[1], STDOUT_FILENO);
dup2(fds[0], STDIN_FILENO);
close(fds[1]);
close(fds[0]);
system(cmd.data());
exit(0); // child terminates
}
return fds;
}
My thought is subprocess returns a pipe (fd_in, fd_out) and parent can write to write-end and read-from read-end afterwards. However when I try an input redirection something like sort < in.txt, the program just hangs. I think there is a deadlock because one waiting other to write, and other one to read, however, after parent writes to write-end it closes, and then read from read-end. How should I consider this case ?
When I did a bit of searching, I saw this answer, which my original thinking was similar except that in the answer it mentions creating two pipes. I did not quite understand this part. Why do we need two separate pipes ?
I need to create pipe/subprocess and get output from it as it will change and pass it as string to main process. It would be great if there is some crossplatform solution to it (but right now windows is main priority)
I tried Observer pattern plus Async from std (I don't think that it's working right).
All in C++ and WinApi (if needed)
Code:
void pipe::Pipe::syncMode()
{
//some code for subproccessing in Windows
}
void pipe::Pipe::asyncMode() {
std::async(std::launch::async, &pipe::Pipe::syncMode, this);
}
Thanks!
CreateProcess, if you prioritize Windows.
Can get you:
1. Standard Output and error from the child process.
2. Return code of the child process.
3. Can set security attributes.
4. Can inherit handles from parent process. And a lot more.
If you needed a cross platform solution, popen is your way to go.
popen is not as sophisticated as createprocess. But if you only want to invoke a child process and get its standard output, it is more than enough.
Sample popen code to create a process and get its standard output
std::wstring ExecuteCmdLine(std::string cmd) // NOLINT
//! Execute given comment in std::string and returns output as std::wstring
{
try
{
printf("##### \t Inside CMcUtil::executeCMDLine \t #####\n");
printf("The commnad to execute is : \"%s\"\n", cmd.c_str());
wchar_t buffer[128];
ATL::CStringW result = "";
FILE* pipe = _wpopen(CA2W(cmd.c_str()), L"r");
if (!pipe)
{
throw std::runtime_error("popen() failed!");
}
try
{
while (!feof(pipe)) {
if (fgetws(buffer, 128, pipe) != nullptr)
result += buffer;
}
}
catch (...)
{
_pclose(pipe);
throw;
}
_pclose(pipe);
return result.GetBuffer();
}
catch (...)
{
ERRMSG("Exception caught while executing command in command prompt\n");
return L"Error";
}
}
I've got a piece of code that executes a process and retrieves the result.
namespace {
FILE* really_popen(const char* cmd, const char* mode) {
#ifdef _MSC_VER
return _popen(cmd, mode);
#else
return popen(cmd, mode);
#endif
}
void really_pclose(FILE* pipe) {
#ifdef _MSC_VER
_pclose(pipe);
#else
pclose(pipe);
#endif
}
std::string ExecuteProcess(std::string cmd) {
FILE* pipe = really_popen(cmd.c_str(), "r");
if (!pipe) throw std::runtime_error("Could not invoke command " + cmd);
char buffer[128];
std::string result = "";
while(!feof(pipe)) {
if(fgets(buffer, 128, pipe) != NULL)
result += buffer;
}
really_pclose(pipe);
return result;
}
}
This works just fine for me on Linux, but on Windows, it has a terrible habit of deadlocking- seems that fgets never returns. I've looked into the CRT sources and fgets eventually delegates to ReadFile, which never returns.
If I invoke the command from the command line, it returns within a second.
How can I read the output on Windows without deadlocking the parent?
If the child hasn't exited, then the call to fgets() won't exit, so you need to resolve why the child isn't exiting. The most likely cause in this sort of situation is that the child has hung because it has no standard input.
If that's the problem, you may be able to resolve it by changing the pipe mode to "rw". You won't typically need to do anything with the extra pipe, it just has to be there.
(As you mention in the comments, the problem can also be resolved by using the command shell redirection to give the child a handle to NUL as standard input.)
In my C++ program, I need to start a very long running new process and monitor its I/O. I cannot modify the source code of the program in question.
I was thinking of create a new thread and starting the process in it and sending the output continuously (which will be coming out asynchronously) to main thread.
My code for creating the process currently looks like this:
std::string SysExec::exec(char* cmd) {
FILE* pipe = popen(cmd, "r");
if (!pipe)
return "ERROR";
char buffer[128];
std::string result = "";
while (!feof(pipe)) {
if (fgets(buffer, 128, pipe) != NULL)
result += buffer;
}
pclose(pipe);
return result;
}
However, if called from main thread, it will make the main program stop (because of while (!feof(pipe))). How should I modify this? Or is there any better way to do this?
I have the helper function below, used to execute a command and get the return value on posix systems. I used to use popen, but it is impossible to get the return code of an application with popen if it runs and exits before popen/pclose gets a chance to do its work.
The following helper function creates a process fork, uses execvp to run the desired external process, and then the parent uses waitpid to get the return code. I'm seeing odd cases where it's refusing to run.
When called with wait = true, waitpid should return the exit code of the application no matter what. However, I'm seeing stdout output that specifies the return code should be non-zero, yet the return code is zero. Testing the external process in a regular shell, then echoing $? returns non-zero, so it's not a problem w/ the external process not returning the right code. If it's of any help, the external process being run is mount(8) (yes, I know I can use mount(2) but that's besides the point).
I apologize in advance for a code dump. Most of it is debugging/logging:
inline int ForkAndRun(const std::string &command, const std::vector<std::string> &args, bool wait = false, std::string *output = NULL)
{
std::string debug;
std::vector<char*> argv;
for(size_t i = 0; i < args.size(); ++i)
{
argv.push_back(const_cast<char*>(args[i].c_str()));
debug += "\"";
debug += args[i];
debug += "\" ";
}
argv.push_back((char*)NULL);
neosmart::logger.Debug("Executing %s", debug.c_str());
int pipefd[2];
if (pipe(pipefd) != 0)
{
neosmart::logger.Error("Failed to create pipe descriptor when trying to launch %s", debug.c_str());
return EXIT_FAILURE;
}
pid_t pid = fork();
if (pid == 0)
{
close(pipefd[STDIN_FILENO]); //child isn't going to be reading
dup2(pipefd[STDOUT_FILENO], STDOUT_FILENO);
close(pipefd[STDOUT_FILENO]); //now that it's been dup2'd
dup2(pipefd[STDOUT_FILENO], STDERR_FILENO);
if (execvp(command.c_str(), &argv[0]) != 0)
{
exit(EXIT_FAILURE);
}
return 0;
}
else if (pid < 0)
{
neosmart::logger.Error("Failed to fork when trying to launch %s", debug.c_str());
return EXIT_FAILURE;
}
else
{
close(pipefd[STDOUT_FILENO]);
int exitCode = 0;
if (wait)
{
waitpid(pid, &exitCode, wait ? __WALL : (WNOHANG | WUNTRACED));
std::string result;
char buffer[128];
ssize_t bytesRead;
while ((bytesRead = read(pipefd[STDIN_FILENO], buffer, sizeof(buffer)-1)) != 0)
{
buffer[bytesRead] = '\0';
result += buffer;
}
if (wait)
{
if ((WIFEXITED(exitCode)) == 0)
{
neosmart::logger.Error("Failed to run command %s", debug.c_str());
neosmart::logger.Info("Output:\n%s", result.c_str());
}
else
{
neosmart::logger.Debug("Output:\n%s", result.c_str());
exitCode = WEXITSTATUS(exitCode);
if (exitCode != 0)
{
neosmart::logger.Info("Return code %d", (exitCode));
}
}
}
if (output)
{
result.swap(*output);
}
}
close(pipefd[STDIN_FILENO]);
return exitCode;
}
}
Note that the command is run OK with the correct parameters, the function proceeds without any problems, and WIFEXITED returns TRUE. However, WEXITSTATUS returns 0, when it should be returning something else.
Probably isn't your main issue, but I think I see a small problem. In your child process, you have...
dup2(pipefd[STDOUT_FILENO], STDOUT_FILENO);
close(pipefd[STDOUT_FILENO]); //now that it's been dup2'd
dup2(pipefd[STDOUT_FILENO], STDERR_FILENO); //but wait, this pipe is closed!
But I think what you want is:
dup2(pipefd[STDOUT_FILENO], STDOUT_FILENO);
dup2(pipefd[STDOUT_FILENO], STDERR_FILENO);
close(pipefd[STDOUT_FILENO]); //now that it's been dup2'd for both, can close
I don't have much experience with forks and pipes in Linux, but I did write a similar function pretty recently. You can take a look at the code to compare, if you'd like. I know that my function works.
execAndRedirect.cpp
I'm using the mongoose library, and grepping my code for SIGCHLD revealed that using mg_start from mongoose results in setting SIGCHLD to SIG_IGN.
From the waitpid man page, on Linux a SIGCHLD set to SIG_IGN will not create a zombie process, so waitpid will fail if the process has already successfully run and exited - but will run OK if it hasn't yet. This was the cause of the sporadic failure of my code.
Simply re-setting SIGCHLD after calling mg_start to a void function that does absolutely nothing was enough to keep the zombie records from being immediately erased.
Per #Geoff_Montee's advice, there was a bug in my redirect of STDERR, but this was not responsible for the problem as execvp does not store the return value in STDERR or even STDOUT, but rather in the kernel object associated with the parent process (the zombie record).
#jilles' warning about non-contiguity of vector in C++ does not apply for C++03 and up (only valid for C++98, though in practice, most C++98 compilers did use contiguous storage, anyway) and was not related to this issue. However, the advice on reading from the pipe before blocking and checking the output of waitpid is spot-on.
I've found that pclose does NOT block and wait for the process to end, contrary to the documentation (this is on CentOS 6). I've found that I need to call pclose and then call waitpid(pid,&status,0); to get the true return value.