Wrote a program that has a method that opens a popen command to a temporary file, reads the output and parses for use somewhere else in the program. If the command succeeds, program works as expected. However, if the popen tries a command that fails, the file still has a valid pointer but when the program tries to read the data with fgets the program seg faults.
Function body:
std::map<std::string,size_t> cols;
const char* command = command_string.c_str();
if (FILE *fp = popen(command,"r")) {
char buff[linesize];
std::vector<std::string> list;
std::cout << "here, popen succeeded\n";
std::cout << fp << '\n';
while (fgets(buff,linesize,fp)) {
std::cout << "here, fgets succeeded\n";
std::string data(buff);
list.push_back(data);
}
parse_cols(list);
pclose(fp);
}
else {
std::cout << "Failed to open bash shell when trying to run command\n";
std::exit(EXIT_FAILURE);
}
With the output:
here, popen succeeded
0x1cc2430
sh: my_command: command not found
Segmentation fault (core dumped)
Is it possible to handle this error? It's a somewhat useful error, but I'd like to be able to handle it rather then just relying on the seg fault. I tried looking at the FILE struct, but seems to be different for different C library versions.
Popen is a beast. It only returns nullptr when fork or pipe fails, but in your case they do not.
However, your program should not segfault. When shell returns failure, you should be reading from a valid (albeit empty) stream. Than fgets() returns NULL because end of file occurs while no characters have been read.
Than you are calling parse_calls - a function we do not see - but I have reasons to believe it can't handle the empty list. Crash stacks could be of further help when it comes to pinpointing the actual problem, which is not in popen or fgets.
Related
I have a simple C++ application which is supposed to read lines from a POSIX named pipe:
#include<iostream>
#include<string>
#include<fstream>
int main() {
std::ifstream pipe;
pipe.open("in");
std::string line;
while (true) {
std::getline(pipe, line);
if (pipe.eof()) {
break;
}
std::cout << line << std::endl;
}
}
Steps:
I create a named pipe: mkfifo in.
I compile & run the C++ code using g++ -std=c++11 test.cpp && ./a.out.
I feed data to the in pipe:
sleep infinity > in & # keep pipe open, avoid EOF
echo hey > in
echo cats > in
echo foo > in
kill %1 # this closes the pipe, C++ app stops on EOF
When doing this under Linux, the application successfully displays output after each echo command as expected (g++ 8.2.1).
When trying this whole process on macOS, output is only displayed after closing the pipe (i.e. after kill %1).
I started suspecting some sort of buffering issue, so i've tried disabling it like so:
std::ifstream pipe;
pipe.rdbuf()->pubsetbuf(0, 0);
pipe.open("out");
With this change, the application outputs nothing after the first echo, then prints out the first message after the second echo ("hey"), and keeps doing so, alwasy lagging a message behind and displaying the message of the previous echo instead of the one executed.
The last message is only displayed after closing the pipe.
I found out that on macOS g++ is basically clang++, as
g++ --version yields: "Apple LLVM version 10.0.1 (clang-1001.0.46.3)".
After installing the real g++ using Homebrew, the example program works, just like it did on Linux.
I am building a simple IPC library built on named pipes for various reasons, so this working correctly is pretty much a requirement for me at this point.
What is causing this weird behaviour when using LLVM? (update: this is caused by libc++)
Is this a bug?
Is the way this works on g++ guaranteed by the C++ standard in some way?
How could I make this code snippet work properly using clang++?
Update:
This seems to be caused by the libc++ implementation of getline().
Related links:
Why does libc++ getline block when reading from pipe, but libstdc++ getline does not?
https://bugs.llvm.org/show_bug.cgi?id=23078
The questions still stand though.
I have worked around this issue by wrapping POSIX getline() in a simple C API and simply calling that from C++.
The code is something like this:
typedef struct pipe_reader {
FILE* stream;
char* line_buf;
size_t buf_size;
} pipe_reader;
pipe_reader new_reader(const char* pipe_path) {
pipe_reader preader;
preader.stream = fopen(pipe_path, "r");
preader.line_buf = NULL;
preader.buf_size = 0;
return preader;
}
bool check_reader(const pipe_reader* preader) {
if (!preader || preader->stream == NULL) {
return false;
}
return true;
}
const char* recv_msg(pipe_reader* preader) {
if (!check_reader(preader)) {
return NULL;
}
ssize_t read = getline(&preader->line_buf, &preader->buf_size, preader->stream);
if (read > 0) {
preader->line_buf[read - 1] = '\0';
return preader->line_buf;
}
return NULL;
}
void close_reader(pipe_reader* preader) {
if (!check_reader(preader)) {
return;
}
fclose(preader->stream);
preader->stream = NULL;
if (preader->line_buf) {
free(preader->line_buf);
preader->line_buf = NULL;
}
}
This works well against libc++ or libstdc++.
As discussed separately, a boost::asio solution would be best, but your question is specifically about how getline is blocking, so I will talk to that.
The problem here is that std::ifstream is not really made for a FIFO file type. In the case of getline(), it is trying to do a buffered read, so (in the initial case) it decides the buffer does not have enough data to reach the delimiter ('\n'), calls underflow() on the underlying streambuf, and that does a simple read for a buffer-length amount of data. This works great for files because the file's length at a point in time is a knowable length, so it can return EOF if there's not enough data to fill the buffer, and if there is enough data, it simply returns with the filled buffer. With a FIFO, however, running out of data does not necessarily mean EOF, so it doesn't return until the process that writes to it closes (this is your infinite sleep command that holds it open).
A more typical way to do this is for the writer to open and close the file as it reads and writes. This is obviously a waste of effort when something more functional like poll()/epoll() is available, but I'm answering the question you're asking.
I want to be able to read and write to a program from C++. It seems like pstream can do the job, but I find the documentation difficult to understand and have not yet find an example.
I have setup the following minimum working example. This opens python, which in turn (1) prints hello (2) ask input, and (3) prints hello2:
#include <iostream>
#include <cstdio>
#include "pstream.h"
using namespace std;
int main(){
std::cout << "start";
redi::pstream proc(R"(python -c "if 1:
print 'hello'
raw_input()
print 'hello2'
")");
std::string line;
//std::cout.flush();
while (std::getline(proc.out(), line)){
std::cout << " " << "stdout: " << line << '\n';
}
std::cout << "end";
return 0;
}
If I run this with the "ask input" part commented out (i.e. #raw_input()), I get as output:
start stdout: hello
stdout: hello2
end
But if I leave the "ask input" part in (i.e. uncommented raw_input()), all I get is blank, not even start, but rather what seems like a program waiting for input.
My question is, how can one interact with this pstream, how can one establish a little read-write-read-write session? Why does the program not even show start or the first hello?
EDIT:
I don't seem to be making much progress. I don't think I really grasp what is going on. Here are some further attempts with commentary.
1) It seems like I can successfully feed raw_inputI prove this by writing to the child's stderr:
int main(){
cout << "start" <<endl;
redi::pstream proc(R"(python -c "if 1:
import sys
print 'hello'
sys.stdout.flush()
a = raw_input()
sys.stdin.flush()
sys.stderr.write('hello2 '+ a)
sys.stderr.flush()
")");
string line;
getline(proc.out(), line);
cout << line << endl;
proc.write("foo",3).flush();
cout << "end" << endl;
return 0;
}
output:
start
hello
end
hello2 foo
But it locks if I try to read from the stdout again
int main(){
...
a = raw_input()
sys.stdin.flush()
print 'hello2', a
sys.stdout.flush()
")");
...
proc.write("foo",3).flush();
std::getline(proc.out(), line);
cout << line << endl;
...
}
output
start
hello
2) I can't get the readsome approach to work at all
int main(){
cout << "start" <<endl;
redi::pstream proc(R"(python -c "if 1:
import sys
print 'hello'
sys.stdout.flush()
a = raw_input()
sys.stdin.flush()
")");
std::streamsize n;
char buf[1024];
while ((n = proc.out().readsome(buf, sizeof(buf))) > 0)
std::cout.write(buf, n).flush();
proc.write("foo",3).flush();
cout << "end" << endl;
return 0;
}
output
start
end
Traceback (most recent call last):
File "<string>", line 5, in <module>
IOError: [Errno 32] Broken pipe
The output contains a Python error, it seems like the C++ program finished while the Python pipe were still open.
Question: Can anyone provide a working example of how this sequential communication should be coded?
But if I leave the "ask input" part in (i.e. uncommented raw_input()), all I get is blank, not even start, but rather what seems like a program waiting for input.
The Python process is waiting for input, from its stdin, which is connected to a pipe in your C++ program. If you don't write to the pstream then the Python process will never receive anything.
The reason you don't see "start" is that Python thinks it's not connected to a terminal, so it doesn't bother flushing every write to stdout. Try import sys and then sys.stdout.flush() after printing in the Python program. If you need it to be interactive then you need to flush regularly, or set stdout to non-buffered mode (I don't know how to do that in Python).
You should also be aware that just using getline in a loop will block waiting for more input, and if the Python process is also blocking waiting for input you have a deadlock. See the usage example on the pstreams home page showing how to use readsome() for non-blocking reads. That will allow you to read as much as is available, process it, then send a response back to the child process, so that it produces more output.
EDIT:
I don't think I really grasp what is going on.
Your problems are not really problems with pstreams or python, you're just not thinking through the interactions between two communicating processes and what each is waiting for.
Get a pen and paper and draw state diagrams or some kind of chart that shows where the two processes have got to, and what they are waiting for.
1) It seems like I can successfully feed raw_input
Yes, but you're doing it wrong. raw_input() reads a line, you aren't writing a line, you're writing three characters, "foo". That's not a line.
That means the python process keeps trying to read from its stdin. The parent C++ process writes the three characters then exits, running the pstream destructor which closes the pipes. Closing the pipes causes the Python process so get EOF, so it stops reading (after only getting three characters not a whole line). The Python process then prints to stderr, which is connected to your terminal, because you didn't tell the pstream to attach a pipe to the child's stderr, and so you see that output.
But it locks if I try to read from the stdout again
Because now the parent C++ process doesn't exit, so doesn't close the pipes, so the child Python process doesn't read EOF and keeps waiting for more input. The parent C++ process is also waiting for input, but that will never come.
If you want to send a line to be read by raw_input() then write a newline!
This works fine, because it sends a newline, which causes the Python process to get past the raw_input() line:
cout << "start" <<endl;
redi::pstream proc(R"(python -c "if 1:
import sys
print 'hello'
sys.stdout.flush()
a = raw_input()
print 'hello2', a
sys.stdout.flush()
")");
string line;
getline(proc, line);
cout << line << endl;
proc << "foo" << endl; // write to child FOLLOWED BY NEWLINE!
std::getline(proc, line); // read child's response
cout << line << endl;
cout << "end" << endl;
N.B. you don't need to use proc.out() because you haven't attached a pipe to the process' stderr, so it always reads from proc.out(). You would only need to use that when reading from both stdout and stderr, where you would use proc.out() and proc.err() to distinguish them.
2) I can't get the readsome approach to work at all
Again, you have the same problem that you're only writing three characters and so the Python processes waits forever. The C++ process is trying to read as well, so it also waits forever. Deadlock.
If you fix that by sending a newline (as shown above) you have another problem: the C++ program will run so fast that it will get to the while loop that calls readsome before the Python process has even started. It will find nothing to read in the pipe, and so the first readsome call returns 0 and you exit the loop. Then the C++ program gets to the second while loop, and the child python process still hasn't started printing anything yet, so that loop also reads nothing and exits. Then the whole C++ program exits, and finally the Python child is ready to run and tries to print "hello" but by then it's parent is gone and it can't write to the pipe.
You need readsome to keep trying if there's nothing to read the first time you call it_, so it waits long enough for the first data to be readable.
For your simple program you don't really need readsome because the Python process only writes a single line at a time, so you can just read it with getline. But if it might write more than one line you need to be able to keep reading until there's no more data coming, which readsome can do (it reads only if there's data available). But you also need some way to tell whether more data is still going to come (maybe the child is busy doing some calculations before it sends more data) or if it's really finished. There's no general way to know that, it depends on what the child process is doing. Maybe you need the child to send some sentinel value, like "---END OF RESPONSE---", which the parent can look for to know when to stop trying to read more.
For the purposes of your simple example, let's just assume that if readsome gets more than 4 bytes it received the whole response:
cout << "start" <<endl;
redi::pstream proc(R"(python -c "if 1:
import sys
print 'hello'
sys.stdout.flush()
a = raw_input()
sys.stdin.flush()
print 'hello2', a
sys.stdout.flush()
")");
string reply;
streamsize n;
char buf[1024];
while ((n = proc.readsome(buf, sizeof(buf))) != -1)
{
if (n > 0)
reply.append(buf, n);
else
{
// Didn't read anything. Is that a problem?
// Need to try to process the content of 'reply' and see if
// it's what we're expecting, or if it seems to be incomplete.
//
// Let's assume that if we've already read more than 4 characters
// it's a complete response and there's no more to come:
if (reply.length() > 3)
break;
}
}
cout << reply << std::flush;
proc << "foo" << std::endl;
while (getline(proc, reply)) // maybe use readsome again here
cout << reply << std::endl;
cout << "end" << endl;
This loops while readsome() != -1, so it keeps retrying if it reads nothing and only stops the loop if there's an error. In the loop body it decides what to do if nothing was read. You'll need to insert your own logic in here that makes sense for whatever you're trying to do, but basically if readsome() hasn't read anything yet, then you should loop and retry. That makes the C++ program wait long enough for the Python program to print something.
You'd probably want to split out the while loop into a separate function that reads a whole reply into a std::string and returns it, so that you can re-use that function every time you want to read a response. If the child sends some sentinel value that function would be easy to write, as it would just stop every time it receives the sentinel string.
I want to extract the framesize of a video from a file. For this purpose, I have launched an ffmpeg command via bash shell, and I want to extract the output. This command is working well in the bash shell, and returns the output as wanted.
ffprobe -v error -count_frames -of flat=s=_ -select_streams v:0 -show_entries stream=nb_read_frames /home/peter/DA/videos/IMG-2014-1-10-10-4-37.avi
I want to call it via C++ and read out the result. I use the IDE Qt 4.8.6 with GCC 4.8 compiler.
For my code, I use this template:
executing shell command with popen
and changed it for my demands to
#include <iostream>
#include <string>
#include <stdio.h>
using namespace std;
int main()
{
FILE* pipe = popen("echo $(ffprobe -v error -count_frames -of flat=s=_ -select_streams v:0 -show_entries stream=nb_read_frames /home/peter/DA/videos/IMG-2014-1-10-10-4-37.avi)", "r");
if(!pipe)
{
cout << "error" << endl;
return 1;
}
char* buffer = new char[512];
string result;
fgets(buffer, sizeof(buffer), pipe) ;
while(!feof(pipe))
{
if(fgets(buffer, sizeof(buffer), pipe) != NULL)
{
cout << buffer << endl;
result += buffer;
}
}
pclose(pipe);
cout << result<< endl;
return 0;
}
The Qt console returned me this warning, and it is rending with return 0:
/home/peter/DA/videos/IMG-2014-1-10-10-4-37.avi: Invalid data found when processing input
and "pipe" is empty.
When I compile the main.cpp file above with g++ in the shell it works nice too.
Old post, but as I see, there are two points here:
Error "Invalid data found when processing input"
That's an ffprobe normal file processing error. Usually it happens when there are errors inside media file, it is not related to c++ program.
ffprobe writes warning/error messages into stderr stream, but popen only captures stdout stream, that's why your program couldn't get that error message trough the pipe.
How get the stdout+stderr in my program
popen allows execute any shell command, so we can use it to redirect stderr into stdout, so your program can get that output too, like this:
FILE *pipe = popen("ffprobe ... 2>&1");
The 2> redirect handle#2 output into current &1 handle#1 output (#1=stdout, #2=stderr).
There's absolute no need to execute FILE *pipe = popen("echo $(ffprobe ...)");, because the final result will be the same: Note that $(...) returns a string with stdout command output, and echo prints it. Totally redundant.
A few observations in order to improve your code:
When a string is too big to be displayed in one screen width, it's better split it into multiple lines (maybe grouping text inside each line within some logic), because that will improve the reading of your code by other people (and eventually by yourself in a few months).
You can do this with a C/C++ compiler feature that concatenates strings separated by spaces (newlines, tab, etc.), ex. "hi " "world" is the same as "hi world" to the compiler.
When your program have to write error messages, use the stderr stream. In c++ that's std::cerr instead std::cout.
Always free memory allocated when it's no loger used (each new has to have a delete)
Avoid use using namespace std;, instead use using std::name; for each standard instance/class that you'll use. Ex. using std::string;, that avoids future problems, specially in big programs. An example of a common error is here. In general avoid using using namespace xxxx;.
Reorganizing your code, we have:
#include <iostream>
#include <stdio.h>
using std::string;
using std::cout;
using std::cerr;
using std::endl;
int main() {
static char ffprobeCmd[] =
"ffprobe " // command
"-v error " // args
"-count_frames "
"-of flat=s=_ "
"-select_streams v:0 "
"-show_entries stream=nb_read_frames "
"/home/peter/DA/videos/IMG-2014-1-10-10-4-37.avi" // file
" 2>&1"; // Send stderr to stdout
FILE *pipe = popen(ffprobeCmd, "r");
if (!pipe) {
perror("Cannot open pipe.");
return 1;
}
char* buffer = new char[512];
string result;
while ((fgets(buffer, sizeof(buffer), pipe)) != NULL) {
result += buffer;
}
// See Note below
int retCode = pclose(pipe);
if (retCode != 0) {
// Program ends with error, and result has the error message
cerr << "retCode: " << retCode << "\nMessage: " << result << endl;
return retCode;
} else {
// Program ends normally, prints: streams_stream_0_nb_read_frames="xxx"
cout << result << endl;
}
delete buffer; // free memory
return 0;
}
Note
pclose is not intended to return the executed program status code, but if you need this value, pclose does it in some c++ versions/systems, so check it. Anyway it will be zero only if everything was OK.
Trying to write .ply parser to use .ply models in OpenGL.
Trying to begin to read the .ply file and write all the lines of it out.
My program does this but when it print out the last line i get Unhandled exception:
Unhandled exception at 0x62aad540 (msvcr100d.dll) in PLY parser.exe: 0xC0000005: Access violation reading location 0x00000000.
This is my code:
#include <iostream>
#include <fstream>
#include <string>
#include <stdio.h>
using namespace std;
int main ()
{
char buffer[10000];
FILE * myFile;
myFile = fopen("walkman.ply", "r");
if(myFile != NULL)
{
while (!feof(myFile))
{
cout<<fgets(buffer, 10000, myFile);
}
fclose(myFile);
}
else
{
cout<<"file not found"<<endl;
}
system("pause");
return 0;
}
This may be foolish error in my code, but it would be great if someone can spot the error causing this.
Before we get into what the bug is, you should know that the "Unhandled exception ... Access violation reading location 0x00000000" message you get is not caused by a C++ exception; it's the Windows equivalent of "Segmentation fault". Your code tried to dereference the NULL pointer.
Now, you have made one of the classic mistakes in working with FILE objects. feof(fp) does not become true when you reach the end of the file. It only becomes true after you have attempted to read past the end of the file at least once. Thus, your read loop will iterate until after fgets attempts to read past the end of the file. And when fgets attempts to read past the end of the file, it fails, and returns a NULL pointer, which you blindly passed to cout. Kaboom.
(By the way, this is also how istream::eof() works.)
The correct way to write this loop is
while (fgets(buffer, 10000, myFile))
cout << buffer;
(Or, even better, one of these:
while (fgets(buffer, 10000, myFile))
fputs(buffer, stdout));
while(myFile.get(buffer, 10000))
cout << buffer;
it being a little weird to mix stdio.h FILEs and iostreams as you are doing.)
feof() tells you that you've tried to read past the end of file, not that you've reached the end of file. fgets() returns NULL when you're at the end of file and there's no more data to read. That's where the exception is coming from. At the end of file, feof() will return false and fgets() will return NULL, which will cause the exception when your program tries to execute cout << NULL;.
This is the idiomatic way to write it in C style:
char buffer[10000];
FILE* myFile = fopen("walkman.ply", "r");
if (myFile != NULL) {
while (fgets(buffer, sizeof(buffer), myFiles) {
fputs(buffer, stdout);
}
fclose(myFile);
}
or in C++ style:
std::string buffer;
std::ifstream myFile("walkman.ply");
if (myFile.is_open()) {
while (std::getline(myFile, buffer)) {
std::cout << buffer << '\n';
}
}
EDIT: my prognosis was wrong but still read the following if you were meant to use streams.
Also please consider using streams (in ) and vectors, these methods are a lot less error prone and fit in with the C++ style and ethos.
std::ifstream in("walkman.ply", std::ios::binary);
std::vector<char> v(istream_iterator<char>(in),
istream_iterator<char>( ));
or if that is supposed to a string value.
std::ifstream in("walkman.ply");
std::string str(istream_iterator<char>(in),
istream_iterator<char>( ));
std::cout << str << std::endl;
My program crashes on this part of code:
if(fclose(_device) != SUCCESS){
cerr << "Output device library error CLOSING FILE\n";
exit(1);
}
It doesn't print anything, and when i write instead this line:
cout << fclose(_device)<<endl;
It doesn't print anything either, and just crashes my program with no further comments.
In an earlier part of my program, i initialize the file with this line:
_device = fopen ((char*)filename , "a");
What can cause such a crash to my program?
Also if your program has bugs which result in writing memory randomly, it might be that the information fclose needs to use to close the file gets overwritten.
You could try to use a memory checking tool, like valgrind, to check this is not the case.
Could be a failure on open, after
_device = fopen ((char*)filename , "a");
check that _device != NULL
Edit Since you are checking that _device is valid after being opened, I'd tend to use the debugger to check the value of _device on opening and compare it to the value being passed to _fclose. Out of interest, is _device pointing to a file or a comms devices such as "COM2:" as this could also have some bearing on the problem. Lastly, I'd break down your final statement as follows;
int CloseResult = fclose(_device);
if (CloseResult != 0)
cout << errno << CloseResult << endl;
Reason for this is that you don't know if the fclose or stream output is the cause of your crash. I'm assuming that the stream your outputting to isn't linked to the file your trying to close ;)