Raspberry Pi C++ Read NMEA Sentences from Adafruit's Ultimate GPS Module - c++
I'm trying to read the GPS NMEA sentences from Adafruit's Ultimate GPS module. I'm using C++ on the raspberry pi to read the serial port connection to the module
Here is my read function:
int Linuxutils::readFromSerialPort(int fd, int bufferSize) {
/*
Reading data from a port is a little trickier. When you operate the port in raw data mode,
each read(2) system call will return however many characters are actually available in the
serial input buffers. If no characters are available, the call will block (wait) until
characters come in, an interval timer expires, or an error occurs. The read function can be
made to return immediately by doing the following:
fcntl(fd, F_SETFL, FNDELAY);
The NDELAY option causes the read function to return 0 if no characters are available on the port.
*/
// Check the file descriptor
if ( !checkFileDecriptorIsValid(fd) ) {
fprintf(stderr, "Could not read from serial port - it is not a valid file descriptor!\n");
return -1;
}
// Now, let's wait for an input from the serial port.
fcntl(fd, F_SETFL, 0); // block until data comes in
// Now read the data
int absoluteMax = bufferSize*2;
char *buffer = (char*) malloc(sizeof(char) * bufferSize); // allocate buffer.
int rcount = 0;
int length = 0;
// Read in each newline
FILE* fdF = fdopen(fd, "r");
int ch = getc(fdF);
while ( (ch != '\n') ) { // Check for end of file or newline
// Reached end of file
if ( ch == EOF ) {
printf("ERROR: EOF!");
continue;
}
// Expand by reallocating if necessary
if( rcount == absoluteMax ) { // time to expand ?
absoluteMax *= 2; // expand to double the current size of anything similar.
rcount = 0; // Re-init count
buffer = (char*)realloc(buffer, absoluteMax); // Re-allocate memory.
}
// Read from stream
ch = getc(fdF);
// Stuff in buffer
buffer[length] = ch;
// Increment counters
length++;
rcount++;
}
// Don't care if we return 0 chars read
if ( rcount == 0 ) {
return 0;
}
// Stick
buffer[rcount] = '\0';
// Print results
printf("Received ( %d bytes ): %s\n", rcount,buffer);
// Return bytes read
return rcount;
}
So I kind of get the sentences as you can see below, the problem is I get these "repeated" portions of a complete sentence like this:
Received ( 15 bytes ): M,-31.4,M,,*61
Here is the complete thing:
Received ( 72 bytes ): GPGGA,182452.000,4456.2019,N,09337.0243,W,1,8,1.19,292.6,M,-31.4,M,,*61
Received ( 56 bytes ): GPGSA,A,3,17,07,28,26,08,11,01,09,,,,,1.49,1.19,0.91*00
Received ( 15 bytes ): M,-31.4,M,,*61
Received ( 72 bytes ): GPGGA,182453.000,4456.2019,N,09337.0242,W,1,8,1.19,292.6,M,-31.4,M,,*61
Received ( 56 bytes ): GPGSA,A,3,17,07,28,26,08,11,01,09,,,,,1.49,1.19,0.91*00
Received ( 15 bytes ): M,-31.4,M,,*61
Received ( 72 bytes ): GPGGA,182456.000,4456.2022,N,09337.0241,W,1,8,1.21,292.6,M,-31.4,M,,*64
Received ( 56 bytes ): GPGSA,A,3,17,07,28,26,08,11,01,09,,,,,2.45,1.21,2.13*0C
Received ( 70 bytes ): GPRMC,182456.000,A,4456.2022,N,09337.0241,W,0.40,183.74,110813,,,A*7F
Received ( 37 bytes ): GPVTG,183.74,T,,M,0.40,N,0.73,K,A*34
Received ( 70 bytes ): GPRMC,182453.000,A,4456.2019,N,09337.0242,W,0.29,183.74,110813,,,A*7E
Received ( 37 bytes ): GPVTG,183.74,T,,M,0.29,N,0.55,K,A*3F
Received ( 32 bytes ): 242,W,0.29,183.74,110813,,,A*7E
Received ( 70 bytes ): GPRMC,182452.000,A,4456.2019,N,09337.0243,W,0.33,183.74,110813,,,A*75
Why am I getting the repeated sentences and how can I fix it? I tried flushing the serial port buffers but then things became really ugly! Thanks.
I'm not sure I understand your exact problem. There are a few problems with the function though which might explain a variety of errors.
The lines
int absoluteMax = bufferSize*2;
char *buffer = (char*) malloc(sizeof(char) * bufferSize); // allocate buffer.
seem wrong. You'll decide when to grow the buffer by comparing the number of characters read to absoluteMax so this needs to match the size of the buffer allocated. You're currently writing beyond the end of allocated memory before you reallocate. This results in undefined behaviour. If you're lucky your app will crash, if you're unlucky, things will appear to work but you'll lose the second half of the data you've read since only the data written to memory you own will be moved by realloc (if it relocates your heap cell).
Also, you shouldn't cast the return from malloc (or realloc) and can rely on sizeof(char) being 1.
You lose the first character read (the one that is read just before the while loop). Is this deliberate?
When you reallocate buffer, you shouldn't reset rcount. This causes the same bug as above where you'll write beyond the end of buffer before reallocating again. Again, the effects of doing this are undefined but could include losing portions of output.
Not related to the bug you're currently concerned with but also worth noting is the fact that you leak buffer and fdF. You should free and fclose them respectively before exiting the function.
The following (untested) version ought to fix these issues
int Linuxutils::readFromSerialPort(int fd, int bufferSize)
{
if ( !checkFileDecriptorIsValid(fd) ) {
fprintf(stderr, "Could not read from serial port - it is not a valid file descriptor!\n");
return -1;
}
fcntl(fd, F_SETFL, 0); // block until data comes in
int absoluteMax = bufferSize;
char *buffer = malloc(bufferSize);
int rcount = 0;
int length = 0;
// Read in each newline
FILE* fdF = fdopen(fd, "r");
int ch = getc(fdF);
for (;;) {
int ch = getc(fdF);
if (ch == '\n') {
break;
}
if (ch == EOF) { // Reached end of file
printf("ERROR: EOF!\n");
break;
}
if (length+1 >= absoluteMax) {
absoluteMax *= 2;
char* tmp = realloc(buffer, absoluteMax);
if (tmp == NULL) {
printf("ERROR: OOM\n");
goto cleanup;
}
buffer = tmp;
}
buffer[length++] = ch;
}
if (length == 0) {
return 0;
}
buffer[length] = '\0';
// Print results
printf("Received ( %d bytes ): %s\n", rcount,buffer);
cleanup:
free(buffer);
fclose(fdH);
return length;
}
Maybe you could try to flush serial port buffers before reading from it as shown in this link ?
I would also consider not reopening the serial port every time you call Linuxutils::readFromSerialPort - you could keep the file descriptor open for further reading (anyway the call is blocking so from the caller's point of view nothing changes).
Related
stringstream vs ifstream (ofstream) in c++ using socket programming
I have one question about socket programming in C++. Most of the tutorials I found on web assume that (binding etc. is omitted) there is a string at client process it is saved to a file then the file is sent to server by first reading the file into stream server receives the stream and write it into another file. Then, my question is that what if we can use stringstrem at step 2 instead of saving as a file? File I/O (in C++ ifstream and ofstream are typically used) is generally slow. Is it more efficient if I use stringstream directory?
Your Original Question: "What if we can use stringstrem at step 2 instead of saving as a file?" My Initial Response: stringstream has nothing to do with server sockets and IO files. You are lacking the fundamental idea of IO operations which is the concept of files for IO devices. There is no way around. You save nothing in a logical file stream. Your file bytes are buffered temporarily on your memory and flushed. stringstream is a nice C++ library utility that let's you treat strings as file streams. Just like you read from an input file stream bytes after bytes until EOF/some other errors, or write into an output file stream bytes after bytes, using stringstream you can treat your string like the way you do to file streams. This is really helpful when you want to divide your string in small logical units. For example, suppose you read a string line and want to read each word from that line by treating the string line as a stream of words. Further Instructions To Guide You To The Right Direction: Nothing is "saved" in a logical file stream. Every I/O operation is performed through "logical" files in any computer system. Socket connection has two file descriptors on both ends: one is a client file descriptor and another one is a server file descriptor (connected file descriptor). Server listens for connection requests through a listening file descriptor which actually stays around as long as the lifetime of the server, and when it accepts a connection request, it returns another file descriptor through accept function called connected file descriptor that stays around as long as the client-server connection/transaction is ongoing. int accept(int listenfd, struct sockaddr *addr, int *addrlen); If you want to read from or write into a file stream and also wish to buffer your file bytes, you exactly need to do that- buffer your bytes. This is also very important in the context of servers and short counts because your connection might time out or it might get interrupted by signals. There are several options and techniques that you might implement. However, such discussions are not possible in this small thread. What I'm going to do based on your question is give you an example of how you can buffer your file stream, avoid short count, and handle signal interruptions through following steps: For example, following is a function that reads n bytes and doesn't buffer ssize_t rio_readn(int fd, void *usrbuf, size_t n) { size_t nleft = n; ssize_t nread; char *bufp = usrbuf; while (nleft > 0) { if ((nread = read(fd, bufp, nleft)) < 0) { if (errno == EINTR) /* Interrupted by sig handler return */ nread = 0;/* and call read() again */ else return -1;/* errno set by read() */ } else if (nread == 0) break;/* EOF */ nleft -= nread; bufp += nread; } return (n - nleft);/* Return >= 0 */ } We can implement the following steps to do buffered and robust IO operations (note RIO means robust IO): Step 1: Set up empty read buffer and associate an open file descriptor so that we can implement our robust IO operations #define RIO_BUFSIZE 8192 typedef struct { int rio_fd;/* Descriptor for this internal buf */ int rio_cnt;/* Unread bytes in internal buf */ char *rio_bufptr;/* Next unread byte in internal buf */ char rio_buf[RIO_BUFSIZE]; /* Internal buffer */ } rio_t; //Initialize robust IO buffer void rio_readinitb(rio_t *rp, int fd) { rp->rio_fd = fd; rp->rio_cnt = 0; rp->rio_bufptr = rp->rio_buf; } Step 2: A robust read utility function to handle short count static ssize_t rio_read(rio_t *rp, char *usrbuf, size_t n) { int cnt; while (rp->rio_cnt <= 0) {/* Refill if buf is empty */ rp->rio_cnt = read(rp->rio_fd, rp->rio_buf, sizeof(rp->rio_buf)); if (rp->rio_cnt < 0) { if (errno != EINTR) /* Interrupted by sig handler return */ return -1; } else if (rp->rio_cnt == 0)/* EOF */ return 0; else rp->rio_bufptr = rp->rio_buf; /* Reset buffer ptr */ } /* Copy min(n, rp->rio_cnt) bytes from internal buf to user buf */ cnt = n; if (rp->rio_cnt < n) cnt = rp->rio_cnt; memcpy(usrbuf, rp->rio_bufptr, cnt); rp->rio_bufptr += cnt; rp->rio_cnt -= cnt; return cnt; } Step 3: A robust IO function for buffered reading ssize_t rio_readnb(rio_t *rp, void *usrbuf, size_t n) { size_t nleft = n; ssize_t nread; char *bufp = usrbuf; while (nleft > 0) { if ((nread = rio_read(rp, bufp, nleft)) < 0) { if (errno == EINTR) /* Interrupted by sig handler return */ nread = 0;/* Call read() again */ else return -1;/* errno set by read() */ } else if (nread == 0) break;/* EOF */ nleft -= nread; bufp += nread; } return (n - nleft);/* Return >= 0 */ }
Creating and sending raw data packets in C/C++
I know how my packet looks like. It has 6 header fields (1 byte each, each header has 8 fields) and then it has the payload (data). I would like to build a raw packet in C or C++ (it should look the same I think). Here's what I think I should do: unsigned char packet[11]; packet[0] = (0x81); // first header with 8 fields packet[1] = (0x8c); // second header with 8 fields packet[2] = (0xfe); packet[3] = (0x84); packet[4] = (0x1d); packet[5] = (0x79); packet[6] = (0x96); // payload, the 'h' letter, masked packet[7] = (0xe1); // 'e' packet[8] = (0x71); // 'l' packet[9] = (0x15); // 'l' packet[10] = (0x91);// 'o' Where, for instance, 0x81 is the first byte (I simply converted every field (bit) of my first header to hex). And then, simply, I want to send it to server: send(sockfd, packet, sizeof(packet), 0) to send it. Receiving and printing the response: unsigned char buffer[1024]; if ((recv(sockfd, buffer, len, 0)) == 0) { if (errno != 0) { exit(1); } } int i; for(i = 0; i<len; i++) printf("%x ", buffer[i]); Am I right?
Other than mishandling the return value from recv, your code looks okay. if ((recv(sockfd, buffer, len, 0)) == 0) { if (errno != 0) { exit(1); } } A zero return indicates normal close of the connection. There's no reason to check errno if it returns zero. A return value of -1 indicates an error. In that case, it does make sense to check errno. A value greater than zero indicates that number of bytes have been received. Be aware that it is perfectly normal for recv to return fewer bytes than you asked it for. If you want to receive exactly some number of bytes, you must call recv in a loop. TCP is a byte-stream protocol and has no idea where your "packets" (really, messages) begin and end.
Your code will not appear to be error-prone! But a good practice would be: const std::uint32_t BUFFER_SIZE = 11; std::vector<std::uint8_t> buffer; buffer.reserve(BUFFER_SIZE) buffer = {0x81,0x8c.....}; send( sockfd, reinterpret_cast <const char*> ( buffer.data() ), static_cast <int> ( buffer.size() ), 0 ); Doing so, your code gets more optimized, and avoids possible leaks, using the std vectors. May also benefit from taking a look at ZeroMQ, as an example of a ready-made, high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications.
How do you loop read for c++ sockets?
How do you loop read for c++ sockets. stringstream ss; while (1) { char buf [2]; ssize_t e = read(fd, buf, 2); ss << buf[0]; ss << buf[1]; cout << e << endl; cout << buf[0] << endl; cout << buf[1] << endl; if (e <= 0) { break; } } string msg = ""; ss >> msg; When I telnet and type hello and then enter to test this out, I get 2 h e 2 l l 2 o 1 And it just waits there, no looping. What is going on, and how do I read sockets? How do I signify EOF using telnet buf in the context of write(fd, buf, bufSize);
What do you mean no looping? Typo some more, press enter and you'll see some more output.. If you finished, you have to press Ctrl+D to signal the end of input (EOF).
How do I signify EOF using 1) telnet 2) buf in the context of write(fd, buf, bufSize); First, it's not EOF that you want to signal. It's "end of message" or "the data you have just received is a complete protocol data unit and you should now process it". And the answer is, however you want, so long as both sides agree. For example, you could reserve the ~ character specifically to mark the end of a message. To indicate the end of a message, one side sends a ~. The other side knows that when it receives a ~, everything before that is a single message and everything after it is a new message. You can also send the length of a message prior to sending the actual message. You can do it in text followed by a zero byte or by a line feed. Or you can make each message a line. You can do it however makes the most sense in your specific application, but you actually have to do it. There are two cases where TCP does do it for you: 1) When a side is completely done sending and knows the other side will not send anymore data anyway, it can use close on the socket. The other side will read the rest of the data already sent and then read will return 0. 2) When a side is not going to send any more data but the other side might send more data, it can shutdown its side of the connection. It can never send any more data though. The other side will know about this shutdown, and if specifically coded to do so, can continue to send data until it shuts down its half. Then the other side's read will return zero. (Note that if you do this to a program that wasn't designed to take it, it will likely get confused and not realize it can still send data to you.) Otherwise, it's all up to you.
TCP packets are a stream, not a file. It is up to you how the stream is read. In general, if you write 20 bytes at one end, you'll get 20 bytes at the other end in one read, barring the use of some common tcp/ip options. There are assumptions made below. Some of the assumptions are: Assume all the scaffolding code is there. Assume I made mistakes and debug it yourself. Assume all byte order issues are handled. Assume you're smart enough not to send floating point types as binary. Some programmers choose to use a length prefixed packet and read the length byte(2) and then read as many bytes as that length represents, like so. unsigned char buffer[MAX_CHAR] = ""; unsigned char length = 0; int bytesRead = 0; read( fd, sizeof( length ), &length ); // Handle failure from read while( bytesRead < length ) { int readRv = read( fd, &buffer[bytesRead], length - bytesRead ); if( readRv <= 0 ) // 0 usually means socket was closed. -1 is an erro { // Handle socket error/closed socket } else if( readRv < length ) { bytesRead += readRv; } else if( readRv == length ) { bytesRead = readRv; break; } } Some programmers read what is available and look for an end of packet marker unsigned char buffer[MAX_CHAR] = ""; unsigned char length = 0; int bytesRead = 0; int readRv = read( fd, buffer, sizeof( buffer ) ); int eopFound = 0; if( (eopFound = findEndOfPacketMarker( buffer )) > 0 ) { // Less than 0 = error. } else if( eopFound == 0 ) { // No end of packet, must read more bytes here } else { // Found an end of packet marker } // Here you deal with bytes that are "over read" // Either the // 1) packet was bigger than buffer, // 2) there were bytes left over after the EOP marker // 3) Martians crapped in your code and screwed it all up. :) Some programmers use a fixed sized set of packets and a packet type id int packetType; unsigned char buffer[MAX_CHAR] = ""; unsigned char length = 0; int bytesRead = 0; read( fd, sizeof( packetType ), &packetType ); read( fd, getLengthFromPacketType( packetType ), buffer ); That should cover most things... If you have more questions, please ask in the comments with the #JimR thingy so it shows up as a message for me to read.
how to receive the large data using recv()?
i developed client server program using c++,so i want to receive more than 500kb , my client message is terminated with "!" ,so i want to receive until my last byte(!) receive , this is my code it doesn't work.what is wrong with it. do { int num = recv(*csock, buf, bytesLeft,0); if (num == 0) { break; } else if (num < 0 && errno != EINTR) { fprintf(stderr, "Exit %d\n", __LINE__); exit(1); } else if (num > 0) { numRd += num; buf += num; bytesLeft -= num; fprintf(stderr, "read %d bytes - remaining = %d\n", num, bytesLeft); } } while (bytesLeft != 0); fprintf(stderr, "read total of %d bytes\n", numRd);
While I'm not sure exactly what your problem is because of the wording of your question, you generally can't use strcat to append raw buffers received over the network unless you know specifically they will be NULL-terminated, and even then, that's not really "safe" in the event you get an unexpected data transmission. The assumption with c-strings is that they are NULL-terminated, but a raw network buffer may not be, and using strcat will cause you to over-run the input buffer should it not be NULL-terminated. Instead of strcat, use a known fixed-size buffer of size N bytes for receiving the data into, and increment a temporary pointer through the buffer until you reach the end of the buffer or the end of the packet transmission. That way you will always read from the network up to N bytes and no more, and prevent buffer over-run situations from occuring. For instance, you can do the following (this is not the fastest or more efficient solution because of all the copying, but it works): unsigned char buf[10000]; //10Kb fixed-size buffer unsigned char buffer[MAXRECV]; //temporary buffer unsigned char* temp_buf = buf; unsigned char* end_buf = buf + sizeof(buf); do { iByteCount = recv(GetSocketId(), buffer,MAXRECV,0); if ( iByteCount > 0 ) { //make sure we're not about to go over the end of the buffer if (!((temp_buf + iByteCount) <= end_buf)) break; fprintf(stderr, "Bytes received: %d\n",iByteCount); memcpy(temp_buf, buffer, iByteCount); temp_buf += iByteCount; } else if ( iByteCount == 0 ) { if(temp_buf != buf) { //do process with received data } else { fprintf(stderr, "receive failed"); break; } } else { fprintf(stderr, "recv failed: "); break; } } while(iByteCount > 0 && temp_ptr < end_buf); //check for end of buffer
Do you need all 1MB+ of data in one contiguous byte buffer? If so, and you stick with that protocol that has a terminating '!' and does not have a header that includes the length, then you ar stuck with memcpy() and realloc() a lot or some other buffer type like std::vector which, really just does the same thing. If you don't need all those bytes in one string, you can store them in some other way, eg. a vector of *buffer, and so avoid copying.
Assuming you are using a blocking socket (which is the default mode for sockets), then recv() will block waiting for the full MAXRECV number of bytes to arrive. If the client sends less than that number of bytes, recv() will block waiting for data that does not arrive. To work around that, you need to either: 1) call recv() with a 1-byte buffer, calling recv() until you encounter your ! byte. 2) call select() before calling recv() to detect when the socket actually has data to read, then call ioctlsocket(FIONREAD) to determine how many bytes can actually be read with recv() without blocking, then have recv() read that number of bytes.
C++ - ensuring full serial response
I am trying to read a serial response from a hardware device. The string I read is long and I only need a portion of it. To get to portion of the string I want I use std::string.substr(x,y); . The problem I run into however is sometimes I get an exception error because the buffer I am reading from doesn't have y characters. Here is the code I use now to read values: while(1) { char szBuff[50+1] = {0}; char wzBuff[14] = {"AT+CSQ\r"}; DWORD dZBytesRead = 0; DWORD dwBytesRead = 0; if(!WriteFile(hSerial, wzBuff, 7, &dZBytesRead, NULL)) std::cout << "Write error"; if(!ReadFile(hSerial, szBuff, 50, &dwBytesRead, NULL)) std::cout << "Read Error"; std:: cout << szBuff; std::string test = std::string(szBuff).substr(8,10); std::cout << test; Sleep(500); I am issuing the command "AT+CSQ". This returns: N, N OK It returns two integer values seperated by a comma followed by a new line, followed by "OK". My question is, how can I make sure I read all values from the serial port before grabbing a substring? From what I understand, the last character received should be a new line.
The interface of your ReadFile function seems to provide you with the number of bytes read. If you know the length that is expected, you should loop trying reading from the file (probably port descriptor) until the expected number of bytes is read. If the length of the response is not known, you might have to read and check in the read buffer whether the separator token has been read or not (in this case your protocol seems to indicate that a new-line can be used to determine EOM --end of message) If you can use other libraries, I would consider using boost::asio and the read_until functionality (or the equivalent in whatever libraries you are using). While the code to manage this is not rocket science, in most cases there is no point in reinventing the wheel.
As you said yourself in the last line, you know that the terminator for the response is a new line character. You need to read from the serial until you receive a new line somewhere in the input. Everything you received from the previous new line to the current new line is the response, with everything after the current new line is part of the next response. This is achieved by reading in a loop, handling each response as it is discovered: char* myBigBuff; int indexToBuff = 0; int startNewLine = 0; while (ReadFile(hSerial, myBigBuff + indexToBuff, 100, &dwBytesRead, NULL)) { if (strchr(myBigBuff, '\n') != NULL) { handleResponse(myBigBuff + startNewLine, indexToBuff + dwBytesRead); startNewLine = indexToBuff + dwBytesRead; } // Move forward in the buffer. This should be done cyclically indexToBuff += dwBytesRead; } This is the basic idea. You should handle the left overs characters via any way you choose (cyclic buffer, simple copy to a temp array, etc.)
You should use ReadFile to read a certain amount of bytes per cycle into your buffer. This buffer should be filled until ReadFile reads 0 bytes, you have reached your \n or \r\n characters, or filled your buffer to the max. Once you have done this, there would be no need to substr your string and you can iterate through your character buffer. For example, while (awaitResponse) { ReadFile(hSerial, szBuff, 50, &dwBytesRead, NULL); if (dwBytesRead != 0) { // move memory from szBuff to your class member (e.g. mySerialBuff) } else { // nothing to read if (buffCounter > 0) { // process buffer } else { // zero out all buffers } } }
Old question, but I modified #Eli Iser code to: while (ReadFile(hSerial, myBigBuff + indexToBuff, 1, &dwBytesRead, NULL)) { if (strchr(myBigBuff, '-') != NULL || dwBytesRead < 1) break; // Move forward in the buffer. This should be done cyclically indexToBuff += dwBytesRead; } if (indexToBuff != 0) { //Do whatever with the code, it received successfully. }