Concatenating strings into own protocol - c++

I'm writing networking programming using socket.h to my studies. I have written server and client simple programs that can transfer files between them using buffer size given by user.
Server
void transfer(string name)
{
char *data_to_send;
ifstream myFile;
myFile.open(name.c_str(),ios::binary);
if(myFile.is_open))
{
while(myFile.eof))
{
data_to_send = new char [buffer_size];
myFile.read(data_to_send, buffer_size);
send(data_to_send,buffer_size);
delete [] data_to_send;
}
myFile.close();
send("03endtransmission",buffer_size);
}
else
{
send("03error",buffer_size);
}
}
Client
void download(string name)
{
char *received_data;
fstream myFile;
myFile.open(name.c_str(),ios::out|ios::binary);
if(myFile.is_open())
{
while(1)
{
received_data = new char[rozmiar_bufora];
if((receivedB = recv(sockfd, received_data, buffer_size,0)) == -1) {
perror("recv");
close(sockfd);
exit(1);
}
if(strcmp(received_data,"03endoftransmission") == 0)
{
cout<<"End of transmission"<<endl;
break;
}
else if (strcmp(received_data,"03error") == 0)
{
cout<<"Error"<<endl;
break;
}
myFile.write(received_data,buffer_size);
}
myFile.close();
}
The problem occurs, when I want to implement my own protocol- two chars (control), 32 chars hash, and the rest of package is data. I tried few times to split it and I end up with this code:
Server
#define PAYLOAD 34
void transfer(string name)
{
char hash[] = "12345678901234567890123456789012"; //32 chars
char *data_to_send;
ifstream myFile;
myFile.open(name.c_str(),ios::binary);
if(myFile.is_open))
{
while(myFile.eof))
{
data_to_send = new char [buffer_size-PAYLOAD];
myFile.read(data_to_send, buffer_size-PAYLOAD);
concatenation = new char[buffer_size];
strcpy(concatenation,"02");
strcat(concatenation,hash);
strcat(concatenation,data_to_send);
send(concatenation,buffer_size);
delete [] data_to_send;
delete [] concatenation;
}
myFile.close();
send("03endtransmission",buffer_size);
}
else
{
send("03error",buffer_size);
}
}
Client
void download(string name)
{
char *received_data;
fstream myFile;
myFile.open(name.c_str(),ios::out|ios::binary);
if(myFile.is_open())
{
while(1)
{
received_data = new char[buffer_size];
if((receivedB = recv(sockfd, received_data, buffer_size,0)) == -1) {
perror("recv");
close(sockfd);
exit(1);
}
if(strcmp(received_data,"03endoftransmission") == 0)
{
cout<<"End of transmission"<<endl;
break;
}
else if (strcmp(received_data,"03error") == 0)
{
cout<<"Error"<<endl;
break;
}
control = new char[3];
strcpy(control,"");
strncpy(control, received_data,2);
control[2]='\0';
hash = new char[33];
strcpy(hash,"");
strncpy(hash,received_data+2,32);
hash[32]='\0';
data = new char[buffer_size-PAYLOAD+1];
strcpy(data,"");
strncpy(data,received_data+34,buffer_size-PAYLOAD);
myFile.write(data,buffer_size-PAYLOAD);
}
myFile.close();
}
But this one inputs to file some ^# instead of real data. Displaying "data" to console looks the same on server and client. If you know how I can split it up, I would be very grateful.

You have some issues which may or may not be your problem.
(1) send/recv can return less than you requested. You may ask to receive 30 bytes but only get 10 on the recv call so all of these have to be coded in loops and buffered somewhere until you actually get the number you wanted. Your first set of programs was lucky to work in this regard and probably only because you tested on a limited amount of data. Once you start to push through more data your assumptions on what you are reading (and comparing) will fail.
(2) There is no need to keep allocating char buffers in the loops; allocate them before the loop or just use a local buffer rather than the heap. What you are doing is inefficient and in the second program you have memory leaks because you don't delete them.
(3) You can get rid of the strcpy/strncpy statements and just use memmove()
Your specific problem is not jumping out at me but maybe this will push in the right direction. More information what is being transmitted properly and exactly where in the data you are seeing problems would be helpful.

But this one inputs to file some ^# instead of real data. Displaying
"data" to console looks the same on server and client. If you know how
I can split it up, I would be very grateful.
You say that the data (I presume the complete file rather than the '^#') is the same on both client and server? If this is the case, then your issue is likely writing the data to file, rather than the actual transmission of the data itself.
If this is the case, you'll probably want to check assumptions about how the program writes to file - for example, are you passing in text data to be written to file, or binary data? If you're writing binary data, but it uses the NULL-terminated string, chances are it will quit early treating valid binary information as a NULL.
If it's text mode, you might want to consider initialising all strings with memset to a default character (other than NULL) to see if it's garbage data being out put.
If both server and client display the '^#' (or whatever data), binary based char data would be incompatible with the strcpy/strcat functions as this rely on NULL termination (where-as binary uses size termination instead).
I can't track down the specific problem, but maybe this might offer an insight or two that helps.

Related

Serving HTML as C array from Arduino code - String size limit problem

I've been working on a HTML / websocket server on a Wiznet W5100S-EVB-Pico, programmed in the Arduino IDE. It all worked fine up until now but I'm running into, I think, a string size limit. I guess it is in the way the code handles the const char but I don't know how to do it properly.
I hope someone is willing to help :)
Let me explain:
I convert the index.html to a index_html.h file containing a const char array:
const char c_index_html[] = {
0x3c,0x21,0x44,0x4f,0x43,..., ..., 0x6d,0x6c,0x3e};
In my code I include the index_html.h file:
#include "index_html.h"
Now the code that actually serves the "HTML"
if (web_client){
Serial.println("New client");
// an http request ends with a blank line
bool currentLineIsBlank = true;
while (web_client.connected()){
if (web_client.available()){
char c = web_client.read();
if (c == '\n' && currentLineIsBlank) // if you've gotten to the end of the line (received a newline
{ // character) and the line is blank, the http request has ended,
Serial.println(F("Sending response")); // so you can send a reply
String strData;
strData = c_index_html;
web_client.println(strData);
break;
}
if (c == '\n')
{
// you're starting a new line
currentLineIsBlank = true;
}
else if (c != '\r')
{
// you've gotten a character on the current line
currentLineIsBlank = false;
}
}
}
This is not the prettiest code, it's smashed together from examples and now the main culprit seems to be:
String strData;
strData = c_index_html;
web_client.println(strData);
When I add extra code to the HTML and view the page source, the code is incomplete. I tested reducing the HTML to a minimum and that solves the problem.
So my main question is:
How do I serve the 'const char c_index_html' without use of 'String'?
But also:
How could I prettify the whole 'if (web_client)' serving function?
Thank you very much for making it all the way through this post and if you have a suggestion I would very much appreciate it ;)
Edit: There is a bug in the ethernet library shown in this post.
I don't know if it affects you; you should look at your library implementation.
I'm assuming that web_client is an instance of EthernetClient from the Arduino libraries.
EthernetClient::println is inherited from Print via Stream and is defined in terms of write, which is:
size_t EthernetClient::write(const uint8_t *buf, size_t size)
{
if (_sockindex >= MAX_SOCK_NUM) return 0;
// This library code is not correct:
if (Ethernet.socketSend(_sockindex, buf, size)) return size;
setWriteError();
return 0;
}
So we see that it asks the socket to send the buffer up to some size. The socket can respond with a size or 0 (see edit); if it responds with 0 then there's an error condition to check.
Edit: This is how it's supposed to work. Since write is always returning the requested size and not telling you how much was written, you can't fix your problem using the print/write facilities and need to directly use socketSend.
You're not checking the result of this write (which is supposed to come through println) so you don't know whether the socket sent size bytes, 0 bytes, or some number in between.
In EthernetClient::connect we see that it's opening a TCP stream:
_sockindex = Ethernet.socketBegin(SnMR::TCP, 0);
When you call socketSend you're actually just copying your buffer into a buffer in the network stack. The TCP driver writes out that buffer when it can. If you're writing into that buffer faster than it's being flushed to the network then you'll fill it up and your socketSend calls will start returning < size bytes. See Does send() always send whole buffer?.
So you're probably right that your string is too long. What you need to do is spread your writes out. There are countless tutorials covering this on the web; it's roughly like this in your example:
...
size_t bytesRemaining = 0;
while (web_client.connected()){
if (bytesRemaining > 0) {
// Still responding to the last request
char const* const cursor = c_index_html
+ sizeof(c_index_html)
- bytesRemaining;
size_t const bytesWritten = web_client.write(cursor, bytesRemaining);
if (!bytesWritten) {
// check for error
}
bytesRemaining -= bytesWritten;
if (bytesRemaining == 0) {
// End the message. This might not write!
// We should add the '\n' to the source array so
// it's included in our write-until-finished routine.
web_client.println();
// Stop listening
break;
}
} else if (web_client.available()){
// Time for a new request
char c = web_client.read();
if (c == '\n' && currentLineIsBlank)
{
Serial.println(F("Sending response"));
// Start responding to this request
bytesRemaining = sizeof(c_index_html);
continue;
}
...
This is what I think is going on. I'm not an expert so I might be wrong, but it seems to make sense.
This is not an answer as in "solution" but I found out there is a 2k buffer size limit using the W5100S-EVB-Pico. And indeed, if I keep the HTML below 2k it works. Turns out that I actually got Matt Murphy's suggestion to work but the 2k limit was the problem. It looks like a hardware/library limitation, not completely sure on that.
For now I'll shrink my HTML and Javascript to a minimum and compact it even more with for example textfixer.com. I think I might write some python code to do that
Maybe there is a path to a solution in the link below but at this moment I'll try to live with the limitations
Link:
https://github.com/khoih-prog/EthernetWebServer/issues/7

How to send image data over linux socket

I have a relatively simple web server I have written in C++. It works fine for serving text/html pages, but the way it is written it seems unable to send binary data and I really need to be able to send images.
I have been searching and searching but can't find an answer specific to this question which is written in real C++ (fstream as opposed to using file pointers etc.) and whilst this kind of thing is necessarily low level and may well require handling bytes in a C style array I would like the the code to be as C++ as possible.
I have tried a few methods, this is what I currently have:
int sendFile(const Server* serv, const ssocks::Response& response, int fd)
{
// some other stuff to do with headers etc. ........ then:
// open file
std::ifstream fileHandle;
fileHandle.open(serv->mBase + WWW_D + resource.c_str(), std::ios::binary);
if(!fileHandle.is_open())
{
// error handling code
return -1;
}
// send file
ssize_t buffer_size = 2048;
char buffer[buffer_size];
while(!fileHandle.eof())
{
fileHandle.read(buffer, buffer_size);
status = serv->mSock.doSend(buffer, fd);
if (status == -1)
{
std::cerr << "Error: socket error, sending file\n";
return -1;
}
}
return 0
}
And then elsewhere:
int TcpSocket::doSend(const char* message, int fd) const
{
if (fd == 0)
{
fd = mFiledes;
}
ssize_t bytesSent = send(fd, message, strlen(message), 0);
if (bytesSent < 1)
{
return -1;
}
return 0;
}
As I say, the problem is that when the client requests an image it won't work. I get in std::cerr "Error: socket error sending file"
EDIT : I got it working using the advice in the answer I accepted. For completeness and to help those finding this post I am also posting the final working code.
For sending I decided to use a std::vector rather than a char array. Primarily because I feel it is a more C++ approach and it makes it clear that the data is not a string. This is probably not necessary but a matter of taste. I then counted the bytes read for the stream and passed that over to the send function like this:
// send file
std::vector<char> buffer(SEND_BUFFER);
while(!fileHandle.eof())
{
fileHandle.read(&buffer[0], SEND_BUFFER);
status = serv->mSock.doSend(&buffer[0], fd, fileHandle.gcount());
if (status == -1)
{
std::cerr << "Error: socket error, sending file\n";
return -1;
}
}
Then the actual send function was adapted like this:
int TcpSocket::doSend(const char* message, int fd, size_t size) const
{
if (fd == 0)
{
fd = mFiledes;
}
ssize_t bytesSent = send(fd, message, size, 0);
if (bytesSent < 1)
{
return -1;
}
return 0;
}
The first thing you should change is the while (!fileHandle.eof()) loop, because that will not work as you expect it to, in fact it will iterate once too many because the eof flag isn't set until after you try to read from beyond the end of the file. Instead do e.g. while (fileHandle.read(...)).
The second thing you should do is to check how many bytes was actually read from the file, and only send that amount of bytes.
Lastly, you read binary data, not text, so you can't use strlen on the data you read from the file.
A little explanations of the binary file problem: As you should hopefully know, C-style strings (the ones you use strlen to get the length of) are terminated by a zero character '\0' (in short, a zero byte). Random binary data can contain lots of zero bytes anywhere inside it, and it's a valid byte and doesn't have any special meaning.
When you use strlen to get the length of binary data there are two possible problems:
There's a zero byte in the middle of the data. This will cause strlen to terminate early and return the wrong length.
There's no zero byte in the data. That will cause strlen to go beyond the end of the buffer to look for the zero byte, leading to undefined behavior.

Use stringstream to read from TCP socket

I am using a socket library (I'd rather not not use it) whose recv operations works with std::string, but is just a wrapper for one call of the recv socket function, so it is probably that I only got some part of the message I wanted. My first instinct was to go in a loop and append the received string to another string until I get everything, but this seems inefficient. Another possibility was to do the same with a char array, but this seems messy. (I'd have to check the strings size before adding into the array and if it overflowed I need to store the string somewhere until the array is empty again.. )
So I was thinking about using a stringstream. I use a TLV protocol, so I need to first extract two bytes into an unsigned short, then get a certain amount of bytes from the stringstream and then loop again until I reach a delimiter field.
Is there any better way to do this? Am I completely on the wrong track? Are there any best practices? So far I've always only seen direct use of the socket library with char arrays so I'm curious why using `std::string`` with stringstreams could be a bad idea..
Edit: Replying to the comment below: The library is one we use internally, its not public (its nothing special though, mostly just a wrapper around the socket library to add exceptions, etc.).
I should mention that I have a working prototype using the socket library directly.
This works something like:
int lengthFieldSize = sizeof(unsigned short);
int endOfBuffer= 0;//Pointer to last valid position in buffer.
while(true) {
char buffer[RCVBUFSIZE];
while(true) {
int offset= endOfBuffer;
int rs= 0;
rs= recv(sock, buffer+offset, sizeof(buffer)-offset, 0);
endOfBuffer+= rs;
if(rs < 1) {
// Received nothing or error.
break;
} else if(endOfBuffer == RCVBUFSIZE) {
// Buffer full.
break;
} else if(rs > 0 && endOfBuffer > 1) {
unsigned short msglength= 0;
memcpy((char *) &msglength, buffer+endOfBuffer-lengthFieldSize, lengthFieldSize);
if(msglength == 0) {
break; // Received a full transmission.
}
}
}
unsigned int startOfData = 0;
unsigned short protosize= 0;
while(true) {
// Copy first two bytes into protosize (length field)
memcpy((char *) &protosize, buffer+startOfData, lengthFieldSize);
// Is the last length field the delimiter?
// Then reply and return. (We're done.)
// Otherwise: Is the next message not completely in the buffer?
// Then break. (Outer while will take us back to receiving)
if(protosize == 0) {
// Done receiving. Now send:
SendReplyMsg(sock, lengthFieldSize);
// Clean up.
close(sock);
return;
} else if((endOfBuffer-lengthFieldSize-startOfData) < protosize) {
memmove(buffer, buffer+startOfData, RCVBUFSIZE-startOfData);
//Adjust endOfBuffer:
endOfBuffer-=startOfData;
break;
}
startOfData+= lengthFieldSize;
gtControl::gtMsg gtMessage;
if(!gtMessage.ParseFromArray(buffer+startOfData, protosize)) {
cerr << "Failed to parse gtMessage." << endl;
close(sock);
return;
}
// Move position pointer forward by one message (length+pbuf)
startOfData+= protosize;
PrintGtMessage(&gtMessage);
}
}
So basically I have a big loop which contains a receiving loop and a parsing loop. There's a character array being passed back and forth as I can't be sure to have received everything until I actually parse it. I'm trying to replicate this behaviour using "proper" C++ (i.e. std::string)
My first instinct was to go in a loop and append the received string to another string until I get everything, but this seems inefficient.
String concatenation is technically platform dependent, but probably str1 + str2 will require one dynamic allocation and two copies (from str1 and str2). That's sorta slow, but it's far faster than network access! So my first piece of advice would be to go with your first instinct, to find out whether it's both correct and fast enough.
If it's not fast enough, and your profiler shows that the redundant string copies are to blame, consider maintaining a list of strings (std::vector<string*>, perhaps) and joining all the strings together once at the end. This requires some care, but should avoid a bunch of redundant string copying.
But definitely profile first!

C++ - ensuring full serial response

I am trying to read a serial response from a hardware device. The string I read is long and I only need a portion of it. To get to portion of the string I want I use std::string.substr(x,y); . The problem I run into however is sometimes I get an exception error because the buffer I am reading from doesn't have y characters. Here is the code I use now to read values:
while(1)
{
char szBuff[50+1] = {0};
char wzBuff[14] = {"AT+CSQ\r"};
DWORD dZBytesRead = 0;
DWORD dwBytesRead = 0;
if(!WriteFile(hSerial, wzBuff, 7, &dZBytesRead, NULL))
std::cout << "Write error";
if(!ReadFile(hSerial, szBuff, 50, &dwBytesRead, NULL))
std::cout << "Read Error";
std:: cout << szBuff;
std::string test = std::string(szBuff).substr(8,10);
std::cout << test;
Sleep(500);
I am issuing the command "AT+CSQ". This returns:
N, N
OK
It returns two integer values seperated by a comma followed by a new line, followed by "OK".
My question is, how can I make sure I read all values from the serial port before grabbing a substring? From what I understand, the last character received should be a new line.
The interface of your ReadFile function seems to provide you with the number of bytes read. If you know the length that is expected, you should loop trying reading from the file (probably port descriptor) until the expected number of bytes is read.
If the length of the response is not known, you might have to read and check in the read buffer whether the separator token has been read or not (in this case your protocol seems to indicate that a new-line can be used to determine EOM --end of message)
If you can use other libraries, I would consider using boost::asio and the read_until functionality (or the equivalent in whatever libraries you are using). While the code to manage this is not rocket science, in most cases there is no point in reinventing the wheel.
As you said yourself in the last line, you know that the terminator for the response is a new line character. You need to read from the serial until you receive a new line somewhere in the input. Everything you received from the previous new line to the current new line is the response, with everything after the current new line is part of the next response. This is achieved by reading in a loop, handling each response as it is discovered:
char* myBigBuff;
int indexToBuff = 0;
int startNewLine = 0;
while (ReadFile(hSerial, myBigBuff + indexToBuff, 100, &dwBytesRead, NULL))
{
if (strchr(myBigBuff, '\n') != NULL)
{
handleResponse(myBigBuff + startNewLine, indexToBuff + dwBytesRead);
startNewLine = indexToBuff + dwBytesRead;
}
// Move forward in the buffer. This should be done cyclically
indexToBuff += dwBytesRead;
}
This is the basic idea. You should handle the left overs characters via any way you choose (cyclic buffer, simple copy to a temp array, etc.)
You should use ReadFile to read a certain amount of bytes per cycle into your buffer. This buffer should be filled until ReadFile reads 0 bytes, you have reached your \n or \r\n characters, or filled your buffer to the max.
Once you have done this, there would be no need to substr your string and you can iterate through your character buffer.
For example,
while (awaitResponse) {
ReadFile(hSerial, szBuff, 50, &dwBytesRead, NULL);
if (dwBytesRead != 0) {
// move memory from szBuff to your class member (e.g. mySerialBuff)
} else {
// nothing to read
if (buffCounter > 0) {
// process buffer
}
else {
// zero out all buffers
}
}
}
Old question, but I modified #Eli Iser code to:
while (ReadFile(hSerial, myBigBuff + indexToBuff, 1, &dwBytesRead, NULL)) {
if (strchr(myBigBuff, '-') != NULL || dwBytesRead < 1)
break;
// Move forward in the buffer. This should be done cyclically
indexToBuff += dwBytesRead;
}
if (indexToBuff != 0) {
//Do whatever with the code, it received successfully.
}

Problems with pwrite() to a file in C/C++

I've a bad problem. I'm trying to write to a file via filedescriptor and memalign. I can write to it but only something like an wrong encoded char is written to a file.
Here's my code:
fdOutputFile = open(outputFile, O_CREAT | O_WRONLY | O_APPEND | O_DIRECT, 0644)
void writeThis(char* text) {
while (*text != '\0') {
// if my internal buffer is full -> write to disk
if (buffPositionOutput == outputbuf.st_blksize) {
posix_memalign((void **)&bufferO, outputbuf.st_blksize, outputbuf.st_blksize);
cout << "wrote " << pwrite(fdOutputFile, bufferO, outputbuf.st_blksize, outputOffset*outputbuf.st_blksize) << " Bytes to disk." << endl;
buffPositionOutput = 0;
++outputOffset;
}
// buffer the incoming text...
bufferO[buffPositionOutput] = *text;
++text;
++buffPositionOutput;
}
}
I think it's the alignment - can someone help me?
It writes to the file but not the correct text, just a bunch of '[]'-chars.
Thanks in advance for your help!
Looking at your program, here is what happens:
You fill the memory initially pointed to by buffer0+buffPositionOutput (Which is where, precisely? I don't know based on the code you give.) up to buffer0+outputbuf.st_blksize with data.
You pass the address of the buffer0 pointer to posix_memalign, which ignores its current value and overwrites it with a pointer to outputbuf.st_blksize bytes of newly-allocated memory.
You write data from the newly-allocated block to disk; this might be anything, since you just allocated memory and haven't written anything there yet.
This won't work, obviously. You probably want to initialize your buffer via posix_memalign at the top of your function, and then just overwrite the block's worth of data in it as you use your aligned buffer to repeatedly write data into the file. (Reset buffpositionoutput to zero after each time you write data, but don't re-allocate.) Make sure you free your buffer when you are done.
Also, why are you using pwrite instead of write?
Here's how I would implement writeThis (keeping your variable names so you can match it up with your version):
void writeThis(char *text) {
char *buffer0;
size_t buffPositionOutput = 0;
posix_memalign(&buffer0, outputbuf.st_blksize, outputbuf.st_blksize);
while (*text != 0) {
++text; ++buffPositionOutput;
if (buffPositionOutput == outputbuf.st_blksize) {
write(fdOutputFile, buffer0, outputbuf.st_blksize);
buffPositionOuput = 0;
}
}
if (buffPositionOutput != 0) {
// what do you want to do with a partial block of data? Not sure.
}
}
(For speed, you might consider using memcpy calls instead of a loop. You would need to know the length of the data to write ahead of time though. Worry about that after you have a working solution that does not leak memory.)
You're re-allocating buffer0 every time you try to output it, and not freeing it. That's really not efficient (and leaks memory). I'd suggest you refactor your code a bit, because it's quite hard to follow whether your bounds checking on that buffer is correct or not.
Allocate buffer0 only once somewhere (form that snippet, storing it in outputbuf sounds like a good idea). Also store buffPositionOutput in that struct (or in another struct, but close to that buffer).
// in setup code
int rc = posix_memalign(&(outputbuf.data), outputbuf.st_blksize,
outputbuf.st_blksize);
// check rc!
outputbuf.writePosition = 0;
// in cleanup code
free(outputbuf.data);
Then you can rewrite your function like this:
void writeThis(char *text) {
while (*text != 0) {
outputbuf.data[outputbuf.writePosition] = *text;
outputbuf.writePosition++;
text++;
if (outputbuf.writePosition == outputbuf.block_size) {
int rc = pwrite(...);
// check rc!
std::cout << ...;
outputbuf.writePosition = 0;
}
}
I don't think C/C++ has encodings. ASCII only.
Unless you use wchar http://en.wikipedia.org/wiki/Wide_character