I know how my packet looks like. It has 6 header fields (1 byte each, each header has 8 fields) and then it has the payload (data).
I would like to build a raw packet in C or C++ (it should look the same I think).
Here's what I think I should do:
unsigned char packet[11];
packet[0] = (0x81); // first header with 8 fields
packet[1] = (0x8c); // second header with 8 fields
packet[2] = (0xfe);
packet[3] = (0x84);
packet[4] = (0x1d);
packet[5] = (0x79);
packet[6] = (0x96); // payload, the 'h' letter, masked
packet[7] = (0xe1); // 'e'
packet[8] = (0x71); // 'l'
packet[9] = (0x15); // 'l'
packet[10] = (0x91);// 'o'
Where, for instance, 0x81 is the first byte (I simply converted every field (bit) of my first header to hex).
And then, simply, I want to send it to server: send(sockfd, packet, sizeof(packet), 0) to send it.
Receiving and printing the response:
unsigned char buffer[1024];
if ((recv(sockfd, buffer, len, 0)) == 0)
{
if (errno != 0)
{
exit(1);
}
}
int i;
for(i = 0; i<len; i++)
printf("%x ", buffer[i]);
Am I right?
Other than mishandling the return value from recv, your code looks okay.
if ((recv(sockfd, buffer, len, 0)) == 0)
{
if (errno != 0)
{
exit(1);
}
}
A zero return indicates normal close of the connection. There's no reason to check errno if it returns zero.
A return value of -1 indicates an error. In that case, it does make sense to check errno.
A value greater than zero indicates that number of bytes have been received. Be aware that it is perfectly normal for recv to return fewer bytes than you asked it for. If you want to receive exactly some number of bytes, you must call recv in a loop.
TCP is a byte-stream protocol and has no idea where your "packets" (really, messages) begin and end.
Your code will not appear to be error-prone!
But a good practice would be:
const std::uint32_t BUFFER_SIZE = 11;
std::vector<std::uint8_t> buffer;
buffer.reserve(BUFFER_SIZE)
buffer = {0x81,0x8c.....};
send( sockfd,
reinterpret_cast <const char*> ( buffer.data() ),
static_cast <int> ( buffer.size() ),
0
);
Doing so, your code gets more optimized, and avoids possible leaks, using the std vectors.
May also benefit from taking a look at ZeroMQ, as an example of a ready-made, high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications.
Related
I want to send a string from a server to a client. To account for the case that the string is bigger than the buffer that the client reads into, I send a byte count to the client before the actual string. Then on the client side, I split the read operation into multiple recv() calls if needed.
Server:
// Send byte count to client so it knows how much data to expect
strcpy(buffer, std::to_string(string_size).c_str());
int result{ send(socket, buffer, buffer_size, 0) };
// Send actual data
unsigned int bytes_to_send{ sizeof(s) };
while (bytes_to_send > 0) {
send_bytes = std::min(buffer_size, bytes_to_send);
strcpy(buffer, s.substr(0, send_bytes).c_str());
result = send(socket, buffer, send_bytes, 0);
if (s.size() > send_bytes) {
s = s.substr(send_bytes, s.size() - send_bytes);
}
bytes_to_send -= send_bytes;
return_int += send_bytes;
(In the actual code, I check result to see whether send() succeeded)
Client:
result = recv(socket, buffer, buffer_size, 0);
count = std::stoi(buffer);
while(count > 0){
read_bytes = std::min(count, buffer_size);
result = recv(socket, buffer, read_bytes, 0);
if(result < 1 ) {
// throw exception
}else{
return_string += buffer;
}
count -= read_bytes;
What I expected:
Client blocks on first recv() call until server sends byte count
Client proceeds until second recv() call, then blocks until server sends string
What actually happens:
Client blocks on first recv() call until server sends byte count
Client does not block on second recv() call. recv() return value is equal to read_bytes, but does not fill the buffer with anything
Thanks in advance for any help with this
There are quite a few problems with your code.
strcpy(buffer, std::to_string(string_size).c_str()); is a buffer overflow waiting to happen.
send(socket, buffer, buffer_size, 0) - you did not set buffer_size to the number of bytes copied into buffer by strcpy(). It looks like you are sending the entire buffer without regard to how many bytes it is actually holding.
sizeof(s) does not give you the number of characters in a std::string. It gives you the byte size of the std::string class itself, which is not the same thing. You need to use the string's size() or length() method to get the proper number of characters.
TCP is a byte stream, there is no 1-to-1 relationship between send() and recv(). Both can report fewer bytes were processed than requested, but you are not accounting for that properly.
You are completely ignoring the return value of send(), assuming that all requested bytes have been sent in full on each call, when in actuality fewer bytes may have been sent, or none at all. On the 1st call, you are not accounting for the possibility that the entire size string may not be sent in 1 call and may need to be looped. Inside of the loop, you are completely ignoring the return value of send() for purposes of looping.
Likewise, you are ignoring the return value of recv() on the 1st call, assuming the entire string size is received in 1 call, without regard to how many characters are actually in the string size. Inside the loop, you are paying attention to the return value of recv() for purposes of looping, except that you are not accounting for the possibility that recv() returns 0 on a graceful disconnect. You are not breaking your loop if that happens.
It is good that you want to send the string's size before sending its content. However, you are sending the size as another variable-length string, which really defeats the purpose of sending a size at all. You should send the size using a fixed-length integer in binary form instead.
Try something more like this instead:
void send_raw(int socket, const void *buffer, size_t buffer_size)
{
const char *ptr = (const char*) buffer;
while (buffer_size > 0) {
int sent = send(socket, ptr, buffer_size, 0);
if (sent < 0) {
// throw exception
}
ptr += sent;
buffer_size -= sent;
}
}
void send_u32(int socket, uint32_t value)
{
value = htonl(value);
send_raw(socket, &value, sizeof(value));
}
...
// Send byte count to client so it knows how much data to expect
send_u32(socket, s.size());
// Send actual data
send_raw(socket, s.c_str(), s.size());
void recv_raw(int socket, void *buffer, size_t buffer_size)
{
char *ptr = (char*) buffer;
while (buffer_size > 0) {
int recvd = recv(socket, ptr, buffer_size, 0);
if (recvd <= 0) {
// throw exception
}
ptr += recvd;
buffer_size -= recvd;
}
}
uint32_t recv_u32(int socket)
{
uint32_t value;
recv_raw(socket, &value, sizeof(value));
return ntohl(value);
}
...
size_t count = recv_u32(socket);
return_string.resize(count);
recv_raw(socket, &return_string[0], count);
However, if you really want to stick with sending the string size as another string, you need to delimit it, such as sending its null terminator, eg:
// Send byte count to client so it knows how much data to expect
std::string size_buffer = std::to_string(s.size());
send_raw(socket, size_buffer.c_str(), size_buffer.size()+1);
// Send actual data
send_raw(socket, s.c_str(), s.size());
std::string size_buffer;
char ch;
do {
recv_raw(socket, &ch, 1);
if (ch == '\0') break;
size_buffer += ch;
}
while (true);
size_t count = std::stoi(size_buffer);
return_string.resize(count);
recv_raw(socket, &return_string[0], count);
You can add MSG_WAITALL to the recv() flags and do only one read.
I want to use the function recv(socket, buf, len, flags) to receive an incoming packet. However I do not know the length of this packet prior to runtime so the first 8 bytes are supposed to tell me the length of this packet. I don't want to just allocate an arbitrarily large len to accomplish this so is it possible to set len = 8 have buf be a type of uint64_t. Then afterwards
memcpy(dest, &buf, buf)?
Since TCP is stream-based, I'm not sure what type of packages you mean. I will assume that you are referring to application level packages. I mean packages which are defined by your application and not by underlying protocols like TCP. I will call them messages instead to avoid confusion.
I will show two possibilities. First I will show, how you could read a message without knowing the length before you have finished reading. The second example will do two calls. First it reads the size of the message. Then it read the whole message at once.
Read data until the message is complete
Since TCP is stream-based, you will not loss any data when your buffer is not big enough. So you can read a fixed amount of bytes. If something is missing, you can call recv again. Here is a extensive example. I just wrote it without testing. I hope everything would work.
std::size_t offset = 0;
std::vector<char> buf(512);
std::vector<char> readMessage() {
while (true) {
ssize_t ret = recv(fd, buf.data() + offset, buf.size() - offset, 0);
if (ret < 0) {
if (errno == EINTR) {
// Interrupted, just try again ...
continue;
} else {
// Error occured. Throw exception.
throw IOException(strerror(errno));
}
} else if (ret == 0) {
// No data available anymore.
if (offset == 0) {
// Client did just close the connection
return std::vector<char>(); // return empty vector
} else {
// Client did close connection while sending package?
// It is not a clean shutdown. Throw exception.
throw ProtocolException("Unexpected end of stream");
}
} else if (isMessageComplete(buf)) {
// Message is complete.
buf.resize(offset + ret); // Truncate buffer
std::vector<char> msg = std::move(buf);
std::size_t msgLen = getSizeOfMessage(msg);
if (msg.size() > msgLen) {
// msg already contains the beginning of the next message.
// write it back to buf
buf.resize(msg.size() - msgLen)
std::memcpy(buf.data(), msg.data() + msgLen, msg.size() - msgLen);
msg.resize(msgLen);
}
buf.resize(std::max(2*buf.size(), 512)) // prepare buffer for next message
return msg;
} else {
// Message is not complete right now. Read more...
offset += ret;
buf.resize(std::max(buf.size(), 2 * offset)); // double available memory
}
}
}
You have to define bool isMessageComplete(std::vector<char>) and std::size_t getSizeOfMessage(std::vector<char>) by yourself.
Read the header and check the length of the package
The second possibility is to read the header first. Just the 8 bytes which contains the size of the package in your case. After that, you know the size of the package. This mean you can allocate enough storage and read the whole message at once:
/// Reads n bytes from fd.
bool readNBytes(int fd, void *buf, std::size_t n) {
std::size_t offset = 0;
char *cbuf = reinterpret_cast<char*>(buf);
while (true) {
ssize_t ret = recv(fd, cbuf + offset, n - offset, MSG_WAITALL);
if (ret < 0) {
if (errno != EINTR) {
// Error occurred
throw IOException(strerror(errno));
}
} else if (ret == 0) {
// No data available anymore
if (offset == 0) return false;
else throw ProtocolException("Unexpected end of stream");
} else if (offset + ret == n) {
// All n bytes read
return true;
} else {
offset += ret;
}
}
}
/// Reads message from fd
std::vector<char> readMessage(int fd) {
std::uint64_t size;
if (readNBytes(fd, &size, sizeof(size))) {
std::vector buf(size);
if (readNBytes(fd, buf.data(), size)) {
return buf;
} else {
throw ProtocolException("Unexpected end of stream");
}
} else {
// connection was closed
return std::vector<char>();
}
}
The flag MSG_WAITALL requests that the function blocks until the full amount of data is available. However, you cannot rely on that. You have to check it and read again if something is missing. Just like I did it above.
readNBytes(fd, buf, n) reads n bytes. As far as the connection was not closed from the other side, the function will not return without reading n bytes. If the connection was closed by the other side, the function returns false. If the connection was closed in the middle of a message, an exception is thrown. If an i/o-error occurred, another exception is thrown.
readMessage reads 8 bytes [sizeof(std::unit64_t)] und use them as size for the next message. Then it reads the message.
If you want to have platform independency, you should convert size to a defined byte order. Computers (with x86 architecture) are using little endian. It is common to use big endian in network traffic.
Note: With MSG_PEEK it is possible to implement this functionality for UDP. You can request the header while using this flag. Then you can allocate enough space for the whole package.
A fairly common technique is to read leading message length field, then issue a read for the exact size of the expected message.
HOWEVER! Do not assume that the first read will give you all eight bytes(see Note), or that the second read will give you the entire message/packet.
You must always check the number of bytes read and issue another read (or two (or three, or...)) to get all the data you want.
Note: Because TCP is a streaming protocol and because the packet size "on the wire" varies in accordance with a very arcane algorithm designed to maximize network performance, you could easily issue a read for eight bytes and the read could return having only read three (or seven or ...) bytes. The guarantee is that unless there is an unrecoverable error you will receive at least one byte and at most the number of bytes you requested. Because of this you must be prepared to do byte address arithmetic and issue all reads in a loop that repeats until the desired number of bytes is returned.
Since TCP is streaming there isn't really any end to the data you receive, not until the connection is closed or there is an error.
Instead you need to implement your own protocol on top of TCP, one that either contains a specific end-of-message marker, a length-of-data header field, or possibly a command-based protocol where the data of each command is of a well-known size.
That way you can read into a small fixed-sized buffer and append to a larger (possibly expanding) buffer as needed. The "possibly expanding" part is ridiculously easy in C++, what with std::vector and std::string (depending on the data you have)
There is another important thing to remember, that since TCP is stream-based, a single read or recv call may not actually fetch all the data you request. You need to receive the data in a loop until you have received everything.
In my Personal opinion.
I suggest receive "size of message"(integer 4 byte fixed) first.
recv(socket, "size of message written in integer" , "size of integer")
then
receive real message after.
recv(socket, " real message" ,"size of message written in integer")
This techinique also can be used on "sending files, images ,long messages"
I am writing a simple game named "TicTacToe". I also wanted to create a network function using WinSock. While connection between two users goes without errors, I can't receive any correct data. After receiving it, the buffer is filled with a strange characters like "☺$0/A", etc., while it should receive "Hello!".
Here's my code:
const char buf[] = "Hello!";
char rbuf[16];
int bytes = 0;
memset(buf, 0, sizeof(buf));
// sending
send(sox, buf, sizeof(buf), 0);
// recv
bytes = recv(sox, rbuf, 16, 0)
rbuf[bytes] = '\0';
cout << rbuf << endl;
Well, for one, you're not checking the return values of your send/recv calls, also you can't expect it to always receive 16 bytes. In order to ensure you get what you want to receive, you might have to call recv multiple times. Here's an example solution to that problem:
int FixedRecv(SOCKET sox, char* rbuf, int length)
{
int ref, len = 0;
do {
ref = recv(sox, rbuf + len, length - len, 0);
if(ref == 0) return 0;
if(SOCKET_ERROR != ref)
len += ref;
else return SOCKET_ERROR;
} while(len < length);
return len;
}
Same will also apply to your calls to send but I just want you to get a general idea.
Hope that helps, uses a little bit of pointer arithmetic, but nothing
too extensive.
Also, if you're using nonblocking sockets, you will want to check for WSAEWOULDBLOCK with WSAGetLastError.
How do you loop read for c++ sockets.
stringstream ss;
while (1) {
char buf [2];
ssize_t e = read(fd, buf, 2);
ss << buf[0];
ss << buf[1];
cout << e << endl;
cout << buf[0] << endl;
cout << buf[1] << endl;
if (e <= 0) {
break;
}
}
string msg = "";
ss >> msg;
When I telnet and type hello and then enter to test this out, I get
2
h
e
2
l
l
2
o
1
And it just waits there, no looping. What is going on, and how do I read sockets?
How do I signify EOF using
telnet
buf in the context of write(fd, buf, bufSize);
What do you mean no looping? Typo some more, press enter and you'll see some more output..
If you finished, you have to press Ctrl+D to signal the end of input (EOF).
How do I signify EOF using
1) telnet
2) buf in the context of write(fd, buf, bufSize);
First, it's not EOF that you want to signal. It's "end of message" or "the data you have just received is a complete protocol data unit and you should now process it". And the answer is, however you want, so long as both sides agree.
For example, you could reserve the ~ character specifically to mark the end of a message. To indicate the end of a message, one side sends a ~. The other side knows that when it receives a ~, everything before that is a single message and everything after it is a new message.
You can also send the length of a message prior to sending the actual message. You can do it in text followed by a zero byte or by a line feed. Or you can make each message a line.
You can do it however makes the most sense in your specific application, but you actually have to do it.
There are two cases where TCP does do it for you:
1) When a side is completely done sending and knows the other side will not send anymore data anyway, it can use close on the socket. The other side will read the rest of the data already sent and then read will return 0.
2) When a side is not going to send any more data but the other side might send more data, it can shutdown its side of the connection. It can never send any more data though. The other side will know about this shutdown, and if specifically coded to do so, can continue to send data until it shuts down its half. Then the other side's read will return zero. (Note that if you do this to a program that wasn't designed to take it, it will likely get confused and not realize it can still send data to you.)
Otherwise, it's all up to you.
TCP packets are a stream, not a file. It is up to you how the stream is read.
In general, if you write 20 bytes at one end, you'll get 20 bytes at the other end in one read, barring the use of some common tcp/ip options.
There are assumptions made below. Some of the assumptions are:
Assume all the scaffolding code is there.
Assume I made mistakes and debug it yourself.
Assume all byte order issues are handled.
Assume you're smart enough not to send floating point types as binary.
Some programmers choose to use a length prefixed packet and read the length byte(2) and then read as many bytes as that length represents, like so.
unsigned char buffer[MAX_CHAR] = "";
unsigned char length = 0;
int bytesRead = 0;
read( fd, sizeof( length ), &length );
// Handle failure from read
while( bytesRead < length )
{
int readRv = read( fd, &buffer[bytesRead], length - bytesRead );
if( readRv <= 0 ) // 0 usually means socket was closed. -1 is an erro
{
// Handle socket error/closed socket
}
else if( readRv < length )
{
bytesRead += readRv;
}
else if( readRv == length )
{
bytesRead = readRv;
break;
}
}
Some programmers read what is available and look for an end of packet marker
unsigned char buffer[MAX_CHAR] = "";
unsigned char length = 0;
int bytesRead = 0;
int readRv = read( fd, buffer, sizeof( buffer ) );
int eopFound = 0;
if( (eopFound = findEndOfPacketMarker( buffer )) > 0 )
{
// Less than 0 = error.
}
else if( eopFound == 0 )
{
// No end of packet, must read more bytes here
}
else
{
// Found an end of packet marker
}
// Here you deal with bytes that are "over read"
// Either the
// 1) packet was bigger than buffer,
// 2) there were bytes left over after the EOP marker
// 3) Martians crapped in your code and screwed it all up. :)
Some programmers use a fixed sized set of packets and a packet type id
int packetType;
unsigned char buffer[MAX_CHAR] = "";
unsigned char length = 0;
int bytesRead = 0;
read( fd, sizeof( packetType ), &packetType );
read( fd, getLengthFromPacketType( packetType ), buffer );
That should cover most things...
If you have more questions, please ask in the comments with the #JimR thingy so it shows up as a message for me to read.
i developed client server program using c++,so i want to receive more than 500kb , my client message is terminated with "!" ,so i want to receive until my last byte(!) receive ,
this is my code it doesn't work.what is wrong with it.
do
{
int num = recv(*csock, buf, bytesLeft,0);
if (num == 0)
{
break;
}
else if (num < 0 && errno != EINTR)
{
fprintf(stderr, "Exit %d\n", __LINE__);
exit(1);
}
else if (num > 0)
{
numRd += num;
buf += num;
bytesLeft -= num;
fprintf(stderr, "read %d bytes - remaining = %d\n", num, bytesLeft);
}
}
while (bytesLeft != 0);
fprintf(stderr, "read total of %d bytes\n", numRd);
While I'm not sure exactly what your problem is because of the wording of your question, you generally can't use strcat to append raw buffers received over the network unless you know specifically they will be NULL-terminated, and even then, that's not really "safe" in the event you get an unexpected data transmission. The assumption with c-strings is that they are NULL-terminated, but a raw network buffer may not be, and using strcat will cause you to over-run the input buffer should it not be NULL-terminated. Instead of strcat, use a known fixed-size buffer of size N bytes for receiving the data into, and increment a temporary pointer through the buffer until you reach the end of the buffer or the end of the packet transmission. That way you will always read from the network up to N bytes and no more, and prevent buffer over-run situations from occuring.
For instance, you can do the following (this is not the fastest or more efficient solution because of all the copying, but it works):
unsigned char buf[10000]; //10Kb fixed-size buffer
unsigned char buffer[MAXRECV]; //temporary buffer
unsigned char* temp_buf = buf;
unsigned char* end_buf = buf + sizeof(buf);
do
{
iByteCount = recv(GetSocketId(), buffer,MAXRECV,0);
if ( iByteCount > 0 )
{
//make sure we're not about to go over the end of the buffer
if (!((temp_buf + iByteCount) <= end_buf))
break;
fprintf(stderr, "Bytes received: %d\n",iByteCount);
memcpy(temp_buf, buffer, iByteCount);
temp_buf += iByteCount;
}
else if ( iByteCount == 0 )
{
if(temp_buf != buf)
{
//do process with received data
}
else
{
fprintf(stderr, "receive failed");
break;
}
}
else
{
fprintf(stderr, "recv failed: ");
break;
}
} while(iByteCount > 0 && temp_ptr < end_buf); //check for end of buffer
Do you need all 1MB+ of data in one contiguous byte buffer? If so, and you stick with that protocol that has a terminating '!' and does not have a header that includes the length, then you ar stuck with memcpy() and realloc() a lot or some other buffer type like std::vector which, really just does the same thing.
If you don't need all those bytes in one string, you can store them in some other way, eg. a vector of *buffer, and so avoid copying.
Assuming you are using a blocking socket (which is the default mode for sockets), then recv() will block waiting for the full MAXRECV number of bytes to arrive. If the client sends less than that number of bytes, recv() will block waiting for data that does not arrive.
To work around that, you need to either:
1) call recv() with a 1-byte buffer, calling recv() until you encounter your ! byte.
2) call select() before calling recv() to detect when the socket actually has data to read, then call ioctlsocket(FIONREAD) to determine how many bytes can actually be read with recv() without blocking, then have recv() read that number of bytes.