I'm having a trouble with receiving binary data to my server (python).
It seems that the OS (WIN7) sending big data in several packets without "permission",
so when im trying to send from my client (C++) binary data, I have to do some manipulations that will combine all the data.
I tried several ways but none of those worked.
here is the sending part (C++ - Works fine for sure):
sendbuf = "2011#" + ReadThisFile("C:\\0x3z4.jpg") + "#"; // buffer should be "2011#<Image Data>#"
// ReadThisFile returns string with binary data from file
vector<char> vect(sendbuf.begin(), sendbuf.end()); // Vector with the image data
iResult = send( ConnectSocket, &vect[0], vect.size(), 0 ); // Sending The Image
here is the receiving part (Python - part of threaded function 'Handler'):
while True:
Buffer = Sock.recv(self.BufferSize)
if Buffer[0:4] == "2011":
self.Print(ADDR[0] + " > 2011 > Capture Screen Response.")
# Save Image
Path = datetime.now().strftime("Information\\" + ADDR[0] + "#" + self.Clients[Index].computerName + "\\CaptureScreen.Files\\" + "%d-%m-%Y-%H-%M-%S.png")
f = open(Path,'wb')
f.write(Buffer[5:-1])
data = ""
# I tried to receive data till i'll find the eof
while True:
data += Sock.recv(4096)
if data.find("EOF"):
break
f.write(data)
This question is from Trojan project that me and couple friends working on for our course.
Thanks.
You're not sending a the buffer, you're sending the first vector.size() bytes of a vector. Stop the code in a debugger, inspect the memory starting at &vect[0], and compare it to what you're actually recieving, and I think you'll find that your code is behaving correctly (in the sense that it's doing what it's supposed to, not necessarily what you want).
I don'tknow what ReadThisFile does, but you need to actually put the bytes you want to send into a buffer and use that buffer as the argument to send. FWIW, ReadFile would probably be at least as easy for what it appears you're trying to do. Good luck.
Either ReadThisFile needs to tell you how many bytes it read, or you need to get the filesize another way and extrapolate the length of the buffer
Related
i am trying to send data via tcp socket to a server. The idea behind that is a really simple chat programm.
The string I am trying to send looks like the following:
1:2:e9e633097ab9ceb3e48ec3f70ee2beba41d05d5420efee5da85f97d97005727587fda33ef4ff2322088f4c79e8133cc9cd9f3512f4d3a303cbdb5bc585415a00:2:xc_[z kxc_[z kxc_[z kxc_[==
As you can see there a few unprintable characters which I don't think are a problem here.
To send this data I am using the following code snippet.
bool tcp_client::send_data(string data)
{
if( send(sock , data.c_str(), strlen(data.c_str()) , 0) < 0)
{
perror("Send failed : ");
return false;
}
return true;
}
After a few minutes of trying things out I came up, that data.c_str() cuts my string of.
The result is:
1:2:e9e633097ab9ceb3e48ec3f70ee2beba41d05d5420efee5da85f97d97005727587fda33ef4ff2322088f4c79e8133cc9cd9f3512f4d3a303cbdb5bc585415a00:2:xc_[z
I think that there is some kind of null sequence inside my string which is a problem for the c_str() function.
Is there a way to send the whole string as I mentioned aboved without cutting it off?
Thanks.
Is there a way to send the whole string as I mentioned aboved without cutting it off?
What about:
send(sock , data.c_str(), data.size() , 0);
There are only two sane ways to send arbitrary data (such as a array of characters) over stream sockets:
On the server: close the socket after data was sent (like in ftp, http 0.9, etc). On the client - read until socket is closed in a loop.
On the server: prefix the data with fixed-length size (nowadays people usualy use 64 bit integers for size, watch out for endiannes). On the client - read the size first (in a loop!), than read the data until size bytes are read (in a loop).
Everything else is going to backfire sooner or later.
I am communicating with a microcontroller that automatically initializes its flash memory whenever you open its serial port. So on a serial port read, the microcontroller prints up to 10,000 bytes of data showing the addresses and their initial values. You need to leave the port open for the entirety of the print to ensure that initialization completed. I don't ever perform any writes, just reads.
I modified the pySerial buffer from 4k to 32k since I do not want any breaks between reads (subsequent reads will simply restart the init cycle). Below is a snippet of my code where I read from the microcontroller serial port. When I run this snippet from the interpreter, I can tell from print and sizeof that temp contains all 9956 bytes. However when I run the py file, I get only 296 bytes. I inserted the sleep() method after read() but this did not have any effect. I cannot tell from the microcontroller if the initialization completed.
Is there a robust way to read until the serial buffer is empty? The microcontroller image is application-specific, so I cannot always predict the required read() size or timeout.
Any ideas what I could try? I've searched other threads but haven't found anything specific to this problem.
# Create serial port instance
self.ser_port = serial.Serial()
self.ser_port.port = 0
self.ser_port.timeout = 0
.
.
.
self.ser_port.open()
time.sleep(1)
temp = self.ser_port.read(32768)
time.sleep(4)
self.ser_port.close()
A timeout of 0 tells pySerial to return immediately with whatever data is already available. If you want read() to wait to allow more data to come in, use a non-zero timeout.
The first sleep() could cause you to lose some of the data that comes in after open but before read, because the internal buffer might not be as big as the buffer you are passing to read(). But that is highly application-dependent -- if the device you are communicating with takes awhile after the open() to respond, then it might not be a problem.
The second sleep() does nothing for you, because the read() is already finished at that point.
Note that the timeout value is the maximum total amount of time to wait. read() will return when it has filled its buffer, or it has been this long after it was called, whichever comes first.
Here is some example code that will return under the following circumstances:
a) 10-12 seconds with no data
b) 2-4 silent seconds after last data returned, if at least 100 bytes was received
self.ser_port = serial.Serial()
self.ser_port.port = 0
self.ser_port.timeout = 2
.
.
.
self.ser_port.open()
timeout_count = 0
received = []
data_count = 0
while 1:
buffer = self.ser_port.read(32768)
if buffer:
received.append(buffer)
data_count += len(buffer)
timeout_count = 0
continue
if data_count > 100:
# Break if we received at least 100 bytes of data,
# and haven't received any data for at least 2 seconds
break
timeout_count += 1
if timeout_count >= 5:
# Break if we haven't received any data for at
# least 10 seconds, even if we never received
# any data
break
received = ''.join(received) # If you need all the data
Depending on your microcontroller, you may be able to reduce these timeout values considerably, and still have a fairly robust solution for retrieving all the startup data from the microcontroller.
the key is that I send 4096 bytes but only 119 bytes aprox. carry useful information.
The 100 bytes ends with \r\n\r\n so in the client, when I read \r\n\r\n I want to stop receiving information from that string, and start over.
I don't know if I have to flush, or close the socket, or whatever...
They are sockets TCP.
In the client I do:
buf details[4096];
strcpy(details,"1");
strcat(details,"10/04/12");
strcat(details,"Kevin Fire");
strcat(detils,"abcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcde\r\n\r\n");
nbytes_sent = send(sock,(char *)details,sizeof(details),0);
On the other hand, the server...
char buf[20];
memset(buf,'\0',20);
while(!end){
nbytes_read=recv(sclient,(char *)ress,sizeof(ress),0);
if(strcmp(ress,"1")==0){
printf("Details: %s (%i)\n",buf,nbytes_read);
while(strcmp(buf,"\r\n\r\n") != 0){
nbytes_read=recv(sclient,(char *)buf,sizeof(buf),0);
cout.flush();
printf("Details: %s (%i)\n",buf,nbytes_read);
} }
if(strcmp(buf,"\r\n\r\n")==0) printf("The End\n");
cout.flush();
}
}
I just want to read a new "ress" and not being retrieving the rest of bytes that are not useful.
Thanks in advance.
If you mean you want to discard rest of data and read a new block you can't do it with TCP because it is stream oriented and do not have a concept of message and have no idea about the rest of message that you want to ignore. if you mean something else please describe it more.
but beside that why you use nbytes_sent = send(sock,(char *)details,sizeof(details),0); when only data until \r\n' is important. you can usenbytes_sent = send(sock,(char *)details,strlen(details),0);` that only send valid data and reduce garbage that you send over network and you don't need to start over in the server??
I'm not sure if I'm following your question entirely, but it appears that you can just set end=true whenever you detect the end of the message you're receiving:
char buf[20];
memset(buf,'\0',20);
while(!end)
{
nbytes_read=recv(sclient,(char *)ress,sizeof(ress),0);
if(strcmp(ress,"1")==0)
{
printf("Details: %s (%i)\n",buf,nbytes_read);
while(strcmp(buf,"\r\n\r\n") != 0)
{
nbytes_read=recv(sclient,(char *)buf,sizeof(buf),0);
cout.flush();
printf("Details: %s (%i)\n",buf,nbytes_read);
}
}
if(strcmp(buf,"\r\n\r\n")==0)
{
end = true; // <--- This should do it for you, right?
printf("The End\n");
}
cout.flush();
}
However, if the client is still connected and writing the next message to the socket, then you just need to start reading the next message. So what happens with the client once the message is written? Does it start writing the next message or does it close the socket connection?
In addition: you need to take what's in your buffer and create a message from it. When the current message is done, then consider creating a new message with the contents of the buffer from the next message.
If you design your protocol like HTTP 1.0, where each request opens a new socket, then you close the socket after you've read enough.
Otherwise, you need to keep on reading until you skipped the entire 4096 bytes. The easiest thing to do is to keep on reading till you get 4096 bytes in the first place (you'll need to call recv in a loop), and then parse the contents of the buffer. Then again, you might be better off redesigning your protocol.
My thought would be to just to peek at the first x chars.
The 4 chars could be the size of the buffer expected.
So for example if your message is:
abcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcde\r\n\r\n
It's (to use your schematic) 100 bytes, plus the \r\n\r\n. So it's 100 + 4, so 104.
I would send char(104) at the beginning of your string, as a sentinal value
then the string with it right after so it'd appear similar to
char(104)abcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcde\r\n\r\n
Then use recv's peek_MSG ability to get the first char, make your string size, read only that value and whatever's left get's discarded by a socket flush call you make.
I have a TCP client connecting to my server which is sending raw data packets. How, using Boost.Asio, can I get the "whole" packet every time (asynchronously, of course)? Assume these packets can be any size up to the full size of my memory.
Basically, I want to avoid creating a statically sized buffer.
Typically when you build a custom protocol on the top of TCP/IP you use a simple message format where first 4 bytes is an unsigned integer containing the message length and the rest is the message data. If you have such a protocol then the reception loop is as simple as below (not sure what is ASIO notation, so it's just an idea)
for(;;) {
uint_32_t len = 0u;
read(socket, &len, 4); // may need multiple reads in non-blocking mode
len = ntohl(len);
assert (len < my_max_len);
char* buf = new char[len];
read(socket, buf, len); // may need multiple reads in non-blocking mode
...
}
typically, when you do async IO, your protocol should support it.
one easy way is to prefix a byte array with it's length at the logical level, and have the reading code buffer up until it has a full buffer ready for parsing.
if you don't do it, you will end up with this logic scattered all over the place (think about reading a null terminated string, and what it means if you just get a part of it every time select/poll returns).
TCP doesn't operate with packets. It provides you one contiguous stream. You can ask for the next N bytes, or for all the data received so far, but there is no "packet" boundary, no way to distinguish what is or is not a packet.
c++
#define BUF_LEN 1024
the below code only receives one byte when its called then immediately moves on.
output = new char[BUF_LEN];
bytes_recv = recv(cli, output, BUF_LEN, 0);
output[bytes_recv] = '\0';
Any idea how to make it receive more bytes?
EDIT: the client connecting is Telnet.
The thing to remember about networking is that you will be able to read as much data as has been received. Since your code is asking for 1024 bytes and you only read 1, then only 1 byte has been received.
Since you are using a telnet client, it sounds like you have it configured in character mode. In this mode, as soon as you type a character, it will be sent.
Try to reconfigure your telnet client in line mode. In line mode, the telnet client will wait until you hit return before it sends the entire line.
On my telnet client. In order to do that, first I type ctrl-] to get to the telnet prompt and then type "mode line" to configure telnet in line mode.
Update
On further thought, this is actually a very good problem to have.
In the real world, your data can get fragmented in unexpected ways. The client may make a single send() call of N bytes but the data may not arrive in a single packet. If your code can handle byte arriving 1 by 1, then you know it will work know matter how the data arrives.
What you need to do is make sure that you accumulate your data across multiple receives. After your recv call returns, you should then append the data a buffer. Something like:
char *accumulate_buffer = new char[BUF_LEN];
size_t accumulate_buffer_len = 0;
...
bytes_recv = recv(fd,
accumulate_buffer + accumulate_buffer_len,
BUF_LEN - accumulate_buffer_len,
0);
if (bytes_recv > 0)
accumulate_buffer_len += bytes_recv;
if (can_handle_data(accumulate_buffer, accumulate_buffer_len))
{
handle_data(accumulate_buffer, accumulate_buffer_len);
accumulate_buffer_len = 0;
}
This code keeps accumulating the recv into a buffer until there is enough data to handle. Once you handle the data, you reset the length to 0 and you start accumulating afresh.
First, in this line:
output[bytes_recv] = '\0';
you need to check if bytes_recv < 0 first before you do that because you might have an error. And the way your code currently works, you'll just randomly stomp on some random piece of memory (likely the byte just before the buffer).
Secondly, the fact you are null terminating your buffer indicates that you're expecting to receive ASCII text with no embedded null characters. Never assume that, you will be wrong at the worst possible time.
Lastly stream sockets have a model that's basically a very long piece of tape with lots of letters stamped on it. There is no promise that the tape is going to be moving at any particular speed. When you do a recv call you're saying "Please give me as many letters from the tape as you have so far, up to this many.". You may get as many as you ask for, you may get only 1. No promises. It doesn't matter how the other side spit bits of the tape out, the tape is going through an extremely complex bunch of gears and you just have no idea how many letters are going to be coming by at any given time.
If you care about certain groupings of characters, you have to put things in the stream (ont the tape) saying where those units start and/or end. There are many ways of doing this. Telnet itself uses several different ones in different circumstances.
And on the receiving side, you have to look for those markers and put the sequences of characters you want to treat as a unit together yourself.
So, if you want to read a line, you have to read until you get a '\n'. If you try to read 1024 bytes at a time, you have to take into account that the '\n' might end up in the middle of your buffer and so your buffer may contain the line you want and part of the next line. It might even contain several lines. The only promise is that you won't get more characters than you asked for.
Force the sending side to send more bytes using Nagle's algorithm, then you will receive them in packages.