I'm using C++ and wondering if I can just send an entire int array over a network (using basic sockets) without doing anything. Or do I have to split the data up and send it one at a time?
Yes.
An array will be laid out sequentially in memory so you are free to do this. Simply pass in the address of the first element and the amount of data and you'll send all data.
You could definitely send an array in one send, however you might want to do some additional work. There are issues with interpreting it correctly at the receiving end. For example, if using different machine architectures, you may want to convert the integers to network order (e.g., htonl).
Another thing to keep in mind is the memory layout. If it is a simple array of integers, then it would be contiguous in memory and a single send could successfully capture all the data. If, though, (and this is probably obvious), you have an array with other data, then the layout definitely needs consideration. A simple example would be if the array had pointers to other data such as a character string, then a send of the array would be sending pointers (and not data) and would be meaningless to the receiver.
Related
Before anything, I want to say the title is not the question.
My question is why can't you just send all the bytes of the structure and then cast it into that structure (giving you have the structure defined in both sides, which makes sense you have).
Thank you!
EDIT: Here's my current structure:
struct COMPUTER_INFO
{
const char* Name;
int Brightness;
int Volume;
}
I was thinking that it you can easily calculate the size of all that and then send it trough send().
Name is a pointer (contains an address) that only makes sense to your program on your computer. If you sent this structure as bytes the receiving program would receive just the address not the characters that comprise the name. The received address would also not point to a valid location in memory in the receiving computer.
Brightness and Volume are ints - ints do not have a fixed size they are the "natural" word size of the computer (the standard does impose some restrictions). So the sizeof(int) on the sending and receiving computer may be different. There may also be encoding differences e.g. big vs little endian. See: https://en.wikipedia.org/wiki/Endianness
in general you are right. You can, of course, send the raw byte streams. But how do you want the receiver to recognize the positions of the members of your structure in this byte stream? Especially, the char* in your example is of variable size, so this will not be possible.
I recommend using the Boost serialization package. You can find a detailed tutorial here. The package will take care of "serializing" (packaging your object into a byte stream) and "deserializing" (constructing your object from the byte stream). It is absolutely convenient for nearly all STL containers and can easily be expanded for custom types.
I want to send a vector from one computer to another through internet. I'm looking for a Peer-to-Peer solution in C++. I have made a Winsock2 solution, but it can only send char* with the send and recv-functions which I don't think will work for my project.
Is there a way of using JSON with a P2P-solution in C++? So make a JSON-object of the vector and send it through internet? Or do you know a better solution?
The vector I want to send through internet to another computer looks like this:
Vector<AVpacket> data;
AVpacket is a struct from ffmpeg, consisting 14 data members. https://ffmpeg.org/doxygen/trunk/structAVPacket.html. You don't want to make this to a char*
You can actually send anything using the send and recv functions. You just have to make sure you pass a pointer to the data, and then typecast that pointer as a char * and it will work.
However, you can't send a std::vector as is. Instead you should first send its size (otherwise the receiving end will not know how much data it should receive) then you send the actual data in the vector, i.e. someVector.data() or &someVector[0].
Though in your case it will be even more complicated, as the structures you want to send contains embedded pointers. You can't send pointers over the Internet, it's barely possible to transfer pointers between two processes on the same system. You need to read about serialization and maybe about the related subject marshalling.
In short: You can send any kind of data, it doesn't have to be characters, and for the kind of structures you want to send you have to convert them to a format is transferable through serialization.
You can not simply send a vector. You can not be sure how the std allocator reserved the memory. So it is very likely that the whole memory of the vector is not just one big linear chunk.
In addition to this, as pointed out above, there are pointers in this struct. They point to data in your local memory. These addresses aren't valid on the recipients side, thus you would access invalid memory trying to read this.
I guess, that in order to achieve what you want to do, you have to try a completely different approach. Do not get lost by trying to send the data of the pointers or similar, rather try having parallel data on both machine.
E.g. both load the same video e.g. from a file which both share. Then use a unique identifier for that video to reference the video on both sides.
If I keep sending data to a receiver is it possible for the data sent to overlap such that they accumulate in the buffer and so the next read to the buffer reads also the data of another sent data?
I'm using Qt and readAll() to receive data and parse it. This data has some structure in it so I can know if the data is already complete or if it is valid data at all but I'm worried that other data will overlap with others when I call readAll() and so would invalidate this suppose-to-be valid data.
If it can happen, how do I prevent/control it? Or is that something the OS/API worries about instead? I'm worried partly because of how the method is called. lol
TCP is a stream based connection, not a packet based connection, so you may not assume that what is sent in one time will also be received in one time. You still need some kind of protocol to packetize your stream.
For sending strings, you could use the nul-character as separator or you could begin with a header which contains a magic and a length.
According to http://qt-project.org/doc/qt-4.8/qiodevice.html#readAll this function snarfs all the data and returns it as an array. I don't see how the API raises concerns about overlapping data. The array is returned by value, and given that it represents the entire stream, so what would it even overlap with? Are you worried that the returned object actually has reference semantics (i.e. that it just holds pointers to storage that is re-used in other calls to the same function?)
If send and receive buffers overlap in any system, that's a bug, unless special care is taken that the use is completely serialized. (I.e. a buffer is somehow used only for sending and only for receiving, without any mixup.)
Why dont you use a fixed length header followed by variable length packet with the header holding the information of length of packet.
This way you can avoid worrying about packet boundaries. Say for example instead of just sending the string send the length of the string followed by the string. In the receiver end always read the length and then based on the length read the string.
I'm using libcurl (HTTP transfer library) with C++ and trying to download files from remote HTTP servers. As file is downloaded, my callback function is called multiple times (e.g. every 10 kb) to send me buffer data.
Basically I need something like "string bufer", a data structure to append char buffer to existing string. In C, I allocate (malloc) a char* and then as new buffers come, I realloc and then memcpy so that I can easily copy my buffer to resized array.
In C, there are multiple solutions to achieve this.
I can keep using malloc, realloc, memcpy but I'm pretty sure that they are not recommended in C++.
I can use vector<char>.
I can use stringstream.
My use cases is, I'll append a few thousands of items (chars) at a time, and after it all finishes (download is completed), I will read all of it at once. But I may need options like seek in the future (easy to achieve in array solution (1)) but it is low priority now.
What should I use?
I'd go for stringstream. Just insert into it as you recieve the data, and when you're done you can extract a full std::string from it. I don't see why you'd want to seek into an array? Anyway, if you know the block size, you can calculate where in the string the corresponding block went.
I'm not sure if many will agree with this, but for that use case I would actually use a linked list, with each node containing an arbitrarily large array of char that were allocated using new. My reasoning being:
Items are added in large chunks at a time, one at a time at the back.
I assume this could use quite a large amount of space, so you avoid reallocation events when a vector would otherwise need more space.
Since items are read sequentially, the penalty of link lists being unidirectional doesn't affect you.
Should Seeking through the list become a priority, this wouldn't work though. If it's not a lot of data ultimately, I honestly think a vector would be fine, dispite not being the most efficient structure.
If you just need to append char buffers, you can also simply use std::string and the member function append. On top of that stringstream gives you formatting, functionality, so you can add numbers, padding etc., but from your description you appear not to need that.
I would use vector<char>. But they will all work even with a seek, so your question is really one of style and there are no definitive answers there.
I think I'd use a deque<char>. Same interface as vector, and vector would do, but vector needs to copy the whole data each time an append exceeds its existing capacity. Growth is exponential, but you'd still expect about log N reallocations, where N is the number of equal-sized blocks of data you append. Deque doesn't reallocate, so it's the container of choice in cases where a vector would need to reallocate several times.
Assuming the callback is handed a char* buffer and length, the code to copy and append the data is simple enough:
mydeque.insert(mydeque.end(), buf, buf + len);
To get a string at the end, if you want one:
std::string mystring(mydeque.begin(), mydeque.end());
I'm not exactly sure what you mean by seek, but obviously deque can be accessed by index or iterator, same as vector.
Another possibility, though, is that if you expect a content-length at the start of the download, you could use a vector and reserve() enough space for the data before you start, which avoids reallocation. That depends on what HTTP requests you're making, and to what servers, since some HTTP responses will use chunked encoding and won't provide the size up front.
Create your own Buffer class to abstract away the details of the storage. If I were you I would likely implement the buffer based on std::vector<char>.
For a networked application, the way we have been transmitting dynamic data is through memcpying a struct into a (void*). This poses some problems, like when this is done to an std::string. Strings can be dynamic length, so how will the other side know when the string ends? An idea I had was to use something similiar to Java's DataOuputStream, where I could just pass whatever variables to it and it could then be put into a (void*). If this can't be done, then its cool. I just don't really like memcpying a struct. Something about it doesn't seem quite right.
Thanks,
Robbie
nothing wrong with memcpy on a struct - as lng as the struct is filled with fixed-size buffers. Put a dynamic variable in there and you have to serialise it differently.
If you have a struct with std::strings in there, create a stream operator and use it to format a buffer. You can then memcpy that buffer to the data transport. If you have boost, use Boost::serialize which does all this for you (that link also has links to alternative serialization libs)
Notes: the usual way to pass a variable-size buffer is to begin by sending the length, then that many bytes of data. Occasionally you see data transferred until a delimiter is received (and fields within that data are delimited themselves by another character, eg a comma).
I see two parts of this question:
- serialization of data over a network
- how to pass structures into a network stack
To serialize data over a network, you'll need a protocol. Doesn't have to be difficult; for ASCII even a cr/lf as packet end may do. If you use a framework (like MFC), it may provide serialization functions for you; in that case you need to worry about how to send this in packets. A packetization which often works well for me is :
<length><data_type>[data....][checksum]
In this case the checksum is optional, and also zero-data is possible, for instance if the signal is carried in the data_type (i.e. Ack for acklnowedgement)
If you're working on the memcpy with structures, you'll need to consider that memcpy only makes a shallow copy. A pointer is worthless once transmitted over a network; instand you should transmit the data from that pointer (i.e. the contents of your string example)
For sending dynamic data across the network you have the following options.
First option in the same packet.
void SendData()
{
int size;
char payload[256];
Send(messageType)
Send(size);
Send(payload)
}
Second option:
void SendData()
{
char payload[256];
Send(messageType)
Send(payload)
}
Though in either situation, you will be faced with more of a design choice. In the first example you would send the message type, and the payload size and also then the payload.
The second option you have is you can send the message type and then you can send the string that has a delimiter of null terminator.
Though either option does not cover fully the problem your facing I think. Firstly, you need to determine if you're building a game what type of protocal you will be using, UDP? TCP? The second problem you will be facing is the maximum packet size. Then on top of that you need to have the framework in place so that you can calculate the optimum packet size that will not be fragmented and lost to the inter web. After that you have bandwidth control in regards to how much data you can transmitted and receive between the client and server.
For example the way that most games approach this situation is each packet is identified with the following.
MessageType
MessageSize
CRCCheckSum
MessageID
void buffer[payload]
In situation where you need to send dynamic data you would send a series of packets not just one. For example if you were to send a file accross the network the best option would to use TCP/IP because its a streaming protocal and it garnentees that the complete stream arrives safly to the other end. On the other hand UDP is a packet based protocal and is does not do any checking that all packets arrived in order or at all on the other end.
So in conclusion.
For dynamic data, send multiple packets but with a special flag
to say more data is to arrive to complete this message.
Keep it simple and if your working with C++ dont assume the packet or data
will contain a null terminator and check the size compared to the
payload if you decide to use a null terminator.