I'm working to copy the following structure to a byte array to send over a named pipe. I've found that since switching from a byte array that I had given a static definition, to a vector because my host length will be of varying lengths.
Here is the outline of my structure:
USHORT version; // Header Version
USHORT type; // IPVersion
USHORT count; // Number of IP addresses of remote system
USHORT length; // Header Length (1)
BYTE SysConfigLocIP[4];
BYTE SysConfigRemoteIP[4];
USHORT lengthHost;
std::vector<BYTE>HostName;
later, after filling the structure I copy it to a byte like so:
BYTE Response[sizeof(aMsg)]
memcpy(response, &aMsg, sizeof(aMsg))
I find that my array is vector is holding the correct information for the host when I inspect the container during a debug. However, after the copy to the Response byte array, I'm finding the data that has been copied is drastically different. Is this a valid operation, if so, what can I do correctly copy the data from my vector the BYTE array. If not, what are other strategies I can use to dynamically size the structure to send the hostnames? Thank you for taking the moment of time to read my question, and I appreciate any feedback.
I'm working to copy the following structure to a byte array to send
over a named pipe.
named pipe (or other forms of inter-process or inter-processor communication) does not understand your struct, neither do they understand vector. They just operate on the concept of byte-in-byte-out. It is up to you, the programmer, to assign meaning to those bytes.
As suggested, please read on serialization. Try starting at http://en.wikipedia.org/wiki/Serialization. If permitted you can use the Boost solution, http://www.boost.org/doc/libs/1_55_0/libs/serialization/doc/index.html, but I would still encourage you to understand the basics first.
As an exercise, first try transferring a vector<int> from sender to receiver. The number of elements in the vector must not be implicitly known by the receiver. Once you achieve that, migrating from int to your struct would be trivial.
That memcpy will only work for POD (plain old data) types. A vector is not POD. Instead, write code to put each byte in the buffer exactly where it needs to be. Don't rely on "magic".
99% of the time in C++ there is no reason to use memcpy. It breaks classes. Learn about copy constructors and std::copy and use them instead.
Related
I am wondering if it is possible to create/copy a "virtual" array of a specific member of a struct in another array. Let's say we have a struct
struct foo {
int value;
char character;
};
Now assume there is an array containing this struct foo and I have an operation that needs to add all int value's together. This would normally be very easy with a loop adding all the values with a pointer. Problem is I am using OpenCL and need to copy an array to some device. In OpenCL this is done using
clEnqueueWriteBuffer(cmdQueue, buffer, CL_TRUE, 0, datasize, A, 0, NULL, NULL);
which will copy an array buffer to the device. It doesn't make sense to copy the entire array of structs, since the would take more time, because it also sends the characters which is not needed. It would also take up more space on the OpenCL device. Is it therefore possible to copy the "array" of values from the structs directly as an array to the device?
I know I can create a new array on the host (CPU) with all the values and then copy that array to the OpenCL device, but then I would spend time copying to a local int-array and afterwards copy that array to the OpenCL device.
Would it be possible to copy a "virtual" array of values directly from the array of foo-structs, containing only the int values?
Please beware, that this is a very simplified example of my actual problem and would like to avoid having the values in a separate array from the beginning, which the structs would then point to. I have big doubts that this is possible, and if my explanation even makes sense, but look forward to feedback!
No.
clEnqueueWriteBuffer expects a contiguous container. You cannot create a "virtual" contiguous container.
[I] like to avoid having the values in a separate array from the beginning.
At that point, you must profile and compare two implementations: one copying the array as-is with the superfluous data, and one creating a local copy of the useful data to send. Compare and choose.
If you have an array of structs you would need a staging buffer with just the values, which is extra copies on the CPU side.
Sometimes such work is unavoidable, but if you can, it is better to have multiple arrays of continuous values. Even in pure CPU work this is frequently more efficient for the CPU cache as it avoids read/writes of unneeded members and is often easier for SIMD instruction sets like SSE.
For example you could have int *values and char *chars of the same length (prefer some type like std::vector or std::unique_ptr<T[]> though!), then the copy is easy.
Before anything, I want to say the title is not the question.
My question is why can't you just send all the bytes of the structure and then cast it into that structure (giving you have the structure defined in both sides, which makes sense you have).
Thank you!
EDIT: Here's my current structure:
struct COMPUTER_INFO
{
const char* Name;
int Brightness;
int Volume;
}
I was thinking that it you can easily calculate the size of all that and then send it trough send().
Name is a pointer (contains an address) that only makes sense to your program on your computer. If you sent this structure as bytes the receiving program would receive just the address not the characters that comprise the name. The received address would also not point to a valid location in memory in the receiving computer.
Brightness and Volume are ints - ints do not have a fixed size they are the "natural" word size of the computer (the standard does impose some restrictions). So the sizeof(int) on the sending and receiving computer may be different. There may also be encoding differences e.g. big vs little endian. See: https://en.wikipedia.org/wiki/Endianness
in general you are right. You can, of course, send the raw byte streams. But how do you want the receiver to recognize the positions of the members of your structure in this byte stream? Especially, the char* in your example is of variable size, so this will not be possible.
I recommend using the Boost serialization package. You can find a detailed tutorial here. The package will take care of "serializing" (packaging your object into a byte stream) and "deserializing" (constructing your object from the byte stream). It is absolutely convenient for nearly all STL containers and can easily be expanded for custom types.
I don't quite understand the advantage of using streambuf over the regular array.
Let me explain my problem. I have a network connection which is encrypted using Rijndael 128 ECB + some easy cipher to encrypt remaining data that is shorter than 16 bytes. The packets are structured as length_of_whole_packet+operationcode+data. I have to actually copy all the data from the streambuf so I can apply decryption algorithm? Why making another copy of data I already have?
Same problem I have with the sending data. When in securedmode the packet structure is length_of_whole_packet+crc+data, where crc and data is encrypted. I could make some monstrosity as MakePacket(HEADER, FORMAT, ...) that would allocate array, format the packet, add crc and encrypt it, but I would like to avoid vararg function. I can't use structs as the packets has dynamic length because there can be arrays in it, or strings. If I used MakePacket(unsigned char opcode, &streambuf sb) then there would be problem with the crc again -> have to make a copy to encrypt it.
Should I use the vararg monstrosity for sending using regular array as buffer in combination with unsigned char pbyRecvBuffer[BUFFERMAXLEN] for recv?
I'm not really sure how to design this communication with avoiding copies of data.
Thank you for you answers.
When using streambufs, copying of data can often be minimized by using algorithms that operate on iterators, such as std::istreambuf_iterator or boost::asio::buffers_iterator, rather than copying data from a streambuf into another data structure.
For stream-like application protocols, boost::asio::streambuf is often superior to boost::asio::buffer() compatible types, such as a raw array. For example, consider HTTP, where a delimiter is used to identify boundaries between variable length headers and bodies. The higher-level read_until() operations provide an elegant way to to read the protocol, as Boost.Asio will handle the memory allocation, detect the delimiter, and invoke the completion handler once the message boundary has been reached. If an application used a raw array, it would need to read chunks and copy each fragmented chunk into an aggregated memory buffer until the appropriate delimiter was identified.
If the application can determine the exact amount of bytes to read, then it may be worth considering using boost::array for fixed length portions and std::vector for the variable length portions. For example, an application protocol with a:
fixed length body could be read into a boost::array.
fixed length header that contains enough information to determine the length of the following variable length body could use a std::vector to read the fixed size header, resize the vector once the body length has been determined, then read the body.
In context of the question, if length_of_whole_packet is of a fixed length, the application could read it into std::vector, resize the vector based on the determined body length, then read the remaining data into the vector. The decryption algorithm could then operate directly on the vector and use an output iterator, such as std::back_insert_iterator, with an auxiliary output buffer if the algorithm cannot be done in-place. The same holds true for encrypting data to be written.
My program uses UDP to send data between two programs, it works great however I have added a new vector into the data I want to send, the vector type is another class which looks like this...
class Bullet: public Sprite
{
public:
float speed;
};
The DataPacket...
typedef struct DataPacket
{
int ID; //Player ID
int elapsedTime; //Total elapsed player time
float x, y; //X & Y pos of player
std::vector<Bullet>* pBullets; //Vector containing all the players bullets
};
Is there a way to send this data correctly? The expression cannot be evaluated once the server receives the data from the client, every other part is correct in the received packet.
Basically the server is receiving the positional data of the bullets on the screen which is contained in this Bullet class along with a few other items.
Just to note: pBullet never used to be a pointer but in an attempt to try and figure out what was wrong I changed it to a pointer...it never fixed the issue tho
You cannot do this. A vector is a class which is typically implemented with an internal pointer to dynamic memory which changes location as the vector grows.
When you try and serialize the vector by casting your whole structure, you just serialize the pointer to the memory holding the vector contents, you don't get the contents itself because it's not part of that structure.
You'll have to individually serialize all items in the vector one-by-one and individually add them back in when you deserialize it.
Note that you can change the vector of bullets to a statically sized array internal to the structure and then it would be contiguous in memory and you could just serialize the whole structure - and you can also make the last element of the array an array of one bullet, and then allocate memory for the structure size + (x-1)*sizeof(Bullet), allowing you to overwrite the array of 1 for all of the extra memory you added. This would also be contiguous in memory allowing you to serialize the whole memory region pretty easily.
You should also look in to htonl, ntohl, htons, ntohs and start network-byte-ordering your data as well if you're going to send it over a network to keep the byte endianness from being an issue on some other systems you might end up using.
Use Serialization whenever you have to send data across the network. You may refer to Boost Serialization.
Your DataPacket is something whose size is dynamic. Had it been an array than vector, it might have worked given the same endianness of the machines.
For Game Development I would recommend to use Google ProtoBuf. In comparison with Boost Serialization, it provides binary serialization (Boost Serialization has an example of binary serialization but it's not very portable) and is more convenient for complex data structures.
You need to serialize the data yourself. You can't just send std::vector or pointer thereto.
I am working on a sample application in this application I am serializing some of the data. In client application I am reading the serialized data back. While doing this I observed some strange behavior.
In sample application size of object is different from size of data in client. I think this is because of memory padding. My problem is I am trying to write “BRUSHOBJ” to file. This structure is defined by Microsoft. I can change the declaration of this structure. Please let me know how to solve this problem.
Please let me know how to apply memory padding on slandered data type.
It sounds like you're trying to just cast the address of a struct to
char*, and use ostream::write on it. This simply doesn't work.
There's padding, but there's also the size of different types (which
varies from one platform to the next), byte order, and on some more
exotic platforms (including most mainframes) data representation itself.
Generally, you need a specification of what the output data should look
like, byte by byte, and you have to then write each byte with the
required value.
And this is just for simple types. A quick glance at BRUSHOBJ shows
that it contains a pointer, which you'll probably have to
follow—you'll certainly have to do something with it, since the
receiving end won't be able to do anything with a pointer into your
data. (I suspect, given the description, that you'll have to convert it
into some sort of identifier, and also transmit a dictionary mapping
such identifiers to objects. But I don't know enough about how this
structure is used to be sure.)
you have 2 options
serializing data
modify memory padding via #pragma pack
Serializing data has no relation with memory padding, you are just defining a way to write/ read back memory to/from a memory location (the memory stream).
I see the that _BRUSHOBJ struct has the following definition,
typedef struct _BRUSHOBJ {
ULONG iSolidColor;
PVOID pvRbrush;
FLONG flColorType;
} BRUSHOBJ;
please note that sending a pointer across process is nonsens. serializing a pointer should be done by writing the size of memory and the the memory itself. Anyway if you want to pass this BRUSHOBJ to a windows function you can get undefined behavior. It's not a supported/documented way of passing a BRUSHOBJ across process.
memory padding can by applied like this
#pragma pack(push)
#pragma pack(4)
struct myStruct
{
char Char1
int Int1;
};
#pragma pack(pop)
If you what to modify padding you should doit for a structure that is written by you.