What is the purpose of using MPI_Pack/MPI_Unpack? Is it better than working with mpi structure types or derived types?
Packing & unpacking is great as a serialization tool: if you can turn an object (in an informal sense) into a sequence of "1 int, 3 chars, 2 doubles, 5 chars, 2 shorts" (or whatever) you can pack this into a "self-documenting message". The receiving side can unpack this with very little knowledge of what is in your message.
The only caveat is that the receiving side may not know the size of the buffer that is incoming, so you may need to do a Probe first.
Related
I was surprised to discover, when using Spacetime to profile my OCaml, that my char and even bool arrays used a word to represent each element. That's 8 bytes on my 64 bit machine, and causes way too much memory to be used.
I've substituted char array with Bytes where possible, but I also have char list and dynamic arrays (char BatDynArray). Is there some primitive or general method that I can use across all of these vector data structures and get an underlying 8 bit representation?
Edit: I read your question too fast: it’s possible you already know that; sorry! Here is a more targeted answer.
I think the general advice for storing a varying numbers of chars of varying number (i.e. when doing IO) is to use buffers, possibly resizable. Module Buffer implements a resizable character buffer, which is better than both char list (bad design, except for very short lists perhaps) and char BatDynArray (whose genericity incurs a memory penalty here, as you noticed).
Below is the original answer.
That’s due to the uniform representation of values. Whatever their type , every OCaml value is a machine word: either an immediate value (anything that can fit a 31- or 63-bit integer, so int, char, bool, etc.), or a pointer to a block, i.e. a sequence of machine words (a C-fashion array), prefixed with a header. When the value is a pointer to a block we say that it is “boxed”.
Cells of OCaml arrays are always machine words.
In OCaml, like in C++ but without the ad-hoc overloading, we just define specializations of array in the few cases where we actually want to save space. In your case:
instead of char array use string (immutable) or bytes (mutable) or Buffer.t (mutable appendable and resizable); these types signal to the GC that their cells are never pointers, so they can pack arbitrary binary data;
Unfortunately, the standard library has no specialization for bool array, but we can implement one (e.g. using bytes); you can find one in several third-party libraries, for instance module CCBV (“bitvectors”) in package containers-data.
Finally, you may not have realized it, but floats are boxed! That’s because they require 64 bits (IEEE 754 double-precision), which is more than the 31 or even 63 bits that are available for immediates. Fortunately(?), the compiler and runtime have some adhoc-ery to avoid boxing them as much as possible. In particular float array is specially optimized, so that it stores the raw floating-point numbers instead of pointers to them.
Some more background: we can distinguish between pointers and immediates just by testing one bit. Uniform representation is highly valuable for:
implementing garbage collection,
free parametric polymorphism (no code duplication, by contrast with what you’d get in a template language such as C++).
As far as I understand, the number of bytes used for int is system dependent. Usually, 2 or 4 bytes are used for int.
As per Microsoft's documentation, __int8, __int16, __int32 and __int64 are Microsoft Specific keywords. Furthermore, __int16 uses 16-bits (i.e. 2 bytes).
Question: What are advantage/disadvantage of using __int16 (or int16_t)? For example, if I am sure that the value of my integer variable will never need more than 16 bits then, will it be beneficial to declare the variable as __int16 var (or int16_t var)?
UPDATE: I see that several comments/answers suggest using int16_t instead of __int16, which is a good suggestion but not really an advantage/disadvantage of using __int16. Basically, my question is, what is the advantage/disadvantage of saving 2 bytes by using 16-bit version of an integer instead of int ?
Saving 2 bytes is almost never worth it. However, saving thousands of bytes is. If you have an large array containing integers, using a small integer type can save quite a lot of memory. This leads to faster code, because the less memory one uses the less cache misses one receives (cache misses are a major loss of performance).
TL;DR: this is beneficial to do in large arrays, but pointless for 1-off variables.
The second use of these is if for dealing with binary files and messages. If you are reading a binary file that uses 16-bit integers, well, it's pretty convenient if you can represent that type exactly in your code.
BTW, don't use microsoft's versions. Use the standard versions (std::int16_t)
It depends.
On x86, primitive types are generally aligned on their size. So 2-byte types would be aligned on a 2-byte boundary. This is useful when you have more than one of these short variables, because you will be saving 50% of space. That directly translates to better memory and cache utilization and thus theoretically, better performance.
On the other hand, doing arithmetic on shorter-than-int types usually involves widening conversion to int. So if you do a lot of arithmetic on these types, using int types might result in better performance (contrived example).
So if you care about performance of a critical section of code, profile it to find out for sure if using a certain data type is faster or slower.
A possible rule of thumb would be - if you're memory-bound (i.e. you have lots of variables and especially arrays), use as short a data types as possible. If not - don't worry about it and use int types.
If you for some reason just need a shorter integer type it's already have that in the language - called short - unless you know you need exactly 16 bits there's really no good reason not to just stick with the agnostic short and int types. The broad idea is that these types should align well the target architecture (for example see word ).
That being said, theres no need to use the platform specific type (__int16), you can just use the standard one:
int16_t
See https://en.cppreference.com/w/cpp/types/integer for more information and standard types
Even if you still insist on __int16 you probably want a typedef something ala.:
using my_short = __int16;
Update
Your main question is:
What is the advantage/disadvantage of
saving 2 bytes by using 16-bit version of an integer instead of int ?
If you have a lot of data (In the ballpark of at least some 100.000-1.000.000 elements as a rule of thumb) - then there could be an overall performance saving in terms of using less cpu-cache. Overall there's no disadvantage of using a smaller type - except for the obvious one - and possible conversions as explained in this answer
The main reason for using these types is to make sure about the size of your variable in different architectures and compilers. we call it "code reusability" and "portability"
in higher-level modern languages, all this will handle with compiler/interpreter/virtual machine/etc. that you don't need to worry about, but it has some performance and memory usage costs.
When you have some kind of limitation you may need to optimize everything. The best example is embedded systems that have a very limited size of memory and work at low frequency. In the other hand, there are lots of compilers out there with different implementations. Some of them interpret "int" as a "16bit" value and some as a "32bit".
for example, you receive and specific stream of values over a communication system, you want to save them in a buffer or array and you want to make sure the input data is always interpreted as a 16bit noting else.
I would like to send 1024 bytes of data using Thrift. It must be exactly 1024 bytes because it is a comparative benchmark with other frameworks.
Thrift has two types to represent bytes : 'byte' and 'binary', but I don't know how to use these types.
'binary' type is mapped to std::string which is quite strange (I don't understand why and how to use it).
'byte' type is mapped to a 8 bits integer which seems more logical to me.
To represent 1024 bytes of data, I use : list<byte> byteSequence with a size of 1024.
But a compile warning advises me to use binary instead of list<byte>, but why ? and how ?
I think I will get much better performance with 'binary' because it is strangely slow with a 1024 sequence of bytes.
Thank you.
But a compile warning advises me to use binary instead of list, but why ? and how ?
'byte' type is mapped to a 8 bits integer which seems more logical to me.
And that's exactly why the warning is there. It seems logical but it is the worst choice. Furthermore, byte in Thrift is in fact an i8 - a signed type.
'binary' type is mapped to std::string which is quite strange (I don't understand why).
Don't worry. That's an historical thing. The binarytype was added later and implemented similar to string in some ways to reduce compatibility friction with older versions. That's just an impl detail.
but I don't know how to use these types.
Like any other type:
struct wtf {
1 : binary foo
2 : string bar
3 : byte baz // i8 is replacing byte to indicate the signedness
4 : list<byte> // not recommended, but nevertheless works
}
It probably depends on the language you will be compiling your thrift files to, but binary tells thrift directly that you indeed want to transmit a sequence of raw, unencoded bytes.
It may not change things much at the transport layer in terms of size, but you may run into surprises when you instantiate/de-serialise the objects in your chosen language. In Java, for example, a binary field will be represented with a byte[] whereas list[byte] will give you a List[Byte], which is far less efficient to represent the same thing.
Java might be the only reason for binary, as according to the thrift doc:
binary: a sequence of unencoded bytes
N.B.: This is currently a specialized form of the string type above, added to provide better interoperability with Java. The current plan-of-record is to elevate this to a base type at some point.
I need to send a packet of data over a UDP connection in C++. The first message I need to send is built up of two 32 bit integers and a 64 bit integer. What is the best way to combine multiple variable types into one block of data ready for sending over a UDP connection?
It depends on the requirements for your network. Do you care about endianness? If you do, you should use not just any serialisatioin, but a safe one in regards to endianness.
Generally, each class/struct sendable through the network should have special methods or overloaded operators to stream them in and out. Ultimately you'll have to use macros/functions like hton/ntoh for streaming primitive types eg int, int64, float, double etc.
Upd: also if your network endpoint applications run on different platforms/compilers, you may have different sizes of int, long, short etc. So when serialising, you'll have to convert your integers to some predefined types with sizes guaranteed to be the same on all supported platforms.
I wrote a DNS resolver by hand in C, and structs is indeed the way I did it. Use bit fields to specify sizes of each piece. More about bit fields: http://msdn.microsoft.com/en-us/library/ewwyfdbe.aspx
Make sure to use hton/ntoh to take care of byte order. More information here; http://www.beej.us/guide/bgnet/output/html/multipage/htonsman.html
In fact, peruse beej's guide -- mucho useful information there!
First of all, to clarify my goal: There exist two programs written in C in our laboratory. I am working on a Proxy Server (bidirectional) for them (which will also mainpulate the data). And I want to write that proxy server in Python. It is important to know that I know close to nothing about these two programs, I only know the definition file of the packets.
Now: assuming a packet definition in one of the C++ programs reads like this:
unsigned char Packet[0x32]; // Packet[Length]
int z=0;
Packet[0]=0x00; // Spare
Packet[1]=0x32; // Length
Packet[2]=0x01; // Source
Packet[3]=0x02; // Destination
Packet[4]=0x01; // ID
Packet[5]=0x00; // Spare
for(z=0;z<=24;z+=8)
{
Packet[9-z/8]=((int)(720000+armcontrolpacket->dof0_rot*1000)/(int)pow((double)2,(double)z));
Packet[13-z/8]=((int)(720000+armcontrolpacket->dof0_speed*1000)/(int)pow((double)2,(double)z));
Packet[17-z/8]=((int)(720000+armcontrolpacket->dof1_rot*1000)/(int)pow((double)2,(double)z));
Packet[21-z/8]=((int)(720000+armcontrolpacket->dof1_speed*1000)/(int)pow((double)2,(double)z));
Packet[25-z/8]=((int)(720000+armcontrolpacket->dof2_rot*1000)/(int)pow((double)2,(double)z));
Packet[29-z/8]=((int)(720000+armcontrolpacket->dof2_speed*1000)/(int)pow((double)2,(double)z));
Packet[33-z/8]=((int)(720000+armcontrolpacket->dof3_rot*1000)/(int)pow((double)2,(double)z));
Packet[37-z/8]=((int)(720000+armcontrolpacket->dof3_speed*1000)/(int)pow((double)2,(double)z));
Packet[41-z/8]=((int)(720000+armcontrolpacket->dof4_rot*1000)/(int)pow((double)2,(double)z));
Packet[45-z/8]=((int)(720000+armcontrolpacket->dof4_speed*1000)/(int)pow((double)2,(double)z));
Packet[49-z/8]=((int)armcontrolpacket->timestamp/(int)pow(2.0,(double)z));
}
if(SendPacket(sock,(char*)&Packet,sizeof(Packet)))
return 1;
return 0;
What would be the easiest way to receive that data, convert it into a readable python format, manipulate them and send them forward to the receiver?
You can receive the packet's 50 bytes with a .recv call on a properly connected socked (it might actually take more than one call in the unlikely event the TCP packet gets fragmented, so check incoming length until you have exactly 50 bytes in hand;-).
After that, understanding that C code is puzzling. The assignments of ints (presumably 4-bytes each) to Packet[9], Packet[13], etc, give the impression that the intention is to set 4 bytes at a time within Packet, but that's not what happens: each assignment sets exactly one byte in the packet, from the lowest byte of the int that's the RHS of the assignment. But those bytes are the bytes of (int)(720000+armcontrolpacket->dof0_rot*1000) and so on...
So must those last 44 bytes of the packet be interpreted as 11 4-byte integers (signed? unsigned?) or 44 independent values? I'll guess the former, and do...:
import struct
f = '>x4bx11i'
values = struct.unpack(f, packet)
the format f indicates: big-endian, 4 unsigned-byte values surrounded by two ignored "spare" bytes, 11 4-byte signed integers. Tuple values ends up with 15 values: the four single bytes (50, 1, 2, 1 in your example), then 11 signed integers. You can use the same format string to pack a modified version of the tuple back into a 50-bytes packet to resend.
Since you explicitly place the length in the packet it may be that different packets have different lenghts (though that's incompatible with the fixed-length declaration in your C sample) in which case you need to be a bit more accurate in receiving and unpacking it; however such details depend on information you don't give, so I'll stop trying to guess;-).
Take a look at the struct module, specifically the pack and unpack functions. They work with format strings that allow you to specify what types you want to write or read and what endianness and alignment you want to use.