With the syntax for MPI::Isend as
MPI::Request MPI::Comm::Isend(const void *buf, int count,
const MPI::Datatype& datatype,
int dest, int tag) const;
is the amount of data sent limited by
std::numeric_limits<int>::max()
Many other MPI functions have int parameter. Is this a limitation of MPI?
MPI-2.2 defines data length parameters as int. This could be and usually is a problem on most 64-bit Unix systems since int is still 32-bit. Such systems are referred to as LP64, which means that long and pointers are 64-bit long, while int is 32-bit in length. In contrast, Windows x64 is an LLP64 system, which means that both int and long are 32-bit long while long long and pointers are 64-bit long. Linux for 64-bit x86 CPUs is an example of such a Unix-like system which is LP64.
Given all of the above MPI_Send in MPI-2.2 implementations have a message size limit of 2^31-1 elements. One can overcome the limit by constructing a user-defined type (e.g. a contiguous type), which would reduce the amount of data elements. For example, if you register a contiguous type of 2^10 elements of some basic MPI type and then you use MPI_Send to send 2^30 elements of this new type, it would result in a message of 2^40 elements of the basic type. Some MPI implementations may still fail in such cases if they use int to handle elements count internally. Also it breaks MPI_Get_elements and MPI_Get_count as their output count argument is of type int.
MPI-3.0 addresses some of these issues. For example, it provides the MPI_Get_elements_x and MPI_Get_count_x operations which use the MPI_Count typedef for their count argument. MPI_Count is defined so as to be able to hold pointer values, which makes it 64-bit long on most 64-bit systems. There are other extended calls (all end in _x) that take MPI_Count instead of int. The old MPI_Get_elements / MPI_Get_count operations are retained, but now they would return MPI_UNDEFINED if the count is larger than what the int output argument could hold (this clarification is not present in the MPI-2.2 standard and using very large counts in undefined behaviour there).
As pyCthon has already noted, the C++ bindings are deprecated in MPI-2.2 and were removed from MPI-3.0 as no longer supported by the MPI Forum. You should either use the C bindings or resort to 3rd party C++ bindings, e.g. Boost.MPI.
I haven't done MPI, however, int is the usual limiting size of an array, and I would suspect that is where the limitation comes from.
In practice, this is a fairly high limit. Do you have a need to send more than 4 GB of data? (In a single Isend)
For more information, please see Is there a max array length limit in C++?
Do note that link makes references to size_t, rather than int (Which, for all intents, allows almost unlimited data, at least, in 2012) - however, in the past, 'int' was the usual type for such counts, and while size_t should be used, in practice, a lot of code is still using 'int'.
The maximum size of an MPI_Send will be limited by the maximum amount of memory you can allocate
and most MPI implementations supportsizeof(size_t)
This issue and a number of workarounds (with code) are discussed on https://github.com/jeffhammond/BigMPI. In particular, this project demonstrates how to send more than INT_MAX elements via user-defined datatypes.
Related
I am relatively new to coding and would like to know how do I declare message length (mlen) and ciphertext (clen) length in my c++ code. However, I am not too sure what am I suppose to declare them as. (int? char? unsigned long long?)
The formula that was given to me to include in my code is:
*clen = mlen + CRYPTO_ABYTES
Information that was given to me is:
It operates on a state of 320-bits with message blocks of 64 bits.
UPDATE: Sorry for the bad question, I realized I was given unsigned long long for the message length, it was written in a smaller font that I did not realized it.
If there is no strict requirement regarding how big your type has to be, you should use the native C++ type designed to represent sizes:
std::size_t
The main advantages of std::size_t over types such as unsigned int, unsigned short etc is:
It is capable of holding the maximum possible size of any C++ type, including arrays (since it's defined as the result type of the sizeof operator).
As such, it's often optimized to your architecture (e.g. RAM address space).
It is typically used as the type to represent indices and lengths in new[], std::vector, and large parts of the STL in general. As you are likely going to use such data structures and algorithms, you can avoid useless conversions between different types, making your code more readable, without compiler warnings.
The type std::size_t in your code tells you something about its semantics (indices and sizes; unsigned implied). Other integer types such as unsigned int can be used to represent anything.
For additional information, check out cppreference.com.
On the other hand, if you need to pack your data tightly (e.g. for network protocol), you should use the smallest possible type such as std::uint16_t, std::uint32_t etc.
std::vector::size() returns a size_type which is unsigned and usually the same as size_t, e.g. it is 8 bytes on 64bit platforms.
In constrast, QVector::size() returns an int which is usually 4 bytes even on 64bit platforms, and at that it is signed, which means it can only go half way to 2^32.
Why is that? This seems quite illogical and also technically limiting, and while it is nor very likely that you may ever need more than 2^32 number of elements, the usage of signed int cuts that range in half for no apparent good reason. Perhaps to avoid compiler warnings for people too lazy to declare i as a uint rather than an int who decided that making all containers return a size type that makes no sense is a better solution? The reason could not possibly be that dumb?
This has been discussed several times since Qt 3 at least and the QtCore maintainer expressed that a while ago no change would happen until Qt 7 if it ever does.
When the discussion was going on back then, I thought that someone would bring it up on Stack Overflow sooner or later... and probably on several other forums and Q/A, too. Let us try to demystify the situation.
In general you need to understand that there is no better or worse here as QVector is not a replacement for std::vector. The latter does not do any Copy-On-Write (COW) and that comes with a price. It is meant for a different use case, basically. It is mostly used inside Qt applications and the framework itself, initially for QWidgets in the early times.
size_t has its own issue, too, after all that I will indicate below.
Without me interpreting the maintainer to you, I will just quote Thiago directly to carry the message of the official stance on:
For two reasons:
1) it's signed because we need negative values in several places in the API:
indexOf() returns -1 to indicate a value not found; many of the "from"
parameters can take negative values to indicate counting from the end. So even
if we used 64-bit integers, we'd need the signed version of it. That's the
POSIX ssize_t or the Qt qintptr.
This also avoids sign-change warnings when you implicitly convert unsigneds to
signed:
-1 + size_t_variable => warning
size_t_variable - 1 => no warning
2) it's simply "int" to avoid conversion warnings or ugly code related to the
use of integers larger than int.
io/qfilesystemiterator_unix.cpp
size_t maxPathName = ::pathconf(nativePath.constData(), _PC_NAME_MAX);
if (maxPathName == size_t(-1))
io/qfsfileengine.cpp
if (len < 0 || len != qint64(size_t(len))) {
io/qiodevice.cpp
qint64 QIODevice::bytesToWrite() const
{
return qint64(0);
}
return readSoFar ? readSoFar : qint64(-1);
That was one email from Thiago and then there is another where you can find some detailed answer:
Even today, software that has a core memory of more than 4 GB (or even 2 GB)
is an exception, rather than the rule. Please be careful when looking at the
memory sizes of some process tools, since they do not represent actual memory
usage.
In any case, we're talking here about having one single container addressing
more than 2 GB of memory. Because of the implicitly shared & copy-on-write
nature of the Qt containers, that will probably be highly inefficient. You need
to be very careful when writing such code to avoid triggering COW and thus
doubling or worse your memory usage. Also, the Qt containers do not handle OOM
situations, so if you're anywhere close to your memory limit, Qt containers
are the wrong tool to use.
The largest process I have on my system is qtcreator and it's also the only
one that crosses the 4 GB mark in VSZ (4791 MB). You could argue that it is an
indication that 64-bit containers are required, but you'd be wrong:
Qt Creator does not have any container requiring 64-bit sizes, it simply
needs 64-bit pointers
It is not using 4 GB of memory. That's just VSZ (mapped memory). The total
RAM currently accessible to Creator is merely 348.7 MB.
And it is using more than 4 GB of virtual space because it is a 64-bit
application. The cause-and-effect relationship is the opposite of what you'd
expect. As a proof of this, I checked how much virtual space is consumed by
padding: 800 MB. A 32-bit application would never do that, that's 19.5% of the
addressable space on 4 GB.
(padding is virtual space allocated but not backed by anything; it's only
there so that something else doesn't get mapped to those pages)
Going into this topic even further with Thiago's responses, see this:
Personally, I'm VERY happy that Qt collection sizes are signed. It seems
nuts to me that an integer value potentially used in an expression using
subtraction be unsigned (e.g. size_t).
An integer being unsigned doesn't guarantee that an expression involving
that integer will never be negative. It only guarantees that the result
will be an absolute disaster.
On the other hand, the C and C++ standards define the behaviour of unsigned
overflows and underflows.
Signed integers do not overflow or underflow. I mean, they do because the types
and CPU registers have a limited number of bits, but the standards say they
don't. That means the compiler will always optimise assuming you don't over-
or underflow them.
Example:
for (int i = 1; i >= 1; ++i)
This is optimised to an infinite loop because signed integers do not overflow.
If you change it to unsigned, then the compiler knows that it might overflow
and come back to zero.
Some people didn't like that: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30475
unsigned numbers are values mod 2^n for some n.
Signed numbers are bounded integers.
Using unsigned values as approximations for 'positive integers' runs into the problem that common values are near the edge of the domain where unsigned values behave differently than plain integers.
The advantage is that unsigned approximation reaches higher positive integers, and under/overflow are well defined (if random when looked at as a model of Z).
But really, ptrdiff_t would be better than int.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
size of int, long, etc
Does the size of an int depend on the compiler and/or processor?
I'm not sure if similar questions have been asked before on SO (Atleast, I couldn't find any while searching, so thought of asking myself).
What determines the size of int (and other datatypes) in C. I've read it depends on the machine/operating system/compiler, but haven't come across a clear/detailed enough explanation on things like what overrides the other, etc. Any explanation or pointers will be really helpful.
Ultimately the compiler does, but in order for compiled code to play nicely with system libraries, most compilers match the behavior of the compiler[s] used to build the target system.
So loosely speaking, the size of int is a property of the target hardware and OS (two different OSs on the same hardware may have a different size of int, and the same OS running on two different machines may have a different size of int; there are reasonably common examples of both).
All of this is also constrained by the rules in the C standard. int must be large enough to represent all values between -32767 and 32767, for example.
int is the "natural" size for the platform, and in practice that means one of
the processor's register size, or
a size that's backward compatible with existing code-base (e.g. 32-bit int in Win64).
A compiler vendor is free to choose any size with ≥ 16 value bits, except that (for desktop platforms and higher) a size that doesn't work with OS' API will mean that few if any copies of the compiler are sold. ;-)
The size of C data types is constrained by the C standard, often constraints on the minimum size. The host environment (target machine + OS) may impose further restriction, i.e. constraints on the maximum size. And finally, the compiler is free to choose suitable values between these minimum and maximum values.
Generally, it's considered bad practice to make assumptions about the size of C data types. Besides, it's not necessary, since C will tell you:
the sizeof-operator tells you an object's size in bytes
the macro CHAR_BITS from limits.h tells you the number of bits per byte
Hence, sizeof(foo) * CHAR_BITS tells you the size of type foo, in bits, including padding.
Anything else is just assumptions. Note that the host environment may as well consist of 10.000 Chinese guys with pocket calculators and a huge blackboard, pulling size constraints out of thin air.
SO does not know everything but Wikipedia, almost...
see Integer_(computer_science)
Note (b) says:
"The sizes of short, int, and long in C/C++ are dependent upon the implementation of the language; dependent on data model, even short can be anything from 16-bit to 64-bit. For some common platforms:
On older, 16-bit operating systems, int was 16-bit and long was 32-bit.
On 32-bit Linux, DOS, and Windows, int and long are 32-bits, while long long is 64-bits. This is also true for 64-bit processors running 32-bit programs.
On 64-bit Linux, int is 32-bits, while long and long long are 64-bits."
I have a API like this,
class IoType {
......
StatusType writeBytes(......, size_t& bytesWritten);
StatusType writeObjects(......, size_t& objsWritten);
};
A senior member of the team who I respect seems to have a problem with the type size_t and suggest that I use C99 types. I know it sounds stupid but I always think c99 types like uint32_t and uint64_t look ugly. I do use them but only when it's really necessary, for instance when I need to serialize/deserialize a structure, I do want to be specific about the sizes of my data members.
What are the arguments against using size_t? I know it's not a real type but if I know for sure even a 32-bit integer is enough for me and a size type seems to be appropriate for number of bytes or number of objects, etc.
Use exact-size types like uint32_t whenever you're dealing with serialization of any sort (binary files, networking, etc.). Use size_t whenever you're dealing with the size of an object in memory—that's what it's intended for. All of the functions that deal with object sizes, like malloc, strlen, and the sizeof operator all size_t.
If you use size_t correctly, your program will be maximally portable, and it will not waste time and memory on platforms where it doesn't need to. On 32-bit platforms, a size_t will be 32 bits—if you instead used a uint64_t, you'd waste time and space. Conversely, on 64-bit platforms, a size_t will be 64 bits—if you instead used a uint32_t, your program could behave incorrectly (maybe even crash or open up a security vulnerability) if it ever had to deal with a piece of memory larger than 4 GB.
I can't think of anything wrong in using size_t in contexts where you don't need to serialize values. Also using size_t correctly will increase the code's safety/portability across 32 and 64 bit patforms.
Uhm, it's not a good idea to replace size_t (a maximally portable thing) with a less portable C99 fixed size or minimum size unsigned type.
On the other hand, you can avoid a lot of technical problems (wasted time) by using the signed ptrdiff_t type instead. The standard library’s use of unsigned type is just for historical reasons. It made sense in its day, and even today on 16-bit architectures, but generally it is nothing but trouble & verbosity.
Making that change requires some support, though, in particular a general size function that returns array or container size as ptrdiff_t.
Now, regarding your function signature
StatusType writeBytes(......, size_t& bytesWritten);
This forces the calling code’s choice of type for the bytes written count.
And then, with unsigned type size_t forced, it is easy to introduce a bug, e.g. by checking if that is less or more than some computed quantity.
A grotesque example: std::string("ah").length() < -5 is guaranteed true.
So instead, make that …
Size writeBytes(......);
or, if you do not want to use exceptions,
Size writeBytes(......, StatusType& status );
It is OK to have an enumeration of possible statuses as unsigned type or as whatever, because the only operations on status values will be equality checking and possibly as keys.
Recently, I was challenged in a recent interview with a string manipulation problem and asked to optimize for performance. I had to use an iterator to move back and forth between TCHAR characters (with UNICODE support - 2bytes each).
Not really thinking of the array length, I made a curial mistake with not using size_t but an int to iterate through. I understand it is not compliant and not secure.
int i, size = _tcslen(str);
for(i=0; i<size; i++){
// code here
}
But, the maximum memory we can allocate is limited. And if there is a relation between int and register sizes, it may be safe to use an integer.
E.g.: Without any virtual mapping tools, we can only map 2^register-size bytes. Since TCHAR is 2 bytes long, half of that number. For any system that has int as 32-bits, this is not going to be a problem even if you dont use an unsigned version of int. People with embedded background used to think of int as 16-bits, but memory size will be restricted on such a device. So I wonder if there is a architectural fine-tuning decision between integer and register sizes.
The C++ standard doesn't specify the size of an int. (It says that sizeof(char) == 1, and sizeof(char) <= sizeof(short) <= sizeof(int) <= sizeof(long).
So there doesn't have to be a relation to register size. A fully conforming C++ implementation could give you 256 byte integers on your PC with 32-bit registers. But it'd be inefficient.
So yes, in practice, the size of the int datatype is generally equal to the size of the CPU's general-purpose registers, since that is by far the most efficient option.
If an int was bigger than a register, then simple arithmetic operations would require more than one instruction, which would be costly. If they were smaller than a register, then loading and storing the values of a register would require the program to mask out the unused bits, to avoid overwriting other data. (That is why the int datatype is typically more efficient than short.)
(Some languages simply require an int to be 32-bit, in which case there is obviously no relation to register size --- other than that 32-bit is chosen because it is a common register size)
Going strictly by the standard, there is no guarantee as to how big/small an int is, much less any relation to the register size. Also, some architectures have different sizes of registers (i.e: not all registers on the CPU are the same size) and memory isn't always accessed using just one register (like DOS with its Segment:Offset addressing).
With all that said, however, in most cases int is the same size as the "regular" registers since it's supposed to be the most commonly used basic type and that's what CPUs are optimized to operate on.
AFAIK, there is no direct link between register size and the size of int.
However, since you know for which platform you're compiling the application, you can define your own type alias with the sizes you need:
Example
#ifdef WIN32 // Types for Win32 target
#define Int16 short
#define Int32 int
// .. etc.
#elif defined // for another target
Then, use the declared aliases.
I am not totally aware, if I understand this correct, since some different problems (memory sizes, allocation, register sizes, performance?) are mixed here.
What I could say is (just taking the headline), that on most actual processors for maximum speed you should use integers that match register size. The reason is, that when using smaller integers, you have the advantage of needing less memory, but for example on the x86 architecture, an additional command for conversion is needed. Also on Intel you have the problem, that accesses to unaligned (mostly on register-sized boundaries) memory will give some penality. Off course, on todays processors things are even more complex, since the CPUs are able to process commands in parallel. So you end up fine tuning for some architecture.
So the best guess -- without knowing the architectore -- speeedwise is, to use register sized ints, as long you can afford the memory.
I don't have a copy of the standard, but my old copy of The C Programming Language says (section 2.2) int refers to "an integer, typically reflecting the natural size of integers on the host machine." My copy of The C++ Programming Language says (section 4.6) "the int type is supposed to be chosen to be the most suitable for holding and manipulating integers on a given computer."
You're not the only person to say "I'll admit that this is technically a flaw, but it's not really exploitable."
There are different kinds of registers with different sizes. What's important are the address registers, not the general purpose ones. If the machine is 64-bit, then the address registers (or some combination of them) must be 64-bits, even if the general-purpose registers are 32-bit. In this case, the compiler may have to do some extra work to actually compute 64-bit addresses using multiple general purpose registers.
If you don't think that hardware manufacturers ever make odd design choices for their registers, then you probably never had to deal with the original 8086 "real mode" addressing.