I am using rapidjson on an Arm device and get strange behaviour, when running this code.
#include <document.h>
using namespace std;
int main()
{
const char json []="[{\"Type\":\"float\",\"val_param\" : 12.025 }]";
rapidjson::Document d;
if( d.Parse<0>( json ).HasParseError() ) {
//ErrorCase
}else{
rapidjson:: Value& val_param = d[0]["val_param"];
double tmp_double1 = val_param.GetDouble();
cout << tmp_double1 <<endl; // -9.2559641157289301e+61 instead of 12.025
}
return 0;
}
Before down voting this question. What else information do you need? I really don't know how to isolate this fault. If it occurs because of the embedded device, or rapidjson. And how to solve it.
========================== UPDATE ========================================
What is the device? http://www.keith-koep.com/de/produkte/produkte-trizeps/trizeps-iv-m-eigenschaften/
Does it have a hardware FPU? It is ARMv5 so I don't think so.
What compiler and libraries are you using (version numbers/specific builds)?
What options are you passing to the compiler and linker?
arm-linux-gnueabi-g++ -march=armv5te -marm -mthumb-interwork --sysroot=/usr/local/oecore-x86_64/sysroots/armv5te-linux-gnueabi
This looks like it might be an undefined-behaviour-type bug in RapidJSON.
Since you're targeting ARMv5, you're probably using a software floating-point library using the legacy ARM FPA format (as opposed to the later VFP, which uses IEEE754 format). Crucially, the FPA stores things in a weird middle-endian format, where 64-bit doubles are stored as two little-endian words, but most-significant word first.
(Yes, big-endian ARM is a whole other complicated issue, but I'm deliberately ignoring it here since I don't see an armeb-* triplet or the -mbig-endian option anywhere)
Consider 12.025 as an IEEE754 double:
64-bit value: 0x40280ccccccccccd.
little-endian byte order: cd cc cc cc cc 0c 28 40
as little-endian words: 0xcccccccd 0x40280ccc
Now in FPA format that would be:
as little-endian words: 0x40280ccc 0xcccccccd
byte order: cc 0c 28 40 cd cc cc cc
Trying to interpret that as a pure little-endian 64-bit value yields 0xcccccccd40280ccc, which just so happens to be the IEEE754 representation of -9.255965e+61. Fancy that!
From a quick look around the code, it may strictly be more of an incompatibility than a bug, since RapidJSON does seem to explicitly assume IEEE754 format for floating-point values. To its credit, even though the parsing code looks pretty hairy, I do see unions rather than type-punning pointers. However, even if it's not relying on undefined behaviour, it's still relying on implementation-defined behaviour (the format of floating-point types), and unfortunately this compiler's implementation doesn't match that expectation.
Related
We have some legacy code that at one point in time long data types were refactored to int data types. During this refactor a number of printf / sprintf format statements were left incorrect as %ld instead of changed to %d. For example:
int iExample = 32;
char buf[200];
sprintf(buf, "Example: %ld", iExample);
This code is compiled on both GCC and VS2012 compilers. We use Coverity for static code analysis and code like in the example was flagged as a 'Printf arg type mismatch' with a Medium level of severity, CWE-686: Function Call With Incorrect Argument Type I can see this would be definitely a problem had the format string been that of an signed (%d) with an unsigned int type or something along these lines.
I am aware that the '_s' versions of sprintf etc are more secure, and that the above code can also be refactored to use std::stringstream etc. It is legacy code however...
I agree that the above code really should be using %d at the very least or refactored to use something like std::stringstream instead.
Out of curiosity is there any situation where the above code will generate incorrect results? As this legacy code has been around for quite some time and appears to be working fine.
UPDATED
Removed the usage of the word STL and just changed it to be std::stringstream.
As far as the standard is concerned, the behavior is undefined, meaning that the standard says exactly nothing about what will happen.
In practice, if int and long have the same size and representation, it will very likely "work", i.e., behave as if the correct format string has been used. (It's common for both int and long to be 32 bits on 32-bit systems).
If long is wider than int, it could still work "correctly". For example, the calling convention might be such that both types are passed in the same registers, or that both are pushed onto the stack as machine "words" of the same size.
Or it could fail in arbitrarily bad ways. If int is 32 bits and long is 64 bits, the code in printf that tries to read a long object might get a 64-bit object consisting of the 32 bits of the actual int that was passed combined with 32 bits of garbage. Or the extra 32 bits might consistently be zero, but with the 32 significant bits at the wrong end of the 64-bit object. It's also conceivable that fetching 64 bits when only 32 were passed could cause problems with other arguments; you might get the correct value for iExample, but following arguments might be fetched from the wrong stack offset.
My advice: The code should be fixed to use the correct format strings (and you have the tools to detect the problematic calls), but also do some testing (on all the C implementations you care about) to see whether it causes any visible symptoms in practice. The results of the testing should be used only to determine the priority of fixing the problems, not to decide whether to fix them or not. If the code visibly fails now, you should fix it now. If it doesn't, you can get away with waiting until later (presumably you have other things to work on).
It's undefined and depends on the implementation. On implementations where int and long have the same size, it will likely work as expected. But just try it on any system with 32-bit int and 64-bit long, especially if your integer is not the last format argument, and you're likely to get problems where printf reads 64 bits where only 32 were provided, the rest quite possibly garbage, and possibly, depending on alignment, the following arguments also cannot get accessed correctly.
Immediately warn you that this is a difficult task.
There is a test. The test was the result of parsing a large problem to a bug in which we encountered at work. Construction __ attribute__((noinline)) prohibits the compiler to do the substitution function (for optimizations to something there not imploded). This is the easiest way to optimize guaranteed not to kill an interesting situation.
#include <stdio.h>
double d = 5436277361664796672.000000;
long long ll = 5436277361664796253LL;
int __attribute__((noinline))
func1 (void)
{
double d1 = (double)ll;
if (d > d1)
return 1;
else
return 0;
}
int __attribute__((noinline))
func2 (void)
{
if (d > (double)ll)
return 1;
else
return 0;
}
int
main (void)
{
printf ("%d %d\n", func1(), func2());
return 0;
}
I ran this test on intel and sparc. Gcc used in a mode with optimizations and without optimizations. Obtained the following results:
sparc: "gcc" printed "0 0"
sparc: "gcc -O2" printed "0 0"
intel: "gcc" printed "0 1"
intel: "gcc -O2" printed "1 1"
What is the cause differences? Anyway in the analysis situation would be useful to be able to repeat it all myself, but, of course, almost no one has the possibility to run this code on sparc. Instead sparc can try to run under Windows using microsoft or borland C compiler. I do not know what they will be given the results, but in any case something does not match with anything (because we see three different results)
Edit 1
_attribute_ ((noinline)) - an extension of the compiler gcc (forgot to write about it). Therefore VisualStudio can not compile it.
I note that the declaration of the double constant has 19 significant figures which is more precision than can be represented by a IEEE double (which allows 15 to 17 significant figures). So d cannot hold 5436277361664796672.000000 exactly.
The two constant definition strings become different at the 16th digit, so you are in the region where the inaccuracies in the double are of the same magnitude as the difference between these two numbers. Hence the comparison cannot be relied upon.
I do not know if the C++ standard specifies what happens when an over-precise string is converted to a double, but I would not be surprised if the exact result was either undefined or implementation-dependent.
Seems solved the problem. In general, all written correctly. But actually works correctly sparc version. Because standard to convert int64-> float64 must be a loss of precision. And in the code when you convert (for intel) int64-> float80 loss occurs. Ie intel-based code works with higher accuracy , but it is in contradiction with the standard.
Perhaps it is some sort of agreement for the platform Intel, which is permissible by default to work this way. Surely there are some options on which the code runs in strict accordance with the standard (but becomes slower)
I have Qt 4.4.3 built for ARMv5TE. I try to convert a double to a QString:
#include <QtCore/QtCore>
#include <cmath>
int main(int argc, char** argv)
{
const double pi = M_PI;
qDebug() << "Pi is : " << pi << "\n but pi is : " << QString::number(pi, 'f', 6);
printf("printf: %f\n",pi);
return 0;
}
but get strange output:
Pi is : 8.6192e+97
but pi is : "86191995128153827662389718947289094511677209256133209964237318700300913082475855805240843511529472.0000000000000000"
printf: 3.141593
How do I get the proper string?
This looks to be a sort of endianess issue, but not your plain-vanilla big-endian vs little-endian problem. ARM sometimes uses an unusual byte ordering for double. From "Handbook of Floating-Point Arithmetic" by Jean-Michel Muller, et al.:
... the double-precision number that is closest to
-7.0868766365730135 x 10^-268 is encoded by the sequence of bytes
11 22 33 44 55 66 77 88 in memory (from the lowest to the highest
one) on x86 and Linux/IA-64 platforms (they are said to be
little-endian) and by 88 77 66 55 44 33 22 11 on most PowerPC platforms (they are said to be big-endian). Some architectures, such
as IA-64, ARM, and PowerPC are said to be bi-endian. i.e., they may
be either little-endian or big-endian depending on their
configuration.
There exists an exception: some ARM-based platforms. ARM processors
have traditionally used the floating-point accelerator (FPA)
architecture, where the double-precision numbers are decomposed into
two 32-bit words in the big-endian order and stored according to the
endianess of the machine, i.e., little-endian in general, which means
that the above number is encoded by the sequence 55 66 77 88 11 22 33
44. ARM has recently introduced a new architecture for floating-point
arithmetic: vector floating-point (VFP), where the words are stored
in the processor's native byte order.
When looked at in a big-endian byte order, M_PI will have a representation that looks like:
0x400921fb54442d18
The large number approximated by 8.6192e+97 wil have a representation that looks like:
0x54442d18400921fb
If you look closely, the two 32-bit words are swapped, but the byte order within the 32-bit words is the same. So apparently, the ARM 'traditional' double point format seems to be confusing the Qt library (or the Qt library is misconfigured).
I'm not sure if the processor is using the traditional format and Qt expects it to be in VFP format, or if things are the other way around. But it seems to be one of those two situations.
I'm also not sure exactly how to fix the problem - I'd guess there's some option for building Qt to handle this correctly.
the following snippet will at least tell you what format for double the compiler is using, which may help you narrow down what needs to change in Qt:
unsigned char* b;
unsigned char* e;
double x = -7.0868766365730135e-268;
b = (unsigned char*) &x;
e = b + sizeof(x);
for (; b != e; ++b) {
printf( "%02x ", *b);
}
puts("");
A plain little-endian machine will display:
11 22 33 44 55 66 77 88
Update with a bit more analysis:
At the moment, I'm unable to perform any real debugging of this (I don't even have access to my workstation at the moment), but by looking at the Qt source available on http://qt.gitorious.org here's additional analysis:
It looks like Qt calls in to the QLocalePrivate::doubleToString() function in qlocale.cpp to convert a double to an alphanumeric form.
If Qt is compiled with QT_QLOCALE_USES_FCVT defined, then QLocalePrivate::doubleToString() will use the platform's fcvt() function to perform the conversion. If QT_QLOCALE_USES_FCVT is not defined, then QLocalePrivate::doubleToString() ends up calling _qdtoa() to perform the conversion. That function examines the various fields of the double directly and appears to assume that the double is in a strict big-endian or little-endian form (for example, using the getWord0() and getWord1() functions to get the low and high word of the double respectively).
See http://qt.gitorious.org/qt/qt/blobs/HEAD/src/corelib/tools/qlocale.cpp and http://qt.gitorious.org/qt/qt/blobs/HEAD/src/corelib/tools/qlocale_tools.cpp or your own copy of the files for details.
Assuming that your platform is using the traditional ARM FPA representation for double (where the 32-bit halves of the double are stored in big-endian order regardless of whether the overall system being little-endian), I think you'll need to build Qt with the QT_QLOCALE_USES_FCVT defined. I believe that all you'll need to do is pass the -DQT_QLOCALE_USES_FCVT option to the configure script when building Qt.
The same code produces proper output on an x86 machine (running Windows XP) with Qt 4.7.0.
I see the following possibilities for the source of the problem:
Some bug which is maybe fixed in a newer version of Qt
Something went wrong when compiling for ARM
I found this forum post on a similar problem which supposes it could be a big/little-endian conversion problem.
I can't tell how to fix this as I am not experienced with ARM at all but maybe this information helps you anyway.
I'm trying to get fixed sized floats and ints across all windows computers. As in, I have a program, I compile it and distribute the executable. I want the data types to be of constant bit size/ordering across all windows computers.
My first question is whether windows types defined at http://msdn.microsoft.com/en-us/library/aa383751(v=vs.85).aspx have fixed sizes across all windows computers (let's say running the same OS- Windows 7).
Basically, I'd like to transfer data contained in a struct over a network and I want to avoid having to put it all in a string or encode it into a portable binary form.
Edit: What about floats??
You should use the types from
#include <cstdint>
Like uint64_t, int16_t, int8_t etc.
On the ordering: I'm pretty sure Windows only runs on Big-Endian hardware. Regardless, if platform portability is a concern, why don't you use a proper Serialization library and avoid nasty surprises?
Boost Serialization
Protobuf (Google's data interchange formatting library)
Floating Point
While there is no C++ standard defining the sizes for formats of floating point values Microsoft has specified that they consistently use 4-byte and 8-byte IEEE floating point format for float and double types respectively.
Integrals
As for integral types, Microsoft does have compiler-specific defines for fixed length variables. Some non-Microsoft compilers define fixed-size integral types using the cstdint header. Neither of these are based on official standards.
Serialization
This will be terribly unportable and will most likely turn into a maintenance nightmare as your structs get more complicated. What you are effectively doing is defining an error-prone binary serialization format that must be complied with through convention. This problem has already been solved more effectively.
I would highly recommend using a serialization format like protocol buffers or maybe boost::serialization for communication between machines. If your data is hitting the wire, then the performance of serialization/deserialization is going to be an incredibly small fraction of transmission time.
Alignment
Another serious issue that you'll have is how the struct is packed in memory. Your struct will most likely be laid-out in memory differently in a 32-bit process than it is in a 64-bit process.
In a 32-bit process, your struct members will be aligned on word boundaries, and on doubleword boundaries for 64-bit.
For example, this program outputs 20 on 32-bit and 24 on 64-bit platforms:
#include <iostream>
#include <cstdint>
struct mystruct {
uint32_t y;
double z;
uint8_t c;
float v;
} mystruct_t;
int main() {
std::cout << sizeof(mystruct_t);
}
If your compiler is recent enough to give you the standard <stdint.h> header required by ISO C99 (or <cstdint> in recent C++), you'll better use it (and then, it would make your code portable, w.r.t. this particular issue, even on non Windows systems). So use types like int32_t or int64_t etc.
If serialization across several platforms is a concern, consider using portable binary formats like XDR, ASN1, perhaps using the s11n library, or textual formats like JSON or YAML
How can I test or check C++ code for endian-independence? It's already implemented, I would just like to verify that it works on both little- and big-endian platforms.
I could write unit tests and run them on the target platforms, but I don't have the hardware. Perhaps emulators?
Are there compile time checks that can be done?
If you have access to an x86-based Mac then you can take advantage of the fact that Mac OS X has PowerPC emulation built in as well as developer tool support for both x86 (little endian) and PowerPC (big endian). This enables you to compile and run a big and little endian executable on the same platform, e.g.
$ gcc -arch i386 foo.c -o foo_x86 # build little endian x86 executable
$ gcc -arch ppc foo.c -o foo_ppc # build big endian PowerPC executable
Having built both big endian and little endian executables you can then run whatever unit tests you have available on both, which will catch some classes of endianness-related problems, and you can also compare any data generated by the executables (files, network packets, whatever) - this should obviously match.
You can set up an execution environment in the opposite endianness using qemu. For example if you have access to little-endian amd64 or i386 hardware, you can set up qemu to emulate a PowerPC Linux platform, run your code there.
I read a story that used Flint (Flexible Lint) to diagnose this kind of errors.
Don't know the specifics anymore, but let me google the story back for you:
http://www.datacenterworks.com/stories/flint.html
An Example: Diagnosing Endianness Errors
On a recent engagement, we were porting code from an old Sequent to a SPARC, and after the specific pointer issues we discussed in the Story of Thud and Blunder, we needed to look for other null pointer issues and also endian-ness errors.
I would suggest adapting a coding technique that avoids the problem all together.
First, you have to understand in which situation an endianess problem occurs. Then either find an endianess-agnostic way to write this, or isolate the code.
For example, a typical problem where endianess issues can occur is when you use memory accesses or unions to pick out parts of a larger value. Concretely, avoid:
long x;
...
char second_byte = *(((char *)&x) + 1);
Instead, write:
long x;
...
char second_byte = (char)(x >> 8)
Concatenation, this is one of my favorites, as many people tend to think that you can only do this using strange tricks. Don't do this:
union uu
{
long x;
unsigned short s[2];
};
union uu u;
u.s[0] = low;
u.s[1] = high;
long res = u.x;
Instead write:
long res = (((unsigned long)high) << 16) | low
I could write unit tests and run them on the target platforms, but I don't have the hardware.
You can setup your design so that unit tests are easy to run independent of actually having hardware. You can do this using dependency injection. I can abstract away things like hardware interfaces by providing a base interface class that the code I'm testing talks to.
class IHw
{
public:
virtual void SendMsg1(const char* msg, size_t size) = 0;
virtual void RcvMsg2(Msg2Callback* callback) = 0;
...
};
Then I can have the concrete implementation that actually talks to hardware:
class CHw : public IHw
{
public:
void SendMsg1(const char* msg, size_t size);
void RcvMsg2(Msg2Callback* callback);
};
And I can make a test stub version:
class CTestHw : public IHw
{
public:
void SendMsg1(const char* msg, size_t);
void RcvMsg2(Msg2Callback* callback);
};
Then my real code can us the concrete Hw, but I can simulate it in test code with CTestHw.
class CSomeClassThatUsesHw
{
public:
void MyCallback(const char* msg, size_t size)
{
// process msg 2
}
void DoSomethingToHw()
{
hw->SendMsg1();
hw->RcvMsg2(&MyCallback);
}
private:
IHw* hw;
}
IMO, the only answer that comes close to being correct is Martin's. There are no endianness concerns to address if you aren't communicating with other applications in binary or reading/writing binary files. What happens in a little endian machine stays in a little endian machine if all of the persistent data are in the form of a stream of characters (e.g. packets are ASCII, input files are ASCII, output files are ASCII).
I'm making this an answer rather than a comment to Martin's answer because I am proposing you consider doing something different from what Martin proposed. Given that the dominant machine architecture is little endian while network order is big endian, many advantages arise if you can avoid byte swapping altogether. The solution is to make your application able to deal with wrong-endian inputs. Make the communications protocol start with some kind of machine identity packet. With this info at hand, your program can know whether it has to byte swap subsequent incoming packets or leave them as-is. The same concept applies if the header of your binary files has some indicator that lets you determine the endianness of those files. With this kind of architecture at hand, your application(s) can write in native format and can know how to deal with inputs that are not in native format.
Well, almost. There are other problems with binary exchange / binary files. One such problem is floating point data. The IEEE floating point standard doesn't say anything about how floating point data are stored. It says nothing regarding byte order, nothing about whether the significand comes before or after the exponent, nothing about the storage bit order of the as-stored exponent and significand. This means you can have two different machines of the same endianness that both follow the IEEE standard and you can still have problems communicating floating point data as binary.
Another problem, not so prevalent today, is that endianness is not binary. There are other options than big and little. Fortunately, the days of computers that stored things in 2143 order (as opposed to 1234 or 4321 order) are pretty much behind us, unless you deal with embedded systems.
Bottom line:
If you are dealing with a near-homogenous set of computers, with only one or two oddballs (but not too odd), you might want to think of avoiding network order. If the domain has machines of multiple architectures, some of them very odd, you might have to resort to the lingua franca of network order. (But do beware that this lingua franca does not completely resolve the floating point problem.)
I personally use Travis to test my software hosted on github and it supports running on multiple architectures [1], including s390x which is big endian.
I just had to add this to my .travis.yml:
arch:
- amd64
- s390x # Big endian arch
It's probably not the only CI proposing this, but that's the one I was already using. I run both unit tests and integrated test on both systems which gives me some reasonable confidence that it works fine no matter the endianness.
It's no silver bullet though, I'd like to have an easy way to test it manually too just to ensure there's no hidden error (e.g I'm using SDL, colors could be wrong. I'm using screenshot to validate the output but the code for taking screenshot could have errors compensating the display problem, so the tests could pass with the display being wrong).
[1] https://blog.travis-ci.com/2019-11-12-multi-cpu-architecture-ibm-power-ibm-z