Related
I'm trying to understand what's going on inside with this (weird?) g++ behavior.
#include <iostream>
using namespace std;
int& f(void) {
int a = 9;
int& b = a;
return a;
}
int main(void) {
int& l = f();
cout << ++l << '\n' << l << '\n';
}
When returning a itself and bind it to l, I get a warning (reference to local variable) and a seg.fault if I acess it from l, but when returning b itself not only do I not get a seg.fault but I can access it once from l (UB I am guessing) before the value of l randomly changes. But what exactly happens here?
Aren't the two returns identical? Does g++ automatically mark a's area as unusable after the return, hence the seg.fault while for some reason allowing b to live longer?
Your main question is why gcc doesn't issue a warning in one of the alternatives. Both alternatives are undefined behavior and the only difference is that in one case the compiler can detect it and warn you about it.
The C++ standard does not require a diagnostic for undefined behavior. Any diagnostic to that effect, from your compiler, is just an extra bonus; and although modern C++ compilers are very smart, they can't always figure out that the compiled code will result in demons flying out of your nose.
P.S. gcc 10.2 does issue a warning with the -O3 option, for the return b; alternative. With -Wall only, gcc also issues a 2nd warning for undefined behavior, you can discover what it is by yourself.
No, C++ compilers do not mark areas as unusable.
Segfaults are just one of many ways undefined behavior can exhibit itself. It is in fact one of the friendlier ways, as it makes you notice it early.
You are in charge of lifetime. If you get it wrong, the result is undefined behavior. Not an exception. Not a segfault. Literally anything.
One possibly symptom is "it appears to work". Another is segfault. Others include literal time travel (where UB later in the program makes ealier code behave differently), your computer hard drive being rendered unusable, someone getting your credit card information, your browser history being emailed to your contact list, etc.
Some compilers, in debug mode, mark deallocated memory with a bit pattern to aid debugging.
But what exactly happens here?
Undefined behaviour.
Aren't the two returns identical?
The source code is clearly syntactically different. The programs don't have same semantic meaning because neither program has any semantic meaning because the meaning of the both programs is undefined. As such, the behaviour of the programs is not guaranteed to be the same.
Does g++ automatically mark a's area as unusable after the return
Perhaps. I wouldn't assume this to be the case based on that one observation but this may be true. See the source code of GCC to confirm.
A huge number of operations in C++ result in undefined behavior, where the spec is completely mute about what the program's behavior ought to be and allows for anything to happen. Because of this, there are all sorts of cases where people have code that compiles in debug but not release mode, or that works until a seemingly unrelated change is made, or that works on one machine but not another, etc.
My question is whether there is a utility that looks at the execution of C++ code and flags all instances where the program invokes undefined behavior. While it's nice that we have tools like valgrind and checked STL implementations, these aren't as strong as what I'm thinking about - valgrind can have false negatives if you trash memory that you still have allocated, for example, and checked STL implementations won't catch deleting through a base class pointer.
Does this tool exist? Or would it even be useful to have it lying around at all?
EDIT: I am aware that in general it is undecidable to statically check whether a C++ program may ever execute something that has undefined behavior. However, it is possible to determine whether a specific execution of a C++ produced undefined behavior. One way to do this would be to make a C++ interpreter that steps through the code according to the definitions set out in the spec, at each point determining whether or not the code has undefined behavior. This won't detect undefined behavior that doesn't occur on a particular program execution, but it will find any undefined behavior that actually manifests itself in the program. This is related to how it is Turing-recognizable to determine if a TM accepts some input, even if it's still undecidable in general.
Thanks!
This is a great question, but let me give an idea for why I think it might be impossible (or at least very hard) in general.
Presumably, such an implementation would almost be a C++ interpreter, or at least a compiler for something more like Lisp or Java. It would need to keep extra data for each pointer to ensure you did not perform arithmetic outside of an array or dereference something that was already freed or whatever.
Now, consider the following code:
int *p = new int;
delete p;
int *q = new int;
if (p == q)
*p = 17;
Is the *p = 17 undefined behavior? On the one hand, it dereferences p after it has been freed. On the other hand, dereferencing q is fine and p == q...
But that is not really the point. The point is that whether the if evaluates to true at all depends on the details of the heap implementation, which can vary from implementation to implementation. So replace *p = 17 by some actual undefined behavior, and you have a program that might very well blow up on a normal compiler but run fine on your hypothetical "UB detector". (A typical C++ implementation will use a LIFO free list, so the pointers have a good chance of being equal. A hypothetical "UB detector" might work more like a garbage collected language in order to detect use-after-free problems.)
Put another way, the existence of merely implementation-defined behavior makes it impossible to write a "UB detector" that works for all programs, I suspect.
That said, a project to create an "uber-strict C++ compiler" would be very interesting. Let me know if you want to start one. :-)
John Regehr in Finding Undefined Behavior Bugs by Finding Dead Code points out a tool called STACK and I quote from the site (emphasis mine):
Optimization-unstable code (unstable code for short) is an emerging class of software bugs: code that is unexpectedly eliminated by compiler optimizations due to undefined behavior in the program. Unstable code is present in many systems, including the Linux kernel and the Postgres database server. The consequences of unstable code range from incorrect functionality to missing security checks.
STACK is a static checker that detects unstable code in C/C++ programs. Applying STACK to widely used systems has uncovered 160 new bugs that have been confirmed and fixed by developers.
Also in C++11 for the case of constexpr variables and functions undefined behavior should be caught at compile time.
We also have gcc ubsan:
GCC recently (version 4.9) gained Undefined Behavior Sanitizer
(ubsan), a run-time checker for the C and C++ languages. In order to
check your program with ubsan, compile and link the program with
-fsanitize=undefined option. Such instrumented binaries have to be executed; if ubsan detects any problem, it outputs a “runtime error:”
message, and in most cases continues executing the program.
and Clang Static Analyzer which includes many checks for undefined behavior. For example clangs -fsanitize checks which includes -fsanitize=undefined:
-fsanitize=undefined: Fast and compatible undefined behavior checker. Enables the undefined behavior checks that have small runtime cost and
no impact on address space layout or ABI. This includes all of the
checks listed below other than unsigned-integer-overflow.
and for C we can look at his article It’s Time to Get Serious About Exploiting Undefined Behavior which says:
[..]I confess to not personally having the gumption necessary for cramming GCC or LLVM through the best available dynamic undefined behavior checkers: KCC and Frama-C.[...]
Here is a link to kcc and I quote:
[...]If you try to run a program that is undefined (or one for which we are missing semantics), the program will get stuck. The message should tell you where it got stuck and may give a hint as to why. If you want help deciphering the output, or help understanding why the program is undefined, please send your .kdump file to us.[...]
and here are a link to Frama-C, an article where the first use of Frama-C as a C interpreter is described and an addendum to the article.
Using g++
-Wall -Werror -pedantic-error
(preferably with an appropriate -std argument as well) will pick up quite a few case of U.B.
Things that -Wall gets you include:
-pedantic
Issue all the warnings demanded by strict ISO C and ISO C++; reject
all programs that use forbidden extensions, and some other programs
that do not follow ISO C and ISO C++. For ISO C, follows the
version of the ISO C standard specified by any -std option used.
-Winit-self (C, C++, Objective-C and Objective-C++ only)
Warn about uninitialized variables which are initialized with
themselves. Note this option can only be used with the
-Wuninitialized option, which in turn only works with -O1 and
above.
-Wuninitialized
Warn if an automatic variable is used without first being
initialized or if a variable may be clobbered by a "setjmp" call.
and various disallowed things you can do with specifiers to printf and scanf family functions.
Clang has a suite of sanitizers that catch various forms of undefined behavior. Their eventual goal is to be able to catch all C++ core language undefined behavior, but checks for a few tricky forms of undefined behavior are missing right now.
For a decent set of sanitizers, try:
clang++ -fsanitize=undefined,address
-fsanitize=address checks for use of bad pointers (not pointing to valid memory), and -fsanitize=undefined enables a set of lightweight UB checks (integer overflow, bad shifts, misaligned pointers, ...).
-fsanitize=memory (for detecting uninitialized memory reads) and -fsanitize=thread (for detecting data races) are also useful, but neither of these can be combined with -fsanitize=address nor with each other because all three have an invasive impact on the program's address space.
You might want to read about SAFECode.
This is a research project from the University of Illinois, the goal is stated on the front page (linked above):
The purpose of the SAFECode project is to enable program safety without garbage collection and with minimal run-time checks using static analysis when possible and run-time checks when necessary. SAFECode defines a code representation with minimal semantic restrictions designed to enable static enforcement of safety, using aggressive compiler techniques developed in this project.
What is really interesting to me is the elimination of the runtime checks whenever the program can be proved to be correct statically, for example:
int array[N];
for (i = 0; i != N; ++i) { array[i] = 0; }
Should not incur any more overhead than the regular version.
In a lighter fashion, Clang has some guarantees about undefined behavior too as far as I recall, but I cannot get my hands on it...
The clang compiler can detect some undefined behaviors and warn against them. Probably not as complete as you want, but it's definitely a good start.
Unfortunately I'm not aware of any such tool. Typically UB is defined as such precisely because it would be hard or impossible for a compiler to diagnose it in all cases.
In fact your best tool is probably compiler warnings: They often warn about UB type items (for example, non-virtual destructor in base classes, abusing the strict-aliasing rules, etc).
Code review can also help catch cases where UB is relied upon.
Then you have to rely on valgrind to capture the remaining cases.
Just as a side observation, according to the theory of computability, you cannot have a program that detects all possible undefined behaviours.
You can only have tools that use heuristics and detect some particular cases that follow certain patterns. Or you can in certain cases prove that a program behaves as you want. But you cannot detect undefined behaviour in general.
Edit
If a program does not terminate (hangs, loops forever) on a given input, then its output is undefined.
If you agree on this definition, then determining whether a program terminates is the well-known "Halting Problem", which has been proven to be undecidable, i.e. there exists no program (Turing Machine, C program, C++ program, Pascal program, in whatever language) that can solve this problem in general.
Simply put: there exists no program P that can take as input any program Q and input data I and print as output TRUE if Q(I) terminates, or else print FALSE if Q(I) does not terminate.
For more information you can look at http://en.wikipedia.org/wiki/Halting_problem.
Undefined behaviour is undefined. The best you can do is conform to the standard pedantically, as others have suggested, however, you can not test for what is undefined, because you don't know what it is. If you knew what it was and standards specified it, it would not be undefined.
However, if you for some reason, do actually rely on what the standard says is undefined, and it results in a particular result, then you may choose to define it, and write some unit tests to confirm that for your particular build, it is defined. It is much better, however, to simply avoid undefined behaviour whenever possible.
Take a look at PCLint its pretty decent at detecting a lot of bad things in C++.
Here's a subset of what it catches
Is it possible in either gcc/g++ or ms c++ to set a flag which only allows defined behavior? so something like the below gives me a warning or preferably an error
func(a++, a, ++a)
Undefined and unspecified behavior is designated so in the standard specifically because it could cause undue burden on the implementation to diagnose all examples of it (or it would be impossible to determine).
It's expected that the programmer take care to avoid those areas that are undefined.
For your stated example it should be fairly obvious to a programmer to just not write that code in the first place.
That being said, g++ -Wall will catch some bad code, such as missing return in a non-void function to give one example.
EDIT: #sehe also points out -Wsequence-point which will catch this precise code construct, although there should be a sequence point between evaluation of each argument (the order in which arguments is evaluated is unspecified however).
GNU C++ has the following
-Wsequence-point
Warn about code that may have undefined semantics because of violations of sequence point rules in the C and C++ standards.
This will correctly flag the invocation you showed
-Wstrict-overflow
-Wstrict-overflow
-fstrict-aliasing
-fstrict-overflow
HTH
No. For example, consider the following:
int badfunc(int &a, int &b) {
return func(a++, b++);
}
This has undefined behavior if a and b have the same referand. In general the compiler cannot know what arguments will be passed to a function, so it can't reliably catch this case of undefined behavior. Therefore it can't catch all undefined behavior.
Compiler warnings serve to identify some instances of undefined behavior, but never all.
In theory you could write a C++ implementation that does vast numbers of checks at runtime to ensure that undefined behavior is always identified and dealt with in ways defined by that implementation. It still wouldn't tell you at compile time (see: halting problem), and in practice you'd probably be better off with C#, which was designed to make the necessary runtime checks reasonably efficient...
Even if you built that magical checking C++ implementation, it still might not tell you what you really want to know, which is whether your code is correct. Sometimes (hang on to your seats), it is implementation-defined whether or not behavior is undefined. For a simple example, tolower((char)-1); has defined behavior[*] if the char type is unsigned, but undefined behavior if the char type is signed.
So, unless your magical checking implementation makes all the same implementation choices as the "real" implementation that you want your code to run on, it won't tell you whether the code has defined behavior for the set of implementation choices made in the "real" implementation, only whether it has defined behavior for the implementation choices made in the magical checking implementation.
To know that your code is correct and portable, you need to know (for starters) that it produces no undefined behavior for any set of implementation choices. And, for that matter, for any input, not just the inputs used in your tests. You might think that this is a big deficiency in C++ compared to languages with no undefined behavior. Certainly it is inconvenient at times, and affects how you go about sandboxing programs for security. In practice, though, for you to consider your code correct you don't just need it to have defined behavior, you need the behavior to match the specification document. That's a much bigger problem, and in practice it isn't very much harder to write a bug in (say) Java or Python than it is in C++. I've written countless bugs in all three, and knowing that in Java or Python the behavior was defined but wrong didn't help me all that much.
[*] Well, the result is still implementation-defined, it depends on the execution character set, but the implementation has to return the correct result. If char is signed it's allowed to crash.
This gave me a good laugh. Sorry about that, didn't mean any offense; it's a good question.
There is no compiler on the planet that only allows 100% defined behavior. It's the undefined nature of things that makes it so hard. There are a lot of cases taken up in the standard, but they're often too vague to efficiently implement in a compiler.
I know Clang developers showed some interest to adding that functionality, but they haven't started as far as I know.
The only thing you can do now and in the near/far future is cranking up the warning level and strictness of your compiler. Sadly, even in recent versions, MSVC is a pain in that regard. On warning level 4 and up, it spits some stupid warnings that have nothing to do with code correctness, and you often have to jump through hoops to get them to go away.
GCC is better at that in my personal experience. I personnally use these options, ensuring the strictest checks (I currently know of)
-std=c++0x -pedantic -Wextra -Weffc++ -Wmissing-include-dirs -Wstrict-aliasing
I of course ensure zero warnings, if you want to enforce even that, just add -Werror to the line above and any error will error out. It's mostly the std and pedantic options that enforce Standard behavior, Wextra catches some off-chance semi-errors.
And of course, compile your code with different compilers if possible (and make sure they are correctly diagnosing the problem by asking here, where people know what the Standard says/means).
While I agree with Mark's answer, I just thought I should let you know...
#include <stdio.h>
int func(int a, int b, int c)
{
return a + b + c;
}
int main()
{
int a=0;
printf("%d\n", func(a++, a, ++a)); /* line 11 */
return 0;
}
When compiling the code above with gcc -Wall, I get the following warnings:
test.c:11: warning: operation on ‘a’ may be undefined
test.c:11: warning: operation on ‘a’ may be undefined
because of a++ and ++a, I suppose. So to some degree, it's been implemented. But obviously we can't expect all undefined behavior to be recognized by the compiler.
I’m using g++ with warning level -Wall -Wextra and treating warnings as errors (-Werror).
Now I’m sometimes getting an error “variable may be used uninitialized in this function”.
By “sometimes” I mean that I have two independent compilation units that both include the same header file. One compilation unit compiles without error, the other gives the above error.
The relevant piece of code in the header files is as follows. Since the function is pretty long, I’ve only reproduced the relevant bit below.
The exact error is:
'cmpres' may be used uninitialized in this function
And I’ve marked the line with the error by * below.
for (; ;) {
int cmpres; // *
while (b <= c and (cmpres = cmp(b, pivot)) <= 0) {
if (cmpres == 0)
::std::iter_swap(a++, b);
++b;
}
while (c >= b and (cmpres = cmp(c, pivot)) >= 0) {
if (cmpres == 0)
::std::iter_swap(d--, c);
--c;
}
if (b > c) break;
::std::iter_swap(b++, c--);
}
(cmp is a functor that takes two pointers x and y and returns –1, 0 or +1 if *x < *y, *x == *y or *x > *y respectively. The other variables are pointers into the same array.)
This piece of code is part of a larger function but the variable cmpres is used nowhere else. Hence I fail to understand why this warning is generated. Furthermore, the compiler obviously understands that cmpres will never be read uninitialized (or at least, it doesn’t always warn, see above).
Now I have two questions:
Why the inconsistent behaviour? Is this warning generated by a heuristic? (This is plausible since emitting this warning requires a control flow analysis which is NP hard in the general case and cannot always be performed.)
Why the warning? Is my code unsafe? I have come to appreciate this particular warning because it has saved me from very hard to detect bugs in other cases – so this is a valid warning, at least sometimes. Is it valid here?
An algorithm that diagnoses uninitialized variables with no false negatives or positives must (as a subroutine) include an algorithm that solves the Halting Problem. Which means there is no such algorithm. It is impossible for a computer to get this right 100% of the time.
I don't know how GCC's uninitialized variable analysis works exactly, but I do know it's very sensitive to what early optimization passes have done to the code. So I'm not at all surprised you get false positives only sometimes. It does distinguish cases where it's certain from cases where it can't be certain --
int foo() { int a; return a; }
produces "warning: ‘a’ is used uninitialized in this function" (emphasis mine).
EDIT: I found a case where recent versions of GCC (4.3 and later) fail to diagnose an uninitialized variable:
int foo(int x)
{
int a;
return x ? a : 0;
}
Early optimizations notice that if x is nonzero, the function's behavior is undefined, so they assume x must be zero and replace the entire body of the function with "return 0;" This happens well before the pass that generates the used-uninitialized warnings, so there's no diagnostic. See GCC bug 18501 for gory details.
I bring this up partially to demonstrate that production-grade compilers can get uninitialized-variable diagnostics wrong both ways, and partially because it's a nice example of the point that undefined behavior can propagate backward in execution time. There's nothing undefined about testing x, but because code control-dependent on x has undefined behavior, a compiler is allowed to assume that the control dependency is never satisfied and discard the test.
There was an interesting discussion on clang dev-mailing list related to those heuristics this week.
The bottom line is: it's actually quite difficult to diagnose unitialized values without getting exponential behavior...
Apparently (from the discussion), gcc uses a predicate base approach, but given your experience it seems that it is not always sufficient.
I suspect it's got something to do with the fact that the assignment is mixed within the condition (and after a short-circuiting operator at that...). Have you tried without ?
I think both the gcc and clang folks would be very interested by this example since it's relatively common practice in C or C++ and thus could benefit from some tuning.
The code is correct, but the compiler is failing to identify that the variable is never used without initialization.
I would suggest that it's likely a heuristical error- that's what the "may" is for. I suspect that not many loop conditions look quite like that. That code is not unsafe because in all control paths, cmpres is assigned before use. However, I certainly wouldn't find it wrong to initialize it first.
You could, however, have some kind of variable shadowing going on here. That would be the only explanation I could think of for only one of the two translation units giving errors.
Can't a compiler warn (even better if it throws errors) when it notices a statement with undefined/unspecified/implementation-defined behaviour?
Probably to flag a statement as error, the standard should say so, but it can warn the coder at least. Is there any technical difficulties in implementing such an option? Or is it merely impossible?
Reason I got this question is, in statements like a[i] = ++i; won't it be knowing that the code is trying to reference a variable and modifying it in the same statement, before a sequence point is reached.
It all boils down to
Quality of Implementation: the more accurate and useful the warnings are, the better it is. A compiler that always printed: "This program may or may not invoke undefined behavior" for every program, and then compiled it, is pretty useless, but is standards-compliant. Thankfully, no one writes compilers such as these :-).
Ease of determination: a compiler may not be easily able to determine undefined behavior, unspecified behavior, or implementation-defined behavior. Let's say you have a call stack that's 5 levels deep, with a const char * argument being passed from the top-level, to the last function in the chain, and the last function calls printf() with that const char * as the first argument. Do you want the compiler to check that const char * to make sure it is correct? (Assuming that the first function uses a literal string for that value.) How about when the const char * is read from a file, but you know that the file will always contain valid format specifier for the values being printed?
Success rate: A compiler may be able to detect many constructs that may or may not be undefined, unspecified, etc.; but with a very low "success rate". In that case, the user doesn't want to see a lot of "may be undefined" messages—too many spurious warning messages may hide real warning messages, or prompt a user to compile at "low-warning" setting. That is bad.
For your particular example, gcc gives a warning about "may be undefined". It even warns for printf() format mismatch.
But if your hope is for a compiler that issues a diagnostic for all undefined/unspecified cases, it is not clear if that should/can work.
Let's say you have the following:
#include <stdio.h>
void add_to(int *a, int *b)
{
*a = ++*b;
}
int main(void)
{
int i = 42;
add_to(&i, &i); /* bad */
printf("%d\n", i);
return 0;
}
Should the compiler warn you about *a = ++*b; line?
As gf says in the comments, a compiler cannot check across translation units for undefined behavior. Classic example is declaring a variable as a pointer in one file, and defining it as an array in another, see comp.lang.c FAQ 6.1.
Different compilers trap different conditions; most compilers have warning level options, GCC specifically has many, but -Wall -Werror will switch on most of the useful ones, and coerce them to errors. Use \W4 \WX for similar protection in VC++.
In GCC You could use -ansi -pedantic, but pedantic is what it says, and will throw up many irrelevant issues and make it hard to use much third party code.
Either way, because compilers catch different errors, or produce different messages for the same error, it is therefore useful to use multiple compilers, not necessarily for deployment, but as a poor-man's static analysis. Another approach for C code is to attempt to compile it as C++; the stronger type checking of C++ generally results in better C code; but be sure that if you want C compilation to work, don't use the C++ compilation exclusively; you are likely to introduce C++ specific features. Again this need not be deployed as C++, but just used as an additional check.
Finally, compilers are generally built with a balance of performance and error checking; to check exhaustively would take time that many developers would not accept. For this reason static analysers exist, for C there is the traditional lint, and the open-source splint. C++ is more complex to statically analyse, and tools are often very expensive. One of the best I have used is QAC++ from Programming Research. I am not aware of any free or open source C++ analysers of any repute.
gcc does warn in that situation (at least with -Wall):
#include <stdio.h>
int main(int argc, char *argv[])
{
int a[5];
int i = 0;
a[i] = ++i;
printf("%d\n", a[0]);
return 0;
}
Gives:
$ make
gcc -Wall main.c -o app
main.c: In function ‘main’:
main.c:8: warning: operation on ‘i’ may be undefined
Edit:
A quick read of the man page shows that -Wsequence-point will do it, if you don't want -Wall for some reason.
Contrarily, compilers are not required to make any sort of diagnosis for undefined behavior:
§1.4.1:
The set of diagnosable rules consists of all syntactic and semantic rules in this International Standard except for those rules containing an explicit notation that “no diagnostic is required” or which are described as resulting in “undefined behavior.”
Emphasis mine. While I agree it may be nice, the compiler's have enough problem trying to be standards compliant, let alone teach the programmer how to program.
GCC warns as much as it can when you do something out of the norms of the language while still being syntactically correct, but beyond the certain point one must be informed enough.
You can call GCC with the -Wall flag to see more of that.
If your compiler won't warn of this, you can try a Linter.
Splint is free, but only checks C http://www.splint.org/
Gimpel Lint supports C++ but costs US $389 - maybe your company c an be persuaded to buy a copy? http://www.gimpel.com/