This Github repository added std::regex to the list of regular expression engines and got decimated by the others.
Why is that std::regex - as implemented in libstdc++ - so much slower than others? Is that because of the C++ standard requirements or it is just that that particular implementation is not very well optimized?
Also in the shootout std::regex was unable to compile several regular expressions that all the others accepted, even after adding the flag std::regex::extended. They were (?i)Twain, \b\w+nn\b, (?i)Tom|Sawyer|Huckleberry|Finn, \s[a-zA-Z]{0,12}ing\s, ([A-Za-z]awyer|[A-Za-z]inn)\s and \p{Sm}.
UPDATE: Added comparison with boost::regex.
UPDATE2: added ctre
Is that because of the C++ standard requirements or it is just that that particular implementation is not very well optimized?
The answer is yes. Kinda.
There is no question that libstdc++'s implementation of <regex> is not well optimized. But there is more to it than that. It's not that the standard requirements inhibit optimizations so much as the standard requirements inhibit changes.
The regex library is defined through a bunch of templates. This allows people to choose between char and wchar_t, which is in theory good. But there's a catch.
Template libraries are used by copy-and-pasting the code directly into the code compiling against those libraries. Because of how templates get included, even types that nobody outside of the template library knows about are effectively part of the library's ABI. If you change them, two libraries compiled against different versions of the standard library cannot work with each other. And because the template parameter for regex is its character type, those implementation details touch basically everything about the implementation.
The minute libstdc++ (and other standard library implementations) started shipping an implementation of C++ regular expressions, they bound themselves to a specific implementation that could not be changed in a way that impacted the ABI of the library. And while they could cause another ABI break to fix it, standard library implementers don't like breaking ABI because people don't upgrade to standard libraries that break their code.
When C++11 forbade basic_string copy-on-write implementations, libstdc++ had an ABI problem. Their COW string was widely used, and changing it would make code that compiled against the new one break when used with code compiled against the old one. It took years before libstdc++ bit the bullet and actually implemented C++11 strings.
If Regex had been defined without templates, implementations could use traditional mechanisms to hide implementation details. The ABI for the interface to external code could be fixed and unchanging, with only the implementation of the functions behind that ABI changing from version to version.
From the manual:
In the GCC 5.1 release libstdc++ introduced a new library ABI that includes new implementations of std::string and std::list. These changes were necessary to conform to the 2011 C++ standard which forbids Copy-On-Write strings and requires lists to keep track of their size.
It is possible to use the _GLIBCXX_USE_CXX11_ABI macro to control whter the library headers use the old or the new ABI, independently of which "-std" is being used.
I'd like to know what the implications of using this "compatibility ABI" would be? I guess that the run-time performance of small-string operations will be impacted (negatively I assume), and that list-size access goes from O(1) (C11 ABI) to O(N) (compatibility ABI).
Are my guesses correct and can anyone elaborate?
Are there other implications which I have missed? What about atomics and concurrency features? Any impact?
Your first question is actually answered by the manual itself:
... the choice of ABI to use is independent of the -std option used to
compile your code... This ensures that the -std does not change the ABI, so that it is
straightforward to link C++03 and C++11 code together.
Regarding the second question, I'm afraid it's hard to generalize the impact because it depends on how your code is using the standard library. Does it copy strings a lot? How often a list size is queried? Is the code multi-threaded?
Although atomics and concurrency were introduced in C++11's standard, I'd guess that stdlib++ copy-on-write mechanism already used a variation of it anyhow. Those implementations are typically thread-safe.
Perhaps one thing you didn't directly mention is the impact on other std components that depend on those behaviors, such as list::splice
As a downstream maintainer in a Linux distribution, some of the packages that I usually maintain are starting to use the C++11 features in their code base. All of them depend on different libraries packaged by the Linux distributions.
Problems with the ABI could appear when mixing C++11 code with C++98 and AFAIK, most of the current major Linux Distributions are not enabling the C++11 flag by default when compiling software to generate packages.
The question is: How are the major Linux distributions handling the entry of C++11 code? Is there a decent way of checking or avoiding these problems with the ABI when using system libraries?
Thanks.
The issue has nothing to do with C++11 vs C++98 except that C++11 can motivate binary changes. There is nothing special about binary changes motivated by C++11. They are just as breaking or non-breaking as regular binary changes. Furthermore, they are only changed if the library maintainer specifically chooses to change his binary interface.
In other words, this has nothing to do with the Standard version and everything to do with the library, unless the library explicitly chooses to offer two different binary interfaces to different Standard versions (which is still a library choice). Excepting this case, you are just as broken in C++98 as you are in C++11. Itanium is backwards compatible between the C++11-supporting versions and the C++98-supporting versions, so the compiler ABIs are not broken.
From memory, unless you're using 4.7.0 which they broke for fun and then unbroke, you're pretty much safe with libstdc++- they are storing up ABI breakage for a future release when they can make one big break.
In other words, whilst the transition period to C++11 can introduce additional motivation to break ABI and therefore additional risk, actually using C++11 itself does not introduce any additional risk.
This is a very uninformed question, but:
I would like to start using C++11. Can I continue to use my large collection of libraries which were compiled with my old gcc 4.2.1 compiler or do I need to recompile them all with a new compiler? I would think (or hope) the answer is no, but I am only a dabbler.
So that I may have at least part of my ignorance removed, can you explain why in either case?
Thanks
Yes, you should.
The weaker reason is not binary compatibility, the problem is about expectations. A C++11 enabled compiler will expect a number of features to be available (move constructors among them) and use them when appropriate. And that's only scratching the tip of the iceberg, there are several other incompatibilities (auto, 0 and its interaction with pointers, ...).
It means that any inline method in a header may suddenly be interpreted differently, in light of the C++11 Standard.
The stronger reason is that each version of the compiler comes with its own Standard Library implementation. You don't really want start mixing various versions around and especially not when they have undergone such major changes (once again, rvalue references...).
Believe me, it's simpler to recompile now rather than having that nagging thought that each bug that appear may be due to an incompatibility between old and new libraries...
It's a compiler question. For instance, if you have a single compiler which supports both C++03 and C++11 depending on a compiler switch, you can very likely mix libraries. There's nothing new in C++11 that forces an incompatibility with C++03.
Howeve, you mentioned that your libraries were compiled with GGC 4.2.1. Since C++11 was just an idea back then, it's quite likely that GCC back then was implemented in ways that turned out to be incompatible with C++11.
For instance, std::list::size() must be O(1) in C++11, but it could be O(N) in C++03. GCC chose a O(N) implementation back then, not knowing the future requirements. The current GCC std::list::size implementation is compatible with both C++11 and C++03, since O(1) is better than O(N).
The answer totally depends on the API of your library and its implementation dependencies.
The conditions that guarantee that you don't need to recompile are:
-- Your library doesn't use C++ specific features in its public API.
That implies:
Your library doesn't provide classes/class-templates/function-templates/other C++ specific stuff.
You don't accept/return C++ classes to/from your library functions.
You don't pass function parameters by reference.
You don't provide public inline functions with C++ specific implementations.
You don't throw exceptions from your functions.
You don't (because you have no reason to) include C++ specific library headers in your public headers. (It wouldn't hurt if you do include them but everything must be OK if you remove such includes. It is like an indicator.)
-- Your library depends only on the libraries that binary-compatible with those available in your new build environment.
If those conditions are not satisfied then there is no guarantee that your library will work correctly. In most cases it is much easier to recompile than to make sure that everything works correctly.
Either way if you are going to make binary compatible API that satisfies the conditions above then it is much better to design and implement C language API. This way you automatically satisfy the above conditions and don't fall into sin of writing C-style C++ code.
You can use large parts of C++11 without recompiling (assuming ABI compatability), but a particular important part, for me atleast, will not be accessible to the already compiled code - move semantics.
Move semantics can make your code faster just by recompiling with a C++11 compiler (and preferably, a C++11 stdlib).
There are also other reasons. Maybe your preferred library has become C++11 aware since you last compiled it, and is now more efficient, safer or easier to use if compiled with a C++11 compiler?
For your own libraries, with C++11 you surely can make it them more efficient, safer and easier to use? :)
If your interfaces to the compiled code uses any template which was modified by C++11, then yes, you have to recompile. Otherwise you probably can continue to use your old libraries (unless the compiler provider also decided to make changes of the ABI at the same time, because it's a great opportunity to fix long-standing ABI bugs which are otherwise often refrained from fixing because of binary incompatibilities).
Are standard library required to be standard conformant? I've this feeling that standard library aren't standard conformant. The basis of this feeling is the error messages generated by the compiler(s). For example, sometime GCC gives error messages which starts with prefix __gxx and many others which I don't remember as of now. But seeing them gives me feeling that these are very compiler specific messages, and different compilers wouldn't be able to compile standard library provided by GCC, and vice-versa. Is it true?
The question can be asked in other words as:
Can standard library provided by one compiler be compiled with other compilers?
When we say a particular compiler is standard conformant, does it automatically mean that the stdlib which comes with it is also standard-conformant? Or it simply means that this compiler can compile standard-conformant code written by us, programmers?
Can I use standard library provided by one compiler, in my project which uses a different compiler to compile the project? Is portability same as standard-conformance?
These questions are different angles to look at the same big question. So, please help me understanding what does it exactly mean when we say compiler X is standard-conformant.
The standard library is a detail of implementation. It may not even be 'compiled' in the sense that the standard doesn't require it to consist of 'files' [headers]:
174) A header is not necessarily a source file, nor are the sequences delimited by < and > in header names necessarily valid
source file names (16.2).
The standard carefully eases the requirements on the implementation, so that the library may be 'built-in' to the compiler (a.k.a intrinsics). For example, extending the std namespace, or #defining a name used in the standard library gives you undefined behavior.
Yes, standard libraries must adhere to the standard, but there is quite a bit of flexibility in that. The standard does not require a particular implementation of the functions, and the implementation is free to add internal functions, attributes... as long as the requirements are met.
Note that there is a different concept of the library conforming with the standard and the library being implemented using only standard features.
On the particular questions below:
Can standard library provided by one compiler be compiled with other compilers?
Some will, some won't. The implementation of the standard library can use compiler intrinsics for some operations, of features present only in one platform and not others... Some STL implementations can be compiled with different compilers though, like STLPort, Dinkumware (which is also the one shipped in VS, with some VS modifications)
When we say a particular compiler is standard conformant, does it automatically mean that the stdlib which comes with it is also standard-conformant? Or it simply means that this compiler can compile standard-conformant code written by us, programmers?
It means that the library must be conformant, but again, the standard does not mandate the implementation of the library, and these can use non-standard extensions and the like, which will work in one compiler but possibly not in other compilers. Consider, for example, an implementation of shared_ptr, the reference count has to be updated atomically but there was no atomic operations on integers in the current standard, so it has to be implemented in terms of non-standard features.
Can I use standard library provided by one compiler, in my project which uses a different compiler to compile the project? Is portability same as standard-conformance?
Not necessarily.
Are standard library required to be standard conformant? I've this feeling that standard library aren't standard conformant.
By definition, an implementation must conform to the standard to be standard-conformant, yes.
Otherwise it's not an implementation of the C++ Standard Library, but of some other thing.
When we say a particular compiler is standard conformant, does it automatically mean that the stdlib which comes with it is also standard-conformant?
This would depend on the wording, wouldn't it? If the toolchain purports to include an implementation of the Standard Library, one may reasonably assume that it's compliant. That the compiler executable itself is compliant is not the same thing, unless the library implementation is built into the compiler executable.
But this is all just wordplay.
Can I use standard library provided by one compiler, in my project which uses a different compiler to compile the project? Is portability same as standard-conformance?
Of course not. Two conformant implementations may be completely incompatible with each other.
For example, sometime GCC gives error messages which starts with prefix __gxx and many others which I don't remember as of now.
That's fine. The toolchain may implement the library however it chooses, as long as the interface conforms to the standard. The standard does not say that the implementation cannot use a symbol __gxx in its work.
BTW
Though, it does say that the programmer may not use symbols with leading underscores, in some cases!
You aren't allowed ([lib.requirements]/[requirements]) to use any name which:
contains two consecutive underscores,
begins with an underscore followed by an uppercase letter
begins with an underscore and is in the global namespace
Also, in C++0x, literal suffixes that do not start with an underscore are reserved.
The user-facing API of the library should be standard; the implementation is not required to be, however. You're seeing an example of this, where the standard version of a function is replaced with an optimized variant (with a different name because sometimes the replacement is only possible in some circumstances, such as when appropriate alignment can be proven). This also means that the implementation of one compiler's standard library might not be buildable by a different compiler (which can lead to bootstrapping issues when porting, but that's another issue).
what does it exactly mean when we say compiler X is standard-conformant
It means that the compiler provides the standard library with all the requirements layed upon by the standard and that the implementation itself conformes to all requirements in the standard. The thing is, the "requirements" for the standard library are quite liberal after the general interface.
The Standard library only has to meet a minimum interface. In it's implementation it can do whatever it wants, because, well, it's the implementation, although I'm pretty sure there are some identifier restrictions and such to prevent clashes.
The Standard library implementation does not have to be portable in any way. They can be, but far from necessarily.
The standard places restrictions on the interface of the standard library, not the implementation. It goes out of its way to make it clear that standard library headers can do things user code can not... __MACRO__NAME__ is reserved for the implementation, for instance. And obviously, only the implementation can actually put all these functions and types into namespace std.
Mostly, a "portable" implementation could be written. But it would likely be less performant... as an easy example, consider the traditional implementation of the offsetof macro. It generally involves dereferencing a null pointer, which is formally undefined behavior, but because the implementation knows how its platform works, that's okay. A portable version can not do this, so it would have to actually create a new instance of the provided type to keep everything on the up and up.
Quite a few type traits in C++0x likely require compiler support, which makes a "portable" implementation somewhere between difficult and impossible. There's no standard way to analyze if an arbitrary type is POD, for instance, which is why boost::is_pod requires user support via specialization on some platforms.
There is also the consideration that a lot of the standard library is not header-only. The other bits can be written in a completely different language... so long as it all links together properly, it doesn't matter. If the runtime is implemented in Lisp, obviously it won't be C++ compliant code that can be reliably dropped into a different compiler toolchain.
Can standard library provided by one compiler be compiled with other
compilers?
Assuming that the library is standards conforming, yes (I know this is kind of chicken-and-egg).
When we say a particular compiler is standard conformant, does
it automatically mean that the stdlib
which comes with it is also
standard-conformant? Or it simply
means that this compiler can compile
standard-conformant code written by
us, programmers?
Yes, although I'm not aware of any fully confirming compiler. Note that standards conforming for us as programmers is different from the standard library. For example the implementation is allowed to use identifiers containing __ (double underscore).
Can I use standard library provided by one compiler, in my
project which uses a different
compiler to compile the project? Is
portability same as
standard-conformance?
You should be able to compile and use such a library. You almost certainly wouldn't be able to use any compiled library files (static or dynamic) file because name mangling would be different.