Related
I understand that it's a very hard thing to do, what with #ifdef, #define, and templates, but what is the state of the art of c++ parsers (be it open source, or proprietary?).
I mean, for a university project I'm thinking of creating a tool for analysing c++ code bases, but it seems very hard to find a good parser for it.
Should I give up and settle for java parsers? Similarly, what's the state of the art for java parsers? What about c#?
Also, would ripping the parser part of g++ apart from it ever work for the purposes of code analysis, or is it too much effort trying to do so?
You're in luck! Clang just started being able to parse most c++ programs within the last few months: http://clang.llvm.org/ It's one of the few open source parsers actually able to parse most of C++. (Mostly just GCC and CLANG, I hear Oink(?) Can get pretty good sometimes) And it's built to be used as a library by IDEs and the like, even has architecture built to support code rewriting.
There are some proprietary parsers that get the job done, But none of them are really usable without source access.
Regarding ripping apart gcc, That's not very practical for code analysis depending on what you are trying to do, you could use the new plugin architecture to get some usable information out of it, however at a very early level in parsing, it does something called term folding, where the parser itself will optimize out things like "x = x" (A simplistic example) And other aspects of the compiler expects this to happen, so it's not trivial to remove. Thus making gcc nearly useless for anything resembling source rewriting.
For C++ you can use GCC with -fdump-translation-unit & friends option to get AST from it.
See: http://www.manpagez.com/man/1/g++/
If you can compile something by g++ then you can get tree from it.
The industry stardard C++ parser, widely used in compilers, in EDG's C++ front end. I have no experience with this; but I understand it handles a huge variety of C++ dialects. I understand you can get it free for research purposes.
The open source standard is the GCC compiler. I hear is it difficult to understand and modify.
There's CLANG as mentioned in other answers. I have no experience here. My understanding is that it is fairly sophisticated especially in terms of supporting analysis.
Our proprietary DMS Software Reengineering Toolkit has full C++ parser with full name and type resolution, preprocessor expansion (or retention, which the other tools will not do). The C++ front end handles several dialects of C++: ANSI, GCC, MS Visual Studio. As you might guess, I have a lot of experience with this one.
DMS/CppFrontEnd has been used to carry out program analyses as well as massive program source-to-source transformations on C++ code, enabled by DMS's pattern parser, which will parse any fragment of C++ code. I believe the other C++ front ends don't provide source-to-source transformations. With those you can likely hack at the ASTs procedurally, but this is pretty inconvenient because you have to know the precise AST structure and for C++ this is pretty complicated.
DMS also has full C, Java and COBOL front ends with name and type resolution as well as control and data flow analysis. It has parsers (but not name and type analysis) for many other langauges, including C#.
AFAIK, the other "C++ parsers" can't do this, sort of by definition. One can apply source-to-source transformations on any of these, or any mixture of these.
clang is worth looking into. it's fast and they provide apis to hook into their backend.
Xcode 4 uses clang for tasks such as parsing, error reporting/detection in some cases, auto-completion, and fix-its.
I need a good, stable and, maybe, easy to use C++ parser library with C/C++ interface (C is preferred).
I hear that cint is good c++ interpreter. Can I use it (or some part of it) for this purpose?
Any suggestions?
See: http://clang.llvm.org/
It has both a C++ and a C interface (libclang).
C++ parsing is famously hard. AFAIK there are only three parsers that are acceptable by todays standards: EDG (widely used as a frontend in popular C++ compilers), GCC's and Microsoft's. And apparently, Microsoft has started using EDG's parser in VS2010, for Intellisense.
When you're looking at the free options, you're pretty much stuck at GCC. It can produce XML, though, so the easy part is there. (Easy by C++ parsing standards, that is)
Clang is the most up-to-date and mature option, with a decent C++ API (but no plain C). Elsa is a bit out of date and unmaintained, but still a usable choice. Both could be used as libraries as well as standalone XML frontends.
If you want to parse C or C++ code, there are some options:
http://bellard.org/tcc/
http://students.ceid.upatras.gr/~sxanth/ncc/
If you want to create a parser using C/C++, you can try:
http://boost-spirit.com/home/
http://dinosaur.compilertools.net/ Lex and Yacc
http://www.codeguru.com/csharp/.net/net_general/patterns/article.php/c12805 Flex and Bison
Our C++ Front End is able to parse a variety of C++ dialects (ANSI, GCC, MSVS), automatically builds ASTs whose nodes are marked with precise source positions and are decorated with any nearby comment text, and builds a full symbol table. (EDIT Jan 2013: the C++ front end has been able to handle C++11 for quite awhile now).
The C++ front end is built on top of our DMS Software Reengineering Toolkit, generalized compiler technology for program analysis and transformation, designed to support custom tool building. The C++ front end includes a preprocessor, in which the preprocessor directives can be expanded or not collectively or individually as appropriate for the task. It also includes full symbol construction with all the nasty Koenig lookup stuff.
DMS accepts explicit language definitions (that's how it understands C++; there are also fron ends for C, C#, Java, COBOL, and variety of other languages). DMS provides general parsing, symbol table building, flow analysis machinery, procedural APIs for tree navigation/inspection/modification, source-to-source transformation, and AST-to-source text regeneration including the original comments, number radices, etc. All of these capabilities are available for use by the C++ Front End.
DMS is also designed to handle the scale required for serious tasks. Often you need not just one compilation unit (which is what GCC will give you at best) but access to an entire set. DMS has been used to analyze/transform thousands of C++ compilation units, and literally tens of thousands of C compilation units (on a 25 million line application).
"Easy to use library" is an oxymoron when it comes to program manipulation tools. The langauges themselves are complex (C++ being one of the most difficult and getting worse with C++0X) and that induces complexity in the nature of the questions you can ask and what the answers look like (e.g. "are there any template instantions that can modify local variable X in method Y in class C in any namespace N?"). The questions themselves are hard.
What you want is a library with the necessary complexity to let you carry off your task. DMS has been under continuous development for the last 15 years, to provide that necessary complexity. If you want to do serious program processing, I claim you will need that information.
As proof, DMS has been used to carry out massive automated reengineering of C++-based mission avionics software for Boeing. I don't believe there are any other tools that can do this. (Clang looks to be trying, but only for C++. YMMV).
I don't know for cint, but I heard people use gcc-xml for this.
I have been looking for a good stand-alone library too, but haven't found any.
If you're feeling brave the links in the answer to "is there a yacc-able C++ grammar?" might be helpful. Gcc-xml and clang have already been suggested and Swig also has an XML output which depending on what you're trying to achieve might be relevant.
I did not try it, but I think that best choice will be getting modules for parsing from some popular open source compiler like gcc for C++;
Maybe you'll find something interesting here http://www.nobugs.org/developer/parsingcpp/
I am looking for options for an open source C++ compiler front-end (source parser/analyzer) that I could customize for my requirements. I do not need the back end implementation, just that it would help to find a fast and relatively bug free C++ front end that supports most standard features. Any suggestions?
[I did google, clang seems to be an option but i'd much prefer peer feedback before i begin with it.]
Arpan
Clang and GCC are the two main options. GCC is very complicated (or so I've heard), and Clang is very promising but is immature.
GCC-XML uses GCC's frontend to spit out an XML description of the source. GCC-XML's output is not a full abstract source tree (it doesn't contain function bodies), but it would be a lot easier to work with than GCC itself. (The latest release on the GCC-XML page is horribly out of date; if you don't want to mess around with tracking its CVS yourself, you might try downloading a tarball from, e.g., Debian's gccxml page.)
Depending on your exact requirements, other options might work:
CINT is a C / C++ interpreter. I'm told that it's not very strict in its adherence to C++ standards.
ROSE can take C and C++ source and lets you do a variety of transformations on it. The C and C++ front-end of ROSE is licensed from EDG, so it's not open source, but it is freely redistributable.
Projects such as Doxygen and SWIG include their own limited C++ parsers. Although these are only intended for extracting documentation and generating interfaces, respectively, they may meet your needs.
Edit: For further reading, see "Parsing C++", by Andrew Birkett.
Have you looked at LLVM clang?
For one of the refactoring efforts I've done, we used Elsa:
http://scottmcpeak.com/elkhound/sources/elsa/
with mixed results. Some parts of our code were too complicated or nonstandard for Elsa to deal with, and had to be preprocessed out for the refactoring.
You can use this with Oink:
http://danielwilkerson.com/oink/
if source analysis is what you're in the mood for.
Hope this helps!
The Digital Mars C++ compiler is not open source, but the source code is available for purchase (see http://www.digitalmars.com/shop.html) and you can customize it.
CLang is probably the way to go these days for a comprehensive solution, but if you're looking for something standalone that you can understand then check out this project on github:
https://github.com/robertoraggi/cplusplus
Out of the box it comes with a little command line utility to dump the AST, symbols, and IR code.
This is the C++ front end used in QTCreator. It does a decent job parsing most modern C++ code and even though the cplusplus front end hasn't been updated for a while, QTCreator is very actively used and developed. This code has quite a bit of mileage on it.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
What are some good tools for getting a quick start for parsing and analyzing C/C++ code?
In particular, I'm looking for open source tools that handle the C/C++ preprocessor and language. Preferably, these tools would use lex/yacc (or flex/bison) for the grammar, and not be too complicated. They should handle the latest ANSI C/C++ definitions.
Here's what I've found so far, but haven't looked at them in detail (thoughts?):
CScope - Old-school C analyzer. Doesn't seem to do a full parse, though. Described as a glorified 'grep' for finding C functions.
GCC - Everybody's favorite open source compiler. Very complicated, but seems to do it all. There's a related project for creating GCC extensions called GEM, but hasn't been updated since GCC 4.1 (2006).
PUMA - The PUre MAnipulator. (from the page: "The intention of this project is to
provide a library of classes for the analysis and manipulation of C/C++ sources. For this
purpose PUMA provides classes for scanning, parsing and of course manipulating C/C++
sources."). This looks promising, but hasn't been updated since 2001. Apparently PUMA has been incorporated into AspectC++, but even this project hasn't been updated since 2006.
Various C/C++ raw grammars. You can get c-c++-grammars-1.2.tar.gz, but this has been unmaintained since 1997. A little Google searching pulls up other basic lex/yacc grammars that could serve as a starting place.
Any others?
I'm hoping to use this as a starting point for translating C/C++ source into a new toy language.
Thanks!
-Matt
(Added 2/9): Just a clarification: I want to extract semantic information from the preprocessor in addition to the C/C++ code itself. I don't want "#define foo 42" to disappear into the integer "42", but remain attached to the name "foo". This, unfortunately, excludes several solutions that run the preprocessor first and only deliver the C/C++ parse tree)
Parsing C++ is extremely hard because the grammar is undecidable. To quote Yossi Kreinin:
Outstandingly complicated grammar
"Outstandingly" should be interpreted literally, because all popular languages have context-free (or "nearly" context-free) grammars, while C++ has undecidable grammar. If you like compilers and parsers, you probably know what this means. If you're not into this kind of thing, there's a simple example showing the problem with parsing C++: is AA BB(CC); an object definition or a function declaration? It turns out that the answer depends heavily on the code before the statement - the "context". This shows (on an intuitive level) that the C++ grammar is quite context-sensitive.
You can look at clang that uses llvm for parsing.
Support C++ fully now link
The ANTLR parser generator has a grammar for C/C++ as well as the preprocessor. I've never used it so I can't say how complete its parsing of C++ is going to be. ANTLR itself has been a useful tool for me on a couple of occasions for parsing much simpler languages.
Depending on your problem GCCXML might be your answer.
Basically it parses the source using GCC and then gives you easily digestible XML of parse tree.
With GCCXML you are done once and for all.
pycparser is a complete parser for C (C99) written in Python. It has a fully configurable AST backend, so it's being used as a basis for any kind of language processing you might need.
Doesn't support C++, though. Granted, it's much harder than C.
Update (2012): at this time the answer, without any doubt, would be Clang - it's modular, supports the full C++ (with many C++-11 features) and has a relatively friendly code base. It also has a C API for bindings to high-level languages (i.e. for Python).
Have a look at how doxygen works, full source code is available and it's flex-based.
A misleading candidate is GOLD which is a free Windows-based parser toolkit explicitly for creating translators. Their list of supported languages refers to the languages in which one can implement parsers, not the list of supported parse grammars.
They only have grammars for C and C#, no C++.
Parsing C++ is a very complex challenge.
There's the Boost/Spirit framework, and a couple of years ago they did play with the idea of implementing a C++ parser, but it's far from complete.
Fully and properly parsing ISO C++ is far from trivial, and there were in fact many related efforts. But it is an inherently complex job that isn't easily accomplished, without rewriting a full compiler frontend understanding all of C++ and the preprocessor. A pre-processor implementation called "wave" is available from the Spirit folks.
That said, you might want to have a look at pork/oink (elsa-based), which is a C++ parser toolkit specifically meant to be used for source code transformation purposes, it is being used by the Mozilla project to do large-scale static source code analysis and automated code rewriting, the most interesting part is that it not only supports most of C++, but also the preprocessor itself!
On the other hand there's indeed one single proprietary solution available: the EDG frontend, which can be used for pretty much all C++ related efforts.
Personally, I would check out the elsa-based pork/oink suite which is used at Mozilla, apart from that, the FSF has now approved work on gcc plugins using the runtime library license, thus I'd assume that things are going to change rapidly, once people can easily leverage the gcc-based C++ parser for such purposes using binary plugins.
So, in a nutshell: if you the bucks: EDG, if you need something free/open source now: else/oink are fairly promising, if you have some time, you might want to use gcc for your project.
Another option just for C code is cscout.
The grammar for C++ is sort of notoriously hairy. There's a good thread at Lambda about it, but the gist is that C++ grammar can require arbitrarily much lookahead.
For the kind of thing I imagine you might be doing, I'd think about hacking either Gnu CC, or Splint. Gnu CC in particular does separate out the language generation part pretty thoroughly, so you might be best off building a new g++ backend.
Actually, PUMA and AspectC++ are still both actively maintained and updated. I was looking into using AspectC++ and was wondering about the lack of updates myself. I e-mailed the author who said that both AspectC++ and PUMA are still being developed. You can get to source code through SVN https://svn.aspectc.org/repos/ or you can get regular binary builds at http://akut.aspectc.org. As with a lot of excellent c++ projects these days, the author doesn't have time to keep up with web page maintenance. Makes sense if you've got a full time job and a life.
how about something easier to comprehend like tiny-C or Small C
Elsa beats everything else I know hands down for C++ parsing, even though it is not 100% compliant. I'm a fan. There's a module that prints out C++, so that may be a good starting point for your toy project.
See our C++ Front End
for a full-featured C++ parser: builds ASTs, symbol tables, does name
and type resolution. You can even parse and retain the preprocessor
directives. The C++ front end is built on top of our DMS Software Reengineering
Toolkit, which allows you to use that information to carry out arbitrary
source code changes using source-to-source transformations.
DMS is the ideal engine for implementing such a translator.
Having said that, I don't see much point in your imagined task; I don't
see much value in trying to replace C++, and you'll find building
a complete translator an enormous amount of work, especially if your
target is a "toy" language. And there is likely little point in
parsing C++ using a robust parser, if its only purpose is to produce
an isomorphic version of C++ that is easier to parse (wait, we postulated
a robust C++ already!).
EDIT May 2012: DMS's C++ front end now handles GCC3/GCC4/C++11,Microsoft VisualC 2005/2010. Robustly.
EDIT Feb 2015: Now handles C++14 in GCC and MS dialects.
EDIT August 2015: Now parses and captures both the code and the preprocessor directives in a unified tree.
EDIT May 2020: Has been doing C++17 for the past few years. C++20 in process.
A while back I attempted to write a tool that will automatically generate unit tests for c files.
For preprosessing I put the files thru GCC. The output is ugly but you can easily trace where in the original code from the preprocessed file. But for your needs you might need somthing else.
I used Metre as the base for a C parser. It is open source and uses lex and yacc. This made it easy to get up and running in a short time without fully understanding lex & yacc.
I also wrote a C app since the lex & yacc solution could not help me trace functionality across functions and parse the structure of the entire function in one pass. It became unmaintainable in a short time and was abandoned.
What about using a tool like GNU's CFlow, that can analyse the code and produce charts of call-graphs, here's what the opengroup(man page) has to say about cflow. The GNU version of cflow comes with source, and open source also ...
Hope this helps,
Best regards,
Tom.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
Does anybody know a fully featured refactoring tool for C++ that works reliably with large code bases (some 100.000 lines)?
I tried whatever i can find again and again over the last years: SlickEdit, Eclipse CDT. They all were not at all usable.
SUMMARY:
I took time and evaluated "Visual Assist X" as well as "Refactor for C++". Both have some impressing features, but both as well are far from perfect. Extracting a large block of code usually is not done satisfying without manual modifications - and therefore does not pay off.
"Visual Assist X" has nice features such as much more complete autocompletition etc. But it leads to so much flickering and slows down much at certain points.
By my opinion therefore the answer is: "No, there is no production ready refactoring tool for C++"
UPDATE March 2015
As for hdoghmens reply today i tried Resharper for C++.
His link
https://www.jetbrains.com/resharper/ does not say anything about C++. But i found Resharper C++ that was announced more than a year ago here:
https://www.jetbrains.com/resharper/features/cpp.html
I gave it a try with VC2010 using a code base of 20MB.
Test 1: Extract Method:
results in a Resharper exception. No source code changed.
Test 2: Extract Method with different source:
Works fine
Test 3: Change signature of extracted function: Results in broken C++ code:
bool myclass::do_work123(<unknown long Color>int& Filled*&, long, int&)
Maybe thats why C++ its not listed on the main page.
By my opinion the answer to this question still is "NO".
Visual Assist and Visual Studio make dealing with large codebases much easier. Visual assist is good at tracking down how a class or member is used and is more effective at renaming it without false positives than search and replace.
I find Visual Assist X with Visual Studio very useful. Another choice is Refactor for C++.
I expect clang will significantly change the landscape of C++ refactoring tools out there over the next couple of years. It's an open-source, modular compiler that exposes an API for parsing and semantically analyzing C++ code. IDEs and other tools will be able to use this API rather than doing the difficult work of writing their own parser and semantic analyzer.
Google already made a large-scale refactoring tool using clang.
Mozilla has their own refactoring tool named Pork (Wiki, Developer Wiki). Here is the blog of the developer behind Pork. From what I've read Pork was successfully used in refactorings at Mozilla.
Pork should help if you come from *nix land, for Visual Studio I too recommend Visual Assist.
Our DMS Software Reengineering Toolkit is a transformation engine designed to carry out complex transforms over large bodies of code, including C++. It has been used to make reliable changes on systems of millions of lines of code. It operates by using compiler-accurate langauges analyzers and transformers.
It has a full C++ parser with name and type resolution, builds ASTs of code, can apply procedural or source-to-source transformations (with C++ surface syntax) to revise those trees, and regenerate compilable output with comments preserved. (Edit: 7/1/2011: Now does C++1X to the extent we understand the standard :)
It has been used on large scale reengineering projects, including C++ component re-architecting, and 100% fully automated translations between langauges.
You can read about this at the website.
DMS is also used to build arbitrary source analysis tools. Examples include clone detection, test coverage, smart difference (comparision of source code structures and abstract editing operations rather than lines with simple insert and delete), etc.
What it is not (presently) is an interactive refactoring tool. We believe that to do most refactorings well, you need deep control and data fow analyses. DMS has generic machinery to support this, and that machinery is implemented for C, COBOL and Java at this point, with C++ being next in line. This is a tough job. You won't see a lot of serious C++ refactoring tools from anybody until this kind of problem has been solved well.
First you need a full C++ parser :-}
EDIT 7/5/2011: Looks like we are going to take a run at the interactive version. We have won a Department of Energy Phase I SBIR to investigate how to do this. See http://science.energy.gov/sbir/awards-and-general-stats/fy-2011/phase-i-by-state/?p=1#tx (Look for Semantic Designs under "Texas").
Don't expect a result in a hurry; this is just the start of 3 phase multi-year program to get to a tool.
EDIT 8/11/2011: First progress... we now handle all of C++0x and OpenMP directives.
EDIT 1/4/2012: Does full control flow analysis of C++ code.
EDIT 9/15/2014: Now have C++14 front end parser/transformation engine well in hand. Even does rename pretty reliably at this point :-}
If you're using emacs, try Xrefactory . It supports method extraction, renaming of classes/functions/variables and insert/delete/move parameters.It also has very good/fast code completion engine.
Currently I can't recommend any refactoring tool for C++, certainly not for large code bases of 100k lines and above. I've been hoping this will change, like the OP, and I hope one day there will be something. I fear that the language itself might have to change significantly before we see any really good tools.
btw, has SlickEdit dropped its refactoring features?
I recommend to try rtags if you use emacs and haven't tried it yet (there is also a package for vim available). It is a clang based client/server application that indexes C/C++ code, with these features included:
go to definition/declaration
find all references, go to next/previous
rename symbol
integration with clang’s “fixits”
I decided to give it a try after watching this talk which introduced rtags (and emacs) for me.
(I have to say that I went this far only after my QtCreator failed to rename some symbols properly, which is a show-stopper for my using this great IDE for now)
Besides what is supported by rtags, I also need some additional neat features, including:
create function definition/prototype
extract function
create getter/setter methods
For these, I recommend to use a semantic-refactor package for emacs (not sure if there are alternatives for vim)
Generally, clang based tools looks very promising. If you are interested in more information about clang tools for C++ refactoring, including for projects with large codebase, there are some great talks by Chandler Carruth.
The DMS software rengineering toolkit does this I think. It is a code transformation engine, designed for large scale and handles C++. Have no idea how elegant the output is though.
The problem are C++ templates. As of 2019 I'm not aware of any refactoring tool that supports C++ templates. I've tried VS2019, VisualAssist, Clion, QtCreator.
Consider example:
#include <iostream>
struct foo { void print() {} };
struct bar { void print() {} };
template <typename T>
void call_print(T&& v) { v.print(); }
void print() {}
int main()
{
call_print(foo{});
call_print(bar{});
return 0;
}
If I run Rename Refactoring on foo::print, bar::print should be also renamed automatically. Because they are linked through call_print function template instantiations.
One surely has to mention Klocwork as a commercial code refactoring suite. It does look very promising when you go through the demo video.
Definetely Resharper Ultimate is the way to go. Happiness guaranteed :)
In Beta version as of march 2015.
Sorry to only find this question so late. My students and assistants work on C++ refactoring since about 2006. Most of CDTs refactoring infrastrucure was built by my team at IFS institute of software. since a couple of years we provide Cevelop our version of CDT with support for C++ code modernization refactorings etc. Cevelop can work with large code bases, if workspace is set up correctly. Free available at https://cevelop.com
If you are using Visual C++ (Express Edition is free), you can use Visual Assist from www.wholetomato.com (link to the C++ refactoring features).
It has a 30 day trial period and we have found it to be faster and more feature-full that the built-in intellisense in the Visual C++ product itself.
If your looking to reengineer your codebase: MOOSE. But that's a large collection of analysis and reengineering tools, not an editor.
There is now a C++ refactoring extension for Visual Studio 2013 from Microsoft:
http://visualstudiogallery.msdn.microsoft.com/164904b2-3b47-417f-9b6b-fdd35757d194
CLion looks very promising.
Disclaimer: I've not tried it yet as I need to convert my projects to CMake format in order to use it.
I recommend you try Lattix. It allows you to analyze large C/C++ codebases to discover the archtecture, identify problematic dependencies, and re-engineer the code to improve modularity and reduce technical debt. Lattix also provides a number of algorithms to help in the refactoring process. These algorithms help you figure out how to move elements from one part of the hierarchy to another, to break cycles and to move subsystems so that the coupling and cohesion of subsystems can be improved. Here are the results of Lattix analyzing the Android Kernel (1.6 million LOC of C/C++).
Full disclosure: I work for Lattix
I found the following plugin for Visual Studio 2013:
Visual C++ Refactoring by Microsoft.
It is just a simple rename tool but it works flawlessy. It adds the following context menu after right-clicking on a symbol: