I need a way to grep function definitions from a specific subsystem of the kernel source tree (lets say mm). My original thought was to search for the regular expression ")\n{" which by convention is the starting point of a Linux kernel function.
Sadly though I found out that the newline character is used as a separator in grep, so it can't be used in a regular expression.
Anyone got a work-around or another useful command?
Thanks!
Another approach is to customize the GCC compiler to search what you want.
In particular, recent GCC (that is, version 4.6) can be extended with plugins (coded in C), or with extensions coded in GCC MELT (MELT is a high-level domain specific language to extend GCC).
Honestly, what you are attempting to do is parsing, and regex is the wrong approach. Sure, you
might get something that works 99% of the time, but it will fail you in strange and unexpected ways.
Looking into using a parser
Related
I'm looking for a way to search for a given term in a project's C/C++ code, while ignoring any occurrences in comments and strings.
As the code base is rather large, i am searching for a way to automatically identify the lines of code matching my search term, as they need manual inspection.
If possible I'd like to perform the search on my linux system.
background
the code base in question is a realtime signal processing engine with a large number of 3rd party plugins. plugins are implemented in a variety of languages (mostly C, but also C++ and others; currently I only care for those two), no standards have been enforced.
our code base currently uses the built-in type float for floating-point numbers and we would like to replace that with a typedef that would allow us to use doubles.
we would like to find all occurrences of float in the actual code (ignoring legit uses in comments and printouts).
What complicates things furthermore, is that there are some (albeit few) legit uses of float in the code payload (so we are really looking for a way to identify all places that require manual inspection, rather than run some automatic search-and-replace.)
the code also contains C-style static casts to (float), so relying on compiler warnings to identify type mismatches is often not an option.
the code base consists of more than 3000 (C and C++) files accumulating about 750000 lines of code.
the code is cross-platform (linux, osx, w32 being the main targets; but also freebsd and similar), and is compiled with the various native compilers (gcc/g++, clang/clang++, VisualStudio,...).
so far...
so far I'm using something ugly like:
grep "\bfloat\b" | sed -e 's|//.*||' -e 's|"[^"]*"||g' | grep "\bfloat\b"
but I'm thinking that there must be some better way to search only payload code.
IMHO there is a good answers on a similar question at "Unix & Linux":
grep works on pure text and does not know anything about the
underlying syntax of your C program. Therefore, in order not search
inside comments you have several options:
Strip C-comments before the search, you can do this using gcc
-fpreprocessed -dD -E yourfile.c For details, please see Remove comments from C/C++ code
Write/use some hacky half-working scripts like you have already found
(e.g. they work by skipping lines starting with // or /*) in order to
handle the details of all possible C/C++ comments (again, see the
previous link for some scary testcases). Then you still may have false
positives, but you do not have to preprocess anything.
Use more advanced tools for doing "semantic search" in the code. I
have found "coccigrep": http://home.regit.org/software/coccigrep/ This
kind of tools allows search for some specific language statements
(i.e. an update of a structure with given name) and certainly they
drop the comments.
https://unix.stackexchange.com/a/33136/158220
Although it doesn't completely cover your "not in strings" requirement.
It might practically depend upon the size of your code base, and perhaps also on the editor you are usually using. I am suggesting to use GNU emacs (if possible on Linux with a recent GCC compiler...)
For a small to medium size code (e.g. less than 300KLOC), I would suggest using the grep mode of Emacs. Then (assuming you have bound the next-error Emacs function to some key, perhaps with (global-set-key [f10] 'next-error) in your ~/.emacs...) you can quickly scan every occurrence of float (even inside strings or comments, but you'll skip very quickly such occurrences...). In a few hours you'll be done with a medium sized source code (and that is quicker than learning how to use a new tool).
For a large sized code (millions of lines), it might be worthwhile to customize some static analysis tool or compiler. You could use GCC MELT to customize your GCC compiler on Linux. Its findgimple mode could be inspirational, and perhaps even useful (you probably want to find all Gimple assignments targeting a float)
BTW, you probably don't want to replace all occurrences -but only most of them- of the float type with double (probably suitably typedef-ed...), because very probably you are using some external (or standard) functions requiring a float.
The CADNA tool might also be useful, to help you estimate the precision of results (so help you deciding when using double is sensible).
Using semantical tools like GCC MELT, CADNA, Coccinelle, Frama-C (or perhaps Fluctuat, or Coccigrep mentioned in g0hl1n's answer) would give more precise or relevant results, at the expense of having to spend more time (perhaps days!) in learning and customizing the tool.
The robust way to do this should be with cscope (http://cscope.sourceforge.net/) in line-oriented mode using the find this C symbol option but I haven't used that on a variety of C standards so if that doesn't work for you or if you can't get cscope then do this:
find . -type f -print |
while IFS= read -r file
do
sed 's/a/aA/g; s/__/aB/g; s/#/aC/g' "$file" |
gcc -P -E - |
sed 's/aC/#/g; s/aB/__/g; s/aA/a/g' |
awk -v file="$file" -v OFS=': ' '/\<float\>/{print file, $0}'
done
The first sed replaces all hash (#) and __ symbols with unique identifier strings, so that the preprocessor doesn't do any expansion of #include, etc. but we can restore them after preprocessing.
The gcc preprocesses the input to strip out comments.
The second sed replaces the hash-identifier string that we previously added with an actual hash sign.
The awk actually searches for float within word-boundaries and if found prints the file name plus the line it was found on. This uses GNU awk for word-boundaries \< and \>.
The 2nd sed's job COULD be done as part of the awk command but I like the symmetry of the 2 seds.
Unlike if you use cscope, this sed/gcc/sed/awk approach will NOT avoid finding false matches within strings but hopefully there's very few of those and you can weed them out while post-processing manually anyway.
It will not work for file names that contain newlines - if you have those you can put the body in a script and execute it as find .. -print0 | xargs -0 script.
Modify the gcc command line by adding whatever C or C++ version you are using, e.g. -ansi.
I was wondering what the most suitable tool is for extracting c (and eventually c++) function names, arguments and their types from source code. I would like to use a tool that can be automated as much as possible.
I want to extract the signature types then have the output read by another program written in Erlang. My goal is to use this information to construct the equivalent of the C types/signatures in Erlang (using ei). I would like this process to work for any C file so I can't use any hardcoded stuff.
I have found a lot of tools that look promising like CLang, CScope and ANTLR and so on but I don't know if any of them will work or if there is a better approach out there.
Thanks.
Surely there is something better, but if you don't find it:
gcc -aux-info output demo.c
sed '/include/d' output
Extracts functions form source code skipping standard functions
Sometimes I am reading some code and would like to find the definition for a certain symbol, but it is sprinkled throughout the code to such an extent that grep is more or less insufficient for pointing me to its definition.
For example, I am working with Zlib and I want to figure out what FAR means.
Steven#Steven-PC /c/Users/Steven/Desktop/zlib-1.2.5
$ grep "FAR" * -R | wc -l
260
That's a lot to scan through. It turns out it is in fact #defined to nothing but it took me some time to figure it out.
If I was using Eclipse I would have it easy because I can just hover over the symbol and it will tell me what it is.
What kinds of tools out there can I use to analyze code in this way? Can GCC do this for me? clang maybe? I'm looking for something command-line preferably. Some kind of tool that isn't a full fledged IDE at any rate.
You may want to check out cscope, it's basically made for this, and a command line tool (if you like, using ncurses). Also, libclang (part of clang/llvm) can do so - but that's just a library (but took me just ~100 lines of python to use libclang to emulate basic cscope features).
cscope requires you to build a database first. libclang can parse code "live".
If the variable is not declared in your curernt file, it is declared in an included file, i.e. a .h. So you can limit the amount of data by performing a grep only on those files.
Moreover, you can filter whole word matches with -w option of grep.
Try:
grep -w "FAR" *.h -R | wc -l
Our Source Code Search Engine (SCSE) is kind of graphical grep that indexes a large code base according to the tokens of its language(s) (e.g., C, Java, COBOL, ...). Queries are stated in terms of the tokens, not strings, so finding an identifier won't find it in the middle of a comment. This minimizes false positives, and in a big code base these can be a serious waste of time. Found hits are displayed one per line; a click takes to the source text.
One can do queries from the command line and get grep-like responses, too.
A query of the form of
I=foo*
will find all uses of any identifier that starts with the letters "foo".
Queries can compose mulitiple tokens:
I=foo* '[' ... ']' '='
finds assignments to a subscripted foo ("..." means "near").
For C, Java and COBOL, the SCSE can find reads, writes, updates, and declarations of variables.
D=*baz
finds declarations of variables whose names end in "baz". I think this is what OP is looking for.
While SCSE works for C++, it presently can't find reads/writes/updates/declarations in C++. It does everything else.
The SCSE will handle mixed languages with aplomb. An "I" query will search across all langauges that have identifiers, so you can see cross language calls relatively easily, since the source and target identifiers tend to be the same for software engineering reasons.
gcc can output the pre-processing result, with all macro definitions with gcc -E -dD. The output file would be rather larger, often due to the nested system headers. But the first appearance of a symbol is usually the declaration (definition). The output use #line to show the part pre-processed result belong to source/header file, so you can find where it is originally declared.
To get the exact result when the file is compiled, you may need to add all other parameters used to compile the file, like -I, -D, etc. In fact, I always copy a result compilation command line, and add -E -dD to the beginning, and add (or change) -o in case I accidental overwrite anything.
There is gccxml, but I am not aware of tools that build on top of it. clang and LLVM are suited for such stuff, too; equally, I am not aware of standalone tools that build on them.
Apart from that: QtCreator and code::blocks can find the declartion, too.
So what is it about a "full fledged IDE" you don't want? If its a little speed, I found netbeans somewhat usefull when I was in school, but really for power and speed and general utility I would like to reccomend emacs. It has key board shortcuts for things like this. Keep in mind, its a learning curve to be sure, but once you are over the hump there is no going back.
I wanted to do some regular expressions in C++ so I looked on the interwebz (yes, I am an beginner/intermediate with C++) and found this SO answer.
I really don't know what to choose between boost::regex and boost::xpressive. What are the pros/cons?
I also read that boost::xpressive opposed to boost::regex is a header-only library. Is it hard to statically compile boost::regex on Linux and Windows (I almost always write cross-platform applications)?
I'm also interested in comparisons of compile time. I have a current implementation using boost::xpressive and I'm not too content with the compile times (but I have no comparisons to boost::regex).
Of course I'm open for other suggestions for regex implementations too. The requirements are free (as in beer) and compatible with http://nclabs.org/license.php.
One fairly important difference is that Boost Regex can support linking to ICU for Unicode support (character classes, etc) Boost Regex ICU Support.
As far as I can tell, Boost Xpressive doesn't have this kind of support built-in.
Well if you need to create a regular expression at runtime (i.e. Letting the user type in a regular expression to search for) you can't use xpressive as it is compile time only.
On the other hand, since it is a compile-time construct, it should benefit more from your optimizer than regex does.
I do enough stuff with Boost.MPL, StateChart, and Spirit that 220KB of compiler warning and errors don't really bother me much. If that sounds like hell to you, stick with Boost.Regex.
If you do use xpressive, I highly recommend turning on -Wfatal-errors as this will stop compilation (and further errors) after the first 'error:' line.
For compilation time, it's no contest. Boost.Regex will be faster*. The fact that xpressive uses MPL will cause compile times to be dramatically increased.
*This assumes you only build the dll/so once
When using the Boost libraries I tend to lean toward the use of header only libraries, due to cross platform compatability issues. The down side of that is that when your compiler reports an error related to your use of the the library, the header only output tends toward the arcane.
Assuming you're using a reasonably recent compiler, there's a pretty decent chance that it includes a regex package already. Try just doing #include <regex> and see if the compiler finds it.
The only trick to things is that it could be in either (or both) of two different namespaces. Regexes were included in TR1 of the C++ standard, and are also in (the final drafts of) C++11. The TR1 version is in a namespace named tr1, where the standard version is in std, just like the rest of the library.
FWIW, this is essentially the same as Boost regex, not Boost Xpressive.
I would try to supplement other people answers by get deeper into topic of compile-time regular expressions(CTR) vs run-time(dynamic) regular expressions(RTR) in a more theoretical way(this topic is implied by OP question indirectly IMHO). Run-time regex are more known and popular(most language core-libraries implementations), i suppose due to historical reasons. They are OK when regular expression is determined at run-time, unlike CTR. Both work on finite state machine basis.
RTR are "compiled" and interpreted by some kind of universal finite state machine(universal means its kind of interpreter which scheme is given at run-time, "compiled" in some internal data structure - when you pass regex string, then interpreted at run-time).
But CTR is "compiled" at compile-time and are specific for particular regex, so you can't use them, when regex is given at run-time(applications like text editors, file/internet search engines).
But they are a priori more efficient(theoretically however) as customized in compile-time finite state machine will be efficient, than interpreter with table-preset scheme of this machine(some similar cases are reflection field access vs compile-time access, or specialized function optimized for some fixed parameter as pointed out there). Another advantage is compile-time syntax checking. CTR can be implemented through meta-programming and/or code generation.
As for specific implementations - there are many RTR, but not so numerous CTR. For C++ they are above mentioned Boost and STL C++0x11 implementations. You may need them for optimizing regex perfomance/size of generated code/memory usage, mostly relevant for embedded systems or high perfomance specific applications.
SO question about CTR
Finding CTR-implementations is harder, one example if found is Re2C Code generator project, Java CTR implementation and C# implementation featuring run-time compilation(into IL code, not internal data structure) of Regex [there is SO question about it]
P.S. Sorry, couldn't post some relevant links due to reputation
Many times when I am watching others code I just want to find where and how a variable is defined. Normally what I do now is look for the type of the variable until I find the definition, that is very time consuming. And I guess that there are some tools that can help me in this rutinary situation. Any suggestion in some tools or commands to help me in this task?.
I know that using a GUI and creating a project this is done automatically I am talking of a way to do this without a GUI. I am working with only text mode. I am running under Linux and I am using C/C++, but suggestions for other languages are welcome.
Thanks a lot.
A possible solution
Michel in one of his comments propose a simple an effective solution define again the variable, in that case in compilation time, the compiler will inform where is the previous definiton. Of course to apply this solution we need to think previously in the locality of the variable.
You've already given the most appropriate tool: an IDE. This is exactly the kind of thing which an IDE excels at. Why would you not want to use an IDE if you're finding development painful without one?
Note that Emacs, Vim etc can work as IDEs - I'm not talking about forcing you the world of GUIs if you want to stay in a text-only situation, e.g. because you're SSHing in.
(I'm really not trying to be rude here. I just think you've discounted the obvious solution without explaining why.)
Edit: OK, you say you're using C++. I'm editing my response. I would use the C preprocessor and then grep for the variable. It will appear in the first place.
cpp -I...(preprocessor options here) file.cpp | grep variable
The C preprocessor will join all the includes that the program uses, and the definition has to be before any usage of that variable in the file. Not a perfect thing, but without an IDE or a complete language description/managing tool, you only have the text.
Another option would be using ctags. It understands the C and C++ syntaxes (among others), and can be searched for variables and functions using command line tools, emacs and vi, among others.
I use cscope and ctags-exuberant religiously. Run it once on my code base and then in Vim, I can use various commands like ^] or [D or [I or similar to find any definitions or declarations for a given word.
This is similar to facilities provided by mega-IDEs like Visual Studio and Eclipse.
Cscope also functions as a stand-alone tool that performs these searches.
I use one of three methods:
I will use CTags to process my source tree (nightly) and then can easily use commands in Vim (or other editors) to jump right to the definition.
I will just use grep (linux) or findstr (windows) to look for all occurrences of the variable name or type. The definition is usually quite obvious.
In Vim, you can just search backward in the scope and often find what you are looking for.
Grep for common patterns for variable declarations. Example: *, &, > or an alphanumeric followed by one or more whitespace characters then the name of the variable. Or variable name followed by zero or more whitespace characters, then a left parenthesis or a semicolon. Unless it was defined under really weird circumstances (like with some kind of macro), it works every time.
In VIM you can use gd to see local variable declarations or gD to see global variable declarations, if they're defined in the current file. Reference Go_to_definition_using_g
You can also use [i to see the definition without jumping to it, or [I to see all occurrences of the variable in all the included files as well, which will naturally show the definition as well.
If you work in Microsoft Visual Studio (which I think you could use for C++ as well, but would require working on a Windows workstation) there's an easily accessible right-click menu option for "Go to Definition...", which will take you to the definition of any currently marked variable, type or method.
if you insist on staying text mode, you can do this with either emacs or vi with the appropriate plug-ins.
But really, move into the 21st century.
EDIT: You commented that you are doing this over SSH because you need the build speed of the remote server cluster.
In that case, mount the drive on your local machine and use an IDE, and just SSH in to kick off a build.