I've started long ago to work on a dynamic graph visualizer, editor and algorithm testing platform (graphs with nodes and arcs, not the other kinds).
For the algorithm testing platform i need to let the user write a script or call a script from a file, which will interact with the graph currently loaded. The visualizer would do things like light up nodes while they're being visited by the script algorithm, adding some artificial delay, in order to visualize the algorithm navigating and doing stuff.
Scripts would also be secondly used to add third party features that i could either make available as pre-existing scripts in the program folder OR just integrate inside the program in c++ once they're tested and working.
All my searches for an interpreter to embed in my program sent me to lua;
then i started handwriting my own recursive descent parser for my own C-like syntax scripting language (which i planned to use a subset of C++ grammar so that any code written in my scripting language can be copy-pasted in any C++ code.
It was an interesting crazy idea which i don't regret at all, I have scopes, functions, cycles, gotos, typesafe variables, expressions.
But now that i'm approaching the addition of classes, class methods, inheritance (some default classes would be necessary to interface scripts to the program), i realized it's going to take A LOT of time and effort. A bit too much for a personal project of an ungraduated student with exams to study for… but still i whish to complete this project.
The self-imposed requirement of the scripts being 100% compatible with C++ was all but necessary, it would have been just a little nice extra thing, which i can do without.
Now the question is, is there an alternative to lua with a c-like syntax that supports all i've already done plus classes and inheritance? (being able to add custom "classes" that interface scripts to the program is mandatory)
(i can't assume the user to have a full c++ compiler installed so i cant just compile their "script" at runtime as a dll to load and call it, although i whish i could)
Just-in-time compilation of C++
Parsing C++ is hard. Heck, parsing C is hard. It's difficult to get it right, and there are a lot of edge cases. Thankfully, there are a few libraries out there which can take code and even compile it for you.
libclang
libclang provides a lot of facilities for parsing c++. It's a good, clean library, and it'll parse anything the clang compiler itself will parse. This article here is a good starter
libclang provides a JIT compilation tool that allows you to write and compile C++ at runtime. See this blog post here for a overview of what it does and how to use it. It's very general, very powerful, and user-written code should be fast.
GCC also provides a library called libgccjit for just-in-time compilation during the runtime of a program. libgccjit is a C library, but there's also a C++ wrapper provided by the library maintainers. It can compile abstract syntax trees and link them at runtime, although it's still in Alpha mode.
cppast
If you don't want to use libclang, there's also a library under development called cppast, which is a C++ parser which will give you an abstract syntax tree representation of your c++ code. Unfortunately, it won't parse function bodies.
Other tools
If anyone knows any other libraries for compiling or interpreting C++ at runtime, I encourage them to update this post, or comment them so I can update it!
Here is something that lets you embed a C-like scripting language in your application (and a bunch of other cool things):
http://chaiscript.com/
There is lots of documentation:
https://codedocs.xyz/ChaiScript/ChaiScript/
Related
My question is a little bit stupid but I decided to ask advanced programmers like some of you. So I want to make a "dynamic" C++ program. My idea is to compile it and after compilation (maybe with scripting language like python) to change some how the code of the program. I know you will tell me that after the compilation I can not change the code but is there a way of doing that. Thank you!
You could design a modular architecture using plugins in external libraries. For example you could use the Command Pattern. That way you can dynamically load code that was generated after you main program. You would have to fix an interface though. Functions like GetProcAddress in the Windows api might be a good point to start.
For dynamic coding and rapid prototyping I recommend to have a look at Lua. The engine is very small and easy to integrate in your c++ program.
The only way to do that in C++ is to unload the DLL with the code to be
modified, modify the sources, invoke the compiler to regenerate the DLL,
and reload the DLL. It's very, very heavy weight, and it only works if
the compiler is present on the machines where the code is to be run.
(Usually the case under Unix, rarely the case with Windows.)
Interpreted languages like Python are considerably more dynamic; Python
has a built-in function to execute a string as Python code, for example.
If you need dynamically modifiable code, I'd suggest embedding Python in
your application, and using it for the dynamic parts.
Personally I have never played with re-compiling C++ during runtime, and I do not intend too. However I have been doing a lot of embedding of scripting languages lately.
Someone else mentioned the obvious first choice for embedding: Lua. Lua is a pretty popular language to embed and you will find a bunch of documentation about how to do it. Integrating it into the C++ will allow you to define behavior at runtime like you want.
What I am using is a wonderful langauge called Squirrel. It is a lot like Lua but with native object(class) support and C++-like syntax. I have managed to embed it into a C++ application, and through using the sqrat binding library both languages can share information easily.
I have squirrel building and initializing my UI. As a result, 0 compiling is required in order to craft and structure my UI for my game. I intend to take this a step further and use this gameplay-side in order to create similar behavior you are looking for(changing behavior at runtime through dynamic code usage)
I recommend Checking out squirrel here: http://www.squirrel-lang.org/
I plan on writing tutorials on how to embed squirrel and install the binding library, but I have not started on them yet. If I can remember, I will edit this post in the future (could be a few months) once I have completed them. In the meantime give it a shot yourself. You may find it to your liking.
I came here to ask this question because this site has been very useful to me in the past, seems to have very knowledgeable users who are willing to discuss a question even if it is metaphysical at times. And also because googling it did not work.
Java has a compiler and then it has a JDT library that can compile java on the fly (for example used in JasperReports to turn a report template into Java code).
My question: Does anyone know of a library/project that would offer compiling as a set of library classes in c/c++. For example: a suite of classes to perform Preprocessing, Parsing, CodeOptimization and of course Binary rendering to executable images such as ELF or Win format. Basically something that would allow one to compile c or c++ scriptlets as part of an application.
Yes: llvm. In particular, clang. At least, that's how they advertise the projects. Also, check this question. It might be relevant if you decide to use llvm.
You might be able to adapt something from the LLVM project to your needs.
You can just require that a compiler be installed, then call it. This is a fairly hefty requirement, but about the only way to truly "embed" C or C++. There are interpreters that you may be able to embed, but that currently seems a poor choice, not the least because any libraries used in the script must have development versions (i.e. headers and source/compiled-libraries) installed, and those libraries could be restricted to the feature set supported by the interpreter (e.g. quality of template implementation).
You're better off using a language like Python or Lua to embed.
There is the ch interpreter but I have not used it. Generally for scripting type applications a more natural scripted language is used.
Great. It looks like LLVM is what I was after.
Thanks a lot for your feedback.
I am not primarily after C++ as a scripting language. I have noticed that Python is used as an embedded script engine.
My primary reason is two fold:
Get rid off Make,CMake and the hell that is Autoconf and replace it with something like Scons that binds into and interacts with all phases of compiling
Hook into the compiling process after parsing and auto generate code. Specificaly meta related code. In my case I have been able to implement almost every feature of Java in C++ except one: Reflection.
Why impose on your code uneeded bload like RTTI that is often inadequate. Instead one could selectively generate added features. But developer would have to choice when and how to use this extra code.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I'm getting a little rusty in scripting languages, provided they're popping like mushrooms lately :)
Today I thought that it would be nice to have a scripting language that talks seamlessly to C++, that is, could use C++ classes, and, the most important for me, could be compiled into C++ or some DLL/.SO (plus its .h) so that I could link it into my C++ program and make use of the classes the script defines or implements.
I know I could embed any popular scripting language such as lua, ruby, python... but the interface usually includes some kind of "eval" function that evaluates the provided scripting code. Depending on the tool used to couple C++ and the scripting language, the integration for callbacks of the script into C++ could be more or less easy to write, but I haven't seen any scripting language that actually allows me to write independent modules that are exposed as a .h and .so/dll to my program (maybe along the lines of a scripting language that generates C++ code).
Do you know any such tool/scripting language?
Thanks in advance.
PD. I've been thinking along the lines of Vala or Haskell's GHC. They generate C, but not C++...
UPDATE 2020: Today I would probably go with Lua + Sol2/3 except if I really want to avoid Lua as a language. Chaiscript becomes a good candidate in this case though it is not optimal performance-wise compared to Lua+Sol2/3 (though it was greatly improved through years so it is still good enough in many cases).
Falcon have been dead for some years, RIP.
The following ones are more C++ integration oriented than language bindings :
ChaiScript - trying at the moment in a little project, interesting, this one is MADE with C++ in mind and works by just including a header! Not sure if it's good for a big project yet but will see, try it to have some taste!
(not maintained anymore) Falcon - trying on a big project, excellent; it's not a "one include embed" as ChaiScript but it's because it's really flexible, and totally thought to be used in C++ (only C++ code in libs) - I've decided to stick with it for my biggest project that require a lot of scripting flexibility (comparable to ruby/python )
AngelScript - didn't try yet
GameMonkey - didn't try yet
Io - didn't try yet
For you, if you really want to write your scripting module in C++ and easily expose it to the scripting language, I would recommand going with Falcon. It's totally MADE in C++, all the modules/libraries are written that way.
The question usually asked in this context is: how do I expose my C++ classes so they can be instantiated from script? And the answer is often something like http://www.swig.org/
You're asking the opposite question and it sounds like you're complicating matters a bit. A scripting engine that produced .h files and .so files wouldn't really be a scripting engine - it would be a compiler! In which case you could use C++.
Scripting engines don't work like that. You pass them a script and some callbacks that provide a set of functions that can be called from the script, and the engine interprets the script.
Try lua: http://www.lua.org/
For using C++ classes in lua you can use:
To generate binding use tolua++: http://www.codenix.com/~tolua/
It takes a cleaned up header as input and outputs a c file that does the hard work. Easy, nice and a pleasure to work with.
For using Lua objects in C++ I'd take the approach of writing a generic Proxy object with methods like (field, setField, callMethod, methods, fields).
If you want a dll you could have the .lua as a resource (in Windows, I don't know what could be a suitable equivalent for Linux) and on your DllMain initialize your proxy object with the lua code.
The c++ code can then use the proxy object to call the lua code, with maybe a few introspection methods in the proxy to make this task easier.
You could just reuse the proxy object for every lua library you want to write, just changing the lua code provided to it.
This is slightly outside my area of expertise, but I'm willing to risk the downvotes. :-)
Boost::Python seems to be what you're looking for. It uses a bit of macro magic to do its stuff, but it does expose Python classes to C++ rather cleanly.
I'm the author of LikeMagic, a C++ binding library for the Io language. (I am not the author of Io.)
http://github.com/dennisferron/LikeMagic
One of my explicit goals with LikeMagic is complete and total C++ interoperability, in both directions. LikeMagic will marshal native Io types as C++ types (including converting between STL containers and Io's native List type) and it will represent C++ classes, methods, fields, and arrays within Io. You can even pass a block of Io code out of the Io environment and use it in C++ as a functor!!
Wrapping C++ types up for consumption in Io script is simple, quick and easy. Accessing script objects from C++ does require an "eval" function like you described, but the template based type conversion and marshaling makes it easy to access the result of executing a script string. And there is the aforementioned ability to turn Io block() objects into C++ functors.
Right now the project is still in the early stages, although it is fully operational. I still need to do things like document its build steps and dependencies, and it can only be built with gcc 4.4.1+ (not Microsoft Visual C++) because it uses C++0x features not yet supported in MSVC. However, it does fully support Linux and Windows, and a Mac port is planned.
Now the bad news: Making the scripts produce .h files and .so or .dll files callable from C++ would not only require a compiler (of a sort) but it would also have to be a JIT compiler. That's because (in many scripting languages, but most especially in Io) an object's methods and fields are not known until runtime - and in Io, methods can even be added and removed from live objects! At first I was going to say that the very fact that you're asking for this makes me wonder if perhaps you don't really understand what a dynamic language is. But I do believe in a way of design in which you first try to imagine the ideal or easiest possible way of doing something, and then work backwards from there to what is actually possible. And so I'll admit from an ease-of-use standpoint, what you describe sounds easier to use.
But while it's ideal, and just barely possible (using a script language with JIT compilation), it isn't very practical, so I'm still unsure if what you're asking for is what you really want. If the .h and .so/.dll files are JITted from the script, and the script changes, you'd need to recompile your C++ program to take advantage of the change! Doesn't that violate the main benefit of using script in the first place?
The only way it is practical would be if the interfaces defined the scripts do not change, and you just are making C++ wrappers for script functions. You'd end up having a lot of C++ functions like:
int get_foo() { return script.eval("get_foo()"); }
int get_bar() { return script.eval("get_bar()"); }
I will admit that's cleaner looking code from the point of view of the callers of the wrapper function. But if that's what you want, why not just use reflection in the scripting language and generate a .h file off of the method lists stored in the script objects? This kind of reflection can be easily done in Io. At some point I plan to integrate the OpenC++ source-to-source translator as a callable library from LikeMagic, which means you could even use a robust C++ code generator instead of writing out strings.
You can do this with Lua, but if you have a lot of classes you'll want a tool like SWIG or toLua++ to generate some of the glue code for you.
None of these tools will handle the unusual part of your problem, which is to have a .h file behind which is hidden a scripting language, and to have your C++ code call scripts without knowing that that are scripts. To accomplish this, you will have to do the following:
Write the glue code yourself. (For Lua, this is relatively easy, until you get into classes, whereupon it's not so easy, which is why tools like SWIG and toLua++ exist.)
Hide behind the interface some kind of global state of the scripting interpreter.
Supposing you have multiple .h files that each are implemented using scripts, you have to decide which ones share state in the scripting language and which ones use separate scripting states. (What you basically have is a VM for the scripting language, and the extremes are (a) all .h files use the same VM in common and (b) each .h file has its own separate, isolated VM. Other choices are more complicated.)
If you decide to do this yourself, writing the glue code to turn Lua tables into C++ classes (so that Lua code looks like C++ to the rest of the program) is fairly straightforward. Going in the other direction, where you wrap your C++ in Lua (so that C++ objects look to the scripts like Lua values) is a big pain in the ass.
No matter what you do, you have some work ahead of you.
Google's V8 engine is written in C++, I expect you might be able to integrate it into a project. They talk about doing that in this article.
Good question, I have often thought about this myself, but alas there is no easy solution to this kind of thing. If you are on Windows (I guess not), then you could achieve something like this by creating COM components in C++ and VB (considering that as a scripting language). The talking happens through COM interfaces, which is a nice way to interop between disparate languages. Same holds for .NET based languages which can interop between themselves.
I too am eager to know if something like this exists for C++, preferably open source.
You might check into embedding Guile (a scheme interpreter) or V8 (Google's javascript interpreter - used in Chrome - which is written in C++).
Try the Ring programming language
http://ring-lang.net
(1) Extension using the C/C++ languages
https://en.wikibooks.org/wiki/Ring/Lessons/Extension_using_the_C/C%2B%2B_languages
(2) Embedding Ring Interpreter in C/C++ Programs
https://en.wikibooks.org/wiki/Ring/Lessons/Embedding_Ring_Interpreter_in_C/C%2B%2B_Programs
(3) Code Generator for wrapping C/C++ Libraries
https://en.wikibooks.org/wiki/Ring/Lessons/Code_Generator_for_wrapping_C/C%2B%2B_Libraries
I have fairly large C++ library with several sub-libraries that support it, and I need to turn the whole thing into a python extension. I'm using distutils because it needs to be cross-platform, but if there's a better tool I'm open to suggestions.
Is there a way to make distutils first compile the sub-libraries, and link them in when it creates an extension from the main library?
I do just this with a massive C++ library in our product. There are several tools out there that can help you automate the task of writing bindings: the most popular is SWIG, which has been around a while, is used in lots of projects, and generally works very well.
The biggest thing against SWIG (in my opinion) is that the C++ codebase of SWIG itself is really rather crufty to put it mildly. It was written before the STL and has it's own semi-dynamic type system which is just old and creaky now. This won't matter much unless you ever have to get stuck in and make some modifications to the core (I once tried to add doxygen -> docstring conversion) but if you ever do, good luck to you! People also say that SWIG generated code is not that efficient, which may be true but for me I've never found the SWIG calls themselves to be enough of a bottleneck to worry about it.
There are other tools you can use if SWIG doesn't float your boat: boost.python is also popular and could be a good option if you already use boost libraries in your C++ code. The downside is that it is heavy on compile times since it is pretty much all c++ template based.
Both these tools require you to do some work up-front in order to define what will be exposed and quite how it will be done. For SWIG you provide interface files which are like C++ headers but stripped down, and with some extra directives to tell SWIG how to translate complex types etc. Writing these interfaces can be tedious, so you may want to look at something like pygccxml to help you auto-generate them for you.
The author of that package actually wrote another extension which you might like: py++. This package does two things: it can autogenerate binding definitions that can then be fed to boost.python to generate python bindings: basically it is the full solution for most people. You might want to start there if you no particulrly special or difficult requirements to meet.
Some other questions that might prove useful as a reference:
Extending python - to swig or not to swig
SWIG vs CTypes
Extending Python with C/C++
You may also find this comparison of binding generation tools for Python handy. As Alex points out in the comments though, its rather old now but at least gives you some idea of the landscape...
In terms of how to drive the build, you may want to look at a more advanced built tool that distutils: if you want to stick with Python I would highly recommend Waf as a framework (others will tell you SCons is the way to go, but believe me it's slow as hell: I've been there and back already!)...it takes a little learning, but when you get your head around it is extremely powerful. And since it's pure Python it will integrate perfectly with any other Python code you have as part of your build process (say for example you use Py++ in the end)...
A few semesters back I had a class where we wrote a very rudimentary scheme parser and eventually an interpreter. After the class, I converted my parser into a C++ parser that did a reasonably good job of parsing C++ as long as I didn't do anything fancy with the preprocessor or macros. I could use it to read over my classes and functions and do neat things like automatically generate class readers or writers or set up function callbacks from a text file.
However, my program is pretty limited. I'm sure I could spend some time to make it more robust and do more neat things, but I don't want to spend the time and effort if there are already more robust tools available that do the same thing. I figure there has to be something like this out there since parsers are an essential part of compilers, but I haven't seen tools specifically for automatic code generation that make it easy to go through and play with data structures that represent classes, functions and variables for C++ specifically. Are there tools that do this?
Edit:
Hopefully this will clarify a little bit of what I'm looking for. The program I have runs as a prebuild step in visual studio. It reads over my source files, makes a list of classes, their members, their functions, etc. which is then used to generate new code. Currently I just use it to make it easy to read and write my data structures to a plain text file, but I could do other things as well. The file readers and writers are output into plain .cpp and .h files which I include in the rest of my project just as I would any other file. What I'm looking for are tools that do similar things so I can decide if I should continue to use my own or switch to a some better solution. I'm not looking for anything that generates machine code or edits code that I've written.
A complete parser-building tool like ANTLR or YACC is necessary if you want to parse C++ from scratch, but it's overkill for your purposes.
It reads over my source files, makes a list of classes, their members, their functions, etc. which is then used to generate new code.
Two main options:
GCC-XML can generate a list of classes, members, and functions. The distribution version on their web site is quite old; try the CVS version instead. I don't know about the availability of a Windows port.
Doxygen is designed for producing documentation, but it can also produce an XML output, which you should be able to use to do what you want.
Currently I just use it to make it easy to read and write my data structures to a plain text file...
This is known as serialization. Try Boost.Serialization or maybe libs11n or Google Protocol Buffers. Stack Overflow has further discussion.
...but I could do other things as well.
Other cool applications of this kind of automatic code generation include reflection (inspecting your objects' members at runtime, using duck typing with C++, etc.) and generating wrappers for calling C++ from scripting languages. For a C++ reflection library, see Reflex. For an example of generating wrappers for scripting languages, see Boost.Python or SWIG.
The C++ FAQ Lite has references to YACC grammars for C++. YACC is an old-school parser that was used to generate parser output, clumsy and difficult to learn but very powerful. Nowadays, you'd use Gnu Bison instead of YACC.
Don't forget about Cog. It requires you to know Python. In essence it embeds the output of Python scripts into your code. It's absurdly easy to use, but it takes a totally different approach from things like ANTLR and its purpose is somewhat different.
Maybe Boost::Serialize or ANTLR?
I answered a similar question (re splitting source files into separate header and cpp files) by suggesting the use of lzz.
lzz has a very powerful C++ parser that builds a representation for everything except the bodies of functions. As long as you don't need the contents of the function bodies you you could modify 'lzz' so that it performs the generation step you want.
If you want tools that can parse production C++ code, and carry out arbitrary analyses and transformations, see our DMS Software Reengineering Toolkit and its C++ front end.
It would be straightforward to use the information DMS can provide about C++ code, its structures, types, instances, to generate such access functions. If you wanted to generate access functions in another language, DMS provides means to code transformations from the input language (in this case, C++) to that target language.
Mozilla developed Pork for this kind of thing. I can't say it's easy to use (or even to build), but it is in production.
I've already used professionally the Nvelocity engine combined with C# as a prevoius step to coding, with very good results.