I have a strange request. We're in a position where we need to run a clojure based service in between our python3.3 worker and a database. I'm looking into different ways to interface between python and clojure for this reason. I looked at Thrift, but unfortunately it's doesn't have python3 support yet. 0mq also looked interesting but I'm worried about the req/req pattern blocking the python3 threads (there will be multiple threads on multiple processes needing to use this proxy service).
Are there any other existing libraries in existence that could help me out here? At the end of the day I could write my own service from scratch using aleph and raw sockets, but I feel like that would be re-inventing the wheel too much.
Clojure means JVM. Most languages running on the JVM allow "calling Java" and be "called by Java" which basically means anything else on the JVM. I've never done it, but you can probably call Clojure from Jython. Now Jython doesn't support the Python 3 language, but it does support Pyro.
It's a bit far-fetched but you could try:
JVM
Python-VM Jython Clojure
( Your app -> Pyro ) --> ( Pyro -> Proxy -> Your app )
Arguments of the RMI must probably be translated to Java primitives & Java strings. It might also require some ClassLoader-Voodoo.
I am using ZeroMQ in Python in production quite happily so far; there is Clojure support as well, which I have not really used yet -- it requires native libraries which can be a bit inconvenient. I am going with your ZeroMQ reference even though it's not a full RPC mechanism because often RPC problems can be broken down fairly simply as operations on one or more message queues.
Meanwhile, there is also clojure-py, which would allow you to have your Python and eat your Clojure too. It is still quite young and not without rough edges but most of the core functionality is in place.
Finally, there is always XML-RPC for which both Clojure and Python have libraries, if you just want something simple to glue your Python and Clojure together.
Related
From here:
Storm was designed from the very beginning to be compatible with multiple languages. Nimbus is a Thrift service and topologies are defined as Thrift structures. The usage of Thrift allows Storm to be used from any language.
I see that a topology created in java get deployed by serializing the topology (spouts, bolts , ComponentCommon) as a Thrift datatypes and then gets deployed on Nimbus. In Java it is easy to serialize the object with its methods and data. So on the other side Nimbus just needs to create objects and invoke them. (i might be missing detail here but I hope I got the point correctly)
But I wonder how to write the topology in C++ and deploy it the same way. Does thrift help to serialize the c++ based topology and Nimbus deploys/executes the topology in the same way as for Java?
I have seen the links link1 link2 in this regard and the only solution seems to be using a Shelbolt. which invokes the process and communicates with it over standard i/o.
In order to use the Thrift way, do we need to rewrite the storm core also in C++? Also why use Thrift when it supports only JVM languages? Thrift doesn't seem to be used at all for languages like python/c++.
I'm not sure if I'm understand your question correctly -- in my understanding you're asking Is it possible [without the Shebolt hack] to use Storm [with Thrift as comm protocol] with C++-written bolts and with C++ as the language that creates the topology.
Because of the lack of other answers to this question and based on my own research I assume there is no finished, usable implementation for your problem.
Therefore if you really have to use Storm (its common usecase is the JVM, so even if it could theoretically work with any language, it doesn't mean there is an ecosystem for other languages) and C++, you have no option but to use the Shebolt hack or modify Thrift yourself.
As you know, thrift itself has also been ported to C++. Therefore it is possible to re-build the API calls in C++. Basically, you'd need to port the Java TopologyBuilder. On the C++ side, you could start with the Thrift C++ tutorial.
This is also some kind of a hack, as you basically just rebuild half of the stack (in this case ontop of Thrift), but in general you have very few other options with a system design like Storm.
For example, the MySQL binary protocol has been rebuilt from-scr
Unless anyone has done the work for you (which I would have completely missed in my research) I see no option than to do it yourself (maybe even storm is not the best tool for your usecase!?)
If another hack (which might be even more complex and maybe even slower) besides ShellBolt is good enough for you, you could try starting a JVM from inside C++, e.g. see this SO post. I would not recommend this.
If you need an alternative distributed task queue, I have had good experience with Celery in Python environments, however I have no experience in using it in C++ directly (I usually control Python with ZeroMQ, or write my own ZeroMQ-based queues where necessary, but this is no universal solution).
I know that many large-scale applications such as video games are created using multiple langages. For example, it's likely the game/physics engines are written in C++ while gameplay tasks, GUI are written in something like Python or Lua.
I understand why this division of roles is done; use lower-level languages for tasks that require extreme optimization, tweaking, efficiency and speed, while using higher-level languages to speed up production time, reduce nasty bugs ect.
Recently, I've decided to undertake a larger personal project and would like to divy-up parts of the project similar to above. At this point in time, I'm really confused about how this interoperability between languages (especially compiled vs interpreted) works.
I'm quite familiar with the details of going from ANSCII code test to loading an executable, when written in something like C/C++. I'm very curious at how something like a video game, built from many different languages, works. This is a large/broad question, but specifically I'm interested in:
How does the code-level logic work? I.e. how can I call Python code from a C++ program? Especially since they don't support the same built-in types?
What does the program image look like? From what I can tell, a video game is running in a single process, so what does the runtime image look like when running a C/C++ program that calls a Python function?
If calling code from an interpreted language from a compiled program, what are the sequence of events that occur? I.e If I'm inside my compiled executable, and for some reason have a call to an interpreted language inside a loop, do I have to wait for the interpreter every iteration?
I'm actually finding a hard time finding information on what happening at the machine-level, so any help would be appreciated. Although I'm curious in general about interoperation of software, I'm specifically interested in C++ and Python interaction.
Thank you very much for any insight, even if it's just pointing me to where I can find more information.
In the specific case of python, you have basically three options (and this generally applies across the board):
Host python in C++: From the perspective of the C++ programme, the python interpreter is a C library. On the python side, you may or may not need to use something like ctypes to expose the C(++) api.
Python uses C++ code as DLLs/SOs - C++ code likely knows nothing of python, python definitely has to use a foreign function interface.
Interprocess communication - basically, two separate processes run, and they talk over a socket. These days you'd likely use some kind of web services architecture to accomplish this.
Depending on what you want to do:
Have a look at SWIG: http://www.swig.org/ It's a tool that aims to connect C/C++ code with Python, Tcl, Perl, Ruby, etc. The common use case is a Python interface (graphical or not) that will call the C/C++ code. SWIG will parse the C/C++ code in order to generate the interfaces.
Libpython: it's a lib that allows you to embed Python code. You have some examples here: http://docs.python.org/3.0/extending/embedding.html
If there are parts of my application which are very memory intensive and algorithmic, I would want to implement these in C or Java. What would be the best way to connect this code to my Django/(Rails, Merb etc.) web application?
Have you tried finding/writing a message/instruction queue processing interface for your C/Java code? That way you can pass jobs from your webapp to the message/instruction queue, where your C/Java code would pick up messages, and process them.
For Django, you have the option of using something like Celery.
It depends on how you want to interact with it and what you are most comfortable with. I usually wrap the functionality up in a nice HTTP interface so I can access it from Python (or any other language). Pretty much every language has HTTP libraries and it's pretty easy to make a small REST API. Once it's available over HTTP, it also allows you to use standard HTTP tools for debugging, caching, and load-balancing. For heavier duty processing that can benefit from a queue, Celery/RabbitMQ are powerful and RabbitMQ can be interfaced with from quite a few languages. ZeroMQ can also be a very flexible and lightweight approach.
As an aside, if you the algorithmically intense parts of your application can be formulated into vector operations, check out NumPy. You may not have to leave Python
For Python talking to C/C++, you can use:
Extending Python with C or C++
Boost
The idea in the above 2 solutions is that you will write your memory intensive code in C/C++ and then invoke them from within Python.
My question is pretty straightforward: What are the features of an application that force the developer to exploit OSGI-like system? What are the cases, when such a utility is unneeded?
I just posted the following link to another question (What does OSGi solve), but I think here it is an even better fit. Hal Hildebrand wrote a blog about why and when he thinks OSGi is helping your development/product.
The core essence of the (pretty long) post is that it helps especially when dealing with long running, complex enterprise projects. In particular in the long run, the benefit of having an enforced structure will pay out, but read for yourselves
EDIT: fixed broken link.
Use OSGI if you want some of:
a very modular code structure (extreme case : each install has a different set of modules/plugins)
to start/stop/upgrade a module without stopping the rest of the application
With OSGi you can exchange bundles (manifested jars) in a java program without stopping the java process. This simple thing adds lots of flexibility, but at the same time adds complexity.
It seems nowadays OSGi has lost its relevance, especially in the enterprise and cloud native world. It was designed in the early 2000s to support use cases around embedded devices.
I want to move various parts of my app into simple scripts, to allow people that do not have a strong knowledge of c++ to be able to edit and implement various features.
Because it's a real time app, I need to have some kind of multitasking for these scripts. Ideally I want it so that the c++ app calls a script function which then continues running (under the c++ thread) until either a pause point (Wait(x)), or it returns. In the case of it waiting the state needs to be saved ready for the script to be restarted the next time the app loops after the duration has expired.
The scripts also need to be able to call c++ class methods, ideally using the c++ classes rather than plain wrapper functions around c++ classes.
I don't want to spend a massive amount of time implementing this, so using an existing scripting language is preferred to writing my own. I heard that Python and Lua can be integrated into a c++ app, but I do not know how to do this to achieve my goals.
The scripts must be able to call c++ functions
The scripts must be able to "pause" when certain functions are called (eg. Wait), and be restarted again by the c++ thread
Needs to be fast -- this is for a real time app and there could potentially be a lot of scripts running.
I can probably roll the multitasking code fairly easily, provided the scripts can be saved and restarted (possibly by a different thread to the original).
You can use either Lua or Python. Lua is more "lightweight" than python. It's got a smaller memory footprint than python does and in our experience was easier to integrate (people's mileage on this point might vary). It can support a bunch of scripts running simultaneously. Lua, at least, supports stopping/starting threads in the manner you desire.
Boost.python is nice, but in my (limited) experience, it was difficult for us to get compiling for our different environments and was pretty heavyweight. It has (in my opinion) the disadvantage of requiring Boost. For some, that might not be a problem, but if you don't need Boost (or are not using it), you are introducing a ton of code to get Boost.python working. YMMV.
We have built Lua into apps on multiple platforms (win32, Xbox360 and PS3). I believe that it will work on x64. The suggestion to use Luabind is good. We wound up writing our own interface between the two and while not too complicated, having that glue code will save you a lot of time and perhaps aggravation.
With either solution though, debugging can be a pain. We currently have no good solution for debugging Lua scripts that are embedded into our app. Since we haven't used python in our apps I can't speak to what tools might be available there, but a couple of years ago the landscape was roughly the same -- poor debugging. Having scripting to extend functionality is nice, but bugs in the scripts can cause problems and might be difficult to locate.
The Lua code itself is kind of messy to work with if you need to make changes there. We have seen bugs in the Lua codebase itself that were hard to track down. I suspect that Boost::Python might have similar problems.
And with any scripting language, it's not necessarily a solution for "non-programmers" to extend functionality. It might seem like it, but you will likely wind up spending a fair amount of time either debugging scripts or even perhaps Lua.
That all said, we've been very happy with Lua and have shipped it in two games. We currently have no plans to move away from the language. All in all, we've found it better than other alternatives that were available a couple of years ago. Python (and IronPython) are other choices, but based on experience, they seem more heavy handed than Lua. I'd love to hear about other experiences there though.
I can highly recommend that you take a look at Luabind. It makes it very simple to integrate Lua in your C++ code and vice versa. It is also possible to expose whole C++ classes to be used in Lua.
Your best bet is to embed either lua (www.lua.org) or python (www.python.org). Both are used in the game industry and both access extern "C" functions relatively easily with lua having an edge here (because data types are easier to translate between lua and C). Interfacing to C++ objects will be a bit more work on your end, but you can look up how to do this on Google, or on lua or python discussion forums.
I hope that helps!
You can definitely do what you want with Python. Here are the docs on embedding Python into an application. I'm pretty sure Lua would work too, I'm just less familiar with it.
You're describing cooperative multi-tasking, where the script needs to call a Break or Wait function periodically. Perhaps a better solution would be to run the scripting language in its own thread, and then use mutexes or lock-free queues for the interfaces between the scripting language and the rest of your program. That way a buggy script that doesn't call Break() often enough can't accidentally freeze your program.
Take a look at the Boost.Python library. It looks like it should be fairly straightforward to do what you want.
Take a look at SWIG. I've used it to interface with Python, but it supports many other languages.
One more vote for Lua. It's small, it's fast, it doesnt consume much memory (for games your best bet is to allocate big buffer at the initialization and re-direct all Lua memory allocations there). We used tolua to generate bindings, but there are other options, most of them much smaller/easier to use (IMO) than boost.python.
As for debugging Lua (if you go that route), I have been using DeCoda, and it has not been bad. It pretends to be an IDE, but sorta fails at that, but you can attach The debugging process to visual studio, and go down the call stack at break points. Very handy for Tracking down that bug.
You can also embed C/C++ scripts using Ch. I've been using it for a game project I'm working on, and it does well. Nice blend of power and adaptability.