My goal is ultimately to execute a python script that manipulates values in my C++ program, one line at a time, returning execution to my C++ program between statements in the python script. Right now, I'm been attempting to feed the python interpreter my script one line at a time. But it wants a context, and I have no idea how to construct that. Can anyone point me to some good tutorials (the documentation is not very good for this).
I chose the answer that most closely answered my question, but I believe this may not be enough control for some applications.
An answer that works for those applications might involve lower-level calls in the Python API. Please answer the question if you have an answer that grants more control over execution.
I Asked another question following this one, because I encountered different problems afterwards which are very closely related. Link: Python C API - Stopping Execution (and continuing it later)
I think it will be difficult to feed a script one line at a time. Look into sys.settrace() to set a function that is invoked at each line of execution. You can also set it in the C API using PyEval_SetTrace, in a slightly different form.
I am not 100% sure that I fully understand your goal, but some 8 months ago I wanted to do something similar. I wanted to be able to drive my c++ application from python scripts. I was on win32, Qt, gcc. It turns out that these days gdb, the debugger for gcc, can be scripted via python. It took me a lot of reading and few days of work, but it worked well. I did not have to add any extra code into my c++ sources!
well i tell you what.. i think the best thing to do is to convert your python script to c++ app. this how you'll have working and useful code to embed in your source.
if your c app is not that big, maybe you prefer to make a c-extension for python and turn the python code to the main app..
either way, i would do it in Cython.
you can use the following question to understand how to do this job if you'd like: Convert Python program to C/C++ code?
Related
Suppose I have a simple Hello, World! file in C++ or C (whatever will help me use it easier in Node.js, preferably C) and want to run it from a Node.js file. What is the most efficient way considering that the file will be used to boost the performance (changing CPU intensive functions from Node.js to C/C++)?
I came across the addons, but it seems to me, that in order to use it, I'll have to convert a lot of code to bring it to that format. Is there an easier way?
I don't see why using child_process would be slower than other options.
I recommend:
// myCFile.c
#include <stdio.h>
int main(void){
// Processor-intensive computations
int x = 1+2+3;
// Send to node via standard output
printf("%d", x);
// Terminate the process
return 0;
}
Compile:
gcc -o myExecutable myCFile.c
And use child_process like this:
// File myNodeFile.js
const { exec } = require("child_process");
exec("./myExecutable", (error, stdout, stderr) => console.log(stdout));
For our image segmentation algorithm that I had written in C++, I needed to help the full-stack developer wrap the shared library for Node.js. As far as I can see, from a day of googling around and hacking into Node.js, which is a somewhat unfamiliar world for me, that there are two major options:
using node-ffi, or,
addons as you have already stated.
For 1. above, you do not need to do much. You simply need to require the ffi, ref and ref-array packages/addons in node.js to be able to call the C API of your application code. There is some nice tutorial that I followed, which helped me get going in 15 minutes.
However, I needed to choose 2. above for our project in the end. This was due to the fact that our full-stack developer was relying on some other addons that needed the latest version of Node.js. Apparently, when we check the issue board of node-ffi, as of this answer's posting time, it does not support the v9.x family of node.js. Hence, I went the native addons way. It has taken me roughly 4 hours to understand and write the code. I am not sure if it is the most convenient/efficient way possible, but what I did was to
use buffers to allocate memory in Node.js,
write a simple addon using nan in Node.js that reinterpret_casts the char* buffer of Node.js and calls the very same C API of our shared library, and finally,
link against the shared library we had created using binding.gyp.
Apparently, Native Abstractions for Node.js (aka nan) is supposed to be used by users to avoid the need to handle breaking changes introduced in v8. There is another nice tutorial I have found, which helped me solve my problem easily.
Finally, Scott Frees' blog site seems to have a lot of self-contained articles/examples for those who would like to go deeper. He also argues in which situations you should be preferring one approach over the other (node-ffi over native addons, for instance). Basically, what I understand is that writing native addons will be more efficient, even though for our application it did not matter much. node-ffi gives satisfactory behaviour, too, as we were solving an image segmentation problem (which anyways takes more time than the call overhead).
So, in short,
I came across the addons but it seems to me, that in order to use it I'll have to convert a lot of code to bring it to that format.
Well, not necessarily! It depends on what you are willing to achieve. It can be as easy as compiling your C++ code for a specific C-API shared library, and then writing a 20-liner wrapper in nan, which basically does some reinterpret_cast for in-place memory operations, and finally linking against the library in binding.gyp.
Is there an easier way?
Yes, there is. node-ffi can help you solve the problem under half an hour. But then, it might not be the most efficient for your scenario, or it might not be a viable option for you, as it currently does not build with the v9.x family of Node.js.
There is an option to compile C/C++ with emscripten to WebAssembly and for quick execution on Node. Calling WebAssembly code from JavaScript is not trivial, but allows more flexibility for input and output parameters than communicating with child process.
I've been using LuaJIT for some times now. The tip of the iceberg was enought for my needs until now, but my recent project require me to dig a little deeper.
My actual knowledge of LuaJIT is making function available from C++ to Lua and from Lua to C++. That include passing parameters, tables and retrieving return values.
This is the model I am used to:
I tried to search around for "scoped environement luajit" and multiple variation of the query, but unfortunately I did not find anything relevant. I might not use the right words?
This is the model I want to achieve :
I want to make a "global script environment" that I will share the C++ functions with then make it available to the "scoped script environments".
//push arguments
luaScopedEnvironment1->call("doSomething");
I just want a starting point, help for the terminology and maybe some pointers to related documentation :)
Thanks you for taking time to read me.
I dont think Lua or LuaJIT supports such a thing but if I'm not mistaken, what you are after is called "sandboxing".
It creates a new environment with which you can strip out or add functionality to. Its handy for removing IO and OS functionality.
I want to make a system were I can run a make file and several other gcc related things within a program, basically to use gcc and compile stuff within the program. If I wrote up all the stuff I want to do to a batch file then I'd need to run that batch file from within the program.
Now everyone says System() calls are extremely bad because of security and various other things. So considering I am using c++ what would be a good alternative to System() to run batch files. If preferable I would like the alternative to cross platform.
Thanks
You could look to use the fork and execl family of calls although these are tied down to Unix / Linux and, depending on how you use them, are arguably no safer than system.
I doubt very much that you'll find a common, cross platform way of doing this if only because all platforms will have different and unique ways of doing this. Also, the scripts you're trying to run will no doubt have to be different on different platforms and there may be different ways of specifying things such as directory paths etc.
My suggestion would be to first ask yourself how you'll take the following questions - which would be my main concerns:
How am I going to prevent accidental / intentional misuse?
How am I going to detect errors or success status within the scripts I'm running?
How am I going to provide for dependencies? E.g. script A must run completely and correctly before script B runs.
How am I going to report the success and failure state.
My final question would be why do you want to do this in C++? Is there a specific reason? Naturally I'm a C++ evangelist although I would have thought this would be better tackled by a scripting language such as Perl, Python or possibly Bash unless you're embarking on something far more radical.
I am trying to optimize the interaction between two scripts I have.
Two things I thought of are the c++ program not terminating unless you manually kill it, or generating all info in python before feeding it to c++.
Explanation of the problem:
What the scripts do:
C++ program (not made by me, and I can't program in c++ very well): takes a 7 number array and returns a single number, simple.
Python script (mine, and I can program a bit in python): generates those 7 number arrays, feeds them to the c++ program, waits for an answer and adds it to a list. It then makes the next array.
In theory, this works. However, as it is right now, it opens and closes the c++ program for each call. For one array that is no problem, but I'm trying to upscale to 25k arrays, and in the future to 6+ million arrays. Obviously it is then no longer feasible to open/close it each time, especially since the c++ program first has to load a 130mb VCD file to function.
Two options I thought of myself were to generate all arrays first in python, then feed them to the c++ program and then analyze all results. However, I wouldn't know how to do this with 6M arrays. It is not important however that the results I get back are in the same order as the arrays I feed in.
Second option I thought of was to make the c++ program not quit after each call. I can't program in c++ though so I don't know if this is possible, keeping it 'alive' so you can just feed arrays into it at times and get an answer.
(Note: I cannot program in anything else than python, and want to do this project in python. The c++ program cannot be translated to python for speed reasons.)
Thanks in advance, Max.
Firstly, just to be pedantic, there are no C++ scripts in normal use. C++ compiles, ultimately, to machine code, and the C++ program is properly referred to as a "program" and not a "script".
But to answer your question, you could indeed set up the C++ program to stay in memory, where it listens for connections and sends responses to your Python script. You'd want to study Unix IPC, particularly sockets.
Another way to approach it would be to incorporate what the C++ program does into your Python script, and forget about C++ altogether.
Without the source code or the exact specifications of the Python script and the C++ program, it's difficult to provide more information, but you could modify the C++ code to repeatedly read the array from the standard input and then write the results to standard output.
Then you could use the Python subprocess module to launch the C++ program from your Python script and communicate with it.
Note that simply wrapping a loop around the main() function of the C++ program will not be very helpful, because apparently the main issue is the time the program needs in order to read its data (the VCD that you mentioned).
The loop needs to be strictly around the code that computes the result - which means that you may have to factor everything else out in a way that allows the result computation to be done repeatedly without each run contaminating the next ones.
Okay, your best course of action is probably to write a C/C++ extension to Python that is able to call the C++ code that does the calculation you want. This is not terribly difficult, it will only require a minimal amount of C/C++ coding to make it work. A good explanation of extending Python can be found on the Python page at http://docs.python.org/extending/extending.html
What you in effect do is change your C++ program to be a dynamic library that the Python process can link in and call from the Python script.
If you need a bit of help getting it to work I'm sure we can help you out.
I think the best way is to build C++ extension module for python.
There are lot of ways to do it.
If you have c++ sources you can try SWIG
After that you can use c++ functions/object directly inside python - and manage them by python modules (here processing). It is really simple.
I think you're doing it wrong
What the scripts do: C++ program (not made by me, and I can't program in c++ very well): takes a 7 number array and returns a single number, simple. Python script (mine, and I can program a bit in python): generates those 7 number arrays, feeds them to the c++ program, waits for an answer and adds it to a list. It then makes the next array.
You have this?
python generate_arrays.py | someC++app | python gather_array.py
This allows you to run the three parts in parallel, using every Core of every CPU on the box. The OS makes sure that all three run concurrently.
If you're still not getting 100% CPU Load, you'll have to do something like this.
( python generate_arrays.py --even | someC++app >oneFile ) & ( python generate_arrays.py --odd | someC++app > anotherFile )
python gather_array.py oneFile anotherFile
That will run two copies of python generate_arrays.py and two copies of your magical C++ program.
You'll have to rewrite your generate_arrays.py program so that it takes a command-line option. When the option is --even, you generate 3 million arrays. When the options is --odd you generate the other 3 million arrays.
This (python | c++) & (python | c++) should get to 100% cpu use.
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
Have you used any of the C++ interpreters (not compilers)?
Hi,
I am currently learning C++ and a beginner in programming in general. I've been trying to write some code to a few programming problems from the book I'm using. What I find is that often I make mistakes in what I write and those mistakes come up when the program is run. Its usually quite obvious where in the program I've gone wrong when there is regular output. But in a long computation I'm often not sure why a particular code has acted a certain way. I've also looked at Python recently. Python works with an interpreter, which can take any piece of Python code and compute its output.
I was wondering if there was something similar for C++. Right now when I want to check a line or block of code I have to comment out a lot, save it, compile it, and then run it from a command line. And I have to do that many times for a single error until I've solved it. Is there a way to type code into an active terminal which would run code and show me output? What would be better still would be a way to select a block of code (like you select text) or multiple blocks (to see how a function is being handled) within the IDE and click run to run just that block of code and see its output without having comment out irrelevant lines or to save the file. The compiled code could just reside in memory.
CINT is a c & C++ interpretter that accepts nearly all valid C++. Unfortunately many Linux distros do not offer it, and you'll probably have to build it from source... and that is a non-trivial task.
Typically a debugger is used to step through code line by line, starting at a chosen breakpoint, and keep watch of all variables/values.
Unit testing is a technique to test smaller pieces of code.
A stepping debugger, as found in most IDEs will help you with this.
Here (for example) is a description of how to set the Execution point in In Visual Studio, which sounds like what you want to do.
For certain situations, the "Immediate Window" may be of use to you. It allows you to type in expressions to evaluate immediately.
Rather than just running individual lines independently, or relying on print statements to tell you the state of whatever variables you have decided to print, you can use the debugger to run to the point of interest (where you will have set a breakpoint), then you can examine the state of any in-scope variables, or even alter the normal flow of the program.
There are some solutions that try to do this - the ones I know are Ch and TextTransformer.
However, I doubt that this works very well. C++ is not at all designed to run as an interpreted language.
One of the problems is that C++ is very, very hard to parse. And this makes it very hard to provide certain types of tools that are usual for other languages. For example, I don't think there is any C++ refactoring tool that really works well.
C++ is a compiled language not like python. But there are few c/c++ interpreters out there but not sure about their features. Check these out: Ch interpreter and CINT
If you really want to learn c++ please do not use the c/c++ interpreters.
If you insist on using a interactive interpreter there is since a long time CINT which is the default interpreter used in the ROOT project. It got better over the years, but still has only limited capabilities when dealing with templates. Also, there is a move to replace it with a JIT compiling interpreter based on clang inside the ROOT project.
If I were you I would learn how to run compiler and an interactive debugger like suggested in some comments already.