I am trying to optimize the interaction between two scripts I have.
Two things I thought of are the c++ program not terminating unless you manually kill it, or generating all info in python before feeding it to c++.
Explanation of the problem:
What the scripts do:
C++ program (not made by me, and I can't program in c++ very well): takes a 7 number array and returns a single number, simple.
Python script (mine, and I can program a bit in python): generates those 7 number arrays, feeds them to the c++ program, waits for an answer and adds it to a list. It then makes the next array.
In theory, this works. However, as it is right now, it opens and closes the c++ program for each call. For one array that is no problem, but I'm trying to upscale to 25k arrays, and in the future to 6+ million arrays. Obviously it is then no longer feasible to open/close it each time, especially since the c++ program first has to load a 130mb VCD file to function.
Two options I thought of myself were to generate all arrays first in python, then feed them to the c++ program and then analyze all results. However, I wouldn't know how to do this with 6M arrays. It is not important however that the results I get back are in the same order as the arrays I feed in.
Second option I thought of was to make the c++ program not quit after each call. I can't program in c++ though so I don't know if this is possible, keeping it 'alive' so you can just feed arrays into it at times and get an answer.
(Note: I cannot program in anything else than python, and want to do this project in python. The c++ program cannot be translated to python for speed reasons.)
Thanks in advance, Max.
Firstly, just to be pedantic, there are no C++ scripts in normal use. C++ compiles, ultimately, to machine code, and the C++ program is properly referred to as a "program" and not a "script".
But to answer your question, you could indeed set up the C++ program to stay in memory, where it listens for connections and sends responses to your Python script. You'd want to study Unix IPC, particularly sockets.
Another way to approach it would be to incorporate what the C++ program does into your Python script, and forget about C++ altogether.
Without the source code or the exact specifications of the Python script and the C++ program, it's difficult to provide more information, but you could modify the C++ code to repeatedly read the array from the standard input and then write the results to standard output.
Then you could use the Python subprocess module to launch the C++ program from your Python script and communicate with it.
Note that simply wrapping a loop around the main() function of the C++ program will not be very helpful, because apparently the main issue is the time the program needs in order to read its data (the VCD that you mentioned).
The loop needs to be strictly around the code that computes the result - which means that you may have to factor everything else out in a way that allows the result computation to be done repeatedly without each run contaminating the next ones.
Okay, your best course of action is probably to write a C/C++ extension to Python that is able to call the C++ code that does the calculation you want. This is not terribly difficult, it will only require a minimal amount of C/C++ coding to make it work. A good explanation of extending Python can be found on the Python page at http://docs.python.org/extending/extending.html
What you in effect do is change your C++ program to be a dynamic library that the Python process can link in and call from the Python script.
If you need a bit of help getting it to work I'm sure we can help you out.
I think the best way is to build C++ extension module for python.
There are lot of ways to do it.
If you have c++ sources you can try SWIG
After that you can use c++ functions/object directly inside python - and manage them by python modules (here processing). It is really simple.
I think you're doing it wrong
What the scripts do: C++ program (not made by me, and I can't program in c++ very well): takes a 7 number array and returns a single number, simple. Python script (mine, and I can program a bit in python): generates those 7 number arrays, feeds them to the c++ program, waits for an answer and adds it to a list. It then makes the next array.
You have this?
python generate_arrays.py | someC++app | python gather_array.py
This allows you to run the three parts in parallel, using every Core of every CPU on the box. The OS makes sure that all three run concurrently.
If you're still not getting 100% CPU Load, you'll have to do something like this.
( python generate_arrays.py --even | someC++app >oneFile ) & ( python generate_arrays.py --odd | someC++app > anotherFile )
python gather_array.py oneFile anotherFile
That will run two copies of python generate_arrays.py and two copies of your magical C++ program.
You'll have to rewrite your generate_arrays.py program so that it takes a command-line option. When the option is --even, you generate 3 million arrays. When the options is --odd you generate the other 3 million arrays.
This (python | c++) & (python | c++) should get to 100% cpu use.
Related
This may be kind of basic but... here goes.
If I decide to embed some kind of scripting language like Lua or Ruby into a C++ program by linking it's interpreter what does that allow me to do in C++ then?
Would I be able to write Ruby or Lua code right into the cpp file or simply call scripts from the program?
If the latter is true, how would I do that?
Because they're scripting languages, the code is always going to be "interpreted." In reality, you aren't "calling" the script code inside your program, but rather when you reach that point, you're executing the interpreter in the context of that thread (the thread that reaches the scripting portion), which then reads the scripting language and executes the applicable machine code after interpreting it (JIT compiling kind of, but not really, there's no compiling involved).
Because of this, its basically the same thing as forking the interpreter and running the script, unless you want access to variables in your compiled program/in your script from the compiled program. To access values to/from, because you're using the thread that has your compiled program's context, you should be able to store script variables on the stack as well and access them when your thread stops running the interpreter (assuming you stored the variables on the stack).
Edit: response:
You would have to write it yourself. Think about it this way: if you want to use assembly in c++, you use the asm keyword. You then in the c++ compiler, need to parse the source file, get to the asm keyword, and then switch to the assembly compiler. Then the assembly compiler needs to go until the end bracket of the asm region and compile this code.
If you want to do this,it will be a bit different, since assembly gets compiled, not interpreted (which is what you want to do). What you'll need to do, is change the compiler you're using (lets say c++), so that it recognizes your own user defined keyword. Lets say this keyword is scriptX{}. You need to change the c++'s parser so that when it see's scriptX{}, it stores everything between the brackets in the readonly data section of your compiled program. You then need to add a hook in the compiled assembly file to switch the context of the thread to your script interpreter, and start the program counter at the beginning of your script section (which you put in read only data section of the object file).
Good luck with that...
A common reason to embed a scripting language into a program is to provide for the ability to control the program with scripts provided by the end user.
Probably the simplest example of such a script is a configuration file. Assume that your program has options, and needs to remember the options from run to run. You could write them out to a file as a binary image of your options structure, but that would be fragile, not easy to inspect or edit, and likely not portable across systems. Writing the options out in plain text with some sort of labels for which is which addresses most of those complaints, but now you need to parse that text and recover the options. Then some users want different options on Tuesdays, want to do simple arithmetic to compute one option from another, or to write one configuration file that they can use on both Windows and Linux, and pretty soon you find yourself inventing a little language to express all of those ideas and mechanisms with. At this point, there's a better way.
The languages Lua and TCL both grew out of essentially that scenario. Larger systems needed to be configured and controlled by end users. End users wanted to edit a simple text file and get immediate satisfaction, even (especially) when working with large systems that might have required hours to compile successfully.
One advantage here is that rather than inventing a programming language one feature at a time as user's needs change, you start with a complete language along with its documentation. The language designer has already made a number of tough decisions for you (how do I represent strings and numbers, what about lists, what about named values, what does if look like, etc.) and has generally also brought a carefully designed and debugged implementation to the table.
Lua is particularly easy to integrate. Reading a simple configuration file and extracting the settings from the Lua state can be done using a small subset of its C API. Once you have Lua available, it is attractive to use it for other purposes. In many cases, you will find that it is more productive to write only the innermost loops in C, and use Lua to glue those functions together and provide all the "business logic" of the application. This is how Adobe Lightroom is implemented, as well as many games on platforms ranging from simple set-top-boxes to iOS devices and even PCs.
This question already has answers here:
Why are Python Programs often slower than the Equivalent Program Written in C or C++?
(11 answers)
Closed 9 years ago.
simply run a near empty for loop in python and in C++ (as following), the speed are very different, the python is more than a hundred times slower.
a = 0
for i in xrange(large_const):
a += 1
int a = 0;
for (int i = 0; i < large_const; i++)
a += 1;
Plus, what can I do to optimize the speed of python?
(Addition:
I made a bad example here in the first version of this question, I don't really mean that a=1 so that C/C++ compiler could optimize that, I mean the loop itself consumed a lot of resource (maybe I should use a+=1 as example).. And what I mean by how to optimize is that if the for loop is just like a += 1 that simple, how could it be run in the similar speed as C/C++? In my practice, I used Numpy so I can't use pypy anymore (for now), is there some general methods for making loop far more quickly (such as generator in generating list)?
)
A smart C compiler can probably optimize your loop away by recognizing that at the end, a will always be 1. Python can't do that because when iterating over xrange, it needs to call __next__ on the xrange object until it raises StopIteration. python can't know if __next__ will have side-effect until it calls it, so there is no way to optimize the loop away. The take-away message from this paragraph is that it is MUCH HARDER to optimize a Python "compiler" than a C compiler because python is such a dynamic language and requires the compiler to know how the object will behave in certain circumstances. In
C, that's much easier because C knows exactly what type every object is ahead of time.
Of course, compiler aside, python needs to do a lot more work. In C, you're working with base types using operations supported in hardware instructions. In python, the interpreter is interpreting the byte-code one line at a time in software. Clearly that is going to take longer than machine level instructions. And the data model (e.g. calling __next__ over and over again) can also lead to a lot of function calls which the C doesn't need to do. Of course, python does this stuff to make it much more flexible than you can have in a compiled language.
The typical way to speed up python code is to use libraries or intrinsic functions which provide a high level interface to low-level compiled code. scipy and numpy are excellent examples this kind of library. Other things you can look into are using pypy which includes a JIT compiler -- you probably won't reach native speeds, but it'll probably beat Cpython (the most common implementation), or writing extensions in C/fortran using the Cpython-API, cython or f2py for performance critical sections of code.
Simply because Python is a more high level language and has to do more different things on every iteration (like acquiring locks, resolving variables etc.)
“How to optimise” is a very vague question. There is no “general” way to optimise any Python program (everythng possible was already done by the developers of Python). Your particular example can be optimsed this way:
a = 1
That's what any C compiler will do, by the way.
If your program works with numeric data, then using numpy and its vectorised routines often gives you a great performance boost, as it does everything in pure C (using C loops, not Python ones) and doesn't have to take interpreter lock and all this stuff.
Python is (usually) an interpreted language, meaning that the script has to be read line-by-line at runtime and its instructions compiled into usable bytecode at that point.
C is (usually) a compiled language, so by the time you're running it you're working with pure machine code.
Python will never be as fast as C, for that reason.
Edit: In fact, python compiles INTO C code at run time, that's why you get those .pyc files.
As you go more abstract the speed will go down. The fastest code is assembly code which is written directly.
Read this question Why are Python Programs often slower than the Equivalent Program Written in C or C++?
My goal is ultimately to execute a python script that manipulates values in my C++ program, one line at a time, returning execution to my C++ program between statements in the python script. Right now, I'm been attempting to feed the python interpreter my script one line at a time. But it wants a context, and I have no idea how to construct that. Can anyone point me to some good tutorials (the documentation is not very good for this).
I chose the answer that most closely answered my question, but I believe this may not be enough control for some applications.
An answer that works for those applications might involve lower-level calls in the Python API. Please answer the question if you have an answer that grants more control over execution.
I Asked another question following this one, because I encountered different problems afterwards which are very closely related. Link: Python C API - Stopping Execution (and continuing it later)
I think it will be difficult to feed a script one line at a time. Look into sys.settrace() to set a function that is invoked at each line of execution. You can also set it in the C API using PyEval_SetTrace, in a slightly different form.
I am not 100% sure that I fully understand your goal, but some 8 months ago I wanted to do something similar. I wanted to be able to drive my c++ application from python scripts. I was on win32, Qt, gcc. It turns out that these days gdb, the debugger for gcc, can be scripted via python. It took me a lot of reading and few days of work, but it worked well. I did not have to add any extra code into my c++ sources!
well i tell you what.. i think the best thing to do is to convert your python script to c++ app. this how you'll have working and useful code to embed in your source.
if your c app is not that big, maybe you prefer to make a c-extension for python and turn the python code to the main app..
either way, i would do it in Cython.
you can use the following question to understand how to do this job if you'd like: Convert Python program to C/C++ code?
I'm currently porting some huge arithmetics from MATLAB to C++ including vast amount of data. I would like to step over C++ code in VS and compare contents of the key arrays with ones from MATLAB code being debugged simultaneously. Since there are many steps, it's very ugly to use some real C++ code for exporting values.
So, is there some convenient way to export content of known memory buffer from C++ to whatever? The only thing I can think of is to copy content of watch window. Some better ideas?
Update: Found >d command in Command Window. Almost fit my needs except 1) disturbing metainfo typed alongside with output, 2) only 10000 lines of output is possible :-|
After all, I was able to use >d command along with slight "packing" of my data to fit more in one string and performed several dumps. Since it was the one-moment task, solution is appropriate.
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
Have you used any of the C++ interpreters (not compilers)?
Hi,
I am currently learning C++ and a beginner in programming in general. I've been trying to write some code to a few programming problems from the book I'm using. What I find is that often I make mistakes in what I write and those mistakes come up when the program is run. Its usually quite obvious where in the program I've gone wrong when there is regular output. But in a long computation I'm often not sure why a particular code has acted a certain way. I've also looked at Python recently. Python works with an interpreter, which can take any piece of Python code and compute its output.
I was wondering if there was something similar for C++. Right now when I want to check a line or block of code I have to comment out a lot, save it, compile it, and then run it from a command line. And I have to do that many times for a single error until I've solved it. Is there a way to type code into an active terminal which would run code and show me output? What would be better still would be a way to select a block of code (like you select text) or multiple blocks (to see how a function is being handled) within the IDE and click run to run just that block of code and see its output without having comment out irrelevant lines or to save the file. The compiled code could just reside in memory.
CINT is a c & C++ interpretter that accepts nearly all valid C++. Unfortunately many Linux distros do not offer it, and you'll probably have to build it from source... and that is a non-trivial task.
Typically a debugger is used to step through code line by line, starting at a chosen breakpoint, and keep watch of all variables/values.
Unit testing is a technique to test smaller pieces of code.
A stepping debugger, as found in most IDEs will help you with this.
Here (for example) is a description of how to set the Execution point in In Visual Studio, which sounds like what you want to do.
For certain situations, the "Immediate Window" may be of use to you. It allows you to type in expressions to evaluate immediately.
Rather than just running individual lines independently, or relying on print statements to tell you the state of whatever variables you have decided to print, you can use the debugger to run to the point of interest (where you will have set a breakpoint), then you can examine the state of any in-scope variables, or even alter the normal flow of the program.
There are some solutions that try to do this - the ones I know are Ch and TextTransformer.
However, I doubt that this works very well. C++ is not at all designed to run as an interpreted language.
One of the problems is that C++ is very, very hard to parse. And this makes it very hard to provide certain types of tools that are usual for other languages. For example, I don't think there is any C++ refactoring tool that really works well.
C++ is a compiled language not like python. But there are few c/c++ interpreters out there but not sure about their features. Check these out: Ch interpreter and CINT
If you really want to learn c++ please do not use the c/c++ interpreters.
If you insist on using a interactive interpreter there is since a long time CINT which is the default interpreter used in the ROOT project. It got better over the years, but still has only limited capabilities when dealing with templates. Also, there is a move to replace it with a JIT compiling interpreter based on clang inside the ROOT project.
If I were you I would learn how to run compiler and an interactive debugger like suggested in some comments already.