How to avoid reloading big data on program start - c++

Question

My question is all about tips and tricks. I'm currently working on project, where I have one very big(~1Gb) file with data. First, I need to extract data. This extraction takes 10 mins. Then I do calculations. Next calculation depends on previous. Let's call them calculation1, calculation2 and so on. Assuming, that I've done extraction part right, I currently face two problems:
Every time I launch program it works 10 mins least. I cannot avoid it, so I have to plan debugging.
For every next calculation it takes more time.
Thinking of first problem, I assumed, that some sort of database may help, if database is faster, than reading file, which I doubt.
Second problem might be overcomed if I split my big program in smaller programs, each of which will do: read file-do stuff-write file. So next stage always can read file from previous, for debug. But it introduces many wasted code for file I/O.
I think that both these problems could be solved by some strategy like: write, and test extract module, than launch it and let it extract all data into RAM. Than write calculation1, and launch it to somehow grab data directly from RAM of extract module. And so on with every next calculation. So my questions are:
Are there tips and tricks to minimize loads from files?
Are there ways to share RAM and objects between programs?
By the way I write this task on Perl because I need it quick, but I'll rewrite it on C++ or C# later, so any language-specific or language-agnostic answers welcome.
Thank you!
[EDIT]
File of data does not change it is like big immutable source of knowledge. And it is not exactly 1Gb and it does not take 10 minutes to read it. I just wanted to say, that file is big and time to read it is considerable. On my machine 1 Gb read+parse file into right objects takes about minute. Which is still pretty bad.
[/EDIT]

Answer 1

On my current system Perl copies the whole 1GB file in memory in 2 seconds. So I believe your problem is not reading the file but parsing it.
So the straightforward solution I can think of is to preparse it by, for instance, converting your data into actual code source. I mean, you can prepare your data and hardcode it in your script directly (using another file of course).
However, if reading is an actual problem (which I doubt) you can use database that store the data in the memory (example). It will be faster anyway just because your database reads the data once upon starting and you don't restart your database as often as your program.

Answer 2

The idea for solving this type of problems can be as follows:
Go for 3 programs:
Reader
Analyzer
Writer
and exchange data between them using shared memory.
For that big file I guess you have considerable amount of data of one object type which you can store in circular buffer in shared memory (I recommend using boost::interprocess).
Reader will continuously read data from the input file and store it in shared memory.
In the meantime, once is enough data read for doing calculations, the Analyzer will start processing it and store results into another circular buffer shared memory file.
Once there are some calculations in the second shared memory the Writer will read them and store them into final output file.
You need to make sure all the processes are synchronized properly so that they do their job simultanouesly and you don't lose the data (the data is not being overwritten before is processed or saved into final file).

Answer 3

I like the answer doqtor gives, but to prevent data from being overwritten, a nice helper class to enable and disable critical sections of code within a thread will do the trick.
// Note: I think sealed may be specific to Visual Studio Compiler.
// CRITICAL_SECTION is defined in Windows.h - If on another OS,
// look for similar structure.
class BlockThread sealed {
private:
CRITICAL_SECTION* m_pCriticalSection;
public:
explicit BlockThread( CRITICAL_SECTION& criticalSection );
~BlockThread();
private:
BlockThread( const BlockThread& c );
BlockThread& operator=( const BlockThread& c ); // Not Implement
};
BlockThread::BlockThread( CRITICAL_SECTION& criticalSection ) {
m_pCriticalSection = &criticalSection;
}
BlockThread::~BlockThread() {
LeaveCriticalSection( m_pCriticalSection
}
A class such as this would allow you to block specific threads if you are within the bounds of a critical section where shared memory is being used
and another thread currently has access to it. This will cause this thread
of code to be locked until the current thread is done its work and this
class goes out of scope.
To use this class with in another class is fairly simple: in the class that you
want to block a thread within its .cpp file you need to create a static variable of this type and call the API's function to initialize it. Then
you can use the BlockThread class to lock this thread.
SomeClass.cpp
#include "SomeClass.h"
#include "BlockThread.h"
static CRITICAL_SECTION s_criticalSection;
SomeClass::SomeClass {
// Do This First
InitializeCriticalSection( &s_criticalSection );
// Class Stuff Here
}
SomeClass::~SomeClass() {
// Class Stuff Here
// Do This Last
DeleteCriticalSection( &s_criticalSection );
}
// To Use The BlockThread
SomeClass::anyFunction() {
// When Your Condition Is Met & You Know This Is Critical
// Call This Before The Critical Computation Code.
BlockThread blockThread( s_criticalSection );
}
And that is about it, once this object goes out of scope the static member
is cleaned up within the objects destructor and when this object goes out
of scope so does the BlockThread class and its Destructor cleans it up there.
And now this shared memory can be used. You would usually want to use this class if you are traversing over containers to either add, insert, or find and access elements when this data is a shared type.
As for the 3 different threads running in memory on the same data set a good concept is to have 3 or 4 buffers each about 4MB in size and have them work in a rotating order. Buff1 gets data then Buff2 gets data, while Buff2 is getting data Buff 1 is either parsing the data it or passing it off to be stored for computation, then Buff1 waits until Buff3 or 4 is done, pending on how many buffers you have. Then this process starts again. This is the same principle that is used with Sound Buffers when reading in sound files for doing an Audio Stream, or sending batches of triangles to a graphics card. Another words it is a Batch type process.