I am loading a 123 MB file of unsigned integers that needs to be in memory (for fast look ups for a monte carlo simulation) in C++. Right now I have a global array but i've heard global arrays are frowned upon. What are the best practices for this?
For context, I'm doing Monte Carlo simulations on a poker game and need an array of about 30 million integers to quickly compute the winner of a poker hand. To determine the winner, you first compute the 'handranks' by doing 7 queries of the array. Then to determine the winner, you compare the 'handranks'.
int HR[32487834];
int get_handrank(const std::array<int,7> cards)
{
int p = 53;
for (const auto& c: cards)
p = HR[p + c];
return p;
}
int main()
{
// load the data
memset(HR, 0, sizeof(HR));
FILE * fin = fopen("handranks.dat", "rb");
if (!fin)
std::cout << "error when loading handranks.dat" << std::endl;
size_t bytesread = fread(HR, sizeof(HR), 1, fin);
fclose(fin);
std::cout << "complete.\n\n";
// monte carlo simulations using get_handrank() function
.
.
.
}
Use local variables, and pass them to functions as appropriate. This makes the program easier to reason about.
Use vector instead of an array for this large amount of data, otherwise you might cause stack overflow.
Modify your functions to work with std::span instead of a particular container, as this creates more decoupling.
Create a symbolic constant with a meaningful name for 32487834 instead of using it as a magic constant.
Using local variables and passing them around is always a better practice than having globals IMO. It makes your algorithms more flexible. What if you need to use a different array for some reason later on? You would need to modify all the functions using the global variable. Passing around the array is a bit inconvenient and verbose I agree, but still better than the globals. We will address this problem later on.
So your first option is something like:
// Don't forget to receive the params as references to avoid copying
int get_handrank(const std::array<int,7>& cards, const std::array<int, 32487834>& HR)
{
int p = 53;
for (const auto& c: cards)
p = HR[p + c];
return p;
}
int main()
{
std::array<int, 32487834> HR{}; //zero-inits the array
// or std::vector<int> HR{}; HR.resize(32487834) to avoid stack overflow as #MarkRansom pointed out
FILE * fin = fopen("handranks.dat", "rb");
if (!fin)
std::cout << "error when loading handranks.dat" << std::endl;
size_t bytesread = fread(HR.data(), HR.size() * sizeof(int), 1, fin);
fclose(fin);
}
Second approach, even better: Use classes. You can have a simulation class, and the class can have the HR array as a const private member read and initialized in the constructor. Then get_handrank can be a member function and can access the member HR array:
class Simulation {
public:
int get_handrank(const std::array<int,7>& cards)
{
int p = 53;
for (const auto& c: cards)
p = HR[p + c];
return p;
}
Simulation()
: HR{}
// or HR{readHRFromFileFunction()}
{
FILE * fin = fopen("handranks.dat", "rb");
if (!fin)
std::cout << "error when loading handranks.dat" << std::endl;
size_t bytesread = fread(HR.data(), HR.size() * sizeof(int), 1, fin);
fclose(fin);
}
private:
std::array<int, 32487834> HR;
//or const std::array<int, 32487834> HR; if you use a function to init it
}
Related
I am using the sys/shm library to share memory between two applications.
It works well and as I want it to, however I am unsure of the library is entirely safe in the way I currently have it implemented.
What would happen if a value is changed at the same time the other application tries to read it. Could this even happen in the first place? I dont know.
Here is a simplified version of my two applications:
Send:
int main(int argc, char* argv[])
{
int Delta_x{ atoi(argv[1]) }, Delta_y{ atoi(argv[2]) }, Delta_z{ atoi(argv[3]) };
int shmid;
int* array;
int count = 3;
int i = 0;
int SizeMem;
key_t key = ftok("shmfile", 66);
SizeMem = sizeof(*array) * count;
shmid = shmget(key, count * sizeof(int), IPC_CREAT);
array = (int*)shmat(shmid, 0, 0);
array[0] = Delta_x;
array[1] = Delta_y;
array[2] = Delta_z;
shmdt((void*)array);
return 0;
}
Recieve:
int* Recieve;
int x{ 0 }, y{ 0 }, z{0};
key_t key = ftok("shmfile", 66);
int shmid = shmget(key, 3 * sizeof(int), IPC_EXCL);
Recieve = (int*)shmat(shmid, (void*)0, SHM_RDONLY);
while (Condition)
{
x = Recieve[0];
y = Recieve[1];
z = Recieve[2];
std::cout << "x:" << x << ", y:" << y << ", z:" << z << std::endl;
usleep(50000);
}
shmdt((void*)Recieve);
Is this implementation safe?
It is not safe.
I mean if you put your write code in a loop and the run the send code,the read code may be executed after the first write and before the last write.
It according the thread scheduler.
But your data in share memory are integer values. It can be write and read in one instruction,so the value your read from memory is always a value you wrote in.
And reading/writing a valid integer would never make a program crash.
The situation is exactly the same as in multithreading. To safely read/write data a safety mechanism like mutex is required.
Boost.Interprocess library has both various mutexes, semaphores, and condition variables. Just don't expect them to be efficient as most OSs support for interprocessing has always been very limited.
This was an interview question:
Say there is a class having only an int member. You do not know how many bytes the int will occupy. And you cannot view the class implementation (say it's an API). But you can create an object of it. How would you find the size needed for int without using sizeof.
He wouldn't accept using bitset, either.
Can you please suggest the most efficient way to find this out?
The following program demonstrates a valid technique to compute the size of an object.
#include <iostream>
struct Foo
{
int f;
};
int main()
{
// Create an object of the class.
Foo foo;
// Create a pointer to it.
Foo* p1 = &foo;
// Create another pointer, offset by 1 object from p1
// It is legal to compute (p1+1) but it is not legal
// to dereference (p1+1)
Foo* p2 = p1+1;
// Cast both pointers to char*.
char* cp1 = reinterpret_cast<char*>(p1);
char* cp2 = reinterpret_cast<char*>(p2);
// Compute the size of the object.
size_t size = (cp2-cp1);
std::cout << "Size of Foo: " << size << std::endl;
}
Using pointer algebra:
#include <iostream>
class A
{
int a;
};
int main() {
A a1;
A * n1 = &a1;
A * n2 = n1+1;
std::cout << int((char *)n2 - (char *)n1) << std::endl;
return 0;
}
Yet another alternative without using pointers. You can use it if in the next interview they also forbid pointers. Your comment "The interviewer was leading me to think on lines of overflow and underflow" might also be pointing at this method or similar.
#include <iostream>
int main() {
unsigned int x = 0, numOfBits = 0;
for(x--; x; x /= 2) numOfBits++;
std::cout << "number of bits in an int is: " << numOfBits;
return 0;
}
It gets the maximum value of an unsigned int (decrementing zero in unsigned mode) then subsequently divides by 2 until it reaches zero. To get the number of bytes, divide by CHAR_BIT.
Pointer arithmetic can be used without actually creating any objects:
class c {
int member;
};
c *ptr = 0;
++ptr;
int size = reinterpret_cast<int>(ptr);
Alternatively:
int size = reinterpret_cast<int>( static_cast<c*>(0) + 1 );
I'm Java user coming over to C++, and I am having a hard time understanding what is going wrong with this statement. My program has been segfaulting anywhere I put the push_back command. So I'm wondering what exactly is going on.
class Process {
public:
int nice;
int arrivalTime;
int cpuBursts;
list<int> burstList;
Process() {
burstList.push_back(10); // Segfaults here...
}
};
Here is the full code:
#include<iostream>
#include<stdlib.h>
#include<fstream>
#include<list>
#include<string.h>
using namespace std;
int calcTimeslice(int priority);
int calcOriginalPrio(int nice);
int readFile(int ,char **);
int calcPrioBonus(int,int);
void tokenizeAndAdd(char *);
class Bursts {
public:
int isCPUBurst;
int time;
Bursts() {}
// Constructor to make it easier to add to list
Bursts(int tempIsCPU, int tempTime) {
isCPUBurst = tempIsCPU;
time = tempTime;
}
};
class Process {
public:
int nice;
int arrivalTime;
int cpuBursts;
list<int> burstList;
Process() {
burstList.push_back(10);
}
};
int main(int arg, char **argv) {
// This is if the file was not correctly read into the program
// or it doesnt exist ...
if(readFile(arg,argv)==-1) {
cout << "File could not be read. \n";
return -1;
}
//cout << "Original Calc Whatever: " << calcOriginal(19) << '\n';
return 0;
}
/*
* Calculates the timeslice based on the priority
*/
int calcTimeslice(int priority) {
double finalCalc;
// This is the given function in the prompt
finalCalc = ( (1 - (priority / 140)) * 290 + (.5) ) + 10;
// Cast to int, this will be a truncate
return ((int)finalCalc);
}
int readFile(int arg, char **argv) {
char *temp,*pointer;
int endOfFile = 1;
// While its not the end of the file
while(endOfFile) {
// Read in the input from stdin
fgets(temp,256,stdin);
// Check to see if this line had a * in it
if(*temp =='*')
endOfFile = 0;
else
tokenizeAndAdd(temp);
}
return 0;
}
void tokenizeAndAdd(char *string) {
char *token = strtok(string," \n");
int i = 0;
Process p;
while(token != NULL) {
cout << token << endl;
if(i>2) { // If it is odd (CPU burst)
if(i%2 == 1) {
int tempInt = atoi(token);
//p.burstList.push_back(tempInt);
}
else { // If it is even (IO burst)
int tempInt = atoi(token);
//p.burstLis.push_back(tempInt);
}
}
else if(i==0)
p.nice = atoi(token);
else if(i==1)
p.arrivalTime = atoi(token);
else if(i==2)
p.cpuBursts = atoi(token);
token = strtok(NULL," \n");
i++;
}
//cout << p.nice << " " << p.arrivalTime << " " << p.cpuBursts << "\n";
//i = 0;
//cout << p.burstList.size() << "\n";
// cout <<
//}
return;
}
/*
* Calculates and returns the original priority based on the nice number
* provided in the file.
*/
int calcOriginalPrio(int nice) {
double finalCalc;
// This is the given function from the prompt
finalCalc = (( nice + 20 ) / 39 ) * 30 + 105.5;
// Cast to int, this is a truncate in C++
return ((int)finalCalc);
}
/*
* Calculates the bonus time given to a process
*/
int calcPrioBonus(int totalCPU, int totalIO) {
double finalCalc;
// How to calculate bonus off of the prompt
if(totalCPU < totalIO)
finalCalc = ( (1 - (totalCPU / (double)totalIO)) * (-5)) - .5;
else
finalCalc = ( (1 - (totalIO / (double)totalCPU)) * 5) + .5;
// Cast to int
return ((int)finalCalc);
}
You are using temp uninitialized in the following code:
char *temp;
...
while(endOfFile) {
fgets(temp,256,stdin);
...
This can have any side effect, since it most likely destroys your stack or parts of the heap memory. It could fail immediately (when calling the fgets() function), it could fail later (as in your sample) or it could even run fine - maybe until you upgrade your OS, your compiler or anything else, or until you want to run the same executable on another machine. This is called undefined behaviour.
You need to allocate space for the temp variable, not a pointer only. Use something like
char temp[256];
...
while(endOfFile) {
fgets(temp,256,stdin);
...
For more information, see the fgets() documentation. The first parameter is a pointer to a char array - that is where fgets() will store the bytes which have been read. In your code, you pass an uninitialized pointer which means that fgets() will store the bytes to an undefined memory location - this is catched by the OS which terminates your application with a segmentation fault.
BTW: You should consider enabling pedantic warnings when compiling - I compiled with
g++ -Wall -pedantic -o list list.cpp
which gave me the following warning:
list.cpp: In function 'int readFile(int, char**)':
list.cpp:76:26: warning: 'temp' may be used uninitialized in this function [-Wuninitialized]
This is probably not the actual code with the error you report. But here is one of the problems with give you UB.
char *temp,*pointer; // uninicialized pointer char temp[1000]; could work?
int endOfFile = 1;
// While its not the end of the file
while(endOfFile) {
// Read in the input from stdin
fgets(temp,256,stdin);
The last function call will read a maximum of 256 bytes from stdin and will write it in the memory pointed by pointer tmp. So, you need to first "prepare" that memory. But with char *tmp; you only define a pointer, with no defined value, that is, with point to some possible unexisting or illegal/inaccessible for you memory. In contrary, char tmp[1000]; will define in the "stack memory" a block of 1000 bytes, with you can point to using simple the variable tmp. Hope this is clear for you.
EDIT:
I don't know why that would change the behavior of the list,
You are right. That is Undefined Behavior (UB). When you write in some unknown memory (pointed by an uninitialized pointer) you may overwrite data or even code that will broke somewhere the correct function of your program in an unpredicted way.
You will need to learn more about pointers but better you use std::string, and look how parse your file using string and stringstream. That will manage for you the memmory,
I've a class that consists basically of a matrix of vectors: vector< MyFeatVector<T> > m_vCells, where the outer vector represents the matrix. Each element in this matrix is then a vector (I extended the stl vector class and named it MyFeatVector<T>).
I'm trying to code an efficient method to store objects of this class in binary files.
Up to now, I require three nested loops:
foutput.write( reinterpret_cast<char*>( &(this->at(dy,dx,dz)) ), sizeof(T) );
where this->at(dy,dx,dz) retrieves the dz element of the vector at position [dy,dx].
Is there any possibility to store the m_vCells private member without using loops? I tried something like: foutput.write(reinterpret_cast<char*>(&(this->m_vCells[0])), (this->m_vCells.size())*sizeof(CFeatureVector<T>)); which seems not to work correctly. We can assume that all the vectors in this matrix have the same size, although a more general solution is also welcomed :-)
Furthermore, following my nested-loop implementation, storing objects of this class in binary files seem to require more physical space than storing the same objects in plain-text files. Which is a bit weird.
I was trying to follow the suggestion under http://forum.allaboutcircuits.com/showthread.php?t=16465 but couldn't arrive into a proper solution.
Thanks!
Below a simplified example of my serialization and unserialization methods.
template < typename T >
bool MyFeatMatrix<T>::writeBinary( const string & ofile ){
ofstream foutput(ofile.c_str(), ios::out|ios::binary);
foutput.write(reinterpret_cast<char*>(&this->m_nHeight), sizeof(int));
foutput.write(reinterpret_cast<char*>(&this->m_nWidth), sizeof(int));
foutput.write(reinterpret_cast<char*>(&this->m_nDepth), sizeof(int));
//foutput.write(reinterpret_cast<char*>(&(this->m_vCells[0])), nSze*sizeof(CFeatureVector<T>));
for(register int dy=0; dy < this->m_nHeight; dy++){
for(register int dx=0; dx < this->m_nWidth; dx++){
for(register int dz=0; dz < this->m_nDepth; dz++){
foutput.write( reinterpret_cast<char*>( &(this->at(dy,dx,dz)) ), sizeof(T) );
}
}
}
foutput.close();
return true;
}
template < typename T >
bool MyFeatMatrix<T>::readBinary( const string & ifile ){
ifstream finput(ifile.c_str(), ios::in|ios::binary);
int nHeight, nWidth, nDepth;
finput.read(reinterpret_cast<char*>(&nHeight), sizeof(int));
finput.read(reinterpret_cast<char*>(&nWidth), sizeof(int));
finput.read(reinterpret_cast<char*>(&nDepth), sizeof(int));
this->resize(nHeight, nWidth, nDepth);
for(register int dy=0; dy < this->m_nHeight; dy++){
for(register int dx=0; dx < this->m_nWidth; dx++){
for(register int dz=0; dz < this->m_nDepth; dz++){
finput.read( reinterpret_cast<char*>( &(this->at(dy,dx,dz)) ), sizeof(T) );
}
}
}
finput.close();
return true;
}
A most efficient method is to store the objects into an array (or contiguous space), then blast the buffer to the file. An advantage is that the disk platters don't have waste time ramping up and also the writing can be performed contiguously instead of in random locations.
If this is your performance bottleneck, you may want to consider using multiple threads, one extra thread to handle the output. Dump the objects into a buffer, set a flag, then the writing thread will handle the output, releaving your main task to perform more important tasks.
Edit 1: Serializing Example
The following code has not been compiled and is for illustrative purposes only.
#include <fstream>
#include <algorithm>
using std::ofstream;
using std::fill;
class binary_stream_interface
{
virtual void load_from_buffer(const unsigned char *& buf_ptr) = 0;
virtual size_t size_on_stream(void) const = 0;
virtual void store_to_buffer(unsigned char *& buf_ptr) const = 0;
};
struct Pet
: public binary_stream_interface,
max_name_length(32)
{
std::string name;
unsigned int age;
const unsigned int max_name_length;
void load_from_buffer(const unsigned char *& buf_ptr)
{
age = *((unsigned int *) buf_ptr);
buf_ptr += sizeof(unsigned int);
name = std::string((char *) buf_ptr);
buf_ptr += max_name_length;
return;
}
size_t size_on_stream(void) const
{
return sizeof(unsigned int) + max_name_length;
}
void store_to_buffer(unsigned char *& buf_ptr) const
{
*((unsigned int *) buf_ptr) = age;
buf_ptr += sizeof(unsigned int);
std::fill(buf_ptr, 0, max_name_length);
strncpy((char *) buf_ptr, name.c_str(), max_name_length);
buf_ptr += max_name_length;
return;
}
};
int main(void)
{
Pet dog;
dog.name = "Fido";
dog.age = 5;
ofstream data_file("pet_data.bin", std::ios::binary);
// Determine size of buffer
size_t buffer_size = dog.size_on_stream();
// Allocate the buffer
unsigned char * buffer = new unsigned char [buffer_size];
unsigned char * buf_ptr = buffer;
// Write / store the object into the buffer.
dog.store_to_buffer(buf_ptr);
// Write the buffer to the file / stream.
data_file.write((char *) buffer, buffer_size);
data_file.close();
delete [] buffer;
return 0;
}
Edit 2: A class with a vector of strings
class Many_Strings
: public binary_stream_interface
{
enum {MAX_STRING_SIZE = 32};
size_t size_on_stream(void) const
{
return m_string_container.size() * MAX_STRING_SIZE // Total size of strings.
+ sizeof(size_t); // with room for the quantity variable.
}
void store_to_buffer(unsigned char *& buf_ptr) const
{
// Treat the vector<string> as a variable length field.
// Store the quantity of strings into the buffer,
// followed by the content.
size_t string_quantity = m_string_container.size();
*((size_t *) buf_ptr) = string_quantity;
buf_ptr += sizeof(size_t);
for (size_t i = 0; i < string_quantity; ++i)
{
// Each string is a fixed length field.
// Pad with '\0' first, then copy the data.
std::fill((char *)buf_ptr, 0, MAX_STRING_SIZE);
strncpy(buf_ptr, m_string_container[i].c_str(), MAX_STRING_SIZE);
buf_ptr += MAX_STRING_SIZE;
}
}
void load_from_buffer(const unsigned char *& buf_ptr)
{
// The actual coding is left as an exercise for the reader.
// Psuedo code:
// Clear / empty the string container.
// load the quantity variable.
// increment the buffer variable by the size of the quantity variable.
// for each new string (up to the quantity just read)
// load a temporary string from the buffer via buffer pointer.
// push the temporary string into the vector
// increment the buffer pointer by the MAX_STRING_SIZE.
// end-for
}
std::vector<std::string> m_string_container;
};
I'd suggest you to read C++ FAQ on Serialization and you can choose what best fits for your
When you're working with structures and classes, you've to take care of two things
Pointers inside the class
Padding bytes
Both of these could make some notorious results in your output. IMO, the object must implement to serialize and de-serialize the object. The object can know well about the structures, pointers data etc. So it can decide which format can be implemented efficiently.
You will have to iterate anyway or has to wrap it somewhere. Once you finished implementing the serialization and de-serialization function (either you can write using operators or functions). Especially when you're working with stream objects, overloading << and >> operators would be easy to pass the object.
Regarding your question about using underlying pointers of vector, it might work if it's a single vector. But it's not a good idea in the other way.
Update according to the question update.
There are few things you should mind before overriding STL members. They're not really a good candidate for inheritance because it doesn't have any virtual destructors. If you're using basic data types and POD like structures it wont make much issues. But if you use it truly object oriented way, you may face some unpleasant behavior.
Regarding your code
Why you're typecasting it to char*?
The way you serialize the object is your choice. IMO what you did is a basic file write operation in the name of serialization.
Serialization is down to the object. i.e the parameter 'T' in your template class. If you're using POD, or basic types no need of special synchronization. Otherwise you've to carefully choose the way to write the object.
Choosing text format or binary format is your choice. Text format has always has a cost at the same time it's easy to manipulate it rather than binary format.
For example the following code is for simple read and write operation( in text format).
fstream fr("test.txt", ios_base::out | ios_base::binary );
for( int i =0;i <_countof(arr);i++)
fr << arr[i] << ' ';
fr.close();
fstream fw("test.txt", ios_base::in| ios_base::binary);
int j = 0;
while( fw.eof() || j < _countof(arrout))
{
fw >> arrout[j++];
}
It seems to me, that the most direct root to generate a binary file containing a vector is to memory map the file and place it in the mapped region. As pointed out by sarat, you need to worry about how pointers are used within the class. But, boost-interprocess library has a tutorial on how to do this using their shared memory regions which include memory mapped files.
First off, have you looked at Boost.multi_array? Always good to take something ready-made rather than reinventing the wheel.
That said, I'm not sure if this is helpful, but here's how I would implement the basic data structure, and it'd be fairly easy to serialize:
#include <array>
template <typename T, size_t DIM1, size_t DIM2, size_t DIM3>
class ThreeDArray
{
typedef std::array<T, DIM1 * DIM2 * DIM3> array_t;
array_t m_data;
public:
inline size_t size() const { return data.size(); }
inline size_t byte_size() const { return sizeof(T) * data.size(); }
inline T & operator()(size_t i, size_t j, size_t k)
{
return m_data[i + j * DIM1 + k * DIM1 * DIM2];
}
inline const T & operator()(size_t i, size_t j, size_t k) const
{
return m_data[i + j * DIM1 + k * DIM1 * DIM2];
}
inline const T * data() const { return m_data.data(); }
};
You can serialize the data buffer directly:
ThreeDArray<int, 4, 6 11> arr;
/* ... */
std::ofstream outfile("file.bin");
outfile.write(reinterpret_cast<char*>(arr.data()), arr.byte_size());
I have a function similar to the following that takes an integer as a parameter, and using this integer I want to to access a specific variable.
Of course, I can just casebash using the case/switch statement, or if/else, but this is far too dirty and I'm guessing there must be some form of elegant solution.
static unsigned char frame1[] = {...};
static unsigned char frame2[] = {...};
/*...*/
static unsigned char frame30[] = {...};
int select_frame(int frame_number){
/* How can I use frame_number to select the correct frame variable without case/switch, if at all? */
}
I feel as though there could potentially be a stringstream method (or similar), though I'm not sure.
Any suggestions? I'm happy to see solutions in either of C or C++.
EDIT: I feel I should note that most of these arrays will have over 500 individual elements in them, and thus it becomes a little impractical to combine them all into one giant array. If it's possible, I'm looking for a solution that avoids this method, though I'm currently seriously considering it.
If you want to access one of the frames based on an integer, then simply put the frames into an array:
static unsigned char* frames[30];
frames[0] = (unsigned char*)strdup("first frame's char");
frames[1] = (unsigned char*)strdup("second frame's char");
Then you can directly and efficiently index to frames[i].
In C++, a more flexible approach is to use a std::vector<std::string> (as long as your numbers are contiguous) or even a std::map<int, std::string> (good if you have numbers with gaps, or want to insert/erase single elements often during runtime.
EDIT: helper function to create string literal source code from existing unsigned char arrays (off-the-top-of-my-head, tweaks may be needed):
void f(std::ostream& os, unsigned char* p)
{
os << '"';
for ( ; *p; ++p)
{
if (*p < ' ' || *p >= 127)
os << '\\' << std::setw(3) << std::setfill('0') << std::oct << *p;
else if (*p == '\\')
os << "\\\\";
else
os << *p;
}
os << '"';
}
Then just call f(frame1); f(frame2); etc...
I don't think that's possible in C or C++. If you can, why not have a
static unsigned char frame[30][]
You can use 2D array and make it a single frames array variable. Or, if you are ok with using char array then you can use std::string.
static string frames[30] = { "frame1 string", "frame2 string", ..., "frame30 string" };
int select_frame (int frame_number)
{
string &check = frames[frame_number];
}
If your C compiler is C99 conforming, and the array is defined outside
functions, compound literal like the following might meet the purpose.
static unsigned char *frames[] = {
(unsigned char[]){ 1, 2, ... },
(unsigned char[]){ 3, 4, ... },
...
};
int select_frame(int frame_number){
... frames[ frame_number ] ...
}