I wanted to read an array of double values from a file to an array. I have like 128^3 values. My program worked just fine as long as I stayed at 128^2 values, but now I get an "segmentation fault" error, even though 128^3 ≈ 2,100,000 is by far below the maximum of int. So how many values can you actually put into an array of doubles?
#include <iostream>
#include <fstream>
int LENGTH = 128;
int main(int argc, const char * argv[]) {
// insert code here...
const int arrLength = LENGTH*LENGTH*LENGTH;
std::string filename = "density.dat";
std::cout << "opening file" << std::endl;
std::ifstream infile(filename.c_str());
std::cout << "creating array with length " << arrLength << std::endl;
double* densdata[arrLength];
std::cout << "Array created"<< std::endl;
for(int i=0; i < arrLength; ++i){
double a;
infile >> a;
densdata[i] = &a;
std::cout << "read value: " << a << " at line " << (i+1) << std::endl;
}
return 0;
}
You are allocating the array on the stack, and stack size is limited (by default, stack limit tends to be in single-digit megabytes).
You have several options:
increase the size of the stack (ulimit -s on Unix);
allocate the array on the heap using new;
move to using std::vector.
Related
Before my program can free up memory and end it crashes. Crashes seem to happen on transition from the function UserDataCollection and back to main. This is only my second program using pointers so I'm still quite the newbie considering the whole point of c++ is to use pointers.
Here is the aforementioned code:
#include <iostream>
//Prototypes
void UserDataCollection(int * &movieData_ptr, int &numSurveyed); // Movie Statistics
void DisplayOutput(int *movieData_ptr, int numSurveyed); //Mean, Median, Mode (Display To Console)
//Global Constants
int main()
{
//Variables
int numSurveyed = 0;
//Pointers
int * movieData_ptr = nullptr;
movieData_ptr = new int[numSurveyed];
//"Program Start"
std::cout << "Program start...\n\n";
UserDataCollection(movieData_ptr, numSurveyed);
DisplayOutput(movieData_ptr, numSurveyed);
//Release Memory
delete[] movieData_ptr;
std::cout << "Memory Cleared.";
return 0;
}
void UserDataCollection(int * &movieData_ptr, int &numSurveyed)
{
//Get Number of Students Surveyed
std::cout << "How many students were surveyed: ";
std::cin >> numSurveyed;
//Student Data Input Loop
for (int i = 0; i < numSurveyed; i++)
{
//Get Student Data
std::cout << "Enter How many movies student " << i + 1 << " has seen in ONE month: ";
std::cin >> *(movieData_ptr + i);
//Validation Check
while (*(movieData_ptr + i) >= 337)
{
std::cout << "\nImpossible value!" << std::endl
<< "Hours in a month: 730. Average movie length: 130 minutes."
<< "Total Possible movies: 337";
std::cout << "\n\nEnter How many movies student " << i + 1 << " has seen in ONE month: ";
std::cin >> *(movieData_ptr + i);
} //end while (Validation Check)
} // end for (Data Input)
}
void DisplayOutput(int *movieData_ptr, int numSurveyed)
{
//Display loop for pointer array
for (int i = 0; i < numSurveyed; i++)
{
std::cout << *(movieData_ptr + i) << " ";
}
//End Message
std::cout << "\n\nProgram end.";
}
You never allocated any memory.
int numSurveyed = 0;
//Pointers
int * movieData_ptr = nullptr;
movieData_ptr = new int[numSurveyed];
This is the equivalent of
int *movieData_ptr = new int[0];
You are allocating size of 0 ints. This is undefined behaviour. You can't do anything useful with that pointer without a segmentation fault. You need to either pre-allocate a certain amount, and make sure you don't overflow, or dynamically allocate every time you plan to add data.
Since this is C++, it's probably better not to use raw pointers, but use vector or something instead.
Sorry:
From 5.3.4/7
When the value of the expression in a direct-new-declarator is zero, the allocation function is called to allocate an array with no elements.
From 3.7.3.1/2
The effect of dereferencing a pointer returned as a request for zero size is undefined.
I'm writing a code where I need to write a large string into memory.
I used a stringstream object to do so, but something odd to me happens: even if the size of the underlying string buffer has not exceeded the maximum size of a string object my program crashes with a BAD_ACCESS error.
I've created a test program like this:
#include <sstream> // std::stringstream
#include <iostream> // std::cout
int main(int argc, const char * argv[]) {
std::stringstream stream;
std::string string;
std::cout << "Max string size: " << string.max_size() << "\n";
for (int i = 0; true; i++) {
if (i >= 644245094) {
stream.seekg(0, std::ios::end);
std::stringstream::pos_type size = stream.tellg();
stream.seekg(0, std::ios::beg);
std::cout << "Size of stringstream: " << size << "\n";
}
stream << "hello";
}
return 0;
}
That if (i >= 644245094) inside the loop is only used to print the size of the stringstream buffer just before the program crashes. I've used my debugger to see which was the number of the last iteration and used it to print the size of the buffer just before the crash happens.
This is the output I get:
Max string size: 18446744073709551599
Size of stringstream: 3221225470
After this the program crashes.
I thought that the cause might be because the program fills up my computer's RAM but the memory used by this program is ~6.01GB so not enough to fill up my RAM. For the record I have a 16GB RAM Macbook Pro.
What could be the problem? Am I missing something about how << operator works?
Thank you in advance!
The behaviour of a std::stringstream when it gets full and fails may not be consistent across all platforms.
I modified your code and rain it on Yocto 3.19.0-32 64-bit, with gcc 5.4.1. I did not get an exception thrown, rather the stream set one of its failure mode bits.
The code I ran was:
#include <sstream> // std::stringstream
#include <iostream> // std::cout
std::stringstream::pos_type get_size(std::stringstream& stream)
{
stream.seekg(0, std::ios::end);
std::stringstream::pos_type size = stream.tellg();
stream.seekg(0, std::ios::beg);
return size;
}
int main(int argc, const char * argv[])
{
std::stringstream stream;
std::string string;
std::cout << "Max string size: " << string.max_size() << std::endl;
std::stringstream::pos_type size;
for (unsigned long i = 0; true; ++i)
{
size = get_size(stream);
stream.write("x", 1);
if (stream.fail())
{
std::cout << "Fail after " << i + 1 << " insertions" << std::endl;
std::cout << "Size of stringstream just before fail: " << size << std::endl;
break;
}
}
size = get_size(stream);
std::cout << "Size of stringstream just after fail: " << size << std::endl;
return 0;
}
And I got the following output, which shows that my stringstream filled and failed 56 bytes short of 8GB:
Max string size: 4611686018427387897
Fail after 8589934536 insertions
Size of stringstream just before fail: 8589934535
Size of stringstream just after fail: -1
Can you not use a different container and pre-allocate the memory, instead of using such a large stringstream?
I am learning how vectors work in c++, and wrote a sample program to try to learn how memory with vectors are handled.
#include <iostream>
#include <vector>
int main()
{
//Test 1:
double n = 3.5;
std::vector<double> test;
std::cout << sizeof(test) << std::endl;
test.push_back(n);
std::cout << sizeof(test) << std::endl;
std::cout << std::endl;
std::cout << std::endl;
std::cout << std::endl;
//Test 2
std::vector<int> test2;
std::cout << sizeof(test2) << std::endl;
for (int i = 0; i < 1000; i++) {
test2.push_back(i);
}
std::cout << sizeof(test2) << std::endl;
}
Interestingly, the program prints out 24 as the number of bytes stored each-time. Despite adding new elements to the vector. How is the amount of memory that the vector occupies when it is initially declared the same as after I have added elements to the vector?
Internally, the vector object has a pointer to dynamically-allocated memory that contains the elements. When you use sizeof(test) you're just getting the size of the structure that contains the pointer, the size of the memory that it points to is not included.
This memory has to be dynamically-allocated so that the vector can grow and shrink as needed. It's not possible for a class object to change its size.
To get the amount of memory being used by the data storage, use sizeof(double) * test.capacity().
I'm working on a project that involves binary files.
So I started researching about binary files but I'm still confused about how to write and fill a vector from that binary file that I wrote before
Here's code: for writing.
void binario(){
ofstream fout("./Binario/Data.AFe", ios::out | ios::binary);
vector<int> enteros;
enteros.push_back(1);
enteros.push_back(2);
enteros.push_back(3);
enteros.push_back(4);
enteros.push_back(5);
//fout.open()
//if (fout.is_open()) {
std::cout << "Entre al if" << '\n';
//while (!fout.eof()) {
std::cout << "Entre al while" << '\n';
std::cout << "Enteros size: "<< enteros.size() << '\n';
int size1 = enteros.size();
for (int i = 0; i < enteros.size(); i++) {
std::cout << "for " << i << '\n';
fout.write((char*)&size1, 4);
fout.write((char*)&enteros[i], size1 * sizeof(enteros));
//cout<< fout.get(entero[i])<<endl;
}
//fout.close();
//}
fout.close();
cout<<"copiado con exito"<<endl;
//}
}
Here's code for reading:
oid leerBinario(){
vector<int> list2;
ifstream is("./Binario/Data.AFe", ios::binary);
int size2;
is.read((char*)&size2, 4);
list2.resize(size2);
is.read((char*)&list2[0], size2 * sizeof(list2));
std::cout << "Size del vector: " << list2.size() <<endl;
for (int i = 0; i < list2.size(); i++) {
std::cout << i << ". " << list2[i] << '\n';
}
std::cout << "Antes de cerrar" << '\n';
is.close();
}
I don't know if I'm writing correctly to the file, this is just a test so I don't mess up my main file, instead of writing numbers I need to save Objects that are stored in a vector and load them everytime the user runs the program.
Nope, you're a bit confused. You're writing the size in every iteration, and then you're doing something completely undefined when you try to write the value. You can actually do this without the loop, when you are using a vector.
fout.write(&size1, sizeof(size1));
fout.write(enteros.data(), size1 * sizeof(int));
And reading in:
is.read(&list2[0], size2 * sizeof(int));
To be more portable you might want to use data types that won't change (for example when you switch from 32-bit compilation to 64-bit). In that case, use stuff from <cctype> -- e.g. int32_t for both the size and value data.
I have a program that currently generates large arrays and matrices that can be upwards of 10GB in size. The program uses MPI to parallelize workloads, but is limited by the fact that each process needs its own copy of the array or matrix in order to perform its portion of the computation. The memory requirements make this problem unfeasible with a large number of MPI processes and so I have been looking into Boost::Interprocess as a means of sharing data between MPI processes.
So far, I have come up with the following which creates a large vector and parallelizes the summation of its elements:
#include <cstdlib>
#include <ctime>
#include <functional>
#include <iostream>
#include <string>
#include <utility>
#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/tuple/tuple_comparison.hpp>
#include <mpi.h>
typedef boost::interprocess::allocator<double, boost::interprocess::managed_shared_memory::segment_manager> ShmemAllocator;
typedef boost::interprocess::vector<double, ShmemAllocator> MyVector;
const std::size_t vector_size = 1000000000;
const std::string shared_memory_name = "vector_shared_test.cpp";
int main(int argc, char **argv) {
int numprocs, rank;
MPI::Init();
numprocs = MPI::COMM_WORLD.Get_size();
rank = MPI::COMM_WORLD.Get_rank();
if(numprocs >= 2) {
if(rank == 0) {
std::cout << "On process rank " << rank << "." << std::endl;
std::time_t creation_start = std::time(NULL);
boost::interprocess::shared_memory_object::remove(shared_memory_name.c_str());
boost::interprocess::managed_shared_memory segment(boost::interprocess::create_only, shared_memory_name.c_str(), size_t(12000000000));
std::cout << "Size of double: " << sizeof(double) << std::endl;
std::cout << "Allocated shared memory: " << segment.get_size() << std::endl;
const ShmemAllocator alloc_inst(segment.get_segment_manager());
MyVector *myvector = segment.construct<MyVector>("MyVector")(alloc_inst);
std::cout << "myvector max size: " << myvector->max_size() << std::endl;
for(int i = 0; i < vector_size; i++) {
myvector->push_back(double(i));
}
std::cout << "Vector capacity: " << myvector->capacity() << " | Memory Free: " << segment.get_free_memory() << std::endl;
std::cout << "Vector creation successful and took " << std::difftime(std::time(NULL), creation_start) << " seconds." << std::endl;
}
std::flush(std::cout);
MPI::COMM_WORLD.Barrier();
std::time_t summing_start = std::time(NULL);
std::cout << "On process rank " << rank << "." << std::endl;
boost::interprocess::managed_shared_memory segment(boost::interprocess::open_only, shared_memory_name.c_str());
MyVector *myvector = segment.find<MyVector>("MyVector").first;
double result = 0;
for(int i = rank; i < myvector->size(); i = i + numprocs) {
result = result + (*myvector)[i];
}
double total = 0;
MPI::COMM_WORLD.Reduce(&result, &total, 1, MPI::DOUBLE, MPI::SUM, 0);
std::flush(std::cout);
MPI::COMM_WORLD.Barrier();
if(rank == 0) {
std::cout << "On process rank " << rank << "." << std::endl;
std::cout << "Vector summing successful and took " << std::difftime(std::time(NULL), summing_start) << " seconds." << std::endl;
std::cout << "The arithmetic sum of the elements in the vector is " << total << std::endl;
segment.destroy<MyVector>("MyVector");
}
std::flush(std::cout);
MPI::COMM_WORLD.Barrier();
boost::interprocess::shared_memory_object::remove(shared_memory_name.c_str());
}
sleep(300);
MPI::Finalize();
return 0;
}
I noticed that this causes the entire shared object to be mapped into each processes' virtual memory space - which is an issue with our computing cluster as it limits virtual memory to be the same as physical memory. Is there a way to share this data structure without having to map out the entire shared memory space - perhaps in the form of sharing a pointer of some kind? Would trying to access unmapped shared memory even be defined behavior? Unfortunately the operations we are performing on the array means that each process eventually needs to access every element in it (although not concurrently - I suppose its possible to break up the shared array into pieces and trade portions of the array for those you need, but this is not ideal).
Since the data you want to share is so large, it may be more practical to treat the data as a true file, and use file operations to read the data that you want. Then, you do not need to use shared memory to share the file, just let each process read directly from the file system.
ifstream file ("data.dat", ios::in | ios::binary);
file.seekg(someOffset, ios::beg);
file.read(array, sizeof(array));