I have a global vector that I load data into, which is then read from later in my program.
If I have say, 1000000 elements pushed back into this vector, will it cause any problems such as those created by overflowing the stack? How much memory space is a available in the global scope?
As per C++11 section 23, unless your type provides a specialised allocator, sequence containers such as vector will use std::allocator, which gets its memory using new. In other words, using dynamic memory allocation functions ("from the heap" in layman's parlance).
So, provided you follow the rules, there's no way to corrupt the stack using that container, as might be the case if you did something like:
void function(void) {
int xyzzy[999999999];
:
}
That's not to say you can't run out of memory, the heap isn't infinite in size, as shown in the following code:
#include <iostream>
#include <vector>
int main (void) {
std::vector<const char*> *v = new std::vector<const char*>();
long count = 0;
while (1) {
try {
v->push_back("xyzzy");
count++;
} catch (std::exception &e) {
std::cout << e.what() << '\n';
break;
}
}
std::cout << count << " pushbacks done.\n";
return 0;
}
which outputs (on my system):
std::bad_alloc
134217728 pushbacks done.
But getting an exception because you're run out of memory is a far cry from corruption caused by stack overflow or running out of static storage duration ("global") space.
Question:
If I have say, 1000000 elements pushed back into this vector, will it cause any problems such as those created by overflowing the stack?
No, it won't. When you create an std::vector, the memory for the data is allocated from the heap, not from the memory reserved for global data.
Question:
How much memory space is a available in the global scope?
I don't have an answer to that. It might be irrelevant given the answer to the first question.
You might find this answer to another SO post relevant.
Related
The following code gives me a segmentation fault when run on a 2Gb machine, but works on a 4GB machine.
int main()
{
int c[1000000];
cout << "done\n";
return 0;
}
The size of the array is just 4Mb. Is there a limit on the size of an array that can be used in c++?
You're probably just getting a stack overflow here. The array is too big to fit in your program's stack region; the stack growth limit is usually 8 MiB or 1 MiB for user-space code on most mainstream desktop / server OSes. (Normal C++ implementations use the asm stack for automatic storage, i.e. non-static local variables arrays. This makes deallocating them happen for free when functions return or an exception propagates through them.)
If you dynamically allocate the array you should be fine, assuming your machine has enough memory.
int* array = new int[1000000]; // may throw std::bad_alloc
But remember that this will require you to delete[] the array manually to avoid memory leaks, even if your function exits via an exception. Manual new/delete is strongly discouraged in modern C++, prefer RAII.
A better solution would be to use std::vector<int> array (cppreference). You can reserve space for 1000000 elements, if you know how large it will grow. Or even resize it to default-construct them (i.e. zero-initialize the memory, unlike when you declare a plain C-style array with no initializer), like std::vector<int> array(1000000)
When the std::vector object goes out of scope, its destructor will deallocate the storage for you, even if that happens via an exception in a child function that's caught by a parent function.
In C or C++ local objects are usually allocated on the stack. You are allocating a large array on the stack, more than the stack can handle, so you are getting a stackoverflow.
Don't allocate it local on stack, use some other place instead. This can be achieved by either making the object global or allocating it on the global heap. Global variables are fine, if you don't use the from any other compilation unit. To make sure this doesn't happen by accident, add a static storage specifier, otherwise just use the heap.
This will allocate in the BSS segment, which is a part of the heap. Since it's in static storage, it's zero initialized if you don't specify otherwise, unlike local variables (automatic storage) including arrays.
static int c[1000000];
int main()
{
cout << "done\n";
return 0;
}
A non-zero initializer will make a compiler allocate in the DATA segment, which is a part of the heap too. (And all the data for the array initializer will take space in the executable, including all the implicit trailing zeros, instead of just a size to zero-init in the BSS)
int c[1000000] = {1, 2, 3};
int main()
{
cout << "done\n";
return 0;
}
This will allocate at some unspecified location in the heap:
int main()
{
int* c = new int[1000000]; // size can be a variable, unlike with static storage
cout << "done\n";
delete[] c; // dynamic storage needs manual freeing
return 0;
}
Also, if you are running in most UNIX & Linux systems you can temporarily increase the stack size by the following command:
ulimit -s unlimited
But be careful, memory is a limited resource and with great power come great responsibilities :)
You array is being allocated on the stack in this case attempt to allocate an array of the same size using alloc.
Because you store the array in the stack. You should store it in the heap. See this link to understand the concept of the heap and the stack.
Your plain array is allocated in stack, and stack is limited to few magabytes, hence your program gets stack overflow and crashes.
Probably best is to use heap-allocated std::vector-based array which can grow almost to size of whole memory, instead of your plain array.
Try it online!
#include <vector>
#include <iostream>
int main() {
std::vector<int> c(1000000);
std::cout << "done\n";
return 0;
}
Then you can access array's elements as usual c[i] and/or get its size c.size() (number of int elements).
If you want multi-dimensional array with fixed dimensions then use mix of both std::vector and std::array, as following:
Try it online!
#include <vector>
#include <array>
#include <iostream>
int main() {
std::vector<std::array<std::array<int, 123>, 456>> c(100);
std::cout << "done\n";
return 0;
}
In example above you get almost same behavior as if you allocated plain array int c[100][456][123]; (except that vector allocates on heap instead of stack), you can access elements as c[10][20][30] same as in plain array. This example above also allocates array on heap meaning that you can have array sizes up to whole memory size and not limited by stack size.
To get pointer to the first element in vector you use &c[0] or just c.data().
there can be one more way that worked for me!
you can reduce the size of array by changing its data type:
int main()
{
short c[1000000];
cout << "done\n";
return 0;
}
or
int main()
{
unsigned short c[1000000];
cout << "done\n";
return 0;
}
The following code gives me a segmentation fault when run on a 2Gb machine, but works on a 4GB machine.
int main()
{
int c[1000000];
cout << "done\n";
return 0;
}
The size of the array is just 4Mb. Is there a limit on the size of an array that can be used in c++?
You're probably just getting a stack overflow here. The array is too big to fit in your program's stack region; the stack growth limit is usually 8 MiB or 1 MiB for user-space code on most mainstream desktop / server OSes. (Normal C++ implementations use the asm stack for automatic storage, i.e. non-static local variables arrays. This makes deallocating them happen for free when functions return or an exception propagates through them.)
If you dynamically allocate the array you should be fine, assuming your machine has enough memory.
int* array = new int[1000000]; // may throw std::bad_alloc
But remember that this will require you to delete[] the array manually to avoid memory leaks, even if your function exits via an exception. Manual new/delete is strongly discouraged in modern C++, prefer RAII.
A better solution would be to use std::vector<int> array (cppreference). You can reserve space for 1000000 elements, if you know how large it will grow. Or even resize it to default-construct them (i.e. zero-initialize the memory, unlike when you declare a plain C-style array with no initializer), like std::vector<int> array(1000000)
When the std::vector object goes out of scope, its destructor will deallocate the storage for you, even if that happens via an exception in a child function that's caught by a parent function.
In C or C++ local objects are usually allocated on the stack. You are allocating a large array on the stack, more than the stack can handle, so you are getting a stackoverflow.
Don't allocate it local on stack, use some other place instead. This can be achieved by either making the object global or allocating it on the global heap. Global variables are fine, if you don't use the from any other compilation unit. To make sure this doesn't happen by accident, add a static storage specifier, otherwise just use the heap.
This will allocate in the BSS segment, which is a part of the heap. Since it's in static storage, it's zero initialized if you don't specify otherwise, unlike local variables (automatic storage) including arrays.
static int c[1000000];
int main()
{
cout << "done\n";
return 0;
}
A non-zero initializer will make a compiler allocate in the DATA segment, which is a part of the heap too. (And all the data for the array initializer will take space in the executable, including all the implicit trailing zeros, instead of just a size to zero-init in the BSS)
int c[1000000] = {1, 2, 3};
int main()
{
cout << "done\n";
return 0;
}
This will allocate at some unspecified location in the heap:
int main()
{
int* c = new int[1000000]; // size can be a variable, unlike with static storage
cout << "done\n";
delete[] c; // dynamic storage needs manual freeing
return 0;
}
Also, if you are running in most UNIX & Linux systems you can temporarily increase the stack size by the following command:
ulimit -s unlimited
But be careful, memory is a limited resource and with great power come great responsibilities :)
You array is being allocated on the stack in this case attempt to allocate an array of the same size using alloc.
Because you store the array in the stack. You should store it in the heap. See this link to understand the concept of the heap and the stack.
Your plain array is allocated in stack, and stack is limited to few magabytes, hence your program gets stack overflow and crashes.
Probably best is to use heap-allocated std::vector-based array which can grow almost to size of whole memory, instead of your plain array.
Try it online!
#include <vector>
#include <iostream>
int main() {
std::vector<int> c(1000000);
std::cout << "done\n";
return 0;
}
Then you can access array's elements as usual c[i] and/or get its size c.size() (number of int elements).
If you want multi-dimensional array with fixed dimensions then use mix of both std::vector and std::array, as following:
Try it online!
#include <vector>
#include <array>
#include <iostream>
int main() {
std::vector<std::array<std::array<int, 123>, 456>> c(100);
std::cout << "done\n";
return 0;
}
In example above you get almost same behavior as if you allocated plain array int c[100][456][123]; (except that vector allocates on heap instead of stack), you can access elements as c[10][20][30] same as in plain array. This example above also allocates array on heap meaning that you can have array sizes up to whole memory size and not limited by stack size.
To get pointer to the first element in vector you use &c[0] or just c.data().
there can be one more way that worked for me!
you can reduce the size of array by changing its data type:
int main()
{
short c[1000000];
cout << "done\n";
return 0;
}
or
int main()
{
unsigned short c[1000000];
cout << "done\n";
return 0;
}
The following code gives me a segmentation fault when run on a 2Gb machine, but works on a 4GB machine.
int main()
{
int c[1000000];
cout << "done\n";
return 0;
}
The size of the array is just 4Mb. Is there a limit on the size of an array that can be used in c++?
You're probably just getting a stack overflow here. The array is too big to fit in your program's stack region; the stack growth limit is usually 8 MiB or 1 MiB for user-space code on most mainstream desktop / server OSes. (Normal C++ implementations use the asm stack for automatic storage, i.e. non-static local variables arrays. This makes deallocating them happen for free when functions return or an exception propagates through them.)
If you dynamically allocate the array you should be fine, assuming your machine has enough memory.
int* array = new int[1000000]; // may throw std::bad_alloc
But remember that this will require you to delete[] the array manually to avoid memory leaks, even if your function exits via an exception. Manual new/delete is strongly discouraged in modern C++, prefer RAII.
A better solution would be to use std::vector<int> array (cppreference). You can reserve space for 1000000 elements, if you know how large it will grow. Or even resize it to default-construct them (i.e. zero-initialize the memory, unlike when you declare a plain C-style array with no initializer), like std::vector<int> array(1000000)
When the std::vector object goes out of scope, its destructor will deallocate the storage for you, even if that happens via an exception in a child function that's caught by a parent function.
In C or C++ local objects are usually allocated on the stack. You are allocating a large array on the stack, more than the stack can handle, so you are getting a stackoverflow.
Don't allocate it local on stack, use some other place instead. This can be achieved by either making the object global or allocating it on the global heap. Global variables are fine, if you don't use the from any other compilation unit. To make sure this doesn't happen by accident, add a static storage specifier, otherwise just use the heap.
This will allocate in the BSS segment, which is a part of the heap. Since it's in static storage, it's zero initialized if you don't specify otherwise, unlike local variables (automatic storage) including arrays.
static int c[1000000];
int main()
{
cout << "done\n";
return 0;
}
A non-zero initializer will make a compiler allocate in the DATA segment, which is a part of the heap too. (And all the data for the array initializer will take space in the executable, including all the implicit trailing zeros, instead of just a size to zero-init in the BSS)
int c[1000000] = {1, 2, 3};
int main()
{
cout << "done\n";
return 0;
}
This will allocate at some unspecified location in the heap:
int main()
{
int* c = new int[1000000]; // size can be a variable, unlike with static storage
cout << "done\n";
delete[] c; // dynamic storage needs manual freeing
return 0;
}
Also, if you are running in most UNIX & Linux systems you can temporarily increase the stack size by the following command:
ulimit -s unlimited
But be careful, memory is a limited resource and with great power come great responsibilities :)
You array is being allocated on the stack in this case attempt to allocate an array of the same size using alloc.
Because you store the array in the stack. You should store it in the heap. See this link to understand the concept of the heap and the stack.
Your plain array is allocated in stack, and stack is limited to few magabytes, hence your program gets stack overflow and crashes.
Probably best is to use heap-allocated std::vector-based array which can grow almost to size of whole memory, instead of your plain array.
Try it online!
#include <vector>
#include <iostream>
int main() {
std::vector<int> c(1000000);
std::cout << "done\n";
return 0;
}
Then you can access array's elements as usual c[i] and/or get its size c.size() (number of int elements).
If you want multi-dimensional array with fixed dimensions then use mix of both std::vector and std::array, as following:
Try it online!
#include <vector>
#include <array>
#include <iostream>
int main() {
std::vector<std::array<std::array<int, 123>, 456>> c(100);
std::cout << "done\n";
return 0;
}
In example above you get almost same behavior as if you allocated plain array int c[100][456][123]; (except that vector allocates on heap instead of stack), you can access elements as c[10][20][30] same as in plain array. This example above also allocates array on heap meaning that you can have array sizes up to whole memory size and not limited by stack size.
To get pointer to the first element in vector you use &c[0] or just c.data().
there can be one more way that worked for me!
you can reduce the size of array by changing its data type:
int main()
{
short c[1000000];
cout << "done\n";
return 0;
}
or
int main()
{
unsigned short c[1000000];
cout << "done\n";
return 0;
}
The following code gives me a segmentation fault when run on a 2Gb machine, but works on a 4GB machine.
int main()
{
int c[1000000];
cout << "done\n";
return 0;
}
The size of the array is just 4Mb. Is there a limit on the size of an array that can be used in c++?
You're probably just getting a stack overflow here. The array is too big to fit in your program's stack region; the stack growth limit is usually 8 MiB or 1 MiB for user-space code on most mainstream desktop / server OSes. (Normal C++ implementations use the asm stack for automatic storage, i.e. non-static local variables arrays. This makes deallocating them happen for free when functions return or an exception propagates through them.)
If you dynamically allocate the array you should be fine, assuming your machine has enough memory.
int* array = new int[1000000]; // may throw std::bad_alloc
But remember that this will require you to delete[] the array manually to avoid memory leaks, even if your function exits via an exception. Manual new/delete is strongly discouraged in modern C++, prefer RAII.
A better solution would be to use std::vector<int> array (cppreference). You can reserve space for 1000000 elements, if you know how large it will grow. Or even resize it to default-construct them (i.e. zero-initialize the memory, unlike when you declare a plain C-style array with no initializer), like std::vector<int> array(1000000)
When the std::vector object goes out of scope, its destructor will deallocate the storage for you, even if that happens via an exception in a child function that's caught by a parent function.
In C or C++ local objects are usually allocated on the stack. You are allocating a large array on the stack, more than the stack can handle, so you are getting a stackoverflow.
Don't allocate it local on stack, use some other place instead. This can be achieved by either making the object global or allocating it on the global heap. Global variables are fine, if you don't use the from any other compilation unit. To make sure this doesn't happen by accident, add a static storage specifier, otherwise just use the heap.
This will allocate in the BSS segment, which is a part of the heap. Since it's in static storage, it's zero initialized if you don't specify otherwise, unlike local variables (automatic storage) including arrays.
static int c[1000000];
int main()
{
cout << "done\n";
return 0;
}
A non-zero initializer will make a compiler allocate in the DATA segment, which is a part of the heap too. (And all the data for the array initializer will take space in the executable, including all the implicit trailing zeros, instead of just a size to zero-init in the BSS)
int c[1000000] = {1, 2, 3};
int main()
{
cout << "done\n";
return 0;
}
This will allocate at some unspecified location in the heap:
int main()
{
int* c = new int[1000000]; // size can be a variable, unlike with static storage
cout << "done\n";
delete[] c; // dynamic storage needs manual freeing
return 0;
}
Also, if you are running in most UNIX & Linux systems you can temporarily increase the stack size by the following command:
ulimit -s unlimited
But be careful, memory is a limited resource and with great power come great responsibilities :)
You array is being allocated on the stack in this case attempt to allocate an array of the same size using alloc.
Because you store the array in the stack. You should store it in the heap. See this link to understand the concept of the heap and the stack.
Your plain array is allocated in stack, and stack is limited to few magabytes, hence your program gets stack overflow and crashes.
Probably best is to use heap-allocated std::vector-based array which can grow almost to size of whole memory, instead of your plain array.
Try it online!
#include <vector>
#include <iostream>
int main() {
std::vector<int> c(1000000);
std::cout << "done\n";
return 0;
}
Then you can access array's elements as usual c[i] and/or get its size c.size() (number of int elements).
If you want multi-dimensional array with fixed dimensions then use mix of both std::vector and std::array, as following:
Try it online!
#include <vector>
#include <array>
#include <iostream>
int main() {
std::vector<std::array<std::array<int, 123>, 456>> c(100);
std::cout << "done\n";
return 0;
}
In example above you get almost same behavior as if you allocated plain array int c[100][456][123]; (except that vector allocates on heap instead of stack), you can access elements as c[10][20][30] same as in plain array. This example above also allocates array on heap meaning that you can have array sizes up to whole memory size and not limited by stack size.
To get pointer to the first element in vector you use &c[0] or just c.data().
there can be one more way that worked for me!
you can reduce the size of array by changing its data type:
int main()
{
short c[1000000];
cout << "done\n";
return 0;
}
or
int main()
{
unsigned short c[1000000];
cout << "done\n";
return 0;
}
I am compiling following code VS2012 - 32BIT. I know that max_size() returns
"maximum potential size the container can reach" which is in my case: 1.073.741.823 (yeay)
So how can i know, how many object my container can really store? (I have 64gb RAM)
unsigned int = 100000000;
vector<int*> data;
std::cout<<"max cap: "<<data.max_size()<<std::endl;
for(unsigned int i = 0; i < operationUnit; i++)
data.push_back(new int());
This will end-up in a bad-alloc. However, as i am targetting x64 this problem doesn't occur, as the max-cap is much higher, but i still cannot figure the exact elements, when i would like to reduce it down to clamp user-input.
thanks!
Well, it is OS dependent ofcourse, but the results would be, similar for every one. For example when run as 32bit executable, consistenly a build with VS2012 will stop at 26,906,977 elements in a vector of int*, not posing a threat to your memory (not even by close).
Now it gets interesting when you build a 64bit version, in which case, throwing a bad_alloc happens when (almost) all your memory is drained. In that case, no C++ not any other language can protect you.
In the screenshot that follows I'm posting an example of this happening: by the time bad_alloc gets thrown, the program is in no position to catch it or do anything with it. The OS steps in and kills every process and memory is deallocated at once (see graph). In the respective 32 version the exception was caught normally and deallocation would take about 10 minutes.
Now this is a very simplistic way of seeing this, I'm sure OS gurus could supply more insights but feel free to try this at home (and burn out some memory - I can't stop thinking that I can smell something burnt after this)
the code in the screenshot
#include <iostream>
#include <vector>
#include <exception>
using namespace std;
int main()
{
vector<int*> maxV;
try
{
while (true) maxV.push_back(new int);
}
catch (bad_alloc &e)
{
cout << "I caught bad alloc at element no " << maxV.size() << "\n";
for (auto i : maxV) delete i;
}
catch (exception &e)
{
cout << "Some other exception happened at element no " << maxV.size() << "\n";
for (auto i : maxV) delete i;
}
return 0;
}
You can't. OS could totally run out of memory. You may find that using deque data structure can become larger before error than vector for huge amounts of data, as the data is not contiguous and so it is less effected by memory fragmentation, which can be significant when you end up allocating more than half your entire memory..