I am compiling following code VS2012 - 32BIT. I know that max_size() returns
"maximum potential size the container can reach" which is in my case: 1.073.741.823 (yeay)
So how can i know, how many object my container can really store? (I have 64gb RAM)
unsigned int = 100000000;
vector<int*> data;
std::cout<<"max cap: "<<data.max_size()<<std::endl;
for(unsigned int i = 0; i < operationUnit; i++)
data.push_back(new int());
This will end-up in a bad-alloc. However, as i am targetting x64 this problem doesn't occur, as the max-cap is much higher, but i still cannot figure the exact elements, when i would like to reduce it down to clamp user-input.
thanks!
Well, it is OS dependent ofcourse, but the results would be, similar for every one. For example when run as 32bit executable, consistenly a build with VS2012 will stop at 26,906,977 elements in a vector of int*, not posing a threat to your memory (not even by close).
Now it gets interesting when you build a 64bit version, in which case, throwing a bad_alloc happens when (almost) all your memory is drained. In that case, no C++ not any other language can protect you.
In the screenshot that follows I'm posting an example of this happening: by the time bad_alloc gets thrown, the program is in no position to catch it or do anything with it. The OS steps in and kills every process and memory is deallocated at once (see graph). In the respective 32 version the exception was caught normally and deallocation would take about 10 minutes.
Now this is a very simplistic way of seeing this, I'm sure OS gurus could supply more insights but feel free to try this at home (and burn out some memory - I can't stop thinking that I can smell something burnt after this)
the code in the screenshot
#include <iostream>
#include <vector>
#include <exception>
using namespace std;
int main()
{
vector<int*> maxV;
try
{
while (true) maxV.push_back(new int);
}
catch (bad_alloc &e)
{
cout << "I caught bad alloc at element no " << maxV.size() << "\n";
for (auto i : maxV) delete i;
}
catch (exception &e)
{
cout << "Some other exception happened at element no " << maxV.size() << "\n";
for (auto i : maxV) delete i;
}
return 0;
}
You can't. OS could totally run out of memory. You may find that using deque data structure can become larger before error than vector for huge amounts of data, as the data is not contiguous and so it is less effected by memory fragmentation, which can be significant when you end up allocating more than half your entire memory..
Related
I am learning c++ and in one lesson was a code which is about exception handling. The code is not mine, it is just the example for "try and catch". So this question is not about the code quality
My question to this code is actually: is the output and calculation of the memory size correct?
When I allocate a block of memory with new double(1000), isn't the size then 8000 bytes ?
The cerr output only counts as 1kB instead of 8kB. Am I wrong?
I got the size of 1 double with sizeof(double) to confirm it is 8 bytes.
#include <iostream>
#include <cstdlib>
#include <new>
using namespace ::std;
int main()
{
int i = 0;
double *q;
try
{
while (1)
{
q = new double[1000];
i++;
}
}
catch (bad_alloc &ex)
{
cerr << "The memory is used up. " << i
<< " Kilobyte were available." << endl;
exit(1);
}
}
To summarize what #Peter said in his comment: Your i variable is counting the number of allocations, not the total amount of memory allocated.
Note, however, that even if you "fix" this, what you'll get is not the amount of "available memory", nor even the amount of available memory rounded down to a multiple of 8000. This is because "available memory" is not a very well-defined concept. The operating system may be willing to let you allocate a zillion bytes; but it might not actually do anything visible to other processes until you start writing into that memory. And even if you do write to it - it could swap unused memory pages to the hard disk / SSD, to make room for the pages you're working on.
If you wanted to check what the maximum amount of memory you can allocate using new, you might consider using a binary-search-like procedure to obtain the size; I won't spell it out in case that's your homework. (And of course, this too will not be accurate since other processes' memory use fluctuates.)
Also consider reading: How to get available memory C++/g++?
Finally, some nitpicks:
You're using inconsistent indentation. That's confusing.
i is not such a good name for a variable. num_allocations would fit better. When you use a more meaningful name you also commit to its semantics, which makes it more difficult to get them mixed up.
Try to avoid "magic numbers" like 1000. Define constants using enum or constexpr. For example: enum { Kilo = 1000 };.
There doesn't seem to be a good reason to use double in such a program - which has nothing to do with floating-point arithmetic.
You are absolutely correct. It should be:
cerr << "The memory is used up. " << sizeof(double) * i
<< " Kilobyte were available." << endl;
Having this simple code in C++:
#include <iostream>
#include <memory>
#include <vector>
using namespace std;
class Empty{};
int main() {
array<unique_ptr<Empty>, 1024> empties;
for(size_t i =0; i < 1024; i++){
empties[i] = make_unique<Empty>();
}
for(auto& element : empties){
cout << "ptr: " << element.get() << endl;
}
return 0;
}
when running in ideone.com or Windows we get following result:
ptr: 0x2b601f0c9ca0
ptr: 0x2b601f0c9cc0
ptr: 0x2b601f0c9ce0
ptr: 0x2b601f0c9d00
ptr: 0x2b601f0c9d20
ptr: 0x2b601f0c9d40
ptr: 0x2b601f0c9d60 ...
For me it's totally strange. what kind of allocation algorithms in OS or standard library might cause, that there happened no allocation at address, that ends with number different than 0?
The reason I did this experiment is, that given the OS uses buddy algorithm, that manages pages and allocation request will cause OS to allocate a piece of contiguous memory, then a couple of next allocations (until running out of allocated memory) should be allocated quite nearby. If that would be the case , then probably cache issue with lists would not be so significant in some cases , but I got results I was not expecting anyways.
Also second number from the right in allocated nodes is totally randomly. What may cause such a behavior?
The minimum size of a class in C++ is one byte, if I recall correctly. As there is a consistent 32 byte spacing between the class, it could be that that happens to be the sizeof the empty class you made. To determine this, try adding
std::cout << "Empty class size: " << sizeof(Empty) << std::endl;
It probably won't be 32 bytes, instead, there will probably be some consistent spacing between each object.
Note:
Does this compile for you. It doesn't for me because empties cant be implicitly initialised.
Please notice that the printed pointers is inaccurate, the OS allow you to see them as subsequent pointers when they could be allocated to an entirely different pages.
I have a global vector that I load data into, which is then read from later in my program.
If I have say, 1000000 elements pushed back into this vector, will it cause any problems such as those created by overflowing the stack? How much memory space is a available in the global scope?
As per C++11 section 23, unless your type provides a specialised allocator, sequence containers such as vector will use std::allocator, which gets its memory using new. In other words, using dynamic memory allocation functions ("from the heap" in layman's parlance).
So, provided you follow the rules, there's no way to corrupt the stack using that container, as might be the case if you did something like:
void function(void) {
int xyzzy[999999999];
:
}
That's not to say you can't run out of memory, the heap isn't infinite in size, as shown in the following code:
#include <iostream>
#include <vector>
int main (void) {
std::vector<const char*> *v = new std::vector<const char*>();
long count = 0;
while (1) {
try {
v->push_back("xyzzy");
count++;
} catch (std::exception &e) {
std::cout << e.what() << '\n';
break;
}
}
std::cout << count << " pushbacks done.\n";
return 0;
}
which outputs (on my system):
std::bad_alloc
134217728 pushbacks done.
But getting an exception because you're run out of memory is a far cry from corruption caused by stack overflow or running out of static storage duration ("global") space.
Question:
If I have say, 1000000 elements pushed back into this vector, will it cause any problems such as those created by overflowing the stack?
No, it won't. When you create an std::vector, the memory for the data is allocated from the heap, not from the memory reserved for global data.
Question:
How much memory space is a available in the global scope?
I don't have an answer to that. It might be irrelevant given the answer to the first question.
You might find this answer to another SO post relevant.
I am trying to test the speed of various pointer speeds and ran into a really weird problem. When allocating for raw pointers, it runs fine. (There is a memory leak, but that isn't the problem.) When I run the second test with shared_ptr, It runs the fill fine, prints the log and then when it returns, it enters an infinite loop. It appears the ref count is garbage, but I'm doing everything by value.
#include <memory>
#include <vector>
#include <functional>
#include <iostream>
#include <iterator>
#include <algorithm>
#include <string>
#include <Windows.h>
using namespace std;
static const int TOTAL = 1000000;
int* newTest(int i)
{
return new int(i);
}
shared_ptr<int> sharedTest(int i)
{
return shared_ptr<int>(new int(i));
}
template <typename T>
pair<int, vector<typename T::result_type>> Fill(T fn)
{
unsigned long start = GetTickCount();
vector<typename T::result_type> vec;
vec.reserve(TOTAL);
for(int i = 0; i < TOTAL; i++)
{
vec.push_back(fn(i));
}
unsigned long end = GetTickCount();
return make_pair(end - start, move(vec));
}
template <typename T>
void Test(T fn, string name)
{
vector<typename T::result_type> newTest;
int milliseconds = 0;
tie(milliseconds, newTest) = Fill(fn);
cout << "Fill " << name << " Took " << milliseconds << " milliseconds." << endl;
}
int main()
{
function<int*(int)> fn1 = newTest;
Test(fn1, "Raw Ptr");
function<shared_ptr<int>(int)> fn2 = sharedTest;
Test(fn2, "Shared New");
return 0;
}
OK. It appears I have asked a stack overflow question on Stackoverflow....
When I set TOTAL to 10000 it's fine. So, is that just a symptom of something else or do I need to increase my stack size?
Edit from comment:
Tavison: OK. after several minutes it ended. You're right about not being an infinite loop. But, 1043 ms to new and many minutes to delete makes it hard to justify using them. This is not a result I would expect.
There is no infinite loop, you're just being impatient. It's taking time to free all the shared_ptr's.
To be able to claim there is an infinite loop, you need to actually step into the code and take a look. Check that there is some loop somewhere with a condition that will never change. This is not the case here.
Lower your TOTAL, for example, and verify it actually ends. Having more of these will not magically introduce an infinite loop at some number, so if it works at a lower number, it works at a higher one.
Or, don't allocate ints; allocate some test struct that outputs "bye" (along with some counter) when its destructing, and you'll see "post"-test that they're all being deleted. (Of course, performing IO will increase the destruction time, but the point is to verify a condition is moving towards stopping a loop.)
Also, you can cut the allocations in half by using return make_shared<int>(i); rather than newing the int yourself and putting it into a shared_ptr. Always use make_shared.
Lastly, this is only slow if you have a debugger attached, because it'll, well, debug your memory usage. That is, verify that what you're deleting is sensible to delete, isn't corrupting anything, etc.
And for the love of programming use four spaces to indent, not two.
If you run the code in debug or release with debugging, you'll get a debug heap that can be used to track memory errors. Freed memory is filled with debug patterns when this happens, so the code will run slower. Running the code without debugging from Visual Studio ends in less than 200 milliseconds.
I have the following code to test memory deallocation using a std::list container:
#include <iostream>
#include <list>
#include <string>
#include <boost/bind.hpp>
/* count of element to put into container
*/
static const unsigned long SIZE = 50000000;
/* element use for test
*/
class Element
{
public:
Element()
: mId(0)
{}
Element( long id )
: mId(id)
{}
virtual ~Element()
{
}
inline long getId() const
{
return this->mId;
}
inline bool operator<( const Element & rightOperand ) const
{
return this->mId < rightOperand.mId;
}
inline bool isEven() const
{
return 0 == ( this->mId & 1 );
}
private:
long mId;
};
typedef std::list< Element > Elements;
int main( int argc, char * argv[] )
{
std::string dummy;
{
Elements elements;
std::cout << "Inserting "<< SIZE << " elements in container" << std::endl;
std::cout << "Please wait..." << std::endl;
/* inserting elements
*/
for( long i=0; i<SIZE; ++i )
{
elements.push_back( i );
}
std::cout << "Size is " << elements.size() << std::endl;
std::getline( std::cin, dummy); // waiting user press enter
/* remove even elements
*/
elements.remove_if( boost::bind( & Element::isEven, _1 ) );
std::cout << "Size is " << elements.size() << std::endl;
std::getline( std::cin, dummy);
}
std::getline( std::cin, dummy);
return 0;
}
Running this code gives me the following memory profile:
It looks like gcc is defering deallocation and in my test program, at the end it has no choice and deallocate memory before going back to command line.
Why deallocation happens so late ?
I've tried with a vector to test another container and the shrink-to-fit tricks works and deallocate freed memory when I expect it.
gcc 4.5.0, linux 2.6.34
Most operating systems (including Linux) only allow processes to allocate quite large chunks of memory, and not very small ones; even if it is possible, it is most likely more expensive to make many small allocations than a few large ones. Generally, the C++ library will acquire large chunks from the operating system, and use it's own heap manager to allocate small pieces of them to the program. The large chunks will usually not be returned to the operating system once they've been divided up like that; they will remain allocated to the process, and will be reused for future allocations.
list allocates memory in small chunks (one per node), and so usually the allocated memory won't be released until the program exits. vector might get its memory as a single large allocation directly from the operating system, in which case it will be released when its deallocated.
What exactly is your graph showing? The destructor of std::list
deallocates all of the memory, so that it can be reused elsewhere in the
program, but deallocation won't necessarily return the memory to the
system, where it can be used by other processes. Historically, at
least, under Unix, once memory has been allocated to a process, it
remains with that process until the process terminates. Newer
algorithms may be able to actually return memory to the OS, but even
then, things like fragmentation may prevent it from doing so—if
you allocate, then free a really large block, it may be returned, but if
you allocate a lot of little blocks (which is what std::list does),
the runtime will in fact allocate large blocks from the OS, which it
parcels out; such large blocks cannot be returned until all small blocks
in them have been freed, and likely won't be returned even then.
It depends on how you're measuring the memory usage. If it's measuring the process memory in use, this is what you might actually expect.
It's quite common for a program to request memory and have that assigned from the controlling environment (such as an operating system) to the process but, when the memory is freed, it doesn't necessarily get taken away from the process. It may be returned to a free pool within the process.
This was the way allocation used to work in the olden days. A call to brk or sbrk would increase the size of the heap by giving more memory to the process. That memory would be added to the arena from which malloc calls were satisfied.
But, free would return the memory to the arena, not necessarily back to the operating system.
I imagine something similar is happening in this case.
Your memory profile is actually the process's address space consumption (the sum of mmap-ed pages, as e.g. given by /proc/self/statm or /proc/self/maps from the point of view of the process itself).
But when a C or C++ function release memory (previously allocated with malloc or new, which are using mmap to get memory from the Linux kernel) using free or delete, it is not given back to the system (using munmap -because that would be too slow or impractical [fragmentation issues] - but just kept as reusable for future malloc or new.
So deallocation did happen when requested by free but the memory is not given back to the system, but kept for future re-use.
If you really wanted the memory to be given back, write your own allocator (above mmap and munmap) but usually it is not worth the effort.
Perhaps using Boehm's GC could help (it is very useful, to avoid bothering about free-ing or delete -ing) if you explicitly call GC_gcollect() (but I am not sure of that), but you really should not care that much.
And your question is not technically related to gcc (it would be the same with another C++ compiler). It is related to malloc and new (i.e. to standard C & C++ libraries) under Linux.