In my project, there are one million inputs and I am supposed to take different numbers of inputs in order to compare sort/search algorithms. Everything was allright till I tried to take five hundread thousand inputs. Therefore, I have realized that I can't create five hundred thousand pointers to my class or even an integer type by using array. However, I can create five pointers with size of one hundred thousand.
If I didn't explain very well, just look these two codes;
int *ptr[500000]; // it crashes
int *ptr1[100000]; // it runs well
int *ptr2[100000];
int *ptr3[100000];
int *ptr4[100000];
int *ptr5[100000];
What is the reason of crashing? Is there a limiting or is it about memory? And of course, how can I fix it?
You are trying to allocate a 500,000-entry array on the stack. The stack is not really designed for holding large amounts of data like this. In your case, the stack just happens to be big enough to hold 100,000 entries (or even several different lots of 100,000 entries) but not 500,000 in a single block. If you overflow the stack, behaviour is undefined but a crash is likely.
You will get much better results by allocating your array on the heap instead.
int **ptr = malloc(500000*sizeof(int*));
Remember to check for a NULL return value from malloc, and free the memory when you're finished with it.
Related
I've searched far and wide and I cannot work out what this produces. I've seen no other examples where "new int [] is used twice within one array. Can anyone help?
int *t [2] = { new int [2], new int [2] };
t is an array of two int*, which are, generically, pointers to int.
The new operator is allocating an array of 2 consecutive int on the heap, by returning a memory address to that allocated memory (int*). This is done twice, thus allocating two arrays and storing them in the outer array.
Since new int [2] gives you a heap-allocated array of two integers (each time you call it), you'll end up with an array t of integer pointers pointing to distinct arrays of integers, something like this:
(array) (points to) (arrays-on-heap)
t: [0] -> [int0, int1]
[1] -> [int2, int3]
Were you to print out &t, &(t[0]) and &(t[1]), you would find that was an array with the first two items the same and the third slightly higher (the size of an int *). This is because array elements are consecutively placed.
Printing out t[0] and t[1] may have wildly disparate values since they can come from anywhere on the heap. They probably won't be that different simply because consecutive heap allocations tend to be done from consecutive memory(a), but they're likely to be separated by some memory - this is because many common allocation strategies involve allocating blocks of a minimum size/resolution, and with inline control information between blocks.
Printing out &(t[0][0]) and &(t[0][1]) will again give you close values since they form part of an array.
Note that those paragraphs above are not all mandated to be true by the standard, they're just the most common scenarios. It's possible that allocation strategies may involve exact sizes and out-of-line control information, but it would be unlikely.
(a) There may be exceptions to this in optimised allocators if, for example, different-sized requests come from different pools, or there's a separate preferred pool per thread. But, in the general case here, that's unlikely.
I'm very confused with regard to the following instructions:
#include <iostream>
#define MAX_IT 100
using namespace std;
class Integer{
private :
int a;
public:
Integer(int valoare){a=valoare;}
int getA(){return a;}
void setA(int valoare){a=valoare;}
};
int main(){
Integer* a=new Integer(0);
//cout<<a[0].getA();
for(int i=1;i<=MAX_IT;i++)
{
a[i]=*(new Integer(i));
}
for(int i=0;i<=MAX_IT;i++)
cout<<a[i].getA()<<endl;
return 13;
}
It works for small values of MAX_IT, but when I try to set MAX_IT to 1000 it doesn't work anymore.
Initially, I thought "new" operator was supposed to do the job, but after some reading documentation I understood it is not supposed to work at all like this (out of bound array).
So my question is: why is it working for small values of MAX_IT and not for bigger ones?
EDIT:
I am experimenting with this code for a larger program, where I am not allowed to use STL. You have not understood my concern: if I have Integer *var=new Integer[10]; for(int k=1;K<10;k++) *(var+k)=k; //this is perfectly fine, but if I try var[10]=new Integer; //this should not be working and should generate a memory problem //My concern is that it is working if I do it only 100 times or so...The question if why is it working everytime for small number of iterations?
Because by allocating space for one Integer then using it as an array of multiple Integers, your code invokes undefined behavior, meaning that it can do anything, including crashing, working seemingly fine, or pulling demons out of your nose.
And anyways it's leaking memory. If you don't need dynamic memory allocation, then don't use it.
a[i]=*(new Integer(i));
And kaboom, you lost the pointer to the Integer, no chance to delete it later. Leaks.
If you don't need raw arrays, don't use them. Prefer std::vector. Or switch to C if C++ is too hard.
std::vector<Integer> vec;
vec.push_back(Integer(1337));
The reason that things tend to work nicely when you overflow your buffer by just a little bit is... memory fragmentation! Who would have guessed?
To avoid memory fragmentation, allocators won't return you a block of just sizeof (Integer). They'll give you a somewhat larger block, to ensure that if the block is later freed before the adjacent blocks, it's at least big enough to be useful.
Exactly how big this is can vary by architecture, OS, compiler version, or even how much memory is physically present in the machine. You should consider it to be completely unpredictable. Also, some libraries designed to help catch this sort of bug force any small object to be placed at the end of the block instead of the beginning, so the extra bytes could be negative array indices instead of positive.
Therefore, don't ever rely on having spare area given to you for free after (or before) an object.
Guru note: Occasionally someone comes up with a valid use for the extra memory, and asks for a way to discover how large it is. One good example is that the capacity (not size!) of a std::vector could be adjusted to match the actual allocated space instead of the requested space, and therefore reduce (on average) the number of reallocations needed. Such requests usually come paired with other guru allocator APIs, such as the ability to expand an allocation in-place if there happen to be free blocks adjacent.
Note that in your particular case you do still have undefined behavior, because you're calling operator= on a non-POD object which hasn't first been constructed. If you gave class Integer a trivial default constructor that would change.
you actually need
Integer* a=new Integer[MAX_IT];
//cout<<a[0].getA();
for(int i=1;i<MAX_IT;i++) << note < not <=
{
a[i]=i;
}
better would be to use std::vector though
I have a assignment where I need to take a 7 digit input (a phone number) and check if it's found in the digits of pi. The digits of pi are stored in a supplied space separated text file. It seems reasonably straightforward: break the input into an array, read the digits of pi into an array, and check if a match is found. Long story short, I got the program working to my satisfaction. We were supplied text documents with the digits of pi in multiples of 10, 100, and so on up to 1 million digits. My program works up to 100,000 digits. But for whatever reason, on the 1 million digit file, it crashes with a generic windows error. I have no information on why it crashes and no error message is given (except the generic "a problem caused this program to stop working" message).
Noting that limits on the assignment state I cannot use any object-orientated code except for cin, cout, and the file stream objects (this limitation is because we've yet to get into classes and they don't want us using functions without knowing how they work).
Anyway, I'm looking for insight as to why the program is crashing. I'm using long ints on every variable that should need them (including counters and function returns), which should be sufficient, since they can go up to roughly 2 billion and there should not be any numbers larger than a million here.
Thanks for any help. I've been at this the past few hours with no success.
const long int numberOfDigits = 1000000;
int digitsOfPi[numberOfDigits];
int digitsOfPi[numberOfDigits];
The stack does not have enough room to hold such a large array. The stack is where automatic variables (AKA local variables) are stored. Memory is automatically allocated for local variables when execution enters a function and is freed when the function returns. The stack is great because of this automatic memory management, but one restriction is that its size is limited.
Large objects should go on the heap.1 The heap is a gigantic pool of memory from which you can allocate pieces dynamically whenever you like. The difference between the heap and the stack is that you're responsible for allocating and freeing heap memory. It does not get automatically freed for you.
To allocate memory on the heap in C++, use the new operator, with each new having a corresponding delete to free the memory once it's no longer needed. (Or in our case, we use new[] and delete[] since we're dealing with an array.)
// Allocate memory on the heap.
int *digitsOfPi = new int[numberOfDigits];
// Use it.
// Then free it.
delete[] digitsOfPi;
// Or better yet, once you're allowed to use the STL...
std::vector<int> digitsOfPi;
The larger question, though, is why you need to read all the digits of π into memory at once. A better design, though trickier to code, would only need a fixed O(1) amount of memory—say, 7 digits at a time.
See also
What and where are the stack and heap?
1 You could explore your compiler's options to increase the stack size, but that's not the right solution.
I've got a prewritten function in C that fills an 1-D array with data, e.g.
int myFunction(myData **arr,...);
myData *array;
int arraySize;
arraySize = myFunction(&arr, ...);
I would like to call the function n times in a row with slightly different parameters (n is dependent on user input), and I need all the data collected in a single C array afterwards. The size of the returned array is not always fixed. Oh, and myFunction does the memory allocation internally. I want to do this in a memory-efficient way, but using realloc in each iteration does not sound like a good idea.
I do have all the C++ functionality available (the project is in C++, just using a C library), but using std::vector is no good because the collected data is later sent in to a function with a definition similar to:
void otherFunction(myData *data, int numData, ...);
Any ideas? Only things I can think of are realloc or using a std::vector and copying the data into an array afterwards, and those don't sound too promising.
Using realloc() in each iteration sounds like a very fine idea to me, for two reasons:
"does not sound like a good idea" is what people usually say when they have not established a performance requirement for their software, and they have not tested their software against the performance requirement to see if there is any need to improve it.
Instead of reallocating a new block each time, the realloc method will simply keep expanding your memory block which will presumably be at the top of the memory heap, so it won't be wasting any time either traversing memory block lists, or copying data around. This holds true provided that whatever memory allocated by myFunction() gets freed before it returns. You can verify it by looking at the pointer returned by realloc() and seeing that it always (or almost always(*1)) is the exact same pointer as the one you gave it to reallocate.
EDIT (*1) some C++ runtimes implement two heaps, one for small allocations and one for large allocations, so if your block gets allocated in the heap for small blocks, and then it grows large, there is a possibility that it will be moved once to the heap for large blocks. So, don't expect the pointer to always be the same; just most of the time.
Just copy all of the data into an std::vector. You can call otherFunction on a vector v with
otherFunction(&v[0], v.size(), ...)
or
otherFunction(v.data(), v.size(), ...)
As for your efficiency requirement: it looks to me like your optimizing prematurely. First try this option, then measure how fast it is and only look for other solutions if it's really too slow.
If you know that you are going to call the function N times, and returned arrays are always M long, then why don't you just allocate one array M*N initially? Or if you don't know one of M or N, then set a worst case maximum. Or are M and N both dependent on user-input?
Then, change how you call your user-input-getting function, such that the array pointer you pass it is actually an offset into that large array, so that it stores the data in the right location. Then, next iteration, offset further, and call again.
I think best solution would be to write your own 1D array class with some methods which you need.
depending on how you write the class you'll get such result. (sorry bad grammar)..
I have a linked list of structures. Lets say I insert x million nodes into the linked list,
then I iterate trough all nodes to find a given value.
The strange thing is (for me at least), if I have a structure like this:
struct node
{
int a;
node *nxt;
};
Then I can iterate trough the list and check the value of a ten times faster compared to when I have another member in the struct, like this:
struct node_complex
{
int a;
string b;
node_complex *nxt;
};
I also tried it with C style strings (char array), the result was the same: just because I had another member (string), the whole iteration (+ value check) was 10 times slower, even if I did not even touched that member ever! Now, I do not know how the internals of structures work, but it looks like a high price to pay...
What is the catch?
Edit:
I am a beginner and this is the first time I use pointers, so chances are, the mistake is on my part. I will post the code ASAP (not being at home now).
Update:
I checked the values again, and I know see a much smaller difference: 2x instead of 10x.
It is much more reasonable for sure.
While it is certainly possible it was the case yesterday too and I was just so freaking tired last night I could not divide two numbers, I have just made more tests and the results are mind blowing.
The times for a the same number of nodes is:
One int and a pointer the time to iterate trough is 0.101
One int and a string: 0.196
One int and 2 strings: 0.274
One int and 3 strings: 0.147 (!!!)
For two ints it is: 0.107
Look what happens when there is more than two strings in the structure! It gets faster! Did somebody drop LSD into my coffee? No! I do not drink coffee.
It is way too fckd up for my brain at the mo' so I think I will just figure it out on my own instead of draining public resources here at SO.
(Ad: I do not think my profiling class is buggy, and anyway I can see the time difference with my own eyes).
Anyhow, thanks for the help.
Cheers.
I must be related to memory access. You speak of a million linked elements. With just an int and a pointer in the node, it takes 8 bytes (assuming 32 bits pointers). This takes up 8 MB memory, which is around the size of cache memory sizes.
When you add other members, you increase the overall size of your data. It does not fit anymore entirely in the cache memory. You revert to plain memory accesses that are much slower.
This may also be caused because during the iteration you may create a copy of your structures. That is:
node* pHead;
// ...
for (node* p = pHead; p; p = p->nxt)
{
node myNode = *p; // here you create a copy!
// ...
}
Copying a simple structure very fast. But the member you've added is a string, which is a complex object. Copying it is a relatively complex operation, with heap access.
Most likely, the issue is that your larger struct no longer fits inside a single cache line.
As I recall, mainstream CPUs typically use a cache line of 32 bytes. This means that data is read into the cache in chunks of 32 bytes at a time, and if you move past these 32 bytes, a second memory fetch is required.
Looking at your struct, it starts with an int, accounting for 4 bytes (usually), and then std::string (I assume, even though the namespace isn't specified), which in my standard library implementation (from VS2010) takes up 28 bytes, which gives us 32 bytes total. Which means that the initial int and the the next pointer will be placed in different cache lines, using twice as much cache space, and requiring twice as many memory accesses if both members are accessed during iteration.
If only the pointer is accessed, this shouldn't make a difference, though, as only the second cache line then has to be retrieved from memory.
If you always access the int and the pointer, and the string is required less often, reordering the members may help:
struct node_complex
{
int a;
node_complex *nxt;
string b;
};
In this case, the next pointer and the int are located next to each others, on the same cache line, so they can be read without requiring additional memory reads. But then you incur the additional cost once you need to read the string.
Of course, it's also possible that your benchmarking code includes creation of the nodes, or (intentional or otherwise) copies being created of the nodes, which would obviously also affect performance.
I'm not a spacialist at all, but the "cache miss" problem rings in my head while reading your problem.
When you had a member, as it makes the size of the structure get bigger, it also might cache misses when going throught the linked list (that is naturally cache-unfriendly if you don't have nodes allocated in one bloc and not far from each other in memory).
I can't find another explaination.
However, we don't have the creation and the loop provided so it's still hard to guess if you're not just having code that don't perform the list exploration in an efficient way.
Perhaps a solution would be a linked list of pointers to your object. It may make things more complicated (unless you use smart pointers, ect.) but it may increase search time.