Question about the consumption of memory in OpenCV - c++

Question

Cross post here
I have such simple code:
//OpenCV 3.3.1 project
#include<opencv.hpp>
#include<iostream>
using namespace std;
using namespace cv;
Mat removeBlackEdge(Mat img) {
if (img.channels() != 1)
cvtColor(img, img, COLOR_BGR2GRAY);
Mat bin, whiteEdge;
threshold(img, bin, 0, 255, THRESH_BINARY_INV + THRESH_OTSU);
rectangle(bin, Rect(0, 0, bin.cols, bin.rows), Scalar(255));
whiteEdge = bin.clone();
floodFill(bin, Point(0, 0), Scalar(0));
dilate(whiteEdge - bin, whiteEdge, Mat());
return whiteEdge + img;
}
int main() { //13.0M
Mat emptyImg = imread("test.png", 0); //14.7M
emptyImg = removeBlackEdge(emptyImg); //33.0M
waitKey();
return 0;
}
And this is my test.png.
As the Windows show, it's size is about 1.14M. Of course I know it will be larger when it is read in memory. But I fill cannot accept the huge consumption of memory.
If I use F10, I can know the project.exe take up 13.0M. I have nothing to say about it
When I run Mat emptyImg = imread("test.png", 0);, the consumption of memory is 14.7M. It is normal also.
But why when I run emptyImg = removeBlackEdge(emptyImg);, the consumption of memory is up to 33.0M?? It mean function removeBlackEdge cost my extra 18.3M memory. Can anybody tell me some thing? I cannot totally accept such huge consumption of memory. Is there I miss something? I use new emptyImg to replace the old emptyImg. I think I don't need extra memory to cost?
Ps: If I want implement to do same thing by my removeBlackEdge( delete that black edge in the image), how to adjust it to cost minimum consumption of memory?

Answer 1

Below - guesswork and hints. This is a large comment. Please dont mark it as answer even if it was helpful in finding/solving the problem. Rather than that, if you manage to find the cause, write your own answer instead. I'll probably remove this one later.
Most probably, image/bitmap operations like threshold(img, bin, ..) create copies of the original image. Hard to say how many, but it surely creates at least one, as you can see by the second bin variable. Then goes clone and whiteEdge, so third copy. Also, operations like whiteEdge - bin probably create at least one copy (result) as well. rectangle, dilate and floodfill probably work in-place and use one of the Mat variables passed as workarea and output at the same time.
That means you have at least 4 copies of the image, probably a bit more, but let's say it's 4 * 1.7mb, so ~7mb out of 19mb increase you have observed.
So, we've got 12mb to explain, or so it seems.
First of all, it's possible that the image operations allocate some extra buffers and keep for later reuse. That costs memory, but it's "amortized", so it wont cost you again if you do these operations again.
It's also possible that at the moment those image operations were called, that fact might have caused some dynamic libraries to be loaded. That's also a one-time operation that won't cost you the memory if used again.
Next thing to note is that the process and the memory usage as reported by Windows by simple tools is .. not accurate. As the process allocates and relases the memory, the memory consumption reported by Windows usually only increases, and does not decrease upon release. At least not immediatelly. The process may keep some "air" or "vacuum". There are many causes to this, but the point is that this "air" is often reusable when the program tries to allocate memory again. A process, with base memory usage of 30mb, that periodically allocates 20mb and releases 20mb may be displayed by Windows as always taking up 50mb.
Having said that, I have to say I actually doubt in your memory measurements methodology, or at least conclusions from observations. After a single run of your removeBlackEdge you cannot say it consumed that amount of the memory. What you observed is only that the process' working memory pool has grown by that much. This itself does not tell you much.
If you suspect that temporary copies of the image are taking up too much space, try getting rid of them. This may mean things as obvious as writing the code to use less Mat variables and temporaries, or reusing/deallocating bitmaps that are no longer needed and just wait until function ends, or less obvious things like selecting different algorithms or writing your own. You may also try confirming of that's the case by running the program several times with input images of different sizes, and plotting a chart of observed-memory--vs--input-
size. If it grows linearly (i.e. "memory consumption" is always ~10x input size) then probably that's really some copies.
On the other hand, if you suspect a memory leak, run that removeBlackEdge several times. Hundreds or thousands or millions of times. Observe if the "memory consumption" steadily grows overtime or not. If it does, then probably there's a leak. On the other hand, if it grows only once at start and then keeps steady at the same level, then probably there's nothing wrong and it was only some one-time initializations or amortized caches/workspaces/etc.
I'd suggest you do those two tests now. Further work and analysis depends on what you observe during such a long run. Also, I have to note that this piece of code is relatively short and simple. Aren't you optimising a bit too soon? sidenote - be sure to turn on proper optimisations (speed/memory) in the compiler. If you happen to use and observe a "debug" version, than you can dismiss your speed/memory observations immediatelly.