I'm confused about the purpose of having both hard and soft memory limits for ECS task definitions.
IIRC the soft limit is how much memory the scheduler reserves on an instance for the task to run, and the hard limit is how much memory a container can use before it is murdered.
My issue is that if the ECS scheduler allocates tasks to instances based on the soft limit, you could have a situation where a task that is using memory above the soft limit but below the hard limit could cause the instance to exceed its max memory (assuming all other tasks are using memory slightly below or equal to their soft limit).
Is this correct?
Thanks
If you expect to run a compute workload that is primarily memory bound instead of CPU bound then you should use only the hard limit, not the soft limit. From the docs:
You must specify a non-zero integer for one or both of memory or memoryReservation in container definitions. If you specify both, memory must be greater than memoryReservation. If you specify memoryReservation, then that value is subtracted from the available memory resources for the container instance on which the container is placed; otherwise, the value of memory is used.
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html
By specifying only a hard memory limit for your tasks you avoid running out of memory because ECS stops placing tasks on the instance, and docker kills any containers that try to go over the hard limit.
The soft memory limit feature is designed for CPU bound applications where you want to reserve a small minimum of memory (the soft limit) but allow occasional bursts up to the hard limit. In this type of CPU heavy workload you don't really care about the specific value of memory usage for the containers that much because the containers will run out of CPU long before they exhaust the memory of the instance, so you can place tasks based on CPU reservation and the soft memory limit. In this setup the hard limit is just a failsafe in case something goes out of control or there is a memory leak.
So in summary you should evaluate your workload using load tests and see whether it tends to run out of CPU first or out of memory first. If you are CPU bound then you can use the soft memory limit with an optional hard limit just as a failsafe. If you are memory bound then you will need to use just the hard limit with no soft limit.
#nathanpeck is the authority here, but I just wanted to address a specific scenario that you brought up:
My issue is that if the ECS scheduler allocates tasks to instances
based on the soft limit, you could have a situation where a task that
is using memory above the soft limit but below the hard limit could
cause the instance to exceed its max memory (assuming all other tasks
are using memory slightly below or equal to their soft limit).
This post from AWS explains what occurs in such a scenario:
If containers try to consume memory between these two values (or
between the soft limit and the host capacity if a hard limit is not
set), they may compete with each other. In this case, what happens
depends on the heuristics used by the Linux kernel’s OOM (Out of
Memory) killer. ECS and Docker are both uninvolved here; it’s the
Linux kernel reacting to memory pressure. If something is above its
soft limit, it’s more likely to be killed than something below its
soft limit, but figuring out which process gets killed requires
knowing all the other processes on the system and what they are doing
with their memory as well. Again the new memory feature we announced
can come to rescue here. While the OOM behavior isn’t changing, now
containers can be configured to swap out to disk in a memory pressure
scenario. This can potentially alleviate the need for the OOM killer
to kick in (if containers are configured to swap).
Related
Does anyone know if memory usage have on lambda response time? I understand that memory allocation is directly correlated to CPU allocation, but what about % memory utilization. e.g. 100mb allocated but 95mb is being used(for dependencies, that should be in layers). Will that effect the execution time?
The utilisation at runtime does not change the allocated number of virtual CPU cores at runtime.
As you already know: the number of cores depends on the amount of memory you configure. But this is a configuration time allocation and has nothing to do with the runtime.
Commenter Anon Coward already mentioned, high memory utilisation still can have an impact on your Lambdas execution time. But it does not have to. It depends on what your code is actually doing.
The great thing is: you can measure all of this and you can therefore find out what the best memory size is for your Lambda function.
Even better, there are already projects that help you do exactly that:
https://github.com/alexcasalboni/aws-lambda-power-tuning
In AWS Batch, when I specify a memory requirement of e.g. 32000MB, my job ends up getting killed because (a) the actual instance autoselected has 64GB memory and (b) ECS seems to view 32000MB as both a requirement and a hard limit ("If your container attempts to exceed the memory specified here, the container is killed" from https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html). So as soon as my job goes slightly above 32GB, it gets killed, even though I am happy for it to use up to the 64GB.
How do I properly specify a minimum memory requirement without causing AWS Batch to kill jobs that go slightly above that? It seems very strange to me that the "memory" parameter appears to be both a minimum and a maximum.
I assume I'm misunderstanding something.
The memory requirements in the resourceRequirements property are always the maximum/upper bound. You specify there how much memory at max your job container is going to use.
Quote from https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html :
The hard limit (in MiB) of memory to present to the container. If your container attempts to exceed the memory specified here, the container is killed.
A lower/minimum bound would not make much sense, since AWS needs to put your job container onto a host that actually supports the upper bound/limit, because there is no way to tell a priori how much actual memory your container is going to use.
Or put another way: If there were such a thing as a "minimum" requirement and you specified minimum = 1 MiB and maximum = 16 GiB, what is AWS Batch supposed to do with this information? It cannot put your job container onto a host with say 512 MiB of memory because your job container, as it runs, may exceed that, since you said the maximum was 16 GiB (in this example). And AWS Batch is not going to freeze a running job and migrate it onto another host, once the current host's memory is reached.
The fact that AWS Batch decided to put your concrete job container onto an instance with 64 GiB may be coincidental because 32 GiB is just the border of an instance's memory size 32 GiB <-> 64 GiB. And if your job were to use the full 32 GiB then there wouldn't be any memory left for the host (without swapping).
I'm developing a c++ code on Linux which can run out of memory, go into swap and slow down significantly, and sometimes crash. I'd like to prevent this from happening by allowing the user to specify a limit on the proportion of the total system memory that the process can use up. If the program should exceed this limit, then the code could output some intermediate results, and terminate cleanly.
I can determine how much memory is being used by reading the resident set size from /proc/self/stat. I can then sum this up across all the parallel processes to give me a total memory usage for the program.
The total system memory available can be obtained via a call to sysconf(_SC_PHYS_PAGES) (see How to get available memory C++/g++?). However, if I'm running on a parallel cluster, then presumably this figure will only give me the total memory for the current cluster node. I may, for example, be running 48 processes across 4 cluster nodes (each with 12 cores).
So, my real question is how do I find out which processor a given process is running on? I could then sum up the memory used by processes running on the same cluster node, and compare this with the available memory on that node, and terminate the program if this exceeds a specified percentage on any of the nodes that the program is running on. I would use sched_getcpu() for this, but unfortunately I'm compiling and running on a system with version 2.5 of glibc, and sched_getcpu() was only introduced in glibc 2.6. Also, since the cluster is using on old linux OS (version 2.6.18), I can't use syscall() to call getcpu() either! Is there any other way to get the processor number, or any sort of identifier for the processor, so that I can sum memory used across each processor separately?
Or is there a better way to solve the problem? I'm open to suggestions.
A competently-run cluster will put your jobs under some form of resource limits (RLIMIT_AS or cgroups). You can do this yourself just by calling setrlimit(RLIMIT_AS,...). I think you're overcomplicating things by worrying about sysconf, since on a shared cluster, there's no reason to think that your code should be using even a fixed fraction of the physical memory size. Instead, you should choose a sensible memory requirement (if your cluster doesn't already provide one - most schedulers do memory scheduling reasonably well.) Even if you insist on doing it yourself, auto-sizing, you don't need to know about which cores are in use: just figure out how many copies of your process are on the node, and divide appropriately. (you will need to figure out which node (host) each process is running on, of course.)
It's worth pointing out that the kernel RLIMIT_AS, not RLIMIT_RSS. And that when you hit this limit, new allocations will fail.
Finally, I question the design of a program which uses unbounded memory. Are you sure there's not a better algorithm? Users are going to find your program pretty irritating to use if, after investing significant time in a computation, it decides it's going to try allocating more, then fails. Sometimes people ask this sort of question when they are mistakenly thinking that allocating as much memory as possible will give them better IO buffering (which is naive wrt pagecache, etc).
I have an Amazon RDS instance. Freeable Memory has been declining since setup over 1-2 weeks, starting from 15GB of memory down to about 250MB. As it has dropped this low in the last days, it has started to resemble a sawtooth pattern where Freeable Memory drops to this range (250 - 350MB) and then goes back up agin to 500 - 600MB in a sawtooth pattern.
There has not been any notable decline in application quality. However, I am worried that the DB will run out of memory and crash.
Is there a danger that the RDS instance will run out of memory? Is there some setting or parameter I should be looking at to determine if the instance is set up correctly? What is causing this sawtooth pattern?
Short answer - you shouldn't worry about FreeableMemory unless it is became really low (about 100-200 Mb) or significant swapping occur (see RDS SwapUsage metric).
FreeableMemory is not a MySQL metric, but OS metric. It is hard to give precise definition, but you can treat it as memory which OS will be able to allocate to anyone who request it (in your case it likely will be MySQL). MySQL have a set of settings which are restricting it's overall memory usage to some cap(you can use something like this to actually calculate it). It's unlikely that your instance will ever hit this limit, due to the fact that in general you never reach max number of connections, but this is still possible.
Now going back to "decline" in FreeableMemory metric. For the MySQL most of the memory consume by InnoDB buffer pool (see here for details). RDS instances in there config by default have size for this buffer set to 75% of hosts physical memory - which in your case is about 12 GB. This buffer is used for caching all DB data which used in both read and write operations.
So in your case, since this buffer is really big - it is slowly filling with data which cached (it is likely that this buffer is actually big enough to cache all DB). So when you first start you instance this buffer is empty and than once you start reading/writing stuff into DB all this data are bringing into cache. They will stay here up to the time when this cache became full and new request came. At this time least recently used data will be replaced with new data. So initial decline of FreeableMemory after DB instance restart explains with this fact. It is not a bad thing, cause you actually want as much as possible of you data to be cache in order for you DB to work faster. The only thing which can go nasty is when part or all of this buffer will be pushed out of physical memory into swap. At that point you will have huge performance drop.
As a preventive care it might be a good idea to tune MySQL max memory used for different thing in case you FreeableMemory metric is constantly on a level of 100-200 Mb, just to reduce possibility of swapping.
Freeable memory field is used by MySQL for buffering and caching for it`s own processes. It is normal for the amount of Freeable memory to decrease over time. I wouldn't be worried it kicks old info out as it demands more room.
After several support tickets at AWS I found that tuning the parameters groups can help, specially the shared buffer, lowering them to keep a reserved quantity to avoid drops or failovers due to lack of memory
We have an application that could potentially allocate a large number of small objects (depending on user input). Sometimes the application runs out of memory and effectively crashes.
However, if we knew that memory allocations were becoming tight there are some lower-priority objects which could be destroyed and thereby allow us to gracefully degrade the user results.
What's the best way to detect that memory for a process is running low before calls to 'new' actually fail? We could call API functions like GetProcessWorkingSetSize() or GetProcessMemoryInfo() but how do you know when the limits on a given machine are being reached (e.g. with 80% of maximum allocations)?
At start up, allocate a memory reserve.
Then use set_new_handler() to install a hook that will detect allocations failures.
When one happens:
Free the reserve (so you have enough free memory to work with).
Run your code that finds and frees low priority objects.
When that has done its job, try to reallocate the reserve again (for next time).
Finally return to let the original allocation attempt retry.
If it's a 32-bit process then you would want to ensure that you don't use more than 1.6GB, which is 80% of 2.0GB, the max allowed for your process. Calling GlobalMemoryStatusEx will fill in the struct MEMORYSTATUSEX.ullAvailVirtual, when this is only 400MB available (or less) then you are at your threshold.
Check this answer Win32/MFC: How to find free memory (RAM) available?.
You need to periodically find available free memory and stop allocating at some limit.
As explained in above referred answer, you can use GlobalMemoryStatusEx, and/or VirtualQueryEx.