If I setup just one Docker container on my AWS - and use only default configuration - would this docker container use the whole memory and all the processors?
Or do I need to configure it?
Memory
There is no memory limit for the container be default, it can use as much as it can.
You can configure the memory usage as below using "-m" flag of docker run command
-m, --memory="" Memory limit (format: <number><optional unit>, where unit = b, k, m or g)
$docker run -t -i -m 500M centos6_my /bin/bash
Processors
Containers can by default run on any of the available CPUs, CPU usage can be configured using below "-c" and "--cpuset" flag of docker run command
-c, --cpu-shares=0 CPU shares (relative weight)
--cpuset="" CPUs in which to allow execution (0-3, 0,1)
please read Docker documentation for more details : link
Related
I'm running an AWS EC2 m5.large (a none burstable instance). I have setup one of AWS CloudWatch's default metrics (CPU %) + some custom metrics (memory + disk usage) in my dashboard.
But when I compare the numbers CloudWatch report to me they are pretty far from then actually usage of the Ubuntu 20.04 server when I log in to it...
Actual usage:
CPU: ~ 35 %
Memory: ~ 33 %
CloudWatch report:
CPU ~ 10 %
Memory: ~ 50-55
https://www.screencast.com/t/o1nAnOFjVZW
I have followed AWS own instructions to add the metrics for memory and disk usage (Because CloudWatch doesn't out of the box have access to O/S level stuff): https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html
When numbers are so far from each other - then it would be impossible to setup useful alarms and notifications. I can't believe that is what AWS wants to provide to the people who chose to followed their original instructions?
The only thing with match exactly is the disk usage %.
HOW TO INSTALL AWS AGENT AT UBUNTU 20.04 (NEWER WAY INSTEAD OF THE OLD SCRIPT: "CloudWatchMonitoringScripts")
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html
1. sudo wget https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb
2. sudo dpkg -i -E ./amazon-cloudwatch-agent.deb
3. sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
4. Go through all the steps in the wizard (The result is saved here: /opt/aws/amazon-cloudwatch-agent/bin/config.json)
Hint: I answered:
- Default to most questions and otherwise:
- NO --> Do you want to store the config in the SSM parameter store? (Because when I answered YES it failed later on because of some permission-issue and I didn't know how to make it happy and I don't think I need SSM in regards to this)
- YES --> Do you want to turn on StatsD daemon?
- YES --> Do you want to monitor metrics from CollectD?
- NO --> Do you have any existing CloudWatch Log Agent?
Now to prevent this error: Error parsing /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml, open /usr/share/collectd/types.db: no such file or directory
https://github.com/awsdocs/amazon-cloudwatch-user-guide/issues/1
5. sudo mkdir -p /usr/share/collectd/
6. sudo touch /usr/share/collectd/types.db
7. sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
8. /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
{
"status": "running",
"starttime": "2020-06-07T10:04:41+00:00",
"version": "1.245315.0"
}
https://www.screencast.com/t/42VWgoS88Z (Create IAM role, add policies and make it the server default role).
https://www.screencast.com/t/fAUUHCPe (CloudWatch new custom metrics)
https://www.screencast.com/t/8J0Saw0co (data match OK now)
https://www.screencast.com/t/x0PxOa799 (data match OK now)
I realized - that the second I login to the machine the CPU % usage goes up from 10 % to 30% and stays there (of course some increase was to be expected - but not that much in my opinion) which in my case explains the big difference earlier...I honestly don't now if this way in more precise than the older script - but this should be the right way to do it in year 2020 :-) And you get access to 179 custom metrics when selecting "Advanced" during the wizard (even though only few would be valuable to most people)
When I run a new install of WordPress or a simple build command for some of my web apps in Jenkins the server grinds to a halt. In Netdata it appears the culprit is high "iowait".
I know that I can increase the IOPS on the EBS volume but I'd rather just wait a longer time for the process to finish. Is there a way to limit IOPS on a docker container (in this case; my Jenkins container)?
Try --device-read-iops and --device-write-iops option of docker run command.
The command should be something like this
docker run -itd --device-read-iops /dev/sda:100 --device-write-iops /dev/sda:100 image-name
NOTE: /dev/sda is the device name and 100 is number of iops per second
You can also limit io in terms of bytes using
--device-read-bps and --device-write-bps option.
Check this documentation for more info.
https://docs.docker.com/engine/reference/run/
In short, I am looking to see if it is possible to run multiple Docker containers on the same machine via gcloud's create-with-container functionality (or similar). The idea is that there will be some "worker" container (which does some arbitrary work) which runs and completes, followed by a "cleanup" container which subsequently runs performing the same task each time.
Longer explanation:
I currently have an application that launches tasks that run inside Docker containers on Google Cloud. I use gcloud beta compute instances create-with-container <...other args...> to launch the VM, which runs the specified container. I will call that the "worker" container, and the tasks it performs are not relevant to my question. However, regardless of the "worker" container, I would like to run a second, "cleanup" container upon the completion of the first. In this way, developers can write independently write Docker containers that do not have to "repeat" the work done by the "cleanup" container.
Side note:
I know that I could alternatively specify a startup script (e.g. a bash script) which starts the docker containers as I describe above. However, when I first tried that, I kept running into issues where the docker pull <image> command would timeout or fail for some reason when communicating with dockerhub. The gcloud beta compute instances create-with-container <...args...> seemed to have error handling/retries built-in, which seemed ideal. Does anyone have a working snippet that would provide relatively robust error handling in the startup script?
As far as I know the limitation is one container per VM instance. See limitations.
Answer: It is currently not possible to launch multiple containers with the create-with-container functionality.
Alternative: You mentioned that you have already tried launching your containers with a startup script. Another option would be to specify a cloud-init config through instance metadata. Cloud-init is built into Container-Optimized OS (the same OS that you would use with create-with-container).
It works by adding and starting a systemd service, which means that you can:
specify that your service should run after other services: network-online.target and docker.socket
specify a Restart policy for the service to do retries on failure,
add an ExecStopPost specification to run your cleanup (or add a separate service for that in the cloud-init config)
This is a snippet that could be a starting point (you would need to add it under user-data metadata key):
#cloud-config
users:
- name: cloudservice
uid: 2000
write_files:
- path: /etc/systemd/system/cloudservice.service
permissions: 0644
owner: root
content: |
[Unit]
Description=Start a simple docker container
Wants=network-online.target docker.socket
After=network-online.target docker.socket
[Service]
ExecStart=/usr/bin/docker run --rm -u 2000 --name=cloudservice busybox:latest /bin/sleep 180
ExecStopPost=/bin/echo Finished!
Restart=on-failure
RestartSec=30
runcmd:
- systemctl daemon-reload
- systemctl start cloudservice.service
I am running a Google compute instance with a coreos container (image name: coreos-stable-1688-4-0-v20180327). Copying files from Storage to the local filesystem with gsutil seems to work fine -- except that none of the supposedly copied files actually appears on the filesystem. Running the same copy command on a compute instance without using a container does work, so I imagine the issue is with the container. However, I'm not sure what about the container is causing the copy to fail.
The command is gsutil cp -r gs://my-bucket ./.
You're hitting the issue described in https://github.com/GoogleCloudPlatform/gsutil/issues/453. There's an alias set up for gsutil that runs gsutil within a container (which does not have access to the host filesystem), so the files are being copied to that container's filesystem, rather than your GCE host's. Some workarounds are suggested in that thread.
EDIT for better reading
(info from the GitHub issue thread):
Looks like GCE VMs have a nifty alias set up for gsutil:
$ type gsutil
gsutil is aliased to `(docker images google/cloud-sdk || docker pull google/cloud-sdk) > /dev/null;docker run -t -i --net=host -v /home/<USER>/.config:/root/.config google/cloud-sdk gsutil'
Potential workarounds on CoreOS instance:
Clone the gsutil repo, run git checkout <tag> to fetch the commit for the most recent release, install Python via these instructions, then make sure to run the local copy of gsutil directly on the CoreOS host, rather than running the containerized version of gsutil.
Override the gsutil alias, or make a new one, so that it mounts some part of the host filesystem on the container; this allows you to access the newly-written files after the container terminates.
Can I configure Linux swap space for an AWS Elastic Beanstalk environment?
I don't see an option for it in the console. From looking at /proc/meminfo on my running instances in my environment MemAvailable is looking quite low despite there being quite high Inactive values. I suspect there are a few dormant background processes that it would do no harm to page out, and would free up a non-trivial portion of the limited physical memory on the t2.nano I'm using.
I figured out how to do this using the .ebextensions config folder in my Tomcat web app.
Add a file .ebextensions/swap.config:
container_commands:
add-swap-space:
command: "/bin/bash .ebextensions/scripts/add-swap-space.sh"
ignoreErrors: true
Add a file .ebextensions/scripts/add-swap-space.sh:
#!/bin/bash
set -o xtrace
set -e
if grep -E 'SwapTotal:\s+0+\s+kB' /proc/meminfo; then
echo "Positively identified no swap space, creating some."
dd if=/dev/zero of=/var/swapfile bs=1M count=512
/sbin/mkswap /var/swapfile
chmod 000 /var/swapfile
/sbin/swapon /var/swapfile
else
echo "Did not confirm zero swap space, doing nothing."
fi
More docs: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
After a while running this allowed 150MiB to be swapped out on my t2.nano instance which runs the Elastic Beanstalk Java Tomcat platform with default heap options. From what I can see there is no ongoing paging while the application runs. It looks like some dormant data has been pushed to swap and the page cache is significantly larger (up from 30MiB to 180MiB).