I have a few Docker containers running on EC2 instances in AWS. In the past I have had situations where the Docker containers simply exit due to errors on the docker daemon, and they never start up even though the restart policies are in place (daemon is not running so I don't expect them to get up of course).
Since I am going on holiday I want to implement a quick and easy solution that would allow me to be notified if any containers have exited unexpectedly. The only quick solution I could find was using an Amazon Event Bridge rule for running a scheduled task every X minutes and executing a Systems Manager RunDockerAction command (docker ps) on the instances, but this does not give me any output except for the fact that the command has successfully executed on the instance.
Is there any way that I can get the output of such an Event Bridge task to send the results over an SNS topic if things go wrong?
IF you are running Linux on your AWS EC2 instance, then one solution is to use e-mail as a notification system. In that case, I would suggest the following:
On the AWS EC2 instance, create a Bash script that runs docker ps -a and combine that with a grep statement to filter on the docker container IDs that you want to monitor.
In the same Bash script, using echo and mail, you can e-mail yourself with statistics seen in the previous step. For example"
echo "${container} is not running" | mail -s "Alert! Docker container ${container} is not running!" "first.last#domain.com"
(The above relies on $container to be set appropriately. Use grep to filter out data of interest.)
Create a system crontab job (etc/crontab) and schedule the Bash script to run at your wanted interval.
This is only one possible solution, one that I use myself for quick checks at times.
When I run a new install of WordPress or a simple build command for some of my web apps in Jenkins the server grinds to a halt. In Netdata it appears the culprit is high "iowait".
I know that I can increase the IOPS on the EBS volume but I'd rather just wait a longer time for the process to finish. Is there a way to limit IOPS on a docker container (in this case; my Jenkins container)?
Try --device-read-iops and --device-write-iops option of docker run command.
The command should be something like this
docker run -itd --device-read-iops /dev/sda:100 --device-write-iops /dev/sda:100 image-name
NOTE: /dev/sda is the device name and 100 is number of iops per second
You can also limit io in terms of bytes using
--device-read-bps and --device-write-bps option.
Check this documentation for more info.
https://docs.docker.com/engine/reference/run/
Is it possible to install additional libs/dependencies after the cluster is already up and running?
Things I've done that are related to this:
I've already done the pre-creation bootstrapping process (this is a different solution altogether)
alternatively, SSH'd into each node and installed dependencies after the cluster comes up
I think the post-startup installation solution would involve being able to fire off a command to all the executors from the driver. Is there a way to accomplish this?
If you want additional dependencies to your steps, you can add them in the command of the step ( i.e. for Spark use the --jars option )
So my setup on AWS is 1 master node and 2 executor nodes.
I'd expect both 2 executor nodes would work on my task but I can see only one gets registered normally, the other one as ApplicationMaster. I can also see that 16 partitions at the time are processed.
I use spark-shell for now. All the default settings, EMR 4.3. Command to start the shell:
export SPARK_EXECUTOR_MEMORY=20g
export SPARK_DRIVER_MEMORY=4g
spark-shell --num-executors 2 --executor-cores 16 --packages com.databricks:spark-redshift_2.10:0.6.0 --driver-java-options "-Xss100M" --conf spark.driver.maxResultSize=0
Any ideas where to start debugging this? Or is it correct behaviour?
I think the issue is that you are running in 'cluster' mode and the spark driver is running inside an application master on one of the executor nodes, and using 1 core. Therefore because your executors require 16 cores, one of the nodes only has 15 cores available and does not have the required resources to launch a second executor. You can verify this by looking at "Nodes" in the YARN UI.
The solution may be to launch the spark shell in client mode --deploy-mode client or change the number of executor cores.
In my Django project, I use Celery and Rabbitmq to run tasks in background.
I am using celery beat scheduler to run periodic tasks.
How can i check if celery beat is up and running, programmatically?
Make a task to HTTP requests to a Ping URL at regular intervals. When the URL is not pinged on time, the URL monitor will send you an alert.
import requests
from yourapp.celery_config import app
#app.task
def ping():
print '[healthcheck] pinging alive status...'
# healthchecks.io works for me:
requests.post("https://hchk.io/6466681c-7708-4423-adf0-XXXXXXXXX")
This celery periodic task is scheduled to run every minute, if it doesn't hit the ping, your beat service is down*, the monitor will kick in your mail (or webhook so you can zapier it to get mobile push notifications as well).
celery -A yourapp.celery_config beat -S djcelery.schedulers.DatabaseScheduler
*or overwhelmed, you should track tasks saturation, this is a nightmare with Celery and should be detected and addressed properly, happens frequently when the workers are busy with blocking tasks that would need optimization
Are you use upstart or supervison or something else to run celery workers + celery beat as a background tasks? In production you should use one of them to run celery workers + celery beat in background.
Simplest way to check celery beat is running: ps aux | grep -i '[c]elerybeat'. If you get text string with pid it's running. Also you can make output of this command more pretty: ps aux | grep -i '[c]elerybeat' | awk '{print $2}'. If you get number - it's working, if you get nothing - it's not working.
Also you can check celery workers status: celery -A projectname status.
If you intrested in advanced celery monitoring you can read official documentation monitoring guide.
If you have daemonized celery following the tutorial of the celery doc, checking if it's running or not can be done through
sudo /etc/init.d/celeryd status
sudo /etc/init.d/celerybeat status
You can use the return of such commands in a python module.
You can probably look up supervisor.
It provides a celerybeat conf which logs everything related to beat in /var/log/celery/beat.log.
Another way of going about this is to use Flower. You can set it up for your server (make sure its password protected), it somewhat becomes easier to notice in the GUI the tasks which are being queued and what time they are queued thus verifying if your beat is running fine.
I have recently used a solution similar to what #panchicore suggested, for the same problem.
Problem in my workplace was an important system working with celery beat, and once in a while, either due to RabbitMQ outage, or some connectivity issue between our servers and RabbitMQ server, due to which celery beat just stopped triggering crons anymore, unless restarted.
As we didn't have any tool handy, to monitor keep alive calls sent over HTTP, we have used statsd for the same purpose. There's a counter incremented on statsd server every minute(done by a celery task), and then we setup email & slack channel alerts on the grafana metrics. (no updates for 10 minutes == outage)
I understand it's not purely a programatic approach, but any production level monitoring/alerting isn't complete without a separate monitoring entity.
The programming part is as simple as it can be. A tiny celery task running every minute.
#periodic_task(run_every=timedelta(minutes=1))
def update_keep_alive(self):
logger.info("running keep alive task")
statsd.incr(statsd_tags.CELERY_BEAT_ALIVE)
A problem that I have faced with this approach, is due to STATSD packet losses over UDP. So use TCP connection to STATSD for this purpose, if possible.
You can check scheduler running or not by the following command
python manage.py celery worker --beat
While working on a project recently, I used this:
HEALTHCHECK CMD ["stat celerybeat.pid || exit 1"]
Essentially, the beat process writes a pid file under some location (usually the home location), all you have to do is to get some stats to check if the file is there.
Note: This worked while launching a standalone celery beta process in a Docker container
The goal of liveness for celery beat/scheduler is to check if the celery beat/scheduler is able to send the job to the message broker so that it can be picked up by the respective consumer. [Is it still working or in a hung state]. The celery worker and celery scheduler/beat may or may not be running in the same pod or instance.
To handle such scenarios, we can create a method update_scheduler_liveness with decorator #after_task_publish.connect which will be called every time when the scheduler successfully publishes the message/task to the message broker.
The method update_scheduler_liveness will update the current timestamp to a file every time when the task is published successfully.
In Liveness probe, we need to check the last updated timestamp of the file either using:
stat --printf="%Y" celery_beat_schedule_liveness.stat command
or we can explicitly try to read the file (read mode) and extract the timestamp and compare if the the timestamp is recent or not based on the liveness probe criteria.
In this approach, the more minute liveness criteria you need, the more frequent a job must be triggered from the celery beat. So, for those cases, where the frequency between jobs is pretty huge, a custom/dedicated liveness heartbeat job can be scheduled every 2-5 mins and the consumer can just process it. #after_task_publish.connect decorator provides multiple arguments that can be also used for filtering of liveness specific job that were triggered
If we don't want to go for file based approach, then we can rely on Redis like data-source with instance specific redis key as well which needs to be implemented on the same lines.