Alert email when worker fails on AWS EC2 - amazon-web-services

Question

I have an EC2 instance in AWS with Centos 6 and I only have supervisor on it which maintains a single PHP script. In some cases this script fails and I can see something like this:
$ sudo /usr/local/bin/supervisorctl status
my-worker EXITED Aug 19 10:19 AM
I would like to receive alert email about it because my script hasn't worked since Aug 19.
I try to find something related to health checks, but health check available only for load balancers. Also I tried to find something in CloudWatch but couldn't find a relevant metric for me.
Any idea, how i can receive email when my worker fall down?

Answer 1

There isn't an out of the box metric for something like that as Cloudwatch by default only has access to hypervisor level metrics rather than OS based metrics such as RAM usage or process related statistics.
To augment the data in Cloudwatch you could write a small script that checks whether the process is running and then calls PutMetricData to upload that metric to Cloudwatch.
Something like this should work:
#!/bin/bash
${process_name}=$1
DATE=`date +%Y-%m-%dT%H:%M:%S.000Z`
processes_running=`pidof ${process_name} | wc -w`
aws cloudwatch put-metric-data --metric-name ${process_name}_running --namespace "MyService" --value ${processes_running} --timestamp $DATE
Then just call that with cron or something every minute (or however often you want to update Cloudwatch - max resolution is 1 minute though, more frequent calls will be aggregated)
Then you just need to create an alarm that performs some action (such as using SNS to send an email to all subscribed addresses but potentially also performing some action such as rebooting the instance).