Callbacks for AWS EBS Backups or Snapshots - amazon-web-services

I want to make a backup of several EBS volumes attached to a EC2. There are Snapshots and AWS Backups available.
Issue is, before I can do a backup, I must execute a special script to freeze my application.
Why: it's somewhat im-memory, so this call forces all writes to disk to comlpete and also prevents new disk writes for a duration of a backup.
Is there a way to execute an arbitrary bash script before the backup job/snapshot?

Unfortunately, there's no native --pre-snapshot-script option for creating an EBS snapshot.
However, you could use a Lambda function, that is triggered based on a scheduled EventBridge rule, to trigger a script to be run before you then programmatically take the EBS snapshot. You'd need the SSM agent to be installed on your EC2 instance(s).
The idea is to use ssm:SendCommand and the AWS-RunShellScript managed document to run Linux shell commands on your EC2 instance(s).
You have a couple of options, specifically:
if you want to inline all of your 'special script' in your Lambda function, or download it from S3 during your user data script, or manually transfer it or ...
if you can run the Boto3 create_snapshot() (or the AWS CLI ec2 create-snapshot command) as part of your 'special script'. If not & you want to do it separately after doing send_command, you will also have to use the Boto3 ssm.get_command_invocation (or the AWS CLI ssm get-command-invocation command) to poll+wait for the command to finish, before you create your snapshot
What you decide to do is dependent on your specific requirements and how much infrastructure you want to manage, but the essence remains the same.
This could be a good starting point:
import boto3
def lambda_handler(event, context):
ssm = boto3.client('ssm')
commands = ["echo 'Command xxxxx of special script'"]
commands.append("aws ec2 create-snapshot --volume-id vol-yyyyy")
ssm.send_command(
InstanceIds=['i-zzzzz'],
DocumentName='AWS-RunShellScript',
Parameters={'commands': commands}
)

Related

How to know EC2 instance stopped time?

I really need to know about the stopped time of AWS EC2 instances. I have checked with AWS cloudtrail, but its not easy to find the exact stopped EC2 instance. Is possible to see exact time of stopped EC2 instances by aws-cli commands or any boto3 script?
You can get this info from StateTransitionReason in describe-instances AWS CLI when you search for stopped instances:
aws ec2 describe-instances --filter Name=instance-state-name,Values=stopped --query 'Reservations[].Instances[*].StateTransitionReason' --output text
Example output:
User initiated (2020-12-03 07:16:35 GMT)
AWS Config keeps track of the state of resources as they change over time.
From What Is AWS Config? - AWS Config:
AWS Config provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time.
Thus, you could look back through the configuration history of the Amazon EC2 instance and extract times for when the instance changed to a Stopped state.
Sometimes time is missing from StateTransitionReason, you can use CloudTrail and search for Resource Name = instance ID to find out StopInstance(s) API calls.
By default you can track back 90 days, or indefinitely if you create your own trail.

Automating the installation of CloudWatch agent

I just want to know if there are other ways to approach this problem:
I have an AWS multi account setup. The EC2's are going to be monitored over all the accounts and when alerts are triggered via sns there is a mail going to be sent. For all EC2's with Windows Server 2016 and later, Amazon Linux and Ubuntu 16.04 and 18.04 the SSM agents come pre-installed. That way I can push the CloudWatch agent via System Manager Run Command to the EC2's per AWS account.
I was wondering is there a more simple way that i can force that CloudWatch Agent is installed with every new EC2 deployed in an AWS account, without installing the agent manually on the instance or via Run Command?
I was thinking working with tags, something like: "IsMonitored" and as value true or false. for example everyday at 17hr a Lambda function will go over all the instances in that account and search for IsMonitored = false, Get that instance ID and with a (boto3?) cript push the agent on that instance. This seemed to complicated so i wanted to check if there is maybe other simple solutions that would do the same.
Thanks in advance,
Iman
To install a cloudwatch agent in each instance particular region you can implement by shell script.
The approach is:
Manual work is create some default configuration file in parameter store for both the type of instance a. for windows b. for linux based
In shell script
For particular region
Get the total number of ec2 instance id list
Check the platform which type of machine is using Windows or Linux based
If the platform is Windows then add Windows type configuration file from parameter store else add Linux configuration file
For getting platform name :
platform=$(aws ec2 describe-instances --instance-ids <instance id> --query 'Reservations[*].Instances[*].[Platform]' --output text)
For installing packages :
aws ssm send-command --instance-ids <instance id> --document-name "AWS-ConfigureAWSPackage" --parameters "name=AmazonCloudWatchAgent,action=Install,installationType=Uninstall and reinstall" --comment "Install CloudWatch Agent on EC2 Windows/Linux machine"
For start CWagent :
aws ssm send-command --instance-ids $one_instance --document-name "AmazonCloudWatch-ManageAgent" --parameters "mode=ec2,optionalRestart=yes,optionalConfigurationSource=ssm,action=configure,optionalConfigurationLocation=AmazonLinuxCloudWatchAgentConfig" --comment "Configure CloudWatch Agent on EC2 Linux machine"
For more reference you can use this link.
One simpler approach could be using prebaked AMI. First, spin up an EC2 with the normal AMI you use. Next, install the CloudWatch agent and create an image. From now on, you can spin up EC2's using the new AMI which has CloudWatch agent preinstalled.
If prebaked AMI doesn't work for you, I recommend using an infrastructure-as-code (IaC) tool like Ansible to automate the installation process.

Copying File From S3 To EC2 by User Data Approach

I have been searching solution for this task, all I find CLI approaches which I don't want.
I simply want:
I have an S3 Bucket, which has one private file, file can be an image/zip file anything.
And I want when I launch any EC2 instance it should have taken that file from S3 bucket to EC2 instance directory.
And for this, I want to use only EC2 User Data Approach.
The User Data field in Amazon EC2 passes information to the instance that is accessible to applications running on the instance.
Amazon EC2 instances launched with Amazon-provided AMIs (eg Amazon Linux 2) include a program called Cloud-Init that looks at the User Data and, if a script is provided, runs that script the first time that the instance is booted.
Therefore, you can configure a script (passed via User Data) that will run when the instance is first launched. The script will run as the root user. Your script could copy a file from Amazon S3 by using the AWS Command-Line Interface (CLI), like this:
#!
aws s3 cp s3://my-bucket/foo.txt /home/ec2-user/foo.txt
chown ec2-user foo.txt
Please note that you will need to assign an IAM Role to the instance that has permission to access the bucket. The AWS CLI will use these permissions for the file copy.
You mention that you do not wish to use the AWS CLI. You could, instead, write a program that calls the Amazon S3 API using a preferred programming language (eg Python), but using the CLI is much simpler.

Terminate AWS EC2 instance when SSM Run Command status changes

I would like to (1) launch an AWS EC2 instance, (2) run a shell script (that sends output to an S3 bucket) and (3) terminate the instance automatically when the script terminates, all remotely without logging into the instance. I have managed to get parts (1) and (2) working using the AWS CLI commands aws ec2 run-instances and aws ssm send-command. I am struggling with part (3) - getting the instance to terminate automatically when the script completes.
I have seen in the AWS docs that you can use CloudWatch to monitor the SSM Run Command status, and I thought that this might be a solution - when the status changes, terminate the instance. Is this a feasible option? If so, how do you implement it using AWS CLI?
Within the ssm script, you can issue a command to the operating system to shutdown the computer. If you launched the instance with a Shutdown behavior of Terminate, then this will terminate the instance.
Alternatively, the script can retrieve the Instance ID of the instance it is running on, and issue the aws ec2 terminate-instances command, specifying its own Instance ID.
See: Self-Terminating AWS EC2 Instance?

How to run a script on an EC2 instance remotely?

I have an EC2 instance and I need to download a file from its D drive through my program. Currently, it's a very annoying process because I can't access the instance directly from my local machine. The way what I am doing now is running a script on the instance and the instance uploads the file I need to S3 and my program access S3 to read the file.
Just wonder whether there is any simple way to access the drive on the instance instead of going through S3?
I have used AWS DataPipeline and its task runner to execute scripts on a remote instance. The taskrunner waits for a pipeline event published to its worker group.
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-using-task-runner.html
I use it to execute shell script and commands on a schedule. The script to run should be uploaded to S3, and the Data pipeline template specifies the script's path. Works great for periodic tasks. You can do anything you want on the remote box via the script.
You cannot directly download the file from EC2, but via s3( or maybe using scp command) from your remote ec2.
But to simplify this annoying process you can use AWS Systems Manager.
AWS Systems Manager Run Command allows you to remotely and securely run set of commands on EC2 as well on-premise server. Below are high-level steps to achieve this.
Attach Instance IAM role:
The ec2 instance must have IAM role with policy AmazonSSMFullAccess. This role enables the instance to communicate with the Systems Manager API.
Install SSM Agent:
The EC2 instance must have SSM agent installed on it. The SSM Agent process the run command requests & configure the instance as per command.
Execute command :
Example usage via AWS CLI:
Execute the following command to retrieve the services running on the instance. Replace Instance-ID with ec2 instance id.
aws ssm send-command --document-name "AWS-RunShellScript" --comment "listing services" --instance-ids "Instance-ID" --parameters commands="service --status-all" --region us-west-2 --output text
More detailed information: https://www.justdocloud.com/2018/04/01/run-commands-remotely-ec2-instances/