I would like to setup a batch process as follows on Amazon AWS:
take snapshot of volumes tagged "must_backup"
share those snapshots with account B
make a copy of those snapshots within account B
the purpose of this is to protect the backups in case the first Amazon AWS account gets compromised.
I know how to automate steps 1 & 3, however I cannot find a commandline example on how to perform step 2.
The official documentation https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modifying-snapshot-permissions.html
does not provide any sample and does not clearly state how to specify the target account on the commandline.
I've double checked the previous solution and it's not ok. Basically "sharing" a snapshot means allowing other accounts to create a volume from that snapshot.
This implies adding a value to the "createVolumePermission" attribute
aws ec2 modify-snapshot-attribute --snapshot-id snap-<id> --user-ids <user-id-without-hypens> --attribute createVolumePermission --operation add
the operation might take some time (minutes?) after that you'll be able to query the attribute this way:
aws ec2 describe-snapshot-attribute --snapshot-id snap-<id> --attribute createVolumePermission
PS: for the purposes mentioned in the question this is probably not enough since the 'destination' account will not be able to see any of the tags from the source account, thus it will be impossible to perform a correct backup if the source account shares multiple snapshots with the same size
Example Commands for aws cli: copy ec2 snapshot
aws ec2 modify-snapshot-attribute --snapshot-id snap-1234567890 --user-ids other-amazon-account-id
Related
Does anyone know of an AWS CLI command that will list any running instance (run against a particular region) that doesn't have a snapshot available.
The closest command Ive found to try would be something like:
aws ec2 describe-snapshots --owner-ids self --query 'Snapshots[]' --region=us-east-1
I didn't actually get any return on it - just:
-------------------
|DescribeSnapshots|
+-----------------+
This is supposed to name every EC2 snapshot for each instance -- so I would have to subtract these ones from the entire EC2 inventory to reveal EC2 instances without.
Hence - I would like a command that would show running EC2 instances without any snapshots available -- so I can put something in place going forward.
Amazon EBS Snapshots are associated with Amazon EBS Volumes, which are associated with Amazon EC2 instances.
Therefore, you would need to write a program using an AWS SDK (I'd use Python, but there are many available) that would:
Obtain a list of all EBS Snapshots (make sure you use the equivalent to --owner-ids self), in which the return data will include the associated EBS VolumeId
Obtain a list of all EBS Volumes, in which the return data will include Attachments.InstanceId
Obtain a list of all running EC2 instances
Do a bit of looping logic to find Volumes without Snapshots, and then determine which instances are associated to those Volumes.
Note that rather than finding "instances without snapshots" it has to find "instances that have volumes without snapshots".
I don't think there is by default a CLI command that will allow you to do this. You can tag your snapshots with your instance ids for example then can query snapshots by filtering on the tags. Or you will have to use AWS SDK and create a custom script to allow you the get all instances and then check their volume ids if they have snapshots created or not.
How can I automate EC2 instance snapshots every X time?
By snapshot, I mean an image of all data and state and configuration of the virtual machine, so I can recover it quickly. Is there an AWS service for this purpose? What's the best way?
My EC2 instance type is m5.2xlarge
You may want to investigate a service called AWS Backup. I have only read abou it, as it's relatively new, and I had imlemented a custom solution using a Lambda function before it became available,
If I were to do it again, I'd use AWS Backup.
Some of the options are;
Use AWS Back Up service
OR
Schedule AWS Systems Manager Automation AWS-CreateImage or AWS-CreateSnapshot (choose either of these or both depends upon your use-cases) using AWS EventBridge or AWS CloudWatch
AWS has an service called Lifecycle Manager in EC2 dashboard. With the help of it, you can automate the backup procedure of the EBS volumes.
You can define backup and retention schedules for EBS snapshots by
creating lifecycle policies based on tags defined for volumes.
With this feature, you no longer have to rely on custom scripts to
create and manage your backups.
This feature is now available in the US East (N. Virginia), US
West (Oregon), and Europe (Ireland) AWS regions at no additional
cost.
You mention "all data and state and configuration of the virtual machine". This really consists of two parts:
Contents of the disks
Configuration of the Amazon EC2 virtual machine ("instance")
Configuration of virtual machine
Backups normally only consist of the contents of the disks. The configuration of the virtual machine is specified when a replacement machine is launched, including:
Instance Type
Subnet & IP Address
Tags
IAM Role
It is not possible to "back-up" these settings, but you could create a CloudFormation template that launches an instance with matching settings. This can then become a repeatable, automated process.
Contents of the disks
The easiest way to backup the contents of the disks to allow the launch of an equivalent instance would be to create an Amazon Machine Image (AMI). The AMI contains a copy of all disks connected to the instance.
A new Amazon EC2 instance can then be launch from the AMI, and it will contain exactly the same data on the disks. (An AMI consists of Amazon EBS Snapshots, plus some metadata about the instance configuration. A new instance can be launched from an AMI, but not from an EBS Snapshot.)
If you wish to automate the regular creation of an AMI, you can use the Amazon Data Lifecycle Manager.
See: New – Lifecycle Management for Amazon EBS Snapshots | AWS News Blog
I also recommend that you test the backup by launching a new EC2 instance from the AMI.
I configured a set of Lifecycle manager Policies to back-up my EC2 instances last week, but I cannot find any relevant snapshots in the EBS snapshots section. Can someone please advise if I should look for the snapshots somewhere else, or if I should review any specific parameters that regard the policy, or if even I should use another method?
Thank you.
Schedule details
Lifecycle manager will create the backup as a regular EBS snapshot, EBS snapshots are stored in S3 however you do not have access to the snapshot other than through the console/API.
Based on your configuration it will only apply to a tag of Name with the value of Graylog v3.3.2. This will happen once a week at 12:30PM UTC on a Monday.
If the snapshots are not being generated check the following:
Do the target instances have this name and value assigned to them?
Does the execution IAM role have permissions to perform this action? If it has the default permissions then it will be fine to run.
Thank you for the answer and apologies for this issue I'm kind of new to AWS and I managed to solve it.
The issue was simply in the filter applied to the snapshots tab I wasn't seeing the full name. bottom line the policy was working fine after all.
We have an EC2 instance running in AWS EC2 instance. We have our ML algorithms and data that. We have also hosted a web-based interface also in that machine.
Now there are no new developments happening in that EC2 instance. We would like to terminate AWS subscription for a short period of time (for the purpose of cost-reduction and exploring new cloud services). Most importantly, we want to be in a position where we can purchase a new EC2 instance with a fresh AWS subscription, use the backup which we take now, and resume all operations (web-backend, SMS services for our app which is hosted in AWS, etc.).
What is the best way to do it? Is temporary termination of AWS subscription advisable?
There is no concept of an "AWS Subscription". AWS is charged on-demand, which means you only pay when you use resources.
If you temporarily do not want the Amazon EC2 instance, you could:
Stop the instance, which is like turning off the power. You will not be charged for the instance, but you will still pay for the disk storage attached to the instance. You can simply Start the instance again when you wish to use it. You will only be charged while the instance is running. OR
Create an image of the instance, then terminate the instance. This will create an Amazon Machine Image (AMI), which contains a copy of the disks. You can then launch a new Amazon EC2 instance from the AMI when you wish to use it again. This is a lower-cost option compared to simply stopping the instance, but it takes more effort to stop/start.
It is quite common for companies to stop Amazon EC2 instances at night or over the weekend to reduce costs while they are not needed.
EDIT: Just thought of a third option. Will test it and be back. Not worth it; it would involve creating an image from the EC2 instance and then convert that image to a VM image, storing the VM image in S3. There may be some advantages to this, but I do not see them.
I think you have two options, both of them very reasonably priced. If you can separate the data from the operating system, then your best option would be to use an S3 bucket as a file system within the EC2 instance. Your EC2 instance would use this bucket to store all your "ML algorithms and data" and, possibly, even your "web-based interface". Whenever you decide that you no longer need the processing capacity of the EC2, you would unmount the S3 bucket file system from the EC2 instance and terminate that instance. After configuring an appropriate lifecycle rule for the S3 bucket, it would transition to Glacier, or even Glacier Deep Archive [you must considerer the different options of long term storage]. In the future, whenever you want to work with your data again, you would move your data from Glacier back to S3, create a new EC2 instance, install your applications, mount your S3 bucket as a file system and you would have access to all your data. I think this is your least expensive and shortest recovery time objective option. To implement this option, look at my answer to this question; everything you need to use an S3 bucket as a regular folder inside the EC2 instance is there.
The second option provides an integrated solution, meaning the operating system and the data stay together, and allows you to restore everything as it was the day you stopped processing your data. It's made up of the following cycle:
Shutdown your EC2 and make a note of all the specs [you need them further down].
Export your instance to a virtual image, vmdk for example, and store it in your S3 bucket. Something like this:
aws ec2 create-instance-export-task --instance-id i-0d54b0682aa3998a0
--target-environment vmware --export-to-s3-task DiskImageFormat=VMDK,ContainerFormat=ova,S3Bucket=sm-vm-backup,S3Prefix=vms
Configure an appropriate lifecycle rule for the S3 bucket so that it transitions to Glacier, or even Glacier Deep Archive.
Terminate the EC2 instance.
In the future you will need to implement the inverse, so you will need to restore the archived S3 Object [make sure you you can live with the time needed by AWS to do this]
Import the virtual image as an EC2 AMI, something like this [this is not complete - you will need some more options that you saved above]:
aws ec2 import-image --disk-containers
Format=ova,UserBucket="{S3Bucket=sm-vm-backup,S3Key=vmsexport-i-0a1c382e740f8b0ee.ova}"
Create an EC2 instance based on the image and you're back in business.
Obviously you should do some trial runs and even automate the entire process if it's something that will be done frequently. I have a feeling, based on what you said, that the first option is a better option, provided you can easily install whatever applications they use.
I'm assuming that you launched an EC2 instance from a base Amazon Machine Image and then added your own software and models to it. As opposed to launched an EC2 instance from an AWS Marketplace offering.
The simplest thing to do is to create an Amazon Machine Image (AMI) from your running EC2 instance. That will capture the current state of the instance and persist it in your AWS account. Then you can terminate the instance. Later, when you want to recreate it, launch a new instance, selecting the saved AMI instead of a standard AMI.
An alternative is to avoid the need to capture machine state at all, by using standard DevOps practices to revision-control everything you need to recreate the state of a running machine.
Note that there are costs associated with an AMI, though they are minimal ($0.05 per GB-month of data stored, for example).
I had contacted AWS customer care regarding this issue. Given below is the response I received. Please add your comments on which option might be good for me.
Note: I acknowledge the AWS customer care team for their help.
I understand that you require some information on cost saving for your
Instance since you will not be utilizing the service for a while.
To assist you with this I would recommend checking out the Instance
Stop/Start link here:
==>https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html .
When you stop an Instance, you do not lose any data & you are not
charged for the resources any further. However please keep in mind
that you will still be charged for any EBS Storage Volumes attached to
the stopped Instance(s).
I also recommend checking out the below links on how you can reduce
your costs.
==>https://aws.amazon.com/premiumsupport/knowledge-center/reduce-aws-bill/
==>https://aws.amazon.com/blogs/compute/10-things-you-can-do-today-to-reduce-aws-costs/
That being said, please note that as I am in the billing department,
for the best assistance with the various plans you will require the
assistance of our Sales Team.
The Sales Team will be able to assist with ways to save while
maintaining your configurations.
You will be able to reach the Sales Team here:
==>https://aws.amazon.com/websites/contact-us/.
Once you have completed the details in the link, a member of the team
will be in touch with you at their soonest.
I am using AWS to launch a EC2 instance. Fortunately I did it without problem.
What I need now is to make a backup of the data.
I think snapshot is a good way to do it. I have been doing some research and I found a good tool to do it automatically (https://github.com/colinbjohnson/aws-missing-tools/tree/master/ec2-automate-backup).
The problem is that I think it is not enough to make snapshots. In my opinion a copy of the last snapshot needs to be in another region, but I don't know how to do it automatically. I have been searching on internet and only found this:
http://docs.aws.amazon.com/cli/latest/reference/ec2/copy-snapshot.html. The problem is that I don't know the snapshot id (considering it is generated automatically by the first tool I mentioned).
The question is: Do you know any tool that can help me with this problem? If don't, do you know another approach to get a solution.
It is important to know that the service which is given doesn't need to be up 24 hs.
It is my first time using servers so I don't know how long a region in amazon can be down.
You do not need to know a volume ID to use copy-snapshot in the AWS CLI. When executing the command you provide a value to the --source-snapshot-id option. This specifies the ID of the snapshot you want to copy. A snapshot can be copied in the same region or to another region via the --destination-region option.
You can simply call create-snapshot and then copy-snapshot giving it the generated snapshot ID to copy the snapshot to another region. This could be automated via a cron job if necessary.
You can set up a cron job to invoke aws cli which can copy the snapshots to an S3 bucket 'A'. And, cross region bucket replication can be enabled from the source bucket 'A' in region 1 to destination bucket 'B' in region 2. Whenever a snapshot is uploaded to 'A', it'll get replicated to 'B' as well. So, in case first region becomes offline, you can restore volumes from the snapshots in 'B' bucket in region 2.