I checked this answer but its not clear the steps to follow to create a automatic aws snapshot from a EBS volume.
In the AWS documentaion they talked about creating a new Snapshot but not how to do it automatically?
Can anyone tell be the steps to make snapshot automatically on regular basis?
You can use the API and create a windows task locally on your computer to run this command daily or weekly.
for example
ec2-create-snapshot vol-xxxxxxxxx --description "xxxxxxxxxxxxxxx"
You may want to try this versatile script https://github.com/evannuil/aws-snapshot-tool
Related
I am quiet new to AWS and trying to learn things. I currently have a live production environment and trying to create a test environment. I was able to create an image of an ec2 instance in AWS using the 'Create Image' option in the 'Actions'. Now I am trying to create an image/ duplicate of the RDS Database in AWS in a similar way. Is there any easy way to clone the database so that I don't change any data in the original database and perform modifications only in the test database. Thanks for all your help.
Probably worth reading up on creating snapshots and restoring from a snapshot.
To give you a quick summary; you will take a snapshot of your production RDS DB instance. Then you will launch a new DB instance from this snapshot.
As long as you don't need to launch the new instance in a separate AWS Account, the documentation should tell you all you need to know.
I built and provisioned a AMI using packer and amazon-ebs.
I need to rebuild the AMI weekly. Is there a simple solution for this? Do I need a separate ec2 for jenkins or is that overkill? Would any CI tool be good for this or is there more simple approach? My packer ami code is hosted on github.
In addition, I create a new ec2 instance from AMI and tear down old one weekly. Whats the best way to schedule ec2 tear-downs and rebuilds automatically?
So 2 issues:
Weekly rebuild of AMI
Weekly rebuild of ec2 based on rebuilt AMI
Im not experienced with any devops things so please excuse me.
I'm assuming this is the only task where you want to use an automation server. In other case, I will suggest you create a Jenkins or any other automation server. It all depends on your need.
To automate this single task, you don't necessarily need an automation server. The one method I'm going to demonstrate is one among many ways you can do it. Below are the AWS resources you require.
A Docker image where packer, aws cli, and any other dependencies installed.
An ECS task configured using the image in #1.
A CloudWatch schedule expression to trigger the ECS task periodically, in this case weekly.
Your docker image should be configured such that container execution does the rebuild of the AMI. You can write a bash script for this and configure the same as the container entry point.
The second point, rebuild of EC2 server is not a best practice. You should have a separate process in place to apply the AMI changes to respective instances. However, you can do this by scheduling a lambda function which will terminate and launch a new instance.
I know this is a broad answer and there are many other ways to do the same.
We are working on setting up a GPC environment with some Windows servers. Traditional backup is to backup our data daily. I know that I can run a disk snapshot.
gcloud compute --project=projectid disks snapshot diskname --zone=zonename --snapshot-names=snapshotname
I also understand that the snapshot is a forever incremental snapshot. However, I want an ability to schedule this. I am not sure what the best approach is for this. I am not sure if this is even the best way to do this.
I appreciate any guidance in regards to do backups of instances. I have this created in AWS using Lambda I am just not sure how to do this in GPC.
Google App Engine can schedule tasks. You could use this to invoke a function running in a GAE app that calls the snapshot API.
You can now set snapshot schedule to backup persistent disks.
https://cloud.google.com/compute/docs/disks/scheduled-snapshots
I am using AWS Data Pipeline to import some CSV data from S3 to Redshift. I also added a ShellCommandActivity to remove all S3 files after the copy activity completed. I attached a picture with the whole process.
Everything works fine but each activity starts it's own EC2 instance. Is it possible that the ShellCommandActivity to reuse the same EC2 instance as the RedshiftCopyActivity, after the copy command completed?
Thank you!
Unless you can do all activities in shell or CLI, it is not possible to do everything in the same instance.
One suggestion I can give is to move on to new technologies. AWS Data Pipeline is outdated (4 years old). You should use AWS Lambda which will cost you a fraction of what you are paying and you can load the files into Redshift as soon as the files are uploaded to S3. Clean up is automatic and Lambda is much more powerful than AWS Data Pipeline. The tutorial A Zero-Administration Amazon Redshift Database Loader is the one you want. Yes, there is some learning curve, but as the title suggest it is a zero administration load.
In order for the ShellCommandActivity to run on the same EC2 instance, I edited my ShellCommandActivity using Architect and for the Runs On option a chose Ec2Instance. The ShellCommandActivity gets mapped automatically to the same EC2Instance as the RedshiftCopyActivity. Now the whole process looks like this:
Thank you!
I am new with AWS services and trying some use-cases. I want to create EMR clusters on demand with some predefined configurations and applications/scripts installed. I was planning to create a snapshot of existing EMR cluster or at-least namenode initially and then use it every-time whenever I want to create other clusters. But after some Google search, I couldn't find any way to capture snapshot of EMR cluster. Is it possible to create snapshot ? or any other alternate way that can help me out with my use-case.
Appreciate any kind of help.
Thanks
It is not possible to create a snapshot of an EMR cluster node and you cannot use a custom AMI when running a cluster. However you can install software on the cluster nodes at the cluster creation time using custom bootstrap actions. You can create your custom bootstrap scripts and use them every time you launch a new cluster. This way you can achieve a similar functionality with the one you are seeking.
For more information using bootstrap actions on EMR please visit: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-bootstrap.html#bootstrapCustom
Let us know if you need any further assistance.