I launched an instance in AWS using Terraform with basic functionalities.Once the instance is launched, i need to capture that instance into image using Packer.
How can i accomplish the same?
Packer is used to make customised AMIs. But if the image is already running and customised then an AMI can be made with standard AWS tools
Once the instance is running and configured with Terraform, take an ami image of it. See this documentation https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html
OTOH if you want to automate the whole process of making the AMI automatically then add your steps that configure the base AMI in the first place to run Terraform within the packer script
Once you have a Packer script like this, the whole process is automatic
To make a new AMI running packer build on your packer config file should do everything automatically
Edit: your comments below hint that maybe what you want to do is use Packer on an already running instance to make an AMI, perhaps with further configuration to add to it. To do this, first make an AMI of the running image. There are instructions for doing this in the link above.
Next, write your Packer script in such a way that it takes the AMI id of the new image as a parameter. Then you can run the Packer script using the new AMI as input and so finish up with an AMI made by Packer but based on the running instance
Sorry it took a while to add this but I can't imagine a use case where this would be a useful thing to do
Related
I have the following doubts about using custom AMI with AWS EB.
Now I have:
a default platform, Python 3.6 + Amazon Linux 1.10.0, and in EB configurations > Instances > AMI I get an ID that I think is the default AMI provided by AWS to launch the platform (and if it is like that than it should get modified at every platform update).
some platform configurations done with .ebextensions files
my Flask app that I deploy from CLI (eb deploy)
So, in order to avoid the .ebextensions configurations time, I'd like to use a custom AMI that includes (1) + (2) and continue to deploy my Flask app like before.
So to build the AMI:
can I stop an EC2 instance of my running env and make an AMI from that one from EC2 console? If I do so, then the AMI would contain even my .ebextensions files and my app, is it a problem?
if the AMI shouldn't include .ebextensions files, then the only way to custom the platform before doing the AMI is to SSH?
after having built the AMI I put its ID in EB console > configurations > instances and then EB takes care of everything, like updating the AMI id in EC2 > autoscaling > launch options?
to do a platofrm update I have first to manually rebuild the AMI starting from the new platform and then update the AMI ID in EB configurations? So it's not possible to update the platform from EB console like I was used to do before and then to save the new AMI?
when I deploy my app it then shouldn't contain .ebextensions files?
if I create the AMI with my app included, then EB autoscaling would even save the time of deploying the app? (Of course in this case to deploy I would have to create a new AMI first).
Thanks for help.
can I stop an EC2 instance of my running env and make an AMI from that one from EC2 console? If I do so, then the AMI would contain even my .ebextensions files and my app, is it a problem?
You don't have to stop it. You can make AMI from running instance. Also your instance its in ASG, so stopping it is not a good idea.
if the AMI shouldn't include .ebextensions files, then the only way to custom the platform before doing the AMI is to SSH?
It shoudn't matter if you have pre-existing app on the ami. New deployment will install your app anyway.
after having built the AMI I put its ID in EB console > configurations > instances and then EB takes care of everything, like updating the AMI id in EC2 > autoscaling > launch options?
Yes,
to do a platofrm update I have first to manually rebuild the AMI starting from the new platform and then update the AMI ID in EB configurations? So it's not possible to update the platform from EB console like I was used to do before and then to save the new AMI?
Probably, have to repeat the process.
when I deploy my app it then shouldn't contain .ebextensions files?
It depends what they do. If they install software which is already on the custom ami, you can remove it.
if I create the AMI with my app included, then EB autoscaling would even save the time of deploying the app? (Of course in this case to deploy I would have to create a new AMI first).
The purpose of the custom ami is to save time on installing and configuring custom software that is normally not on the AWS amis. Its not to replace or elimiate the need of deploying your APP. You still need to do it, but can skip installing custom packages.
You can create a custom AMI from a running EC2 instance from the console, and from the CLI. Any AMI you create is a faithful copy of the instance, so if the instance has ebextensions, then the AMI will do also.
I think I understand that you want to create an AMI from instances being managed by ElasticBeanstalk? If that is so, then there are certain files that need to exist on the ElasticBeanstalk EC2 instance so that ElasticBeanstalk and Cloudformation can manage the environment. The .ebextensions are scripts are used to configure the environment, at least in my experience there are maintained in your repo. If your AMI has .ebextensions then they are most likely needed.
I don't think it is typical to use a custom AMI under ElasticBeanstalk: the whole point is to let AWS manage that layer for you. I would recommend that if you really need a custom AMI, you look at doing what you want to do directly in EC2 and forgo ElasticBeanstalk. ElasticBeanstalk is really only an abstracted 'friendly' interface to EC2 and other services (eg autoscaling and load balancer are actually EC2). Maybe even consider putting your application into a docker?
You can create Custom AMI of EC2 instance which is running for Elastic beanstalk. IF you are going with custom AMI then no need to use .ebextension files because either AMI should include all the changes which has already done when you deployed application along with ebextension file or do the necessary changes in server before creating AMI. But it is good to use default AMI which AWS provides while creation of Elastic Beanstalk and use .ebextension files to do required tasks during deployment.
I built and provisioned a AMI using packer and amazon-ebs.
I need to rebuild the AMI weekly. Is there a simple solution for this? Do I need a separate ec2 for jenkins or is that overkill? Would any CI tool be good for this or is there more simple approach? My packer ami code is hosted on github.
In addition, I create a new ec2 instance from AMI and tear down old one weekly. Whats the best way to schedule ec2 tear-downs and rebuilds automatically?
So 2 issues:
Weekly rebuild of AMI
Weekly rebuild of ec2 based on rebuilt AMI
Im not experienced with any devops things so please excuse me.
I'm assuming this is the only task where you want to use an automation server. In other case, I will suggest you create a Jenkins or any other automation server. It all depends on your need.
To automate this single task, you don't necessarily need an automation server. The one method I'm going to demonstrate is one among many ways you can do it. Below are the AWS resources you require.
A Docker image where packer, aws cli, and any other dependencies installed.
An ECS task configured using the image in #1.
A CloudWatch schedule expression to trigger the ECS task periodically, in this case weekly.
Your docker image should be configured such that container execution does the rebuild of the AMI. You can write a bash script for this and configure the same as the container entry point.
The second point, rebuild of EC2 server is not a best practice. You should have a separate process in place to apply the AMI changes to respective instances. However, you can do this by scheduling a lambda function which will terminate and launch a new instance.
I know this is a broad answer and there are many other ways to do the same.
I'm creating an EC2 instance and loading it up with some "on start" scripts as well as installing desired software. This becomes the base image to be used when spinning up new instances.
Once created, I create an AMI. At this point, there is no need to keep the EC2 image used to create the AMI.
So, the sequence is:
Create EC2
Install packages
Configure packages
Create AMI
Destroy EC2
How do I tell Terraform to then delete the EC2 in the final step?
Have you taken a look at Hashicorp's Packer tool? It specializes in building AMIs for use in Terraform and could save you a few manual steps here.
In the meantime though, you can issue a terraform destroy -target aws_instance.<NAME> (substitute your Terraformed EC2 instance's name) command to destroy your instance.
I would like to create a Managed Compute Environment for AWS Batch, but use EC2 User Data to configure the instances as they are brought into the ECS fleet that Batch is scheduling jobs onto.
It shouldn't matter, but the purpose of the User Data script is to pull down large data files onto an InstanceStore that the Docker containers will reference.
This is possible in ECS, but I have found no way to pass User Data to a Managed Batch Compute Environment.
At most, I can specify the AMI. But since we're going with Managed, we must use the Amazon ECS-optimized AMI.
I'd prefer to use EC2 User Data as the solution, as it gives a entry-point for any other bootstrapping we wish to perform. But I'm open to other hacks or solutions, so long as they are applicable to a Managed Compute Environment.
You can create an AMI based on the AWS provided AMI, and customize it. It will still be managed since the Batch and/or ECS daemon is running on it.
As a side note I’m trying to do the same thing but no luck so far. I may end up creating a custom AMI and include the configure script in the AMI itself in /etc/rc.local. Not ideal but I don’t think Batch can pass a user data script other than what it needs. I am still looking into this.
You can create a launch template containing your user-data. Then assign this launch template to your compute environment. Keep in mind that you might have to clean the cloud init directory in your AMI since it probably was already spun up once (at ami creation).
Launch template userguide
I have an EC2 AMI that I create instances from to be used to execute builds. I now need to modify this AMI because I need an additional program installed on it. What I want to do is make my AMI point at a different snapshot to use as its root drive (a snapshot with the new program installed) and things would be all well and good. But, I can't find a way to do this. Someone from Amazon on the forums said it's not possible, but I'm not so sure. So, I wanted to ask here.
I know I can just take the updated snapshot I want and create an entirely new AMI from it, but this results in a new AMI ID and now I need to go change the AMI ID which my scripts use to launch a new instance. I don't want to do this every time I realize I need a change to my AMI setup.
you can build private AMI from exist ec2 instance (of course you can public it as well). Then you can start a new ec2 instance from "My AMI" with all installed application/package directly.
Take a look on this doc
Amazon Machine Images (AMI)
There are new feature such as Docker introduced by AWS last month, but it is not ready to public to use currently.
Another benefit to create private AMI image is, it will save you much time when you need launch a new instance.
If you need update your configuration file, after new AMI created, I recommend to make a CI trigger (via Jenkins, for example). run a awscli command in script, it will easily update your config file. All these tasks, include create a new AMI, update configuration file, etc can be done under Jenkins/bamboo automatically.
If you are not confident with this way, then think about cloudformation template. it will make big improvement in your system. But if you fully set it, the cloudformation way will save you a lot of time in future changes.
In Cloudformation, you need set launch configuration and its autoscaling group, you still need create private AMI image, but every time after create new image, you need trigger a script to update the ami image in its Launch configuration, after that, any new instance will automatically use the new AMI.