Problem:
I have an EC2 instance running and I have made some modifications to the instance: installed docker, setup directories for certs, etc. Now, I am wanting to create the same instance but use infrastructure as code principals. Instead of remembering all the additions that I have done and creating a template by hand, I am trying to find a way to export my current EC2 instance into a json or yaml format so that I can terminate this instance and create another one that is equivalent to the one running.
I have tried:
aws ec2 describe-instances
Reading through the AWS CLI EC2 docs
Reading through the CloudFormation docs
Searched Google
Searched SO
Since you have no knowledge of how the instance was setup, the only choice is to create an Amazon Machine Image (AMI). This will create an exact copy of the disk, so everything you have installed will be available to any new instances launched from the AMI. The CloudFormation template can then be configured to launch instances using this AMI.
If, on the other hand, you knew all the commands that needed to be run to configure the instance, then you could provide a User Data script that would run when new instances first boot. This would configure the instances automatically and is the recommended way to configure instances because it is easy to modify and allows instances to launch with the latest version of the Operating System.
Such a script can be provided as part of a CloudFormation template.
See: Running commands on your Linux instance at launch - Amazon EC2
One option would be to create AMI from live instance and spin up new CF stack using the AMI.
Other would be importing resource: https://aws.amazon.com/blogs/aws/new-import-existing-resources-into-a-cloudformation-stack/
There is a tool (still in beta) developed by AWS called CloudFormer:
CloudFormer is a template creation beta tool that creates an AWS CloudFormation template from existing AWS resources in your account. You select any supported AWS resources that are running in your account, and CloudFormer creates a template in an Amazon S3 bucket.
The CloudFormer is an AWS managed template. Once you launch it, the template will create an AWS::EC2::Instance for you along with a number of other related resources. You will access the instance using URL through browser, and an AWS wizard will guide you from there.
Its tutorial even shows how to create a CloudFormation template from an existing EC2 instance.
Import the EC2 instance into CloudFormation then copy it’s template.
Read more: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html
Related
I'm quite new to AWS and I have this issue that I'm confused about: basically we are trying to add AWS Cloudwatch logging to our EC2 instances. I was able to set it up on Staging instance with manually adding IAM Role for Cloudwatch and running those commands described in this documentation page. It made logging work. And then I was able to set it up for our currently active production EC2 instances also adding those rules and running those commands from that article in CLI while connected to the instance via SSH.
The problem, however, is that there are new production instances being generated each time we redeploy our app to production. Unlike the Staging, where simply pull latest changes and run commands to restart Docker, and the EC2 instance is always running and it's always same. Production instances are being generated using a Launch Template. Basically, I create an AMI image of Staging instance, then update the Launch Template to use the latest AMI, then I run instance refresh for the auto-scaling group that uses that Launch Template, which terminates old production EC2 instances and creates the new ones.
But instead of applying those roles and running those commands manually each time to those newly created production EC2 instances I would like to set it up to be applied automatically each time they are being generated since after I do that I no longer see the IAM Roles attached to those new production instances and logging fails. Could you tell me please how could I set it up to have those roles and any necessary configurations applied to the newly generated production EC2 instances?
UPDATE:
I tried changing "User Data" of the Launch Template to those 2 options, however in (1) case it just didn't attach any role and logging failed and in (2) case auto-scaling would fail:
I've faced with the problem while using AWS SDK. Currently I am using SDK for golang, but solutions from other languages are welcome too!
I have ECS cluster created via SDK
Now I need to add EC2 containers for this cluster. My problem is that I can't use Amazon ECS Agent to specify cluster name via config:
#!/bin/bash
echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config
or something like that. I can use only SDK.
I found method called RegisterContainerInstance.
But it has note:
This action is only used by the Amazon ECS agent, and it is not
intended for use outside of the agent.
It doesn't look like working solution.
I need to understand how (if it's possible) to create working ECS clusterusing SDK only.
UPDATE:
My main target is that I need to start specified count of servers from my Docker image.
While I am investigating this task i've found that I need:
create ECS cluster
assign to it needed count of ec2 instances.
create Task with my Docker image.
run it on cluster manually or as service.
So I:
Created new cluster via CreateCluster method with name "test-cluster".
Created new task via RegisterTaskDefinition
Created new EC2 instance with ecsInstanceRole role with ecs-optimized AMI type, that is correct for my region.
And there place where problems had started.
Actual result: All new ec2 instances had attached to "default" cluster (AWS created it and attach instance to it).
If I am using ECS agent I can specify cluster name by using ECS_CLUSTER config env. But I am developing tool that use only SDK (without any ability of using ECS agent).
With RegisterTaskDefinition I haven't any possibility to specify cluster, so my question, how I can assign new EC2 instance exactly to specified cluster?
When I had tried to just start my task via RunTask method (with hoping that AWS somehow create instances for me or something like that) I receive an error:
InvalidParameterException: No Container Instances were found in your cluster.
I actually can't sort out which question you are asking. Do you need to add containers to the cluster, or add instances to the cluster? Those are very different.
Add instances to the cluster
This is not done with the ECS API, it is done with the EC2 API by creating EC2 instances with the correct ecsInstanceRole. See the Launching an Amazon ECS Container Instance documentation for more information.
Add containers to the cluster
This is done be defining a task definition, then running those tasks manually or as services. See the Amazon ECS Task Definitions for more information.
Now, i'm creating AWS AMI manually from an EC2 instance. and i would like to automate the process using Jenkins build process.
I've configured the jenkins-cloudformation plugin with the credentials and tried to trigger the cloud formation template to launch the EC2 instance. From here how can i proceed the automation process to create the AMI with in the cloud formation template?
Can someone help me on this?
This is an old question but here is some info for anyone trying to do such automation. You might use HashiCorp Packer for creating the image but, if you know your way around lambdas and AWS API, you do not need packer.
You can create a new AMI by launching an instance from a source AMI, customizing it the way you want, and then call AWS api to make an AMI out of the instance. Here are steps you might follow for this:
first, you need to find a source image. You can specify aws ec2 describe_images filters to do this.
once you have the image, you need launch an instance from it. Here is boto3 api to make the call.
while launching the instance, you will want to pass 'UserData' to it. You user data may be a few simple lines of installing packages or do advanced stuff. You can put it all into a script, host it in s3, and make UserData download and execute your script.
Once you are done with your work on the instance, it is time to capture it as a new AMI.
So, how would you do these and where is the glue?
You can use AWS lambda to manage these steps. One lambda can find the source AMI and launch and instance from it. Another lambda can capture the image.
Once your instance is customized, you would trigger the lambda that will capture it as an AMI. You might do that by directly invoking lambda. Depending on re-usability requirements you have, you might want to trigger that lambda from SNS or CloudWatch, in that case you would send an SNS message to your SNS topic or enable/trigger your CloudWatch rule.
You cloudformation would install these lambdas and any other components that would trigger them (SNS and CloudWatch).
I wonder what is the best way to add a post config step after instance creation when instance are automatically created by an ECS Cluster.
It seems there is no way to add user-data to ECS instance ?
Note : the instance are created automatically by the ECS Cluster itself.
EDIT:
When using ECS, you configure a Cluster. While configuring the cluster you select instance type and other stuff (ssh key, ...) but there is nowhere to give some user-data to the instances that will be created by ECS. So the question is how to do some post-configuration on instances automatically created with ECS.
When using the management console, it's more of a wizard that creates everything needed for you, including the instances using the Amazon Linux ECS optimized AMI, and doesn't give you a whole lot of control beyond that.
To get more fine-grained control, you would have to use another method of creating your cluster, such as the AWS CLI or CloudFormation. These methods allow you (or require you, actually) to create each piece at a time.
Example:
$ aws ecs create-cluster --cluster-name MyEcsCluster
The above command creates you a cluster, and cluster only. You would still have to create an ECS task definition, ECS service—although you could still use the management console for those—and (here's the real answer to your question) the EC2 instances which you want to attach to the cluster (either individually or through an Auto Scaling group). You could create instances from the Amazon Linux ECS optimized AMI, but also add user-data at that time to further configure them (you would also probably use the user-data in this scenario to create the /etc/ecs/ecs.config file to make sure it attaches to the ECS cluster you've created, e.g. echo "ECS_CLUSTER=MyEcsCluster" > /etc/ecs/ecs.config).
The short answer is, it's a more work to gain that sort of flexibility, but it is doable.
Edit: Thinking about it further, you could likely use the management console wizards to create everything once, then manually terminate the instances it created for the cluster (or, rather, delete the Auto Scaling group that creates them) and add your own. This would save you some work.
I want to use an AMI that have not been created from a snapshot. The AMI was created using a virtual machine and then I used CLI tools to create the instance and the AMI.
This instance have installed cloud-init. Could I use this AMI to create instances using cloudformation?
Do I need to install more software to use that service?
Thanks!
Yes, definitely, that is a very common use case:
Amazon Web Services (AWS) publishes many Amazon Machine Images (AMIs)
that contain common software configurations for public use. In
addition, the AWS developer community has published many custom AMIs.
You can also create your own custom AMIs so that you can quickly and
easily start new instances that have everything you need for your
application. For example, if your application is a website or a web
service, your AMI could include a web server, the associated static
content, and the code for the dynamic pages. After you launch an
instance with this AMI, your application is running and ready to
accept requests.
https://s3.amazonaws.com/cloudformation-examples/BoostrappingApplicationsWithAWSCloudFormation.pdf
Should you plan to use Cloudformation:Init inside your template to add additional configuration when your instance will start, be sure to install cfn tools as well (cfn-init, cfn-hup, cfn-signal...)
More details about Cloudformation:Init : http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html
How to install cfn helper scripts : http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-helper-scripts-reference.html
Yes.
When you are specifying the instance in the cloud formation script you can use "ImageID": "" <- put in the AMI ID
ImageId
Provides the unique ID of the Amazon Machine Image (AMI) that was assigned during registration.
Required: Yes
Type: String
Update requires: Replacement