I have a bunch of Amazon ECS services that I want to re-name. I like their task definitions, so I'm open to cloning options, I just don't see in the UpdateService API a way to rename a service. Is there an API for renaming? If not, what combination of APIs could I invoke (I figure python or awscli scripting would be the easiest option).
Considerations
I would also want to consider whether it would be possible to gracefully roll the "old name" to "new name"
Ensure security groups were applied correctly.
If possible, facilitate the mutative change via CloudFormation
I don't see any options to rename an ECS service in CLI, API, even the console. I think you're going to have to remove the service with the old name and create a new one with the new name.
There are lots of ways to make removing and creating a little easier on you:
If you are using Terraform to create and manage your ECS services, you can simply rename the service in Terraform, and it will drop and recreate the service when you apply the change.
AWS cli could certainly make this easier than doing it manually -- you could even create a wrapper script to do a rename, perhaps.
You could definitely drop and create the ECS service using Python and Boto3 (I don't use Python/Boto3 to rename services, but I do use it to drop and create services).
It's possible that CloudFormation could make this easier? I don't use CloudFormation, but since Terraform can make renaming an ECS service easier, it seems like CloudFormation might do the same.
Related
Is there some way to 'dehydrate' or extract an entire AWS setup? I have a small application that uses several AWS components, and I'd like to put the project on hiatus so I don't get charged every month.
I wrote / constructed the app directly through the various services' sites, such as VPN, RDS, etc. Is there some way I can extract my setup into files so I can save these files in Version Control, and 'rehydrate' them back into AWS when I want to re-setup my app?
I tried extracting pieces from Lambda and Event Bridge, but it seems like I can't just 'replay' these files using the CLI to re-create my application.
Specifically, I am looking to extract all code, settings, connections, etc. for:
Lambda. Code, Env Variables, layers, scheduling thru Event Bridge
IAM. Users, roles, permissions
VPC. Subnets, Route tables, Internet gateways, Elastic IPs, NAT Gateways
Event Bridge. Cron settings, connections to Lambda functions.
RDS. MySQL instances. Would like to get all DDL. Data in tables is not required.
Thanks in advance!
You could use Former2. It will scan your account and allow you to generate CloudFormation, Terraform, or Troposphere templates. It uses a browser plugin, but there is also a CLI for it.
What you describe is called Infrastructure as Code. The idea is to define your infrastructure as code and then deploy your infrastructure using that "code".
There are a lot of options in this space. To name a few:
Terraform
Cloudformation
CDK
Pulumi
All of those should allow you to import already existing resources. At least Terraform has a import command to import an already existing resource into your IaC project.
This way you could create a project that mirrors what you currently have in AWS.
Excluded are things that are strictly taken not AWS resources, like:
Code of your Lambdas
MySQL DDL
Depending on the Lambdas deployment "strategy" the code is either on S3 or was directly deployed to the Lambda service. If it is the first, you just need to find the S3 bucket etc and download the code from there. If it is the second you might need to copy and paste it by hand.
When it comes to your MySQL DDL you need to find tools to export that. But there are plenty tools out there to do this.
After you did that, you should be able to destroy all the AWS resources and then deploy them later on again from your new IaC.
I am familiar with AWS SDKs (Python/Java) and the need for the same for a lot of services like S3, DDB, KMS etc.
Are there any valid use cases for using the AWS Java SDK e.g. to programmatically spin off an EC2 instance or an EMR cluster or Redshift cluster or RDS instance or for that matter any resource that requires setting up of an infrastructure/cluster?
If AWS console is not an option, and programmatic access is all we have then, don't we always end up using AWS CLI for corresponding services or CloudFormation or Terraform for that matter?
Generally its best practice to manage any infrastructure/service deployments by using an infrastructure as code solutions such as CloudFormation, CDK (which generates CloudFormation stacks under the hood) or Terraform.
Whilst you could use the SDK to create this services (and create a solution that is similar to the solutions above) you will be building a lot of functionality that other services have already created which would put more ownership on you to fix it if you want to support another service etc.
The SDKs simply provide every AWS API interaction in a programmatic way, even under the hood CloudFormation and Terraform will likely be using the SDKs to programmatically create the resources (although I am speculating) but would then add additional functionality such as state management and drift detection on top of this.
I only have seen services being created via the SDKs when that service is not available in the selected tool, and even then it would generally be wrapped within that tool (such as custom resources for CloudFormation).
In summary, yes you could use the SDK to generate these but unless there is a specific usecase to use the SDK I would advise using a tool that already manages this so you can focus more on your infrastructure/applications.
The AWS CLI is built using the AWS SDK for Python. Terraform is built using the AWS SDK for GoLang. You may want to stick with higher level infrastructure-as-code tools, but those tools wouldn't exist without the SDKs, and if you wanted to build a tool like that you would most likely build it on top of one of the SDKs.
There are also quite a few use-cases I've seen discussed here on StackOverflow for performing infrastructure automation through AWS Lambda, for example periodically stopping and starting RDS instances, where using one of the AWS SDKs in the Lambda code would likely be much easier than trying to get Terraform to run inside a Lambda function.
Is there any known downside to creating resources on aws through the CLI? Is it more reliable/easier/error prone/largely accepted/recommended to use one method over the other? While setting up recurring scripts, is there a reason why i would want to use CloudFormation or the AWS Console over the AWS CLI to run commands directly?
For example, if I were to create an ECS Fargate Task Definition, is there any reason why I might want to use AWS CloudFormation or the Console over AWS CLI? Cli syntax is straightforward and easy to use, and there are a few things (like setting up event rules/targets for a fargate task specifically) that are not supported via cloudformation yet.
The AWS CLI and AWS CloudFormation are two different tools that can be used to create infrastructure on AWS. The CLI is more powerful and has finer grained control than CloudFormation. CloudFormation makes it very easy to use yaml or json text files that can describe an entire enterprise in the cloud.
One of the strong benefits of CloudFormation is the automatic support for rolling back changes if anything fails while deploying a stack. The CLI in comparison would require you to figure out the details of what went wrong and how to get back to where your state was. Updating infrastructure using CloudFormation is another benefit. Make the change in the template and update the stack.
For small setups, using the CLI is fine. However, once you get past launching an EC2 instance and start building VPCs, Instances, KeyPairs, Security Groups, RDS, etc. etc. you will find that the CLI has some real limitations: mostly being too manual of a process, not easily repeatable, difficult to put the process into version control, ....
If you are constantly building, testing and deleting complex setups, CloudFormation is absolutely one of the best tools from AWS. Note that there are a number of third party solutions that have a huge number of followers such as Bamboo, Octopus, Jenkins, Chef, etc.
If your job is SysOps or DevOps then you absolutely want to master the CLI and CloudFormation. These are amazing tools for working with AWS. Also master Beanstalk, maybe OpsWorks and one of the third party tools like Jenkins.
I am trying to setup DEIS in AWS. So I am in the process of learning and it is expected that I will have to provision and setup many times before I master things.
Let's say I run provision-ec2-cluster and for some reason, I want to revert everything that was done in AWS (delete VPC, dele instances, scaling rules, security groups, etc.).
What is the easiest way to do that? Does Deis come with an script that can help me in this respect?
This is an ideal use case for AWS CloudFormation. Either use the official Deis published info on CloudFormation at http://docs.deis.io/en/latest/installing_deis/aws/. Note that with CloudFormation, you can with a single command launch all resources, or destroy all resources.
Or, alternatively, for a more sophisticated example with Terraform, a third-party alternative to CloudFormation, you can use something like https://github.com/brandfolder/infrastructure.
I am new to AWS (Amazon Web Services) as well as our own custom boto based python deployment scripts, but wanted to ask for advice or best practices for a simple configuration management task. We have a simple web application with configuration data for several different backend environments controlled by a command line -D defined java environment variable. Sometimes, the requirement comes up that we need to switch from one backend environment to another due to maintenance or deployment schedules of our backend services.
The current procedure requires python scripts to completely destroy and rebuild all the virtual infrastructure (load balancers, auto scale groups, etc.) to redeploy the application with a change to the command line parameter. On a traditional server infrastructure, we would log in to the management console of the container, change the variable, bounce the container, and we're done.
Is there a best practice for this operation on AWS environments, or is the complete destruction and rebuilding of all the pieces the only way to accomplish this task in an AWS environment?
It depends on what resources you have to change. AWS is evolving everyday in a fast paced manner. I would suggest you to take a look at the AWS API for the resources you need to deal with and check if you can change a resource without destroying it.
Ex: today you cannot change a Launch Group once it is created. you must delete it and create it again with the new configurations. but if you have one auto scaling group attached to that launch group you will have to delete the auto scaling group and so on.
IMHO a see no problems with your approach, but as I believe that there is always room for improvement, I think you can refactor it with the help of AWS API documentation.
HTH
I think I found the answer to my own question. I know the interface to AWS is constantly changing, and I don't think this functionality is available yet in the Python boto library, but the ability I was looking for is best described as "Modifying Attributes of a Stopped Instance" with --user-data as being the attribute in question. Documentation for performing this action using HTTP requests and the command line interface to AWS can be found here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_ChangingAttributesWhileInstanceStopped.html