I have a scenario where I want to enable flow-logs for all the existing VPCs(by existing I mean ones which are not created by Cloud-formation stack) which are there in my AWS Account. For these I would need to describe all the existing VPC from my AWS Account(let say region specific) and get their Vpc-id which then I will fed it to create-flow-log.
Is it even possible to describe existing resources through Cloud Formation Stack, because I didn't find any AWS Documentation for this purpose? Is there any other possibility by which this can be done and still using Cloud-Formation?
Malay - It is possible to refer existing resources which are not created by CFN. But, it's not possible to modify them.
Here's an example to take the existing VPC IDs as an input parameter:
List<AWS::EC2::VPC::Id>
An array of VPC IDs, such as vpc-a123baa3, vpc-b456baa3.
Hope it helps.
Related
I'm actually using a Cloudformation stack in Francfurt region. I wanted to create 2 SSL certificates, and validate them. I suceeded, however I discovered later that I can't use them with my Cloudfront distribution as they need to be created in N. Virginia.
Now I'm facing this problem. How to create a resource in a different region ? I'm aware that a Cloudformation stack can only create resources in its own region, however, while searching for some solutions, I saw different persons mentioning Cloudformation StackSets.
I took a quick look, but I'm still questioning, can it really solve my issue ?
From what I've read, it seems it allows to create identical resources in different regions. But it doesn't seem that from one stack, create some resources in a region, and some others in another.
Can someone experimented with Cloudformation StackSets notice me if it is possible or not ?
Thank you a lot.
StackSets does not help in this case as it is used to create the same stack in multiple regions.
However, you can use Custom Resources, which provides a way for you to write custom provisioning logic in CloudFormation template. A Lambda function can be triggered by Cloudformation, which creates the certificates in us-east-1 region by using one of the AWS SDKs.
Azure Resource group is one thing which ties one complete stack together, so in case you want to know what your complete stack looks like just open up your RG and all resources are there(not talking about very big stacks ).
Do we have any service in AWS like this ??
AWS also has Resource Groups. It's not identical to Azure resource groups. But it will work for your need. You need to use tags for the resources and then group them using AWS Resource Groups. Tags are very powerful and widely used in AWS.
An AWS Cloudformation template creates a "Stack" with all the resources defined in the template. In the AWS console what you see under the Cloudformation service are these stacks. They seem to me to be very much like an Azure Resource Group. The life cycle of the resources in the template are managed by the stack. If you delete the stack all the resources are deleted; very much like an Azure Resource Group.
As mentioned in comments you can group together AWS resources using resource groups. If you are looking for more than mere grouping then there is AWS OpsWorks Stacks. AWS OpsWorks Stacks is nothing but chef under the hood. Here you can get the full view of associated resources and you can manage as well.
I've been playing in Azure for a while with Kubernetes (AKS) and Terraform. I was used to deploying an Azure resource group and everything in it:
Key vault
storage account
DataBricks
Data Factory
SQL Server (or some other database)
Simple enough. So, I just tried to deploy a similar data stack in AWS. It's not at all the same, and a lot more effort.
The VPC seems to be the first place to start. You'll also have to
think about CIDR addresses too.
Subnets (you'll be defining CIDR blocks here too) and deciding availability zones come next.
You'll need security group
You'll also need IAM resources and to attach the relevant roles/policies. In Azure, once permissions and Service Principals are in place, you don't have to worry about these things.
All this is before you've deployed anything.
Now that's my immediate feeling trying to replicate what I was doing on Azure. I didn't let how long it takes to deploy these resources on AWS (in the EU region - YMMV) compared to Azure 🙄 cloud my judgement either...
A Cloudformation 'stack' was created when I tested using the AWS EKSCLI command line tool to provision a kubernetes cluster and nodes. Deleting a stack on Cloudformation will remove all the associated resources, like if you were to remove a resource group on Azure.
I think using Terraform is probably a good idea, which is the conclusion you may arrive at after having to chase down why you can't delete a certain resource because something is still using it (and it's not all in the same place (resource group) like in Azure.
Just a quick question, does anyone know if Terraform will wipe out existing resources on AWS?
For example if I already have an existing VPC with resources, or S3/EFS storage will Terraform ignore these resources when I run it with my configuration files to deploy say another VPC?
Or as Terraform is looking for a desired state will it wipe anything existing?
Am hoping unless you specifically import existing resources Terraform will just leave them alone?
Thanks
It Depends.
It varies from case to case as the responses will be coming from the cloud providers (AWS, Azure).
Ex.
If you create a VPC in terraform, it will generate a new VPC ID (terraform won't allow to use VPC ID in coding). So, it won't affect your existing resources.
If you write a Route53 record in terraform, it could overwrite existing Route53 entries.
But, If you import terraform state form existing resources, it will import its state and map it with the terraform resources. In that case, destroying the resource will remove the actual cloud resource.
Hope I understood your question and answered it.
Hi I already have one VPC in my aws for production. Now I want to create same vpc for test environment also. Is there any way to create a mirror image of VPC . Like creating one more VPC with identical of old VPC.
There's no API for this, but you can set up a script pretty easily.
Alternatively, instead of creating the first one manually, you can create it with CloudFormation so you can make multiple identical copies (even in different Regions) whenever you want.
Terraform from hashicorp is the best way to do that in my opinion. You can also use the terraforming from dtan4 at this link to export the existing resources and adjust them to create another environment. For example you may want to go for another IP range, name it different etc.
You should use Cloudfomer to "reverse-engineer" your VPC setup, and there is a nice layout as well. Nevertheless, you need special IAM roles to do this.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cloudformer.html
Because it is "reverse-engineer", all the VPC setting will be similar(same VPC IP/CIDR , subnets) ) , AWS will assign new ID for individual component. To avoid maintenance nightmare, You should assign different tag name for your production and test environment.
I am creating an AWS stack inside a VPC using CloudFormation and need to create ElastiCache clusters on it. I have investigated and there is no support in CloudFormation to create cache clusters in VPCs.
Our "workaround" was to to create the cache cluster when some "fixed" instance (like a bastion for example) bootstrap using CloudInit and AWS AmazonElastiCacheCli tools (elasticache-create-cache-subnet-group, elasticache-create-cache-cluster). Then, when front end machines bootstrap (we are using autoscaling), they use elasticache-describe-cache-clusters to get cache cluster nodes and update configuration.
I would like to know if you have different solutions to this problem.
VPC support has now been added for Elasticache in Cloudformation Templates.
To launch a AWS::ElastiCache::CacheCluster in your VPC, create a AWS::ElastiCache::SubnetGroup that defines which subnet in your VPC you want Elasticache and assign it to the CacheSubnetGroupName property of AWS::ElastiCache::CacheCluster.
You workaround is a reasonable one (and shows that you seem to be in control of your AWS operations already).
You could improve on your custom solution eventually by means of the dedicated CustomResource type, which are special AWS CloudFormation resources that provide a way for a template developer to include resources in an AWS CloudFormation stack that are provided by a source other than Amazon Web Services. - the AWS CloudFormation Custom Resource Walkthrough provides a good overview of what this is all about, how it works and what's required to implement your own.
The benefit of using this facade for a custom resource (i.e. the Amazon ElastiCache cluster in your case) is that its entire lifecycle (create/update/delete) can be handled in a similar and controlled fashion just like any officially supported CloudFormation resource types, e.g. resource creation failures would be handled transparently from the perspective of the entire stack.
However, for the use case at hand you might actually just want to wait for official support becoming available:
AWS has announced VPC support for ElastiCache in the context of the recent major Amazon EC2 Update - Virtual Private Clouds for Everyone!, which boils down to Default VPCs for (Almost) Everyone.
We want every EC2 user to be able to benefit from the advanced networking and other features of Amazon VPC that I outlined above. To enable this, starting soon, instances for new AWS customers (and existing customers launching in new Regions) will be launched into the "EC2-VPC" platform. [...]
You don’t need to create a VPC beforehand - simply launch EC2
instances or provision Elastic Load Balancers, RDS databases, or
ElastiCache clusters like you would in EC2-Classic and we’ll create a
VPC for you at no extra charge. We’ll launch your resources into that
VPC [...] [emphasis mine]
This update sort of implies that any new services will likely be also available in VPC right away going forward (else the new EC2-VPC platform wouldn't work automatically for new customers as envisioned).
Accordingly I'd expect the CloudFormation team to follow suit and complete/amend their support for deployment to VPC going forward as well.
My solution for this has been to have a controller process that polls a message queue, which is subscribed to the SNS topic which I notify CloudFormation events to (click advanced in the console when you create a CloudFormation stack to send notifications to an SNS Topic).
I pass the required parameters as tags to AWS::EC2::Subnet and have the controller pick them up, when the subnet is created. I execute the set up when a AWS::CloudFormation::WaitConditionHandle is created, and use the PhysicalResourceId to cURL with PUT to satisfy a AWS::CloudFormation::WaitCondition.
It works somewhat, but doesn't handle resource deletion in ElastiCache, because there is no AWS::CloudFormation::WaitCondition analogue for stack deletion. That's a manual operation procedure wth my approach.
The CustomResource approach looks more polished, but requires an endpoint, which I don't have. If you can put together an endpoint, that looks like the way to go.