I am creating a EMR cluster through cloud formation .
I have already created security configuration from AWS management console .
I am not able to find any way where i can add this security config while creating EMR from cloud formation.
Alternatively i can leverage lambda function , but how to add a EMR security configuration after cluster is created ?
any help would be appreciated...
Unfortunately this is not yet possible with cloudformation. Security Configurations were released September of last year and Cloudformation has yet to support it. At this time the only way would be to do it manually after the creation or via the lambda or other method.
As of today, this is only possible using Lambda backed custom resources. This is what you'll need to do on a high-level:
Create EMR cluster using CloudFormation supported AWS::EMR::Cluster resource type.
Define a Lambda backed custom resource, say with resource type Custom::EMRSecurityConfiguration.
Define a Lambda function that'll be used in step #2 and do the needful by using AWS SDK (e.g. boto3 if you're writing your Lambda in Python) and actually create/update/delete the security configuration based on the kind of EventType (the value for this will be passed in by CloudFormation to your function).
Lambda function defined in #3 will be triggered by CloudFormation everytime you create/update/delete the custom resource defined in #2. I'd recommend looking at AWS docs for Lambda-backed custom resources and security configuration.
Related
I'm new to Cloud Formation. I want to update the settings of already created a lot of RDS instances using Cloud Formation. I don't have the info about either those resources were created through CF or manually. Is it possible to update such resources with CF?
I can think of another way like I can use AWS SDK (boto3) but doing it with CF is perefrable.
The only way to do this from CloudFormation (CF) is to develop your own CF custom resource. This will be a lambda function which will use AWS SDK to query the state of your RDS databases, and perform any actions you want.
Since its fully custom, you can program any logic which satisfies your requirements.
If the resources were created manually, you can also import them to CF, and then update using CF.
I have use case in amazon cloud, i'm using fargate cluster and cloudformation.
I want to do continuous deployment i.e on new image upload trigger i want to update the cloudformation stack with this new image, also run this automated deployment when client wants using manual trigger.
What should i use for continuous deployment, aws code deploy or aws lambda.
aws CodeDeploy has a provider CloudFormation with limited option and less control i believe.
aws lambda has a great control over CloudFormation client through its boto api.
I also read somewhere that when you get some limitations in CodeDeploy or CodePipeline you can integrate lambda to get rid of this limitation. So why not use lambda in the first place for continuous deployment only.
I'm very convinced about aws lambda over aws CodeDeploy after doing some research, However, i'm open for comments and suggestions.
You can use both of them to achieve perfect CI-CD implementation
If image gets uploaded the Lambda will be triggered and Lambda will be having your configurations and parameters
Using that it will call CodeDeploy to build your ECR images and It will get deployed to your Farget cluster
You can also achieve your second need using this implementation, manual trigger when client wants
In lambda you can trigger manualy passing parameters runtime
I hope this helps you
As part of infra automation we are using cloudformation for automating the AWS infrastructure. We are utilising the service ElaticTranscoder as well, as i understand cloudformation yet does not provide support for ElasticTranscoder, is there any efficient way to automate ElasticTranscoder using cloudformation.
Custom resources provide a way for you to write custom provisioning logic in AWS CloudFormation template and have AWS CloudFormation run it during a stack operation, such as when you create, update or delete a stack.
Check out this example.
Also, a quick google search gives me this result.
Another option is not using CloudFormation. You can use Terraform which does support Elastic Transcoder.
https://www.terraform.io/docs/providers/aws/r/elastic_transcoder_pipeline.html
Ansible also has third-party support for it.
https://github.com/wimnat/ansible-modules/blob/master/elastictranscoder/elastictranscoder.py
Last but not least, you can vote for this feature in AWS wish list by liking or retweeting the request.
https://twitter.com/search?q=%23awswishlist%20transcoder&src=typd
https://awswishlist.com/
Now, i'm creating AWS AMI manually from an EC2 instance. and i would like to automate the process using Jenkins build process.
I've configured the jenkins-cloudformation plugin with the credentials and tried to trigger the cloud formation template to launch the EC2 instance. From here how can i proceed the automation process to create the AMI with in the cloud formation template?
Can someone help me on this?
This is an old question but here is some info for anyone trying to do such automation. You might use HashiCorp Packer for creating the image but, if you know your way around lambdas and AWS API, you do not need packer.
You can create a new AMI by launching an instance from a source AMI, customizing it the way you want, and then call AWS api to make an AMI out of the instance. Here are steps you might follow for this:
first, you need to find a source image. You can specify aws ec2 describe_images filters to do this.
once you have the image, you need launch an instance from it. Here is boto3 api to make the call.
while launching the instance, you will want to pass 'UserData' to it. You user data may be a few simple lines of installing packages or do advanced stuff. You can put it all into a script, host it in s3, and make UserData download and execute your script.
Once you are done with your work on the instance, it is time to capture it as a new AMI.
So, how would you do these and where is the glue?
You can use AWS lambda to manage these steps. One lambda can find the source AMI and launch and instance from it. Another lambda can capture the image.
Once your instance is customized, you would trigger the lambda that will capture it as an AMI. You might do that by directly invoking lambda. Depending on re-usability requirements you have, you might want to trigger that lambda from SNS or CloudWatch, in that case you would send an SNS message to your SNS topic or enable/trigger your CloudWatch rule.
You cloudformation would install these lambdas and any other components that would trigger them (SNS and CloudWatch).
I want to migrate my existing VPC, subnets etc from one amazon account to another amazon account using cloud formation.
How can i do this?
If you have a CloudFormation template for your VPC environment already, then you can simply create a new stack using that same template in another AWS account.
However, this will create a copy of your VPC environment as it was when it was initially created. Any changes done to the VPC since it was created using CloudFormation will not be included. This will include the acquisition of data in a database, for example.
If you do not already have a CloudFormation template, you can try to create one using AWS Cloud Former. Cloud Former can be used to examine your AWS environment and create a CloudFormation template from what it sees.
Instructions for running AWS Cloud Former can be found in the AWS Documentation: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cloudformer.html