I am creating an AWS EMR cluster running Spark using a Cloud Formation template. I am using Cloud Formation because that's how we create reproducible environments for our applications.
When I create the cluster from the web dashboard one of the options is to add a Key Pair. This is necessary in order to access via ssh the nodes of the cluster. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/EMR_CreateJobFlow.html
I can't see how to do the same when using Cloud Formation templates.
The template structure (see below) doesn't have the same attribute.
Type: "AWS::EMR::Cluster"
Properties:
AdditionalInfo: JSON object
Applications:
- Applications
BootstrapActions:
- Bootstrap Actions
Configurations:
- Configurations
Instances:
JobFlowInstancesConfig
JobFlowRole: String
LogUri: String
Name: String
ReleaseLabel: String
ServiceRole: String
Tags:
- Resource Tag
VisibleToAllUsers: Boolean
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-emr-cluster.html#d0e76479
I had a loook to the attribute JobFlowRole that is a reference to an instance profile (http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-instanceprofile.html). Again, no sign of the KeyName.
Did anyone solved this problem before?
Thanks,
Marco
I solved this problem. I was just confused by the lack of naming consistency in Cloud Formation templates.
What is generally referred as KeyName becomes Ec2KeyName under
the JobFlowInstancesConfig.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-jobflowinstancesconfig.html#cfn-emr-cluster-jobflowinstancesconfig-ec2keyname
Related
I have a UI5 web application which has no access to AWS Parameter store. However, it is deployed by CloudFormation, by an account which has this access - I already collect some values from SSM during deployment and use them as parameters for resources which are being deployed by template.yml file.
However, now I need some of these parameters not only during deployment, but also during actual run of the app. Does someone familiar with AWS and UI5 know, how can I store these values during deployment so I can use them later, during the run of the web application? Thank you.
If you can edit the CloudFormation template then you can add resources of type AWS:SSM:Parameter to the template which will create SSM Parameters as part of the deployment.
Example:
Resources:
BasicParameter:
Type: AWS::SSM::Parameter
Properties:
Name: /my/basic/parameter
Type: String
Value: my-basic-parameter-value
Description: A description of the parameter
I would like to perform the following operations in order with CloudFormation.
Start up an EC2 instance.
Give it privileges to access the full internet using security group A.
Download particular versions of Java and Python
Remove its internet privileges by removing security group A and adding a security group B.
I observe that there is a DependsOn attribute for specifying the order in which to create resources, but I was unable to find a feature that would allow me to update the security groups on the same EC2 instance twice over the course of creating a stack.
Is this possible with CloudFormation?
Not in CloudFormation natively, but you could launch the EC2 instance with a configured userdata script that itself downloads Java/Python and the awscli, as necessary, and then uses the awscli to switch security groups for the current EC2 instance.
However, if all you need is Java and Python pre-loaded then why not simply create an AMI with them already installed and launch from that AMI?
The best way out is to utilise a Cloudformation custom resource here. You can create a lambda function that does exactly what you need. This lambda function can then be called as a custom resource function in the cloud formation template.
You can pass your new security group ID and instance ID to the lambda function and code the lambda function to use AWS SDK and do the modifications that you need.
I have leveraged it to post an update to my web server about the progress of the cloud formation template. Below is the sample code of the template.
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles: [!Ref 'EC2Role']
MarkInstanceProfileComplete:
Type: 'Custom::EC2InstanceProfileDone'
Version: '1.0'
DependsOn: EC2InstanceProfile
Properties:
ServiceToken: !Ref CustomResourceArn
HostURL: !Ref Host
LoginType: !Ref LoginType
SecretId: !Ref SecretId
WorkspaceId: !Ref WorkspaceId
Event: 2
Total: 3
Here the resource MarkInstanceProfileComplete is a custom resource that calls a Lambda function. It takes the event count and total count as input and processes them to calculate percentage progress. Based on that it sends out a request to my web server. For all we care, this Lambda function can do potentially anything you want it to do.
I have a CloudFormation template that creates my RDS cluster using aurora serverless. I want the cluster to be created with the data API enabled.
The option exists on the web console:
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html
But I can't find it on the CloudFormation documentation.
How can I turn this option on from the template?
Set the EnableHttpEndpoint property to true, e.g.:
AWSTemplateFormatVersion: '2010-09-09'
Description: Aurora PostgreSQL Serverless Cluster
Resources:
ServerlessWithDataAPI:
Type: AWS::RDS::DBCluster
Properties:
Engine: aurora-postgresql
EngineMode: serverless
EnableHttpEndpoint: true
ScalingConfiguration:
...
You can enable the Data API from CloudFormation by creating a custom resource backed lambda and enable it using any of the available SDK.
I use boto3 (python), so the lambda would have code similar as below:
import boto3
client = boto3.client('rds')
response = client.modify_db_cluster(
DBClusterIdentifier='string',
EnableHttpEndpoint=True|False
)
Obviously, you need to handle different custom resource request types and return from the lambda with success or failure. But to answer your question, this is the best possible way to set up data API via CloudFormation, for now, IMHO.
For more information about the function (Boto3):
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds.html#RDS.Client.modify_db_cluster
Enabling the Data API is currently only possible in the web console. This feature is still in beta so things like CloudFormation support and availability outside of us-east-1 are still pending, and using the Data API in production should be done with caution as it may still change.
I'm using AWS Codestar setup and I would like to add a database.config to my .ebextentions folder in my rails project.
If you're wondering why I'm not adding database trough console, the Codestar's pipeline fails at the final ExecuteChangeSet stage for CloudFormation changes and throws a 404 error, I assume CodePipeline looking for the previous instance.
Error Message I've been receiving AWS suggests I edit Elastic Beanstalk directly. Really somewhat lost how I can add a database to my project using Elastic Beanstalk while not breaking Codestars CodePipline ExecuteChangeSet.
You specified the 'AWSEBRDSDBInstance' resource in your configuration to create a database instance,
without the corresponding database security group 'AWSEBRDSDBSecurityGroup'. For a better way to add
and configure a database to your environment, use 'eb create --db' or the Elastic Beanstalk console
instead of using a configuration file.
My .ebextensions/database.config file so far.
Resources:
AWSEBRDSDatabase:
Type: AWS::RDS::DBInstance
Properties:
AllocatedStorage: 5
DBInstanceClass: db.t2.micro
DBName: phctest
Engine: postgresql
EngineVersion: 10.4
MasterUsername: username
MasterUserPassword: password
I could also make a separate RDS database on it's own I thought about that, but like to leave it for Elastic Beanstalk.
Is it possible to specify Point In Time Recovery for an Amazon DynamoDB table created via CloudFormation? I've been looking through the docs and user guide and haven't found anything useful yet.
This is now possible with cloud formation by adding a PointInTimeRecoverySpecification to your cloud formation template.
Example:
Type: 'AWS::DynamoDB::Table'
Properties:
TableName: MyTable
PointInTimeRecoverySpecification:
PointInTimeRecoveryEnabled: true
...