I have added an EMR cluster to a stack. After updating the stack successfully (CloudFormation), I can see the master and slave nodes in EC2 console and I can SSH into the master node. But AWS console does not show the new cluster. Even aws emr list-clusters doesn't show the cluster. I have triple checked the region and I am certain I'm looking at the right region.
Relevant CloudFormation JSON:
"Spark01EmrCluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Name": "Spark01EmrCluster",
"Applications": [
{
"Name": "Spark"
},
{
"Name": "Ganglia"
},
{
"Name": "Zeppelin"
}
],
"Instances": {
"Ec2KeyName": {"Ref": "KeyName"},
"Ec2SubnetId": {"Ref": "PublicSubnetId"},
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": "m4.large",
"Name": "Master"
},
"CoreInstanceGroup": {
"InstanceCount": 1,
"InstanceType": "m4.large",
"Name": "Core"
}
},
"Configurations": [
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"ConfigurationProperties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
]
}
],
"BootstrapActions": [
{
"Name": "InstallPipPackages",
"ScriptBootstrapAction": {
"Path": "[S3 PATH]"
}
}
],
"JobFlowRole": {"Ref": "Spark01InstanceProfile"},
"ServiceRole": "MyStackEmrDefaultRole",
"ReleaseLabel": "emr-5.13.0"
}
}
The reason is missing VisibleToAllUsers property, which defaults to false. Since I'm using AWS Vault (i.e. using STS AssumeRole API to authenticate), I'm basically a different user every time, so I couldn't see the cluster. I couldn't update the stack to add VisibleToAllUsers either as I was getting Job flow ID does not exist.
The solution was to login as root user and fix things from there (I had to delete the cluster manually, but removing it from the stack template JSON and updating the stack would probably have worked if I hadn't messed things up already).
I then added the cluster back to the template (with VisibleToAllUsers set to true) and updated the stack as usual (AWS Vault).
Related
I'm creating my first Cloudformation template using an archived Github project from an AWS Blog:
https://aws.amazon.com/blogs/devops/part-1-develop-deploy-and-manage-for-scale-with-elastic-beanstalk-and-cloudformation-series/
https://github.com/amazon-archives/amediamanager
The template amm-elasticbeanstalk.cfn.json declares an Elastic Beanstalk resource, outlined here:
"Resources": {
"Application": {
"Type": "AWS::ElasticBeanstalk::Application",
"Properties": {
"ConfigurationTemplates": [{...}],
"ApplicationVersions": [{...}]
}
}
}
From the documentation I'm under the impression that AWS::ElasticBeanstalk::ApplicationVersion and AWS::ElasticBeanstalk::ConfigurationTemplate must be defined as separate resources, yet the example I'm working from is using the same AWSTemplateFormatVersion as the documentation. Is this a "shorthand" where namespaces can be nested if they have the same parent (i.e. AWS::ElasticBeanstalk)? Is it documented somewhere?
In the same file AWS::ElasticBeanstalk::Environment is defined as a separate resource - is this just a stylistic choice, perhaps because the environment configuration is so long?
Elastic Beanstalk consists of Applications and Environments components. Basically each environment runs only one application version at a time, however, you can run the same application version in many environments at the same time. Application versions and Saved configurations are part of the Application resource that's why it's possible to define it within the AWS::ElasticBeanstalk::Application resource properties. Environment however is a separate logical component of Elastic Beanstalk so it's impossible to declare it from within the Application resource.
For better readability I would suggest declaring all the resources separately as per this example. Also when using this approach you can directly reference the TemplateName and VersionLabel in the AWS::ElasticBeanstalk::Environment resource.
Alternatively if you want to stick to the github example you can adjust the above example to look like this:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"sampleApplication": {
"Type": "AWS::ElasticBeanstalk::Application",
"Properties": {
"Description": "AWS Elastic Beanstalk Sample Application",
"ApplicationVersions": [{
"VersionLabel": "Initial Version",
"Description": "Initial Version",
"SourceBundle": {
"S3Bucket": {
"Fn::Sub": "elasticbeanstalk-samples-${AWS::Region}"
},
"S3Key": "php-newsample-app.zip"
}
}],
"ConfigurationTemplates": [{
"TemplateName": "DefaultConfiguration",
"Description": "AWS ElasticBeanstalk Sample Configuration Template",
"OptionSettings": [
{
"Namespace": "aws:autoscaling:asg",
"OptionName": "MinSize",
"Value": "2"
},
{
"Namespace": "aws:autoscaling:asg",
"OptionName": "MaxSize",
"Value": "6"
},
{
"Namespace": "aws:elasticbeanstalk:environment",
"OptionName": "EnvironmentType",
"Value": "LoadBalanced"
},
{
"Namespace": "aws:autoscaling:launchconfiguration",
"OptionName": "IamInstanceProfile",
"Value": {
"Ref": "MyInstanceProfile"
}
}
],
"SolutionStackName": "64bit Amazon Linux 2018.03 v2.9.11 running PHP 5.5"
}]
}
},
"sampleEnvironment": {
"Type": "AWS::ElasticBeanstalk::Environment",
"Properties": {
"ApplicationName": {
"Ref": "sampleApplication"
},
"Description": "AWS ElasticBeanstalk Sample Environment",
"TemplateName": "DefaultConfiguration",
"VersionLabel": "Initial Version"
}
},
"MyInstanceRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"ec2.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Description": "Beanstalk EC2 role",
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/AWSElasticBeanstalkWebTier",
"arn:aws:iam::aws:policy/AWSElasticBeanstalkMulticontainerDocker",
"arn:aws:iam::aws:policy/AWSElasticBeanstalkWorkerTier"
]
}
},
"MyInstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Roles": [
{
"Ref": "MyInstanceRole"
}
]
}
}
}
}
Just want to point out that AWS silently phased out the option of having the ApplicationVerions key under an AWS::ElasticBeanstalk::Application's Properties. It was still working in July 2022 but I noticed it stopped some time in August 2022, giving the error in the CloudFormation stack's Event tab:
Properties validation failed for resource TheEBAppResName with message: #: extraneous key [ApplicationVersions] is not permitted
where TheEBAppResName is the name of your AWS::ElasticBeanstalk::Application resource.
The only solution now is to follow the current AWS example and use a separate AWS::ElasticBeanstalk::ApplicationVersion resource.
Interestingly, I can't seem to find any documentation on the obsolete ApplicationVerions property anymore and the AWS blog that you linked to is no longer available, but I did find it cached on the Wayback machine. Even the earliest AWS doc on GitHub for AWS::ElasticBeanstalk::Application doesn't mention the ApplicationVerions property. Seems like AWS silently deprecated it sometime between when the blog was posted in April 2014 and that earliest GitHub doc page in December 2017, but didn't actually remove the option until last month, August 2022.
I am new to AWS and am currently working on simple tasks.
I have created a free tier EC2 instance using a cloudformation template. Now my next task is to write a simple application that uses respective AWS SDK to call CloudFormation API to create a stack from the template.
Here is the cloudformation template:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Ec2 Template",
"Metadata": {
"Copyright":[
"Copyright 2017, All rights reserved"
],
"Comments":[
"Create an EC2"
]
},
"Parameters": {
"KeyName": {
"Type": "AWS::EC2::KeyPair::KeyName",
"Description": "Name of an existing EC2 KeyPair to enable access to join ECS instances."
},
"InstanceTypeParameter":{
"Type": "String",
"Default": "t2.micro",
"AllowedValues": [
"t2.micro",
"m1.small",
"m1.large"
],
"Description": "Enter t2.micro, m1.small, or m1.large. Default is t2.micro."
},
"EcsSecurityGroupLb":{
"Type": "AWS::EC2::SecurityGroup::Id",
"Description":"The ECS ELB Security Group."
},
"vpcid":{
"Type": "AWS::EC2::VPC::Id"
},
"mySubnetIDs": {
"Description":"Subnet IDs",
"Type":"AWS::EC2::Subnet::Id"
}
},
"Resources":{
"Ec2Instance":{
"Type":"AWS::EC2::Instance",
"Properties":{
"ImageId": "ami-bf4193c7",
"KeyName": {
"Ref": "KeyName"
},
"InstanceType":{
"Ref": "InstanceTypeParameter"
},
"NetworkInterfaces":[
{
"AssociatePublicIpAddress":"true",
"DeviceIndex":"0",
"SubnetId":{
"Ref":"mySubnetIDs"
},
"GroupSet":[
{
"Ref": "EcsSecurityGroupLb"
}
]
}
],
"BlockDeviceMappings":[
{
"DeviceName": "/dev/sdc",
"VirtualName":"ephemeral0"
}
]
}
}
},
"Outputs":{
"Ec2Instance":{
"Description": "InstanceId of newly created EC2 instance",
"Value": {
"Ref": "Ec2Instance"
}
},
"InstanceIPAddress":{
"Value":{ "Fn::GetAtt": ["Ec2Instance", "PublicIp"]},
"Description": "Public IP address of instance"
}
}
}
I have gone through a lot of documentation but haven't really understood as to how to proceed. I would like to know if there are any good tutorials on this.
Looking for suggestions with respect to the steps as well.
Thanks!
Since you have to (as a task requirement) write the application yourself, you'll need to use one of the AWS SDKs that are available.
The SDK you choose will depend on what programming language you are most comfortable using (or required by your task).
Roughly, your program will need to do the following:
With the AWS SDK, create an AWS session which uses your IAM user's API keys.
Grab the Cloudformation template off of your local system.
With the AWS SDK, invoke Cloudformation to create the resource stack with your template.
(Optionally) Wait until the stack is complete and output a report on the stack status.
Good luck!
I'm setting up a pipeline to automate cloudformation stack templates deployment.
The pipeline itself is created in the aws eu-west-1 region, but cloudformation stacks templates would be deployed in any other region.
Actually I know and can execute pipeline action in a different account, but I don't see where to specify the region I would like my template to be deployed in, like we do with aws cli : aws --region cloudformation deploy.....
Is there anyway to trigger a pipeline in one region and execute a deploy action in another region please?
The action configuration properties don't offer such possibility...
A workaround would be to run aws cli deploy command from cli in the codebuild container and speficy the good region, But I would like to know if there is a more elegant way to do it
If you're looking to deploy to multiple regions, one after the other, you could create a Code Pipeline pipeline in every region you want to deploy to, and set up S3 cross-region replication so that the output of the first pipeline becomes the input to a pipeline in the next region.
Here's a blog post explaining this further: https://aws.amazon.com/blogs/devops/building-a-cross-regioncross-account-code-deployment-solution-on-aws/
Since late Nov 2018, CodePipeline supports cross regional deploys. However it still leaves a lot to be desired as you need to create artifact buckets in each region and copy over the deployment artifacts (e.g. in the codebuild container as you mentioned) to them before the Deploy action is triggered. So it's not as automated as it could be, but if you go through the process of setting it up, it works well.
CodePipeline now supports cross region deployment and for to trigger the pipeline in different region we can specify the "Region": "us-west-2" property in the action stage for CloudFormation which will trigger the deployment in that specific region.
Steps to follow for this setup:
Create two bucket in two different region which for example bucket in "us-east-1" and bucket in "us-west-2" (We can also use bucket already created by CodePipeline when you will setup pipeline first time in any region)
Configure the pipeline in such a way that is can use respective bucket while taking action in respective account.
specify the region in the action for CodePipeline.
Note: I have attached the sample CloudFormation template which will help you to do the cross region CloudFormation deployment.
{
"Parameters": {
"BranchName": {
"Description": "CodeCommit branch name for all the resources",
"Type": "String",
"Default": "master"
},
"RepositoryName": {
"Description": "CodeComit repository name",
"Type": "String",
"Default": "aws-account-resources"
},
"CFNServiceRoleDeployA": {
"Description": "CFN service role for create resourcecs for account-A",
"Type": "String",
"Default": "arn:aws:iam::xxxxxxxxxxxxxx:role/CloudFormation-service-role-cp"
},
"CodePipelineServiceRole": {
"Description": "Service role for codepipeline",
"Type": "String",
"Default": "arn:aws:iam::xxxxxxxxxxxxxx:role/AWS-CodePipeline-Service"
},
"CodePipelineArtifactStoreBucket1": {
"Description": "S3 bucket to store the artifacts",
"Type": "String",
"Default": "bucket-us-east-1"
},
"CodePipelineArtifactStoreBucket2": {
"Description": "S3 bucket to store the artifacts",
"Type": "String",
"Default": "bucket-us-west-2"
}
},
"Resources": {
"AppPipeline": {
"Type": "AWS::CodePipeline::Pipeline",
"Properties": {
"Name": {"Fn::Sub": "${AWS::StackName}-cross-account-pipeline" },
"ArtifactStores": [
{
"ArtifactStore": {
"Type": "S3",
"Location": {
"Ref": "CodePipelineArtifactStoreBucket1"
}
},
"Region": "us-east-1"
},
{
"ArtifactStore": {
"Type": "S3",
"Location": {
"Ref": "CodePipelineArtifactStoreBucket2"
}
},
"Region": "us-west-2"
}
],
"RoleArn": {
"Ref": "CodePipelineServiceRole"
},
"Stages": [
{
"Name": "Source",
"Actions": [
{
"Name": "SourceAction",
"ActionTypeId": {
"Category": "Source",
"Owner": "AWS",
"Version": 1,
"Provider": "CodeCommit"
},
"OutputArtifacts": [
{
"Name": "SourceOutput"
}
],
"Configuration": {
"BranchName": {
"Ref": "BranchName"
},
"RepositoryName": {
"Ref": "RepositoryName"
},
"PollForSourceChanges": true
},
"RunOrder": 1
}
]
},
{
"Name": "Deploy-to-account-A",
"Actions": [
{
"Name": "stage-1",
"InputArtifacts": [
{
"Name": "SourceOutput"
}
],
"ActionTypeId": {
"Category": "Deploy",
"Owner": "AWS",
"Version": 1,
"Provider": "CloudFormation"
},
"Configuration": {
"ActionMode": "CREATE_UPDATE",
"StackName": "cloudformation-stack-name-account-A",
"TemplatePath":"SourceOutput::accountA.json",
"Capabilities": "CAPABILITY_IAM",
"RoleArn": {
"Ref": "CFNServiceRoleDeployA"
}
},
"RunOrder": 2,
"Region": "us-west-2"
}
]
}
]
}
}
}
}
I have created a simple ShellCommandActivity which echos some text. It runs on a plain ec2 (vpc) instance. I see that the host has spinned up but it never executes the tasks and the task remains in WAITING_FOR_RUNNER status. After all the retries I get this error
Resource is stalled. Associated tasks not able to make progress.
I followed this troubleshoot-link but it didn't resolve my problem.
Here is the json description of the pipeline:
{
"objects": [
{
"resourceRole": "DataPipelineDefaultResourceRole",
"role": "DataPipelineDefaultRole",
"name": "ec2-compute",
"id": "ResourceId_viWO9",
"type": "Ec2Resource"
},
{
"failureAndRerunMode": "CASCADE",
"resourceRole": "DataPipelineDefaultResourceRole",
"role": "DataPipelineDefaultRole",
"pipelineLogUri": "s3://xyz-logs/",
"scheduleType": "ONDEMAND",
"name": "Default",
"id": "Default"
},
{
"name": "EchoActivity",
"id": "ShellCommandActivityId_kc8xz",
"runsOn": {
"ref": "ResourceId_viWO9"
},
"type": "ShellCommandActivity",
"command": "echo HelloWorld"
}
],
"parameters": []
}
What could be the problem here?
Thanks in advance.
I figured this out. The routing table in the VPC subnets was not properly configured.
To be specific, in my case the routing table didn't have 0.0.0.0/0 mapped to an internet-gateway. When I added this mapping, everything started working.
While using CloudFormation to create EC2 instance along with an autoscaling group, I face the error:
The following resource(s) failed to create: [WebsInstanceServerGroup].
image of CloudFormation Group output
The failure is seen while creating auto scaling group, but when I check the auto scaling group console, it says that the creation was 'successful.' (The 'in-progress' deletion happens after a 15 minute time out value from CloudFormation).
image of AutoScaling output
What could be the reason CloudFormation is not acknowledging that the AutoScale group is created successfully?
The error also says something about WebInstanceServerGroup, so I checked my template for that, but saw nothing suspicious.
"WebsInstanceServerGroup": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": {
"Fn::GetAZs": "AWS::Region"
},
"VPCZoneIdentifier": {
"Ref": "WebsELBSubnetId"
},
"LoadBalancerNames": [
{
"Ref": "WebsELB"
}
],
"LaunchConfigurationName": {
"Ref": "WebsEC2Instance"
},
"Cooldown": 300,
"HealthCheckGracePeriod": 600,
"HealthCheckType": "EC2",
"Tags": [
{
"Key": "Name",
"Value": {
"Ref": "WebsInstanceName"
},
"PropagateAtLaunch": "true"
},
{
"Key": "Service",
"Value": {
"Ref": "ServiceTag"
},
"PropagateAtLaunch": "true"
}
],
"MinSize": {
"Ref": "ASGMin"
},
"DesiredCapacity": {
"Ref": "ASGDesired"
},
"MaxSize": {
"Ref": "ASGMax"
}
},
"CreationPolicy": {
"ResourceSignal": {
"Count": {
"Ref": "ASGMin"
},
"Timeout": "PT15M"
}
}
}
Please let me know if more information is required, thanks in advance.
Looks like your EC2 instances in your autoscaling group are not sending the required success signals.
CloudFormation will wait for you to send ASGMin signals before considering your WebsInstanceServerGroup to be successfully created. So if ASGMin is set to 3, each of your 3 EC2 instances should send a signal.
To send the signal you can either use the cfn-signal helper, or with the AWS CLI:
aws cloudformation signal-resource \
--stack-name {your stack name here} \
--status SUCCESS \
--logical-resource-id WebsInstanceServerGroup \
--unique-id {the instance ID for the EC2 instance that is sending the signal}
Use this command at the end of your User Data script, when you consider your EC2 instance to be fully provisioned and ready to go.