Creating an ec2 instance along with IAM roles using cloud formation - amazon-web-services

I'm very new to Amazon cloudformation technique. I'm trying to launch an ec2 instance along with the IAM roles.
I have cloudformation script for this. But the problem I face is the IAM roles and Ec2 instances are created, but they aren't tied with each other.
I did create the IAM-roles using AWS::IAM::Role and AWS::IAM::InstanceProfile.
Is there any other command that I should use?
Thanks in advance.

Had to dig to get the final result, but here's an example of
Defining an access role (this will allow the EC2 instance to step into / assume to role),
Defining a policy for the role (i.e. when the EC2 assumes the role, what resources does it have access to),
Defining the instance profile (that is referenced by the EC2 instance, and has the access role mapped in)
"S3AccessRole" : {
"Type" : "AWS::IAM::Role",
"Properties" : {
"AssumeRolePolicyDocument" : {
"Statement" : [ {
"Effect" : "Allow",
"Principal" : {
"Service" : [ "ec2.amazonaws.com" ]
},
"Action" : [ "sts:AssumeRole" ]
} ]
},
"Path" : "/"
}
},
"S3RolePolicies" : {
"Type" : "AWS::IAM::Policy",
"Properties" : {
"PolicyName" : "s3access",
"PolicyDocument" : {
"Statement" : [ {
"Effect" : "Allow",
"Action" : "s3:*",
"Resource" : "*"
}]
},
"Roles" : [ { "Ref" : "S3AccessRole" } ]
}
},
"S3InstanceProfile" : {
"Type" : "AWS::IAM::InstanceProfile",
"Properties" : {
"Path" : "/",
"Roles" : [ { "Ref" : "S3AccessRole" } ]
}
}
The policy above allows all access to s3 resources. Adjust according to your needs. The IamInstanceProfile reference in the EC2 instance properties would refer be { "Ref" : "S3InstanceProfile" }
Note that as of May 2015, when you creating a stack that creates IAM roles, you need to check a box acknowledging such creation, otherwise you'll get a "Stack creation error: Requires capabilities : [CAPABILITY_IAM]" error.

The easiest way to solve such problems is to use CloudFormer. CloudFormer is a tool that creates a starting point template from the AWS resources you already have running in your environment.
The CloudFormer tool is packaged as a standalone application that you
can launch inside your AWS environment. The application is started on
a t1.micro Amazon EC2 instance via AWS CloudFormation.
Once you have launched Cloud Former, you will get a web interface (check the URL in the Output section of the launched stack), that will be able to describe all your resources in a specific region. It will lead you through which resources you wish in each category (DNS, Network, Compute...). At the end you can see the template and copy it, or save it in S3.
But if you wish to do it manually, you need to add the AWS::IAM::InstanceProfile you created to the Properties of AWS::EC2::Instance as IamInstanceProfile
{
"Type" : "AWS::EC2::Instance",
"Properties" : {
"AvailabilityZone" : String,
"BlockDeviceMappings" : [ EC2 Block Device Mapping, ... ],
"DisableApiTermination" : Boolean,
"EbsOptimized" : Boolean,
"IamInstanceProfile" : String,
"ImageId" : String,
"InstanceType" : String,
...
"UserData" : String,
"Volumes" : [ EC2 MountPoint, ... ]
}
}
See more details on AWS::EC2::Instance here

Suppose the AWS::IAM::InstanceProfile resource you create is called MyNewRole. To create an instance with that role (in the same CloudFormation template) set the EC2 resource's IamInstanceProfile property to a Ref to that resource. Here's an example (with lots of other details left out):
"Resources": {
"MyNewRole": {
"Type": "AWS::IAM::InstanceProfile",
... more stuff here
},
"MyNewEc2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"IamInstanceProfile": { "Ref": "MyNewRole" },
... more stuff here
}
}
}

Related

AWS Api Gateway attach existing policy

Normally if I want to create a private AWS::ApiGateway::RestApi with policy only allowing VPC traffic to invoke any resources on the API I'd do something like so:
"ApiGatewayRestApi": {
"Type": "AWS::ApiGateway::RestApi",
"Properties": {
"Name": "api-foo-bar",
"EndpointConfiguration": {
"Types": [
"PRIVATE"
]
},
"Policy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": [
"execute-api:Invoke"
],
"Resource": "execute-api:/*",
"Condition": {
"StringEquals": {
"aws:SourceVpc": "vpc-000000000000"
}
}
}
]
}
}
}
I have been asked if we can create a policy and then reuse it for different Api Gateways we may create? something amongst the lines:
"ApiGatewayRestApi": {
"Type": "AWS::ApiGateway::RestApi",
"Properties": {
"Name": "api-foo-bar",
"EndpointConfiguration": {
"Types": [
"PRIVATE"
]
},
"Policy": "arn:aws:*whatever*"
}
},
And I have no idea! Nor can I find any documentation or examples showing that. Has anyone done this? Is it at all doable? Thanks :)
No, it's not doable currently. The policy you are attaching to the Api Gateway is Resource-based policy.
From aws documentation,
With resource-based policies, you can specify who has access to the
resource and what actions they can perform on it.
Aws documentation shows the type that each property in cloudformation can take. The following is the properties and the types allowed in "AWS::ApiGateway::RestApi"
{
"Type" : "AWS::ApiGateway::RestApi",
"Properties" : {
"ApiKeySourceType" : String,
"BinaryMediaTypes" : [ String, ... ],
"Body" : Json,
"BodyS3Location" : S3Location,
"CloneFrom" : String,
"Description" : String,
"DisableExecuteApiEndpoint" : Boolean,
"EndpointConfiguration" : EndpointConfiguration,
"FailOnWarnings" : Boolean,
"MinimumCompressionSize" : Integer,
"Mode" : String,
"Name" : String,
"Parameters" : {Key : Value, ...},
"Policy" : Json,
"Tags" : [ Tag, ... ]
}
}
Notice that Policy property takes the type of JSON. Furthermore, the documentation writes the following for the Policy property:
A policy document that contains the permissions for the RestApi
resource.
and gives us a hint that Policy property does not take the following form: "Policy": "arn:aws:*whatever*" and only accepts a policy document in the form of JSON as Api Gateway's resource-based policy.
ref:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-restapi.html#cfn-apigateway-restapi-policy
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_identity-vs-resource.html
you can find the docs here.
Policy
A policy document that contains the permissions for the RestApi resource. To set the ARN for the policy, use the !Join intrinsic function with "" as delimiter and values of "execute-api:/" and "*".

How to include cloudWatch Log Group subscription in cloudFormation template?

Say I have a Log Group in cloudWatch, which I want to subscribe to a lambda with a filter (Subscriptions->Stream to AWS Lambda).
I want to achieve it with cloudFormation template, but from cloudFormation doc, it seems that the only two available cloudWatch resources are Alarm/Dashboard.
Questions is:
Is there any way to write cloudWatch Log Group subscription in
cloudFormation?
If not, any alternative way (say from lambda
resource configuration in cloudFormation template)?
Oh, that's a tricky one. I only figured it out by creating one in the console and reverse enginerring - ick. But you're lucky - I have it on hand :P This is the json I was using for subscribing a lambda to a vpc flow log.
Note that the 'VPCFlowLogsGroup' is the logical Id of the log group, the 'FlowLogsCollector' that of the lambda.
"FlowLogsCollectorEventPermission": {
"Type" : "AWS::Lambda::Permission",
"Properties" : {
"Principal" : { "Fn::Sub": "logs.${AWS::Region}.amazonaws.com" },
"Action" : "lambda:InvokeFunction",
"FunctionName" : { "Fn::GetAtt": [ "FlowLogsCollector", "Arn" ] },
"SourceAccount": { "Ref": "AWS::AccountId" },
"SourceArn" : { "Fn::GetAtt": [ "VPCFlowLogsGroup", "Arn" ] }
}
},
"FlowLogsCollectorSubscription": {
"Type" : "AWS::Logs::SubscriptionFilter",
"DependsOn": "FlowLogsCollectorEventPermission",
"Properties" : {
"LogGroupName" : { "Ref" : "VPCFlowLogsGroup" },
"FilterPattern" : "",
"DestinationArn" : { "Fn::GetAtt" : [ "FlowLogsCollector", "Arn" ] }
}
},

AWS Service Unable To Assume Role

I've two AWS Cloudformation stacks, one for IAM roles and the second to create an AWS service and import the respective roles into it using Cloudformation.
When 10+ services are deployed the following error appears randomly on 1 or 2 of the services -
AWS::ECS::Service service Unable to assume role and validate the
listeners configured on your load balancer. Please verify that the ECS
service role being passed has the proper permissions.
If all the services are torn down and the services redployed to the ECS cluster, the error appears but for different services.
The AWS fix for this can be seen here
If the 1 or 2 broken services are torn down and redeployed the services deploy without issue. So the problem appears to only occur when many services are deployed at the same time - this indicates it may be an IAM propagation timing issue within Cloudformation.
I've tried adding depends on in the service definition -
"service" : {
"Type" : "AWS::ECS::Service",
"DependsOn" : [
"taskdefinition",
"ECSServiceRole"
],
"Properties" : {
"Cluster" : { "Ref": "ECSCluster"},
"Role" : {"Ref" : "ECSServiceRole"},
etc...
}
}
But this doesn't work.
As you can note, I've also removed the IAM import value for the ECSServiceRole and replaced it with an inline resource policy seen here -
"ECSServiceRole" : {
"Type" : "AWS::IAM::Role",
"Properties" : {
"AssumeRolePolicyDocument" : {
"Statement" : [
{
"Sid": "",
"Effect" : "Allow",
"Principal" : {
"Service" : [
"ecs.amazonaws.com"
]
},
"Action" : [
"sts:AssumeRole"
]
}
]
},
"Path" : "/",
"Policies" : [
{
"PolicyName" : "ecs-service",
"PolicyDocument" : {
"Statement" : [
{
"Effect" : "Allow",
"Action" : [
"ec2:Describe*",
"ec2:AuthorizeSecurityGroupIngress",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:Describe*",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:RegisterTargets",
"sns:*"
],
"Resource" : "*"
}
]
}
}
]
}
}
But again - the inline policy doesn't fix the issue either.
Any ideas or pointers would be much appreciated!
In reply to answer 1.
Thank you - I wasn't aware of this improvment.
Is this the correct way to associate the service linked role for ECS?
"ECSServiceRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": [
"ecs.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName": "CreateServiceLinkedRoleForECS",
"PolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:CreateServiceLinkedRole",
"iam:PutRolePolicy",
"iam:UpdateRoleDescription",
"iam:DeleteServiceLinkedRole",
"iam:GetServiceLinkedRoleDeletionStatus"
],
"Resource": "arn:aws:iam::*:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS*",
"Condition": {
"StringLike": {
"iam:AWSServiceName": "ecs.amazonaws.com"
}
}
}
]
}
}
]
}
}
Final Answer
After months of intermittent on-going issues with AWS regarding this matter AWS came back to say they were throttling us in the background, on the ELB. This is why the random and varied issues were appearing when deploying 3+ docker services via Cloudformation at the same time. The solution was nothing to do with IAM permissions, rather it was to increase the rate limit on the ELB via the "AWS Service Team".
So the fix was to continue using the two stack approach in Cloudformation, one with the IAM roles, which in turn were imported into the service layer stack. The fix was to add a depends on in the service definition for all of the other stack resources in the service layer script. By doing this it allows sufficient time for the IAM roles to be imported and executed by the service, thus this was a Cloudformation resource creation timing issue.
"service" : {
"Type" : "AWS::ECS::Service",
"DependsOn" : [
"TaskDefinition",
"EcsElasticLoadBalancer",
"DnsRecord"
],
"Properties" : {
etc...
}
}
UPDATE:
As of July 19th 2018, it is now possible to create a IAM Service-Linked Roles using CloudFormation https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-servicelinkedrole.html.
EcsServiceLinkedRole:
Type: "AWS::IAM::ServiceLinkedRole"
Properties:
AWSServiceName: "ecs.amazonaws.com"
Description: "Role to enable Amazon ECS to manage your cluster."
OLD ANSWER:
Creating your own ECSServiceRole is no longer required. By not specifying a role for your service, AWS will default on using the ECS Service-Linked role. If your AWS account is recent enough, or you have already created a cluster via the console you don't have to do anything for this to work. If not, run the following command to create the role: aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com.

DeletionPolicy:Snapshot cannot be specified for a cluster instance, use deletion policy on the cluster instead

I am trying to create RDS cluster and aurora instance using the cloudoformation template below:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "example setup",
"Parameters" : {
"DBInstanceIdentifier" : {
"Type": "String",
"Description": "Name for the DB instance."
},
"DBUser" : {
"Type": "String",
"Description": "Master user"
},
"DBPassword" : {
"Type": "String",
"Description": "Pass"
},
"DBModel" : {
"Type": "String",
"Description": "Instance model to be used for the DB."
}
},
"Resources": {
"RDSCluster": {
"Type": "AWS::RDS::DBCluster",
"Properties": {
"MasterUsername": { "Ref" : "DBUser" },
"MasterUserPassword": { "Ref" : "DBPassword" },
"Engine": "aurora",
"DBClusterParameterGroupName": "default.aurora5.6",
"VpcSecurityGroupIds": [{"Fn::GetAtt" : [ "DBFromSiteSecurityGroup" , "GroupId" ]}]
}
},
"AuroraInstance": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
"DBInstanceIdentifier": { "Ref" : "DBInstanceIdentifier" },
"DBParameterGroupName": "default.aurora5.6",
"Engine": "aurora",
"DBClusterIdentifier": {
"Ref": "RDSCluster"
},
"PubliclyAccessible": "true",
"DBInstanceClass": { "Ref" : "DBModel" }
}
},
"DBFromSiteSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Enable MySQL",
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : "3306", "ToPort" : "3306", "CidrIp" : "195.171.102.98/32"}
]
}
},
"DBFromSiteSecurityGroupIngress1" : {
"Type" : "AWS::EC2::SecurityGroupIngress",
"Properties" : {
"GroupName" : { "Ref" : "DBFromSiteSecurityGroup" },
"IpProtocol" : "tcp",
"ToPort" : "3306",
"FromPort" : "3306",
"SourceSecurityGroupName" : { "Ref" : "DBFromSiteSecurityGroup" }
}
}
}
}
The db_model parameter I am passing is "db.t2.medium". The cluster gets created successfully in the cloudformation console however the AWS::RDS::DBInstance creation fails with the following error
"DeletionPolicy:Snapshot cannot be specified for a cluster instance, use deletion policy on the cluster instead."
What's more weird that when I try to run the same CF template in say eu london region, it works fine!!! Is there something wrong with the EU ireland region and aurora?
From AWS Support
This is a known issue and has been reported by other customers as well. The service team is currently working on the fix for this but there is no ETA as to when that would be pushed.
The work-around in the meanwhile is to specify a DeletionPolicy inside the DB instance resource definition that is failing to create, with the value of 'Delete'. [1]
An example below:
"Resources": {
"Database1": {
"DeletionPolicy": "Delete",
"Properties": {...},
"Type": "AWS::RDS::DBInstance"
}
}
References:
[1] DeletionPolicy - http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html#w2ab2c19c23c11c17
An update from AWS Support:
When creating an Amazon Aurora DBInstance in a DB Cluster using AWS
CloudFormation, CloudFormation applies a default Deletion policy of
“Delete”, if a deletion policy is not specified. If a Deletion Policy
of “Snapshot” is specified for an Amazon Aurora DBInstance,
CloudFormation returns an error, because instances in a DB Cluster
cannot be snapshotted individually; Snapshotting must be done at the
DB Cluster level.
As part of a recent deployment, we inadvertently changed the default
deletion policy for an Amazon Aurora DBInstance to “Snapshot”. This
caused our template validation to fail. To remedy this, CloudFormation
is reverting the value of default DeletionPolicy for Amazon Aurora
DBInstances to “Delete”. This fix will be completed by 21st July,
2017. Until this fix is completely rolled out, customers can explicitly override our incorrect defaults by specifying a deletion
policy of “Delete” for Amazon Aurora DBInstances.
We have corrected the gap in our testing that led to this situation,
and will continue to improve our testing to prevent recurrences. We
recognize how critical it is for us to preserve existing behavior for
our customers, and apologize for this inconvenience.

AWS cloudformation - how to attach EBS volume to EC2 windows instance

Here is a working Amazon CloudFormation JSON template that creates an Amazon EC2 Windows 2016 instance.
I want to attach an EBS volume that is backed-up on an S3 bucket. How can I do this? Any pointers please?
{
"Parameters" :{
"KeyName" : {
"Description" : "Name of the existing EC2 KeyPair",
"Type" : "String"
}
},
"Mappings" : {
"RegionMap" : {
"us-east-1" : {
"AMI" : "ami-48b4bf31"
},
"us-west-1": {
"AMI" : "ami-48b4bf31"
},
"us-west-2":{
"AMI" : "ami-48b4bf31"
}
}
},
"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties": {
"KeyName" : {"Ref" : "KeyName"},
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}
}
}
},
"Outputs" : {
"AvailablityZone" : {
"Description" : "Availability Zone of the newly created EC2 instance",
"Value" : { "Fn::GetAtt" : [ "Ec2Instance", "AvailabilityZone" ] }
},
"PublicIp" :{
"Description" : "Public IP is",
"Value": {"Fn::GetAtt": ["Ec2Instance", "PublicIp"] }
}
}
}
There are probably two parts to this question:
1) How do I specify details about the ebs volumes of my windows host in cloudformation
I think you are looking for something like this block device mappings (ephemeral disk or ebs) or Volumes (EBS only)
"BlockDeviceMappings" : [
{
"DeviceName" : "/dev/sdc",
"Ebs" : { "VolumeSize" : "50" }
},
{
"DeviceName" : "/dev/sdd",
"Ebs" : { "VolumeSize" : "100" }
}
]
or volumes
"Volumes" : [
{ "VolumeId" :
{ "Ref" : "NewVolume" }, "Device" : "/dev/sdk" }
]
and secondly
2) How do I back up my windows host.
The easiest way is probably with the aws cli snapshot command.
aws ec2 create-snapshot --volume-id vol-1234567890abcdef0 --description "Backup of my windows server"
Sometimes people drop that in cron on a linux box, wrap it with a little logic and loop it over the whole infrastructure. Nice, easy, cheap backups of AWS infrastructure. If you ever need to restore you can create a new image from snapshot.
Backup: Your best strategy is to create an AMI on a regular basis. An AMI is a snapshot of the volumes attached to an Amazon EC2 instance -- yes, it can include ALL the disks attached to an instance. An AMI is actually just a collection of EBS snapshots, plus some metadata.
Restore: Launch a new Amazon EC2 instance from that AMI. It will contain all data, on all disks, that was present when the AMI was created. (It creates new EBS volumes, but they will contain the same data as when the AMI was created.)
Each time you create a new AMI, it will receive a new AMI-ID. Therefore, I suggest that your CloudFormation template accepts the AMI-ID as a parameter that can be entered when the stack is created. You would simply paste in the AMI-ID of the latest AMI and the instance would use that AMI.
Also, please note that AMIs are created in one region only. You can copy the AMI to a different region, but it will receive a different AMI-ID in the new region.
Frankly, your CloudFormation templates appears to be merely launching the EC2 instance, which you could do just as easily in the console without using CloudFormation.