Cloudformation script taskset cannot pull image from ECR - amazon-web-services

In my current setup I use cloudformation to setup an environment. All the steps configured so far seem to work, except for the taskset. I use an image on AWS ECR repo to launch a Fargate instance, which is put into a new cluster with a service.
When I create the stack all the steps succeed but the creation of the service is stuck in "CREATE_IN_PROGRESS" forever. When I look at the created service and more specifically the task, I found that cloudformation was unable to fetch the image. The error is as follows:
STOPPED (CannotPullContainerError: Error response from daem)
Now when I manually try to an instance using the same image on said cluster, it works just fine. After some googling I came upon certain solutions. I have implemented all of them, such as:
Create a network with security groups which allows outgoing traffic to all destinations
Set the flag AssignPublicIp to 'ENABLED' in the awsvpc configuration
Set the user execution role to the same one I used in manual creation
Note that the task does get created and it does receive a public and private IP address. Also, I have tried to throw away the ECR instance and create a new one with a new name (and of course tagged and pushed the image).
The following snippet is my service and taskset definition (renamed some sensitive info):
"MyVPC": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock": "10.0.1.0/16",
"Tags": [
{"Key":"Name", "Value":"MyVPC"
}
]
},
},
"MyInternetGateway" : {
"Type" : "AWS::EC2::InternetGateway",
"Properties" : {
}
},
"MySubPublic": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"AvailabilityZone": "eu-central-1a",
"CidrBlock": "10.0.1.0/28",
"MapPublicIpOnLaunch": true,
"VpcId": {
"Ref": "MyVPC"
}
},
"DependsOn": "MyInternetGateway"
},
"MySecGroup":
{
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "My security group for all incoming and outgoing.",
"GroupName" : "MySecGroup",
"SecurityGroupEgress" : [ {
"CidrIp" : "0.0.0.0/0",
"Description" : "Allow machine to reach internet.",
"FromPort" : -1,
"IpProtocol" : -1,
"ToPort" : -1
} ],
"SecurityGroupIngress" : [ {
"CidrIp" : "0.0.0.0/0",
"Description" : "Allow machine to be reached from the entire internet.",
"FromPort" : -1,
"IpProtocol" : -1,
"ToPort" : -1
} ],
"VpcId" : {"Ref": "MyVPC"}
},
"DependsOn": "MyVPC"
},
"MyCluster": {
"Type": "AWS::ECS::Cluster",
"Properties": {
"ClusterName": "MyCluster"
},
"DependsOn": [
"MyVPC"
]
},
"MyDockerTask": {
"Type": "AWS::ECS::TaskDefinition",
"Properties": {
"ContainerDefinitions": [
{
"Cpu": 512,
"Image": "<NRHERE>.dkr.ecr.eu-central-1.amazonaws.com/mysqs",
"Memory": 1024,
"MemoryReservation": 1024,
"Name": "MyContainer"
}
],
"Cpu": "512",
"ExecutionRoleArn": "arn:aws:iam::<NRHERE>:role/ecsTaskExecutionRole",
"Family": "MyFam",
"Memory": "1024",
"NetworkMode": "awsvpc",
"RequiresCompatibilities": [
"FARGATE",
"EC2"
],
"TaskRoleArn": "arn:aws:iam::<NRHERE>:role/ecsTaskExecutionRole"
},
},
"MyService": {
"Type" : "AWS::ECS::Service",
"Properties" : {
"Cluster" : {"Fn::GetAtt": ["MyCluster", "Arn"]},
"DesiredCount" : 1,
"LaunchType" : "FARGATE",
"NetworkConfiguration" : {
"AwsvpcConfiguration" : {
"AssignPublicIp" : "ENABLED",
"SecurityGroups" : [ {"Fn::GetAtt": ["MySecGroup", "GroupId"]} ],
"Subnets" : [ {"Ref": "MySubPublic"}]
}
},
"SchedulingStrategy" : "REPLICA",
"ServiceName" : "MyService",
"TaskDefinition": {"Ref": "MyDockerTask"}
},
"DependsOn": "MySubPublic"
},
"MyTaskSet": {
"Type": "AWS::ECS::TaskSet",
"Properties": {
"Cluster": {
"Ref": "MyCluster"
},
"LaunchType": "FARGATE",
"NetworkConfiguration" : {
"AwsvpcConfiguration" : {
"AssignPublicIp" : "ENABLED",
"SecurityGroups" : [ {"Fn::GetAtt": ["MySecGroup", "GroupId"]} ],
"Subnets" : [ {"Ref": "MySubPublic"}]
}
},
"Service": {"Ref": "MyService"},
"TaskDefinition": {
"Ref": "MyDockerTask"
}
},
"DependsOn": [
"MyCluster"
]
},
None of the above provided the solution. I'm open to any suggestion/solution

From the CFN template provided I can conclude that MySubPublic has no internet access.
Specifically, I see that you are creating the following VPC-related resources:
AWS::EC2::VPC
AWS::EC2::InternetGateway
AWS::EC2::Subnet
AWS::EC2::SecurityGroup
However, the problem is the lack of:
AWS::EC2::VPCGatewayAttachment which will connect the InternetGateway to your VPC
Definitions of AWS::EC2::RouteTable and AWS::EC2::Route which will be used to make your MySubPublic route traffic to the AWS::EC2::InternetGateway.
General steps to enable internet access in a subnet are described here:
Attach an internet gateway to your VPC.
Add a route to your subnet's route table that directs internet-bound traffic to the internet gateway. If a subnet is
associated with a route table that has a route to an internet gateway,
it's known as a public subnet. If a subnet is associated with a route
table that does not have a route to an internet gateway, it's known as
a private subnet.
Ensure that instances in your subnet have a globally unique IP address (public IPv4 address, Elastic IP address, or IPv6 address).
Ensure that your network access control lists and security group rules allow the relevant traffic to flow to and from your instance.

Related

How to restrict the default AWS VPC ingress to block all traffic

I have a VPC that gets created through CloudFormation, but it also creates a default Security Group with inbound and outbound access. I don't use this default security group, and would like to restrict it. I can't find anywhere that gives me a good reference on how to lock it down through CloudFormation. Egress I have setup, but not Ingress. The best I can do is to manually remove it once it's created, but I'm hoping to find a config that will do the same when it gets created. This is what I have, but I don't know if it's effective. How do I completely block all ingress and egress for this SG?:
"VPCDefaultSecurityGroupIngress": {
"Type" : "AWS::EC2::SecurityGroupIngress",
"Properties" : {
"GroupId": { "Fn::GetAtt" : ["Vpc", "DefaultSecurityGroup"] },
"CidrIp": "127.0.0.1/32",
"IpProtocol": "-1"
},
"DependsOn" : [
"Vpc"
]
},
"VPCDefaultSecurityGroupEgress": {
"Type" : "AWS::EC2::SecurityGroupEgress",
"Properties" : {
"GroupId": { "Fn::GetAtt" : ["Vpc", "DefaultSecurityGroup"] },
"CidrIp": "127.0.0.1/32",
"IpProtocol": "-1"
},
"DependsOn" : [
"Vpc"
]
},

attribute publicip was not found for resource during aws cloudformation

I'm very new to aws cloudformation, I try to launch a EC2 with Neo4j install in a private VPC, I have found someone who has already created a cloudformation template for instance with Neo4j, but that instance is for public VPC, so I have modified the template to suit my purpose, but I got this problem when I launch it: 'attribute publicip was not found for resource'
Here is some part of the script (without the neo4j bash script and EBS volume setup):
"Mappings" : {
"AWSRegionArch2AMI" : {
"eu-west-1" : { "64" : "ami-58d7e821" }
}
},
"Parameters": {
"InstanceType" : {
"Description" : "EC2 instance type",
"Type" : "String",
"Default" : "m5.large",
"ConstraintDescription" : "Must be a valid EC2 instance type."
},
"SSHKeyName": {
"Description": "Name of the SSH key that you will use to access the server (must be on AWS Availability Zone already)",
"Type": "String"
},
"NetworkWhitelist": {
"Description": " The IP address range that can be used to connect to the Neo4j server (by REST or SSH)",
"Type": "String",
"MinLength": "9",
"MaxLength": "18",
"Default": "",
"AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
"ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x."
},
"SubnetId" : {
"Type" : "AWS::EC2::Subnet::Id",
"Description" : "SubnetId of an existing subnet (for the primary network) in your Virtual Private Cloud (VPC)"
},
"SecurityGroupIds": {
"Type": "AWS::EC2::SecurityGroup::Id",
"Description" : "Existing SecurityGroups ID"
},
"AvailabilityZone": {
"Type" : "AWS::EC2::AvailabilityZone::Name",
"Description" : "Select the Availability Zone"
}
},
"Resources": {
"Server": {
"Type": "AWS::EC2::Instance",
"Properties": {
"AvailabilityZone": {
"Ref": "AvailabilityZone"
},
"DisableApiTermination": "FALSE",
"ImageId": {
"Fn::FindInMap": [ "AWSRegionArch2AMI", {
"Ref": "AWS::Region"
}, "64"]
},
"InstanceType": {
"Ref": "InstanceType"
},
"KeyName": {"Ref": "SSHKeyName"},
"Monitoring": "false",
"NetworkInterfaces" : [
{
"AssociatePublicIpAddress": false,
"DeleteOnTermination": true,
"DeviceIndex": "0",
"SubnetId": {"Ref": "SubnetId"},
"GroupSet": [ {"Ref": "SecurityGroupIds"} ]
}
],
Can't I lanuch an instance without public ip address just like in 'Configure Instance Details' in 'Launch instance wizard'?
thank you
Do you have the "Auto-assign Public IP" option enabled for the subnet you're trying to create the instance in? Because you're explicitly not associating a public IP address, it might be failing because the resource is expecting to have a public IP address assigned. A surefire way to test this would be to set the SubnetId parameter to the ID of a subnet that does not automatically assign public IP addresses when you deploy the stack.
Perhaps you are tying to do Fn::GetAtt on the Instance logical Id to get the PublicIp somewhere in your code and the Instance doesn't have PublicIp assigned to it.
I experienced this error just a couple of weeks ago, while getting my feet wet with CloudFormation. In my case, I'd dropped the public IP for the interface, in favor of only a private IP, but I still had an output configured in the CloudFormation template that referenced the now non-existent publicid attribute. Removing that output from the template fixed my issue.

DeletionPolicy:Snapshot cannot be specified for a cluster instance, use deletion policy on the cluster instead

I am trying to create RDS cluster and aurora instance using the cloudoformation template below:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "example setup",
"Parameters" : {
"DBInstanceIdentifier" : {
"Type": "String",
"Description": "Name for the DB instance."
},
"DBUser" : {
"Type": "String",
"Description": "Master user"
},
"DBPassword" : {
"Type": "String",
"Description": "Pass"
},
"DBModel" : {
"Type": "String",
"Description": "Instance model to be used for the DB."
}
},
"Resources": {
"RDSCluster": {
"Type": "AWS::RDS::DBCluster",
"Properties": {
"MasterUsername": { "Ref" : "DBUser" },
"MasterUserPassword": { "Ref" : "DBPassword" },
"Engine": "aurora",
"DBClusterParameterGroupName": "default.aurora5.6",
"VpcSecurityGroupIds": [{"Fn::GetAtt" : [ "DBFromSiteSecurityGroup" , "GroupId" ]}]
}
},
"AuroraInstance": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
"DBInstanceIdentifier": { "Ref" : "DBInstanceIdentifier" },
"DBParameterGroupName": "default.aurora5.6",
"Engine": "aurora",
"DBClusterIdentifier": {
"Ref": "RDSCluster"
},
"PubliclyAccessible": "true",
"DBInstanceClass": { "Ref" : "DBModel" }
}
},
"DBFromSiteSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Enable MySQL",
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : "3306", "ToPort" : "3306", "CidrIp" : "195.171.102.98/32"}
]
}
},
"DBFromSiteSecurityGroupIngress1" : {
"Type" : "AWS::EC2::SecurityGroupIngress",
"Properties" : {
"GroupName" : { "Ref" : "DBFromSiteSecurityGroup" },
"IpProtocol" : "tcp",
"ToPort" : "3306",
"FromPort" : "3306",
"SourceSecurityGroupName" : { "Ref" : "DBFromSiteSecurityGroup" }
}
}
}
}
The db_model parameter I am passing is "db.t2.medium". The cluster gets created successfully in the cloudformation console however the AWS::RDS::DBInstance creation fails with the following error
"DeletionPolicy:Snapshot cannot be specified for a cluster instance, use deletion policy on the cluster instead."
What's more weird that when I try to run the same CF template in say eu london region, it works fine!!! Is there something wrong with the EU ireland region and aurora?
From AWS Support
This is a known issue and has been reported by other customers as well. The service team is currently working on the fix for this but there is no ETA as to when that would be pushed.
The work-around in the meanwhile is to specify a DeletionPolicy inside the DB instance resource definition that is failing to create, with the value of 'Delete'. [1]
An example below:
"Resources": {
"Database1": {
"DeletionPolicy": "Delete",
"Properties": {...},
"Type": "AWS::RDS::DBInstance"
}
}
References:
[1] DeletionPolicy - http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html#w2ab2c19c23c11c17
An update from AWS Support:
When creating an Amazon Aurora DBInstance in a DB Cluster using AWS
CloudFormation, CloudFormation applies a default Deletion policy of
“Delete”, if a deletion policy is not specified. If a Deletion Policy
of “Snapshot” is specified for an Amazon Aurora DBInstance,
CloudFormation returns an error, because instances in a DB Cluster
cannot be snapshotted individually; Snapshotting must be done at the
DB Cluster level.
As part of a recent deployment, we inadvertently changed the default
deletion policy for an Amazon Aurora DBInstance to “Snapshot”. This
caused our template validation to fail. To remedy this, CloudFormation
is reverting the value of default DeletionPolicy for Amazon Aurora
DBInstances to “Delete”. This fix will be completed by 21st July,
2017. Until this fix is completely rolled out, customers can explicitly override our incorrect defaults by specifying a deletion
policy of “Delete” for Amazon Aurora DBInstances.
We have corrected the gap in our testing that led to this situation,
and will continue to improve our testing to prevent recurrences. We
recognize how critical it is for us to preserve existing behavior for
our customers, and apologize for this inconvenience.

How do I create an Ec2 Instance with a Public IP automatically **without** declaring an Elastic IP in cloudformation?

In AWS Cloudformation, is there any way to declare an EC2 instance in VPC with a Public IP without the need to declare an Elastic IP and attach to it?
In AWS::AutoScaling::LaunchConfiguration, you could add a property "AssociatePublicIpAddress" to say the instances will accept a Public IP automatically. I am looking for the equivalent for AWS::EC2::Instance
Below is my cloudformation snippet for creating an EC2 instance. I can't any doc that mentions how to add a public IP without having to declare an Elastic IP beforehand.
"MyEc2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"IamInstanceProfile": {
"Ref": "MyEc2InstanceProfile"
},
"ImageId": {
"Fn::FindInMap": [
"MyEc2Box",
{
"Ref": "Region"
},
"ImageId"
]
},
"InstanceType": {
"Fn::FindInMap": [
"MyEc2Box",
{
"Ref": "Region"
},
"InstanceType"
]
},
"KeyName": {
"Ref": "DefaultKeyPair"
},
"Monitoring": "true",
"SecurityGroupIds": [
{
"Ref": "MyEc2SecurityGroup"
}
],
"SubnetId": {
"Ref": "MyBoxSubnet"
},
"Tags": [
{
"Key": "Name",
"Value": "MyBox"
},
]
}
},
Assuming you are starting your instance in a VPC public subnet (i.e. a subnet that has a routing table incl. a rule to send traffic to 0.0.0.0/0 to the Internet Gateway), just define AssociatePublicIpAddress property in the NetworkInterfaces group of your EC2 resource:
"NetworkInterfaces" : [{
"AssociatePublicIpAddress" : "True",
"DeleteOnTermination" : "True",
"SubnetId" : { "Ref" : "PublicSubnet" },
"DeviceIndex" : "0",
"GroupSet" : [ { "Ref" : "SecurityGroup" } ]
}],
See documentation at http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-network-iface-embedded.html
If you are starting your instance in EC2 Classic networking (non VPC), it will receive a public IP address automatically.
I see that this is an old post but i post the answer anyway it can be helpful.
In the subnet you can set : "MapPublicIpOnLaunch" to True so all the instance of this subnet will have a public IP.
MapPublicIpOnLaunch
Indicates whether instances that are launched in this subnet receive a public IP address. By default, the value is false.
Required: No
Type: Boolean
Update requires: No interruption.

How do I assign a created SecurityGroup to an ELB from CloudFormation?

I've got a CloudFormation script that generates a SecurityGroup and an ELB; I'm trying to reference the SecurityGroup in the ELB creation; here's the resources bit:
"ELBSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Security group for the Arena dev stack",
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : 80, "ToPort" : 80, "CidrIp" : { "Ref" : "OfficeIp" }}
]
}
},
"ProjectLoadBalancerTest" : {
"Type" : "AWS::ElasticLoadBalancing::LoadBalancer",
"Properties" : {
"AvailabilityZones" : { "Fn::GetAZs" : "" },
"Instances" : [ ],
"Listeners" : [ {
"LoadBalancerPort" : "80",
"InstancePort" : "12345",
"Protocol" : "HTTP"
} ],
"HealthCheck" : {
"Target" : {
"Fn::Join" : [ "", [ "HTTP:", "12345", "/status.json" ] ]
},
"HealthyThreshold" : "2",
"UnhealthyThreshold" : "5",
"Interval" : "60",
"Timeout" : "30"
},
"SecurityGroups" : [
{ "Ref" : "ELBSecurityGroup" }
]
}
}
Unfortunately, this fails with:
Invalid id: "sebelbtest2-ELBSecurityGroup-1F5Z5DIIVQKD1" (expecting "sg-...")
So how can I reference ELBSecurityGroup for use as a property in the ELB creation?
Thanks!
As mytwocents mentioned, the solution is to use Fn::GetAtt. SecurityGroups are now supported by this function: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html
This works on the ELB:
...
"SecurityGroups" : [
{ "Fn::GetAtt" : [ "ELBSecurityGroup", "GroupId" ] }
]
...
Note. If you're putting this into a non-default VPC you'll also need to specify the VPC for the security group, and a subnet ID for the ELB.
As my CloudFormation script is all done within a VPC, I figured out what the problem was - I was creating the Security group, but not specifying the VpcId for it.
Security groups appear to be either normal security groups, or VPC security groups; if you do { "Ref": "MySecurityGroup" } on a normal one, you get the security group name, but not the ID. If you do { "Ref": "MySecurityGroup" } on a VPC one, you get back the sg-abcdef id, which is what is required for the ELB security group parameter.
So the full answer is:
"ELBSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Security group for the ELB",
"VpcId" : { "Ref" : "VpcId" },
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : 80, "ToPort" : 80, "CidrIp" : { "Ref" : "OfficeIp" }}
]
}
},
"MyELB": {
"Type" : "AWS::ElasticLoadBalancing::LoadBalancer",
"Properties" : {
"AvailabilityZones" : { "Fn::GetAZs" : "" },
"Listeners" : [ {
"LoadBalancerPort" : "80",
"InstancePort" : 8000,
"Protocol" : "HTTP"
} ],
"SecurityGroups" : [ { "Ref" : "ELBSecurityGroup" } ]
}
}
This all works perfectly (provided everything you're doing is within your VPC) and in my configuration, will successfully limit access to whatever OfficeIP is set to.
Hmmm…
It appears as though the template is returning the Security Group's name instead of its ID.
Based on the documentation, it would appear that SecurityGroups can only be attached to a load balancer in a VPC.
If it were me, I would do one of two things (or both):
Post my question to the official AWS CloudFormation forum to try and get it answered from someone on that team.
Use the AWS Console, SDKs or CLI tools to set up your environment the way you want, then use the CloudFormer tool to generate a CloudFormation template that matches your environment.
I had the same issue - but decided that since this is just for initial set-up, the elastic load balancer can be managed seperate from cloud formation until they work this bug out. Also, cloud former is not totally reliable especially when you have security groups referencing other security groups.
I was hopeful that a GetAttr function call for the ID would work, but Security Groups are not in the supported list of GetAttr:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html
Try this for your SecurityGroup:
"ELBSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Security group for the Arena dev stack",
"SecurityGroupIngress" : [ {
"IpProtocol" : "tcp",
"FromPort" : 80,
"ToPort" : 80,
"SourceSecurityGroupOwnerId" : { "Fn::GetAtt" : [ "ProjectLoadBalancerTest", "SourceSecurityGroup.OwnerAlias" ] },
"SourceSecurityGroupName" : { "Fn::GetAtt" : [ "ProjectLoadBalancerTest", "SourceSecurityGroup.GroupName" ] }
} ]
}
},
and remove "SecurityGroups" from your ELB definition. This will allow traffic from your ELB to your app servers.
It looks like you're also looking to limit ELB traffic to your office IP. That would be better done with something like apache directives.