I want to create a simple rds proxy. I use the attached cloudformation template. However aws cannot create resource "AWS::RDS::DBProxyTargetGroup". The error info is not enough for debugging: "Resource timed out waiting for completion". Please anyone provide me anwser?
Target group was created, but info was not updated in cloudformation event
rds_proxy_console
CF_event
failed event
Resources:
RDSProxy:
Type: "AWS::RDS::DBProxy"
Properties:
Auth:
- AuthScheme: SECRETS
IAMAuth: DISABLED
SecretArn: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${SecretsManagerName}"
DBProxyName: !Ref ProxyName
EngineFamily: !Ref ProxyEngineFamily
RoleArn: !GetAtt SecretsManagerRole.Arn
VpcSecurityGroupIds: !Ref ProxyVpcSecurityGroupIds
VpcSubnetIds: !Ref ProxyVpcSubnetIds
RDSProxyTargetGroup:
Type: "AWS::RDS::DBProxyTargetGroup"
Properties:
DBClusterIdentifiers: !Ref ProxyTargetDBClusterIdentifiers
DBProxyName: !Ref RDSProxy
TargetGroupName: default
Related
I have a minimal stack for creating a simple service with a listener. The listener gets created first and succeeds. The service gets initiated next but gets stuck on "CREATE_IN_PROGRESS". Now I have seen this issue on SO but that has a clear reason for it failing. In my occasion the Cloudtrail logs simple show the initiation and 10 minutes later (custom timeout) the delete but nothing in between. The Cloudformation dashboard events also just show initiation and delete thereafter.
The service does not get created during this time either. This I visually checked by going over to the services and having other services there but not my own.
I have trimmed down the cloudformation template to the bare (i.e. only listener and service with reference to existing resources) but it still gets stuck.
Apart from the usual cloudtrail and cloudformation logs, what could I do to identify the problem?
[EDIT]
Here is the template I use. The parameters are based on my current setup.
AWSTemplateFormatVersion: "2010-09-09"
Description: "The Script to configure the RDS services."
Parameters:
ClusterNameARN:
Default: "arn:aws:ecs:eu-central-1:<NR_HERE>:cluster/AmsCluster"
Type: String
StaLBARN:
Default: "arn:aws:elasticloadbalancing:eu-central-1:<NR_HERE>:loadbalancer/app/StaPostgrestLoadBalancer/<ID_HERE>"
Type: String
StaTargetGroupARN:
Default: "arn:aws:elasticloadbalancing:eu-central-1:<NR_HERE>:targetgroup/LBTargetGroupSta/<ID_HERE>"
Type: String
LoadBalancerSG:
Type: 'AWS::EC2::SecurityGroup::Id'
LoadBalancerSubnet1:
Description: Subnet instance.
Type: 'AWS::EC2::Subnet::Id'
LoadBalancerSubnet2:
Description: Subnet region B instance.
Type: 'AWS::EC2::Subnet::Id'
LoadBalancerSubnet3:
Description: Subnet region for public.
Type: 'AWS::EC2::Subnet::Id'
StaTaskDefinitionARN:
Default: "arn:aws:ecs:eu-central-1:<NR_HERE>:task-definition/RDSPostgrestFamily:2"
Type: String
CertificateARN:
Default: "arn:aws:acm:eu-central-1:<NR_HERE>:certificate/<ID_HERE>"
Type: String
Resources:
LBListenerSta:
Type: 'AWS::ElasticLoadBalancingV2::Listener'
Properties:
Certificates:
- CertificateArn: !Ref CertificateARN
DefaultActions:
- Type: forward
TargetGroupArn: !Ref StaTargetGroupARN
LoadBalancerArn: !Ref StaLBARN
Port: 443
Protocol: HTTPS
StaService:
Type: 'AWS::ECS::Service'
Properties:
Cluster: !Ref ClusterNameARN
DesiredCount: 2
LaunchType: 'FARGATE'
LoadBalancers:
- ContainerName: 'Postgrest'
ContainerPort: 3000
TargetGroupArn: !Ref StaTargetGroupARN
NetworkConfiguration:
AwsvpcConfiguration:
SecurityGroups:
- !Ref LoadBalancerSG
Subnets:
- !Ref LoadBalancerSubnet1
- !Ref LoadBalancerSubnet2
- !Ref LoadBalancerSubnet3
ServiceName: StaPostgrestService
TaskDefinition: !Ref StaTaskDefinitionARN
DependsOn:
- LBListenerSta
Outputs:
StaServices:
Description: "The ARN of the service for the STA tasks."
Value: !Ref StaService
Based on the comments.
The issue is with the StaService ECS service. To get more information of possible reason why it fails, one can go to:
ECS Console -> Cluster -> Service -> Events
Based on this, the Events showed that the role used for ECS has incorrect permissions.
I am trying to create the glue security configuration using cloudformation script but I am getting the following error:
Property validation failure: [Value of property {/EncryptionConfiguration/S3Encryptions} does not match type {Array}]
What is the right way to give the S3encryption?
AWSTemplateFormatVersion: 2010-09-09
Description: Script creates resources for GlueSecurityConfiguration
Resources:
GlueSecurityConfiguration:
Type: AWS::Glue::SecurityConfiguration
Properties:
EncryptionConfiguration:
S3Encryptions:
KmsKeyArn: !Ref KMSArn
S3EncryptionMode: SSE-KMS
JobBookmarksEncryption:
KmsKeyArn: !Ref KMSArn
CloudWatchEncryption:
KmsKeyArn: !Ref KMSArn
Name: !Sub '${SystemValue}-${SubSystemValue}'
I think it should be
- KmsKeyArn: !Ref KMSArn
S3EncryptionMode: SSE-KMS
since S3Encryptions expects an array.
What is working:
Using the serverless framework:
I have configured an AWS VPC
I have an Amazon Aurora database configured
for my VPC
I have an AWS API Gateway lambda that is configured for my
VPC
When I deploy my lambda, I am able to access it publicly via the AWS
generated URL: XXX.execute-api.us-east-1.amazonaws.com/prod/outages
In my Lambda,I run a very simple query that proves I can connect to
my database.
This all works fine.
What is NOT working:
I have registered a domain with AWS/Route 53 and added a cert (e.g. *.foo.com)
I use the serverless-domain-manager plugin to make my lambda available via my domain (e.g. api.foo.com/outages resolves to XXX.execute-api.us-east-1.amazonaws.com/prod/outages)
This works fine if my lambda is NOT configured for my VPC
But when my lambda IS configured for my VPC, the custom domain api.foo.com/outages does NOT resolve to XXX.execute-api.us-east-1.amazonaws.com/prod/outages
I other words: I can NOT access api.foo.com/outages publicly.
What I need is:
1 - XXX.execute-api.us-east-1.amazonaws.com/prod/outages is available publicly (this works)
2 - My custom domain, api.foo.com/outages points to the SAME lambda as XXX.execute-api.us-east-1.amazonaws.com/prod/outages (in my VPC) and is available publicly (not working. I get: {"message":"Forbidden"})
virtual-private-cloud.yml
service: virtual-private-cloud
provider:
name: aws
region: us-east-1
stage: ${opt:stage, dev}
custom:
appVersion: 0.0.0
VPC_CIDR: 10
resources:
Resources:
ServerlessVPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: ${self:custom.VPC_CIDR}.0.0.0/16
EnableDnsSupport: true
EnableDnsHostnames: true
InstanceTenancy: default
ServerlessSubnetA:
DependsOn: ServerlessVPC
Type: AWS::EC2::Subnet
Properties:
VpcId:
Ref: ServerlessVPC
AvailabilityZone: ${self:provider.region}a
CidrBlock: ${self:custom.VPC_CIDR}.0.0.0/24
ServerlessSubnetB:
DependsOn: ServerlessVPC
Type: AWS::EC2::Subnet
Properties:
VpcId:
Ref: ServerlessVPC
AvailabilityZone: ${self:provider.region}b
CidrBlock: ${self:custom.VPC_CIDR}.0.1.0/24
ServerlessSubnetC:
DependsOn: ServerlessVPC
Type: AWS::EC2::Subnet
Properties:
VpcId:
Ref: ServerlessVPC
AvailabilityZone: ${self:provider.region}c
CidrBlock: ${self:custom.VPC_CIDR}.0.2.0/24
Outputs:
VPCDefaultSecurityGroup:
Value:
Fn::GetAtt:
- ServerlessVPC
- DefaultSecurityGroup
Export:
Name: VPCDefaultSecurityGroup-${self:provider.stage}
SubnetA:
Description: 'Subnet A.'
Value: !Ref ServerlessSubnetA
Export:
Name: vpc-subnet-A-${self:provider.stage}
SubnetB:
Description: 'Subnet B.'
Value: !Ref ServerlessSubnetB
Export:
Name: vpc-subnet-B-${self:provider.stage}
SubnetC:
Description: 'Subnet C.'
Value: !Ref ServerlessSubnetC
Export:
Name: vpc-subnet-C-${self:provider.stage}
database-service.yml
service: database-service
provider:
name: aws
region: us-east-1
stage: ${opt:stage, dev}
environment:
stage: ${opt:stage, dev}
plugins:
- serverless-plugin-ifelse
custom:
appVersion: 0.0.1
AURORA:
DB_NAME: database${self:provider.stage}
USERNAME: ${ssm:/my-db-username~true}
PASSWORD: ${ssm:/my-db-password~true}
HOST:
Fn::GetAtt: [AuroraRDSCluster, Endpoint.Address]
PORT:
Fn::GetAtt: [AuroraRDSCluster, Endpoint.Port]
serverlessIfElse:
- If: '"${opt:stage}" == "prod"'
Set:
resources.Resources.AuroraRDSCluster.Properties.EngineMode: provisioned
ElseSet:
resources.Resources.AuroraRDSCluster.Properties.EngineMode: serverless
resources.Resources.AuroraRDSCluster.Properties.ScalingConfiguration.MinCapacity: 1
resources.Resources.AuroraRDSCluster.Properties.ScalingConfiguration.MaxCapacity: 4
ElseExclude:
- resources.Resources.AuroraRDSInstanceParameter
- resources.Resources.AuroraRDSInstance
resources:
Resources:
AuroraSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: "Aurora Subnet Group"
SubnetIds:
- 'Fn::ImportValue': vpc-subnet-A-${self:provider.stage}
- 'Fn::ImportValue': vpc-subnet-B-${self:provider.stage}
- 'Fn::ImportValue': vpc-subnet-C-${self:provider.stage}
AuroraRDSClusterParameter:
Type: AWS::RDS::DBClusterParameterGroup
Properties:
Description: Parameter group for the Serverless Aurora RDS DB.
Family: aurora5.6
Parameters:
character_set_database: "utf32"
AuroraRDSCluster:
Type: "AWS::RDS::DBCluster"
Properties:
MasterUsername: ${self:custom.AURORA.USERNAME}
MasterUserPassword: ${self:custom.AURORA.PASSWORD}
DBSubnetGroupName:
Ref: AuroraSubnetGroup
Engine: aurora
EngineVersion: "5.6.10a"
DatabaseName: ${self:custom.AURORA.DB_NAME}
BackupRetentionPeriod: 3
DBClusterParameterGroupName:
Ref: AuroraRDSClusterParameter
VpcSecurityGroupIds:
- 'Fn::ImportValue': VPCDefaultSecurityGroup-${self:provider.stage}
# this is needed for non-serverless mode
AuroraRDSInstanceParameter:
Type: AWS::RDS::DBParameterGroup
Properties:
Description: Parameter group for the Serverless Aurora RDS DB.
Family: aurora5.6
Parameters:
sql_mode: IGNORE_SPACE
max_connections: 100
wait_timeout: 900
interactive_timeout: 900
# this is needed for non-serverless mode
AuroraRDSInstance:
Type: "AWS::RDS::DBInstance"
Properties:
DBInstanceClass: db.t2.small
DBSubnetGroupName:
Ref: AuroraSubnetGroup
Engine: aurora
EngineVersion: "5.6.10a"
PubliclyAccessible: false
DBParameterGroupName:
Ref: AuroraRDSInstanceParameter
DBClusterIdentifier:
Ref: AuroraRDSCluster
Outputs:
DatabaseName:
Description: 'Database name.'
Value: ${self:custom.AURORA.DB_NAME}
Export:
Name: DatabaseName-${self:provider.stage}
DatabaseHost:
Description: 'Database host.'
Value: ${self:custom.AURORA.HOST}
Export:
Name: DatabaseHost-${self:provider.stage}
DatabasePort:
Description: 'Database port.'
Value: ${self:custom.AURORA.PORT}
Export:
Name: DatabasePort-${self:provider.stage}
outage-service.yml
service: outage-service
package:
individually: true
plugins:
- serverless-bundle
- serverless-plugin-ifelse
- serverless-domain-manager
custom:
appVersion: 0.0.12
stage: ${opt:stage}
domains:
prod: api.foo.com
test: test-api.foo.com
dev: dev-api.foo.com
customDomain:
domainName: ${self:custom.domains.${opt:stage}}
stage: ${opt:stage}
basePath: outages
custom.customDomain.certificateName: "*.foo.com"
custom.customDomain.certificateArn: 'arn:aws:acm:us-east-1:395671985612:certificate/XXXX'
createRoute53Record: true
serverlessIfElse:
- If: '"${opt:stage}" == "prod"'
Set:
custom.customDomain.enabled: true
ElseSet:
custom.customDomain.enabled: false
provider:
name: aws
runtime: nodejs12.x
stage: ${opt:stage}
region: us-east-1
environment:
databaseName: !ImportValue DatabaseName-${self:provider.stage}
databaseUsername: ${ssm:/my-db-username~true}
databasePassword: ${ssm:/my-db-password~true}
databaseHost: !ImportValue DatabaseHost-${self:provider.stage}
databasePort: !ImportValue DatabasePort-${self:provider.stage}
functions:
hello:
memorySize: 2048
timeout: 30
handler: functions/handler.hello
vpc:
securityGroupIds:
- 'Fn::ImportValue': VPCDefaultSecurityGroup-${self:provider.stage}
subnetIds:
- 'Fn::ImportValue': vpc-subnet-A-${self:provider.stage}
- 'Fn::ImportValue': vpc-subnet-B-${self:provider.stage}
- 'Fn::ImportValue': vpc-subnet-C-${self:provider.stage}
environment:
functionName: getTowns
events:
- http:
path: outage
method: get
cors:
origin: '*'
headers:
- Content-Type
- authorization
resources:
- Outputs:
ApiGatewayRestApiId:
Value:
Ref: ApiGatewayRestApi
Export:
Name: ${self:custom.stage}-ApiGatewayRestApiId
ApiGatewayRestApiRootResourceId:
Value:
Fn::GetAtt:
- ApiGatewayRestApi
- RootResourceId
Export:
Name: ${self:custom.stage}-ApiGatewayRestApiRootResourceId
I believe you might have missed to add VPC endpoints for API Gateway.
Ensure you create a VPC interface endpoint for API GW and use it in the API you create. This will allow API requests to lambda running within VPC
References:
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-apis.html#apigateway-private-api-create-interface-vpc-endpoint
https://aws.amazon.com/blogs/compute/introducing-amazon-api-gateway-private-endpoints/
Hope this helps !!
SOLUTION: We found a solution, in our case the solution is that you can't use wildcard certificates. So if you make a certificate for the exact API URL it does work. Obviously this is not a nice solution for everyone and everything but when you do have this problem and must deploy you can make it work this way.
Also maybe good to mention we did not have any additional problems with the below mentioned case:
I also wanted to add a post I found that might also have lead to this setup not working even if we do get the network to play nice. But again if you or anyone has a working setup like the above mentioned one let me know how you did it.
TLDR
If anyone wants to understand what was going on with API Gateway, take
a look at this thread.
It basically says that API Gateway processes regular URLs (like
aaaaaaaaaaaa.execute-api.us-east-1.amazonaws.com) differently than how
it processes Custom Domain Name URLs (like api.myservice.com). So when
API Gateway forwards your API request to your Lambda Function, your
Lambda Function will receive different path values, depending on which
type of your URL you used to invoke your API.
source:
Understanding AWS API Gateway Custom Domain Names
I am starting an ECS task with Fargate and the container ends up in a STOPPED state after being in PENDING for a few minutes. The Status gives the following error message:
CannotPullContainerError: context canceled
I am using PrivateLink to allow the ECS host to talk to the ECR registry without having to go via the public Internet and this is how it is configured (Serverless syntax augmenting CloudFormation):
Properties:
PrivateDnsEnabled: true
ServiceName: com.amazonaws.ap-southeast-2.ecr.dkr
SubnetIds:
- { Ref: frontendSubnet1 }
- { Ref: frontendSubnet2 }
VpcEndpointType: Interface
VpcId: { Ref: frontendVpc }
Any ideas as to what is causing the error?
did you also add an S3 endpoint?
Here is a working snippet of my template, I was able to solve the issue with the aws support:
EcrDkrEndpoint:
Type: 'AWS::EC2::VPCEndpoint'
Properties:
PrivateDnsEnabled: true
SecurityGroupIds: [!Ref 'FargateContainerSecurityGroup']
ServiceName: !Sub 'com.amazonaws.${AWS::Region}.ecr.dkr'
SubnetIds: [!Ref 'PrivateSubnetOne', !Ref 'PrivateSubnetTwo']
VpcEndpointType: Interface
VpcId: !Ref 'VPC'
For S3 you need to know that a route table is necessary - normally you would like to use the same as for the internet gateway, containing the route 0.0.0.0/0
S3Endpoint:
Type: 'AWS::EC2::VPCEndpoint'
Properties:
ServiceName: !Sub 'com.amazonaws.${AWS::Region}.s3'
VpcEndpointType: Gateway
VpcId: !Ref 'VPC'
RouteTableIds: [!Ref 'PrivateRouteTable']
Without an endpoint for cloudwatch you will get another failure, it is necessary too:
CloudWatchEndpoint:
Type: 'AWS::EC2::VPCEndpoint'
Properties:
PrivateDnsEnabled: true
SecurityGroupIds: [!Ref 'FargateContainerSecurityGroup']
ServiceName: !Sub 'com.amazonaws.${AWS::Region}.logs'
SubnetIds: [!Ref 'PrivateSubnetOne', !Ref 'PrivateSubnetTwo']
VpcEndpointType: Interface
VpcId: !Ref 'VPC'
EDIT: private route table:
PrivateRoute:
Type: AWS::EC2::Route
DependsOn: InternetGatewayAttachement
Properties:
RouteTableId: !Ref 'PublicRouteTable'
DestinationCidrBlock: '0.0.0.0/0'
GatewayId: !Ref 'InternetGateway'
I found I needed not only vpc endpoints for s3, aws logs and the two ecr endpoints as detailed in #graphik_ 's answer but I also needed to ensure that the security groups on the endpoints allowed ingress access to HTTPS from the security group on the Farscape containers.
The security group on the Farscape containers need egress access via HTTPS to the vpce endpoint security group and also to the pl-7ba54012 IP group which is s3.
This and the route to pl-7ba54012 in the route table seems to be the whole picture.
There are Policies on the vpce too, which I left as the default "All Access" but you could harden these up to only allow access from the Role running the Fargate containers.
Trying to create a ECS Service (on Fargate) with cloudformation but got error:
Invalid service in ARN (Service: AmazonECS; Status Code: 400; Error
Code: InvalidParameterException; Request ID: xxx).
According to error message seems some ARN is wrong, but I didn't find the reason, I checked ARN of IAM roles and its ok. The other ARN are passed with !Ref function (so not a typo error)
All Resources (including from all others nested templates, vpc, cluster, alb etc) are created, except the "Service" resouce (the ECS service).
Below is the template used (nested template). All parameters are ok (passed from root template). Parameters TaskExecutionRole and ServiceRole are ARNs from IAM roles created by ECS wizard:
Description: >
Deploys xxx ECS service, with load balancer listener rule,
target group, task definition, service definition and auto scaling
Parameters:
EnvironmentName:
Description: An environment name that will be prefixed to resource names
Type: String
EnvironmentType:
Description: See master template
Type: String
VpcId:
Type: String
PublicSubnet1:
Type: String
PublicSubnet2:
Type: String
ALBListener:
Description: ALB listener
Type: String
Cluster:
Description: ECS Cluster
Type: String
TaskExecutionRole:
Description: See master template
Type: String
ServiceRole:
Description: See master template
Type: String
ServiceName:
Description: Service name (used as a variable)
Type: String
Default: xxx
Cpu:
Description: Task size (CPU)
Type: String
Memory:
Description: Task size (memory)
Type: String
Conditions:
HasHttps: !Equals [!Ref EnvironmentType, production]
HasNotHttps: !Not [!Equals [!Ref EnvironmentType, production]]
Resources:
ServiceTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub '${EnvironmentName}-${ServiceName}'
VpcId: !Ref VpcId
TargetType: ip
Port: 80
Protocol: HTTP
AlbListenerRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref ServiceTargetGroup
Conditions:
- Field: host-header
Values: [www.mydomain.com] # test
ListenerArn: !Ref ALBListener
Priority: 1
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: !Sub '${EnvironmentName}-${ServiceName}-Task'
ContainerDefinitions:
- Name: !Ref ServiceName
Image: nginx
PortMappings:
- ContainerPort: 80
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref EnvironmentName
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: !Ref ServiceName
NetworkMode: awsvpc
RequiresCompatibilities: [FARGATE]
Cpu: !Ref Cpu
Memory: !Ref Memory
ExecutionRoleArn: !Ref TaskExecutionRole
Service:
Type: AWS::ECS::Service
DependsOn: TaskDefinition
Properties:
Cluster: !Ref Cluster
ServiceName: !Ref ServiceName
TaskDefinition: !Ref TaskDefinition
LaunchType: FARGATE
DesiredCount: 1
LoadBalancers:
- ContainerName: !Ref ServiceName
ContainerPort: 80
TargetGroupArn: !Ref ServiceTargetGroup
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets:
- !Ref PublicSubnet1
- !Ref PublicSubnet2
Role: !Ref ServiceRole
I lost a few hours in this and could not solve it, I reviewed a lot in the documentation but nothing, if someone knows how to help.
Thanks!
The error message is confusing because it does not explain which parameter is wrong. Amazon API expects resource ARNs in several parameters including Cluster, TaskDefinition and TargetGroup. The error happens when one of these parameters are wrong. Please check carefully these parameters and make sure they are valid ARNs.
I had exactly the same error and in my case I made a mistake and provided wrong Cluster value.
And I am posting an answer here because this was the first search result for this error message and it had no answer.
The problem for me was that the default AWS region was set to the wrong one. To fix that, run the following command (using the correct region).
$ aws configure set default.region us-west-2