AWS HTTP API Gateway with custom link to private ALB - amazon-web-services

Question

I have the following setup
Custom Domain api.foo.co.uk-> API Mapping to stage v1 -> HTTP API path ANY /{proxy+} -> Private VPC Link -> ALB Fargate
If I hit the Custom Domain api.foo.co.uk I get a 503 "message": "Service Unavailable"
If I hit the API direct p3dqjsdfszlv7.execute-api.eu-west-1.amazonaws.com/v1/ i get the same
In the CW for the API i see the following:
{
"auth_status":"-",
"aws_endpoint":"-",
"cognito_auth_provider":"-",
"cognito_auth_type":"-",
"cognito_identity_id":"-",
"cognito_identity_pool_id":"-",
"domain_name":"api.foo.co.uk",
"domain_prefix":"api",
"err_msg":"Service Unavailable",
"err_response":"INTEGRATION_NETWORK_FAILURE",
"err_string":" "Service Unavailable"",
"http_method":"GET",
"integration_error":"-",
"integration_error_msg":"-",
"integration_int_status":"200",
"integration_status":"-",
"path":"/v1/",
"principa_ord_id":"-",
"protocol":"HTTP/1.1",
"request_id":"cSJJ2h7BjoEEJ-g=",
"route_key":"ANY /{proxy+}",
"source_ip":"22.22.103.68",
"stage":"v1",
"status":"503",
"time":"16/Mar/2021:14:08:24 +0000",
"user":"-",
"user_agent":"insomnia/2021.1.0"}
Anyone have any idea what the issue might be ? I have looked and can not find anything on the error message err_response":"INTEGRATION_NETWORK_FAILURE" or what the cause might be.
I have also enabled the access logs on the ALB but they are blank so I am assuming its not getting as far as the ALB

Answer 1

I had the same problem of INTEGRATION_NETWORK_FAILURE. I managed to find a more informative error message by including all of the $context.integration* variables in the access logs (see https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-logging-variables.html). In my case there was an error message that said "Request failed due to a network error communicating with the endpoint".
I can't tell what was the exact cause or what made it go away, but I can share some CloudFormation snippets of the final working setup:
ALB setup - listening for HTTP requests on port 80:
ApplicationLoadBalancerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Sub "${EnvironmentName} ALB security group"
GroupName: !Sub "${EnvironmentName}-load-balancer-sg"
VpcId:
'Fn::ImportValue': !Sub "${EnvironmentName}:VPC"
SecurityGroupIngress:
- CidrIp: "0.0.0.0/0"
IpProtocol: "tcp"
FromPort: 80
ToPort: 80
SecurityGroupEgress:
- CidrIp: "0.0.0.0/0"
IpProtocol: "-1"
SharedApplicationLoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: !Sub "${EnvironmentName}-shared-lb"
Scheme: "internal"
Type: "application"
Subnets:
- 'Fn::ImportValue': !Sub "${EnvironmentName}:${LBSubnetType}1"
- 'Fn::ImportValue': !Sub "${EnvironmentName}:${LBSubnetType}2"
SecurityGroups:
- !Ref ApplicationLoadBalancerSecurityGroup
IpAddressType: "ipv4"
SharedApplicationLoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
DependsOn:
- SharedApplicationLoadBalancer
Properties:
LoadBalancerArn: !Ref SharedApplicationLoadBalancer
Protocol: "HTTP"
Port: 80
DefaultActions:
- Type: fixed-response
FixedResponseConfig:
StatusCode: 404
MessageBody: Shared ALB has no such route
ContentType: text/plain
Per-service ALB target group:
SharedAlbServiceXTargetGroup:
Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
Properties:
Name: !Sub "${EnvironmentName}-alb-${ServiceName}-tg"
HealthCheckIntervalSeconds: 30
HealthCheckPath: "/ping"
HealthCheckProtocol: "HTTP"
HealthyThresholdCount: 2
Port: 8080
Protocol: "HTTP"
UnhealthyThresholdCount: 2
VpcId:
"Fn::ImportValue": !Sub "${EnvironmentName}:VPC"
TargetType: "ip"
SharedAlbServiceXListenerRule:
Type: "AWS::ElasticLoadBalancingV2::ListenerRule"
Properties:
Actions:
- Type: "forward"
TargetGroupArn: !Ref SharedAlbServiceXTargetGroup
Conditions:
- Field: "host-header"
HostHeaderConfig:
Values:
- !Ref HttpApiCustomDomain
ListenerArn: !Ref SharedApplicationLoadBalancerListener
Priority: !Ref SharedAlbListenerRulePriority
VPC link:
PrivateApiGatewayVpcLinkSecurityGroup:
Condition: PrivateAccess
Type: "AWS::EC2::SecurityGroup"
Properties:
VpcId:
'Fn::ImportValue': !Sub "${EnvironmentName}:VPC"
GroupName: !Sub "${EnvironmentName}-apigw-vpclink"
GroupDescription: !Sub "SG for API Gateway private VPC link in ${EnvironmentName} environment"
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 80
ToPort: 80
CidrIp: 0.0.0.0/0
PrivateApiGatewayVpcLink:
Condition: PrivateAccess
Type: "AWS::ApiGatewayV2::VpcLink"
Properties:
Name: !Sub "${EnvironmentName}-api-gateway-vpclink"
SecurityGroupIds:
- !Ref PrivateApiGatewayVpcLinkSecurityGroup
SubnetIds:
- "Fn::ImportValue": !Sub "${EnvironmentName}:PrivateSubnet1"
- "Fn::ImportValue": !Sub "${EnvironmentName}:PrivateSubnet2"
API Gateway Integration
HttpApiIntegration:
Type: "AWS::ApiGatewayV2::Integration"
Properties:
ApiId: !Ref HttpApi
Description: !Sub "Private ALB Integration for ${ServiceName} in ${EnvironmentName} env"
IntegrationType: "HTTP_PROXY"
IntegrationMethod: "ANY"
ConnectionType: "VPC_LINK"
ConnectionId: !Ref PrivateApiGatewayVpcLink
IntegrationUri: !Ref SharedApplicationLoadBalancerListener
PayloadFormatVersion: "1.0"