How to Create an API with VPC Link Integration for EKS? - amazon-web-services

I have a working EKS Cluster with some services running in there. The challenge right now, is to have an API gateways to call those services. That is why I started reading "Integrate Amazon API Gateway with Amazon EKS" explaining that this is possible but there are some steps to do in order to make this work. This is what I did but after some progress, I ran into an issue.
Here is one of the command-line statements I executed.
aws apigatewayv2 get-vpc-links --region $AGW_AWS_REGION
This results in the following output:
Apparently, the VPC link is available, a security group Id is available, two SubnetIds are available (in the documented example it is three but that should not matter so much, feel free to comment about this) and the status message says that the available VPC link is ready to route traffic.
There for I decided to continue with the next documented step:
Create an API with VPC Link integration
cat > apigw-api.yaml<<EOF
apiVersion: apigatewayv2.services.k8s.aws/v1alpha1
kind: API
metadata:
name: apitest-private-nlb
spec:
body: '{
"openapi": "3.0.1",
"info": {
"title": "ack-apigwv2-import-test-private-nlb",
"version": "v1"
},
"paths": {
"/\$default": {
"x-amazon-apigateway-any-method" : {
"isDefaultRoute" : true,
"x-amazon-apigateway-integration" : {
"payloadFormatVersion" : "1.0",
"connectionId" : "$(kubectl get vpclinks.apigatewayv2.services.k8s.aws \
nlb-internal \
-o jsonpath="{.status.vpcLinkID}")",
"type" : "http_proxy",
"httpMethod" : "GET",
"uri" : "$(aws elbv2 describe-listeners \
--load-balancer-arn $(aws elbv2 describe-load-balancers \
--region $AGW_AWS_REGION \
--query "LoadBalancers[?contains(DNSName, '$(kubectl get service authorservice \
-o jsonpath="{.status.loadBalancer.ingress[].hostname}")')].LoadBalancerArn" \
--output text) \
--region $AGW_AWS_REGION \
--query "Listeners[0].ListenerArn" \
--output text)",
"connectionType" : "VPC_LINK"
}
}
},
"/meta": {
"get": {
"x-amazon-apigateway-integration": {
"uri" : "$(aws elbv2 describe-listeners \
--load-balancer-arn $(aws elbv2 describe-load-balancers \
--region $AGW_AWS_REGION \
--query "LoadBalancers[?contains(DNSName, '$(kubectl get service echoserver \
-o jsonpath="{.status.loadBalancer.ingress[].hostname}")')].LoadBalancerArn" \
--output text) \
--region $AGW_AWS_REGION \
--query "Listeners[0].ListenerArn" \
--output text)",
"httpMethod": "GET",
"connectionId": "$(kubectl get vpclinks.apigatewayv2.services.k8s.aws \
nlb-internal \
-o jsonpath="{.status.vpcLinkID}")",
"type": "HTTP_PROXY",
"connectionType": "VPC_LINK",
"payloadFormatVersion": "1.0"
}
}
}
},
"components": {}
}'
EOF
Unfortunately, this resulted in the following errors:
I tried to isolate the problem but then I ran into another issue. In the command-line, I just executed a small fragment of what is shown above.
aws elbv2 describe-load-balancers --load-balancer-arn --region $AGW_AWS_REGION query "LoadBalancers[?contains(DNSName, '$(kubectl get service authorservice -o jsonpath="{.status.loadBalancer.ingress[].hostname}")')].LoadBalancerArn" --output text
but that resulted in another error:
An error occurred (ValidationError) when calling the
DescribeLoadBalancers operation: 'query' is not a valid load balancer
ARN
So how to get this working? All I want is to just to create an API with VPC link integration. But the documented way to do so does not work for me. Please let me know how I can fix this way or just do the same in a different way.

I am exactly on the same step - trying to create API for micro service on EKS following the same tutorial. I also started to break down the executions to the small chunks to check if things are still correct from the time the tutorial was written.
So taking the smallest piece which should be executable
kubectl get service authorservice -o jsonpath="{.status.loadBalancer.ingress[].hostname}"
You must have the "authorservice" already configured and running or it might be called something else for your particular case.
For me that lines give me back the needed hostname. If it is not giving you the hostname you better check with
kubectl get service <the_service_you_have> -o json
whether the service you have is properly configured and having at the end of the output something similar to
"status": {
"loadBalancer": {
"ingress": [
{
"hostname": "k8s-XXXXXXXXXXXXXXXXXXXXXX.elb.us-west-2.amazonaws.com"
}
]
}
My advice is to go over all the "kubectl" executions and check if they are working and giving back some meaningful results for your case and adjust if needed. At first it might be hard to get where the "kubectl" commands ends but later it gets easy.

Related

AWS StepFunctionsLocal StepFunctions test with definition substitutions

I have been looking into StepFunctionsLocal (SFL) to test. To get a project bootstrapped, I aws the SAM cli to generate a new project - which comes pre-packed with SFL tests and a make file to run everything.
However, it seems broken out of the box. When running the tests using directions in the README, I get this error:
InvalidDefinition: An error occurred (InvalidDefinition) when calling the CreateStateMachine operation: Invalid State Machine Definition: ''SCHEMA_VALIDATION_FAILED:
Value is not a valid resource ARN at /States/Check Stock Value/Resource','SCHEMA_VALIDATION_FAILED: Value is not a valid resource ARN at /States/Sell Stock/Resource', 'SCHEMA_VALIDATION_FAILED: Value is not a valid resource ARN at /States/Buy Stock/Resource', 'SCHEMA_VALIDATION_FAILED: Value is not a valid resource ARN at /States/Record Transaction/Resource''
And, indeed, the state machine definition is provided as a file that uses DefinitionSubstitutions:
{
"Comment": "A state machine that does mock stock trading.",
"StartAt": "Check Stock Value",
"States": {
"Check Stock Value": {
"Type": "Task",
"Resource": "${StockCheckerFunctionArn}", <--
"Retry": [
{
"ErrorEquals": [
"States.TaskFailed"
],
"IntervalSeconds": 15,
"MaxAttempts": 5,
"BackoffRate": 1.5
}
],
"Next": "Buy or Sell?"
},
...
The CloudFormation template injects those values
StockTradingStateMachine:
Type: AWS::Serverless::StateMachine # More info about State Machine Resource: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-statemachine.html
Properties:
DefinitionUri: statemachine/stock_trader.asl.json
DefinitionSubstitutions:
StockCheckerFunctionArn: !GetAtt StockCheckerFunction.Arn <--
the makefile commands to run the test
run:
docker run -p 8083:8083 -d \
--mount type=bind,readonly,source=$(ROOT_DIR)/statemachine/test/MockConfigFile.json,destination=/home/StepFunctionsLocal/MockConfigFile.json \
-e SFN_MOCK_CONFIG="/home/StepFunctionsLocal/MockConfigFile.json" \
amazon/aws-stepfunctions-local
create:
aws stepfunctions create-state-machine \
--endpoint-url http://localhost:8083 \
--definition file://statemachine/stock_trader.asl.json \
--name "StockTradingLocalTesting" \
--role-arn "arn:aws:iam::123456789012:role/DummyRole" \
--no-cli-pager \
--debug
happypathsellstocktest:
aws stepfunctions start-execution \
--endpoint http://localhost:8083 \
--name HappyPathSellStockTest \
--state-machine arn:aws:states:us-east-1:123456789012:stateMachine:StockTradingLocalTesting#HappyPathSellStockTest \
--no-cli-pager
It appears that nothing provides the definition substitutions. I've come up dry when combing through the AWS docs for how to provide those substitutions through the API, maybe I just don't know what to look for. Any clues?
I did make an issue to fix the template: https://github.com/aws/aws-sam-cli-app-templates/issues/342
Unfortunately, DefinitionSubstitutions is a feature of the CloudFormation resource and not supported directly in the Step Functions API. You would need to parse and replace the substitution variables in your own code before you call Create State Machine in your test.

How to change AWS API Gateway method integration type to HTTP_PROXY with CLI?

In my case, AWS API Gateway should proxy every request and response through as they are. The API was created by importing an OpenAPI definition. I know how to manually set the method integration in the AWS console to HTTP proxy and set the endpoint URI.
I want to get rid of manual work and create a PowerShell script that loops through every method and sets those settings. But I haven't found the right command to update the integration type and URI.
After reading AWS CLI reference I have tried this command:
update-integration --rest-api-id x0cm5muxxx --resource-id xvxxxx --http-method GET --integration-type HTTP_PROXY --profile user2
But it gives an error:
"Unknown options: --integration-type, HTTP_PROXY"
Another CLI command I have tried without success is update-method with --patch-operations.
After the changes, the method's integration should look like this example:
PS C:\WINDOWS\system32> aws apigateway get-integration --rest-api-id x0cm5muxxx --resource-id x1lxxx --http-method GET --profile user2
{
"type": "HTTP_PROXY",
"httpMethod": "GET",
"uri": "https://${stageVariables.Url}/api/v1/Productlist/CanOrder/{fnprodId}",
"connectionType": "INTERNET",
"requestParameters": {
"integration.request.path.fnprodId": "method.request.path.fnprodId"
},
"passthroughBehavior": "WHEN_NO_MATCH",
"timeoutInMillis": 29000,
"cacheNamespace": "01lxxx",
"cacheKeyParameters": [],
"integrationResponses": {
"200": {
"statusCode": "200"
}
}
}
The next problem will be how to loop through every single method in the API for this update.
I got help and correct command to change methods integration type is:
aws apigateway put-integration --rest-api-id $RestAPIid --resource-id $ResourceId --http-method GET --type HTTP_PROXY --integration-http-method GET --profile $Profile --uri $uri
If someone is trying to loop Rest API resources with Powershell it goes like this simplified..
$array = Aws apigateway get-resources --rest-api-id $RestAPIid --profile $MyProfile --query [items[*].id]
$array | sort
"Resources:"
$array
""
"Amount of resources:"
echo $array.Length
""
foreach ($arrayItem in $array) {
"$arrayItem = " + $arrayItem.length
}

Need arn values from aws configservice

I am running
aws configservice get-compliance-details-by-config-rule --config-rule-name required-tags --compliance-types NON_COMPLIANT
with output
{
"EvaluationResults": [
{
"EvaluationResultIdentifier": {
"EvaluationResultQualifier": {
"ConfigRuleName": "required-tags",
"ResourceType": "AWS::ACM::Certificate",
"ResourceId": "arn:aws:acm:us-east-1:***:certificate/d9863cca-9e7b-460b-b9f8-bee23e8fb607"
},
"OrderingTimestamp": "2022-08-10T12:46:18.247000+05:30"
},
"ComplianceType": "NON_COMPLIANT",
"ResultRecordedTime": "2022-08-10T13:12:00.037000+05:30",
"ConfigRuleInvokedTime": "2022-08-10T13:11:59.841000+05:30"
},
{
"EvaluationResultIdentifier": {
"EvaluationResultQualifier": {
"ConfigRuleName": "required-tags",
"ResourceType": "AWS::EC2::Instance",
"ResourceId": "i-069c8d8c72ae8db8c"
},
"OrderingTimestamp": "2022-08-10T12:46:18.784000+05:30"
},
"ComplianceType": "NON_COMPLIANT",
"ResultRecordedTime": "2022-08-10T13:11:54.648000+05:30",
"ConfigRuleInvokedTime": "2022-08-10T13:11:54.449000+05:30
I need arn names of all the resources under the rule
and need to run
aws tag-resources
--resource-arn-list <value>
--tags <value>
[--cli-input-json <value>]
[--generate-cli-skeleton <value>]
THe problem is the the aws configservice get-compliance-details-by-config-rule command doen't list the arn of the resources and i need arn of each resource to be tagged.
What can i do?
You can construct Amazon EC2 instances ARNs using aws ec2 describe-instances command line. Please note that I used jq for that so you will need it if you want to use this method.
You can aws ec2 describe-instances with a filter by instance-id. In this case the instance-id corresponds to the ResourceId in your output. It should return the all data regarding the specified instance.
Your command line should look like aws ec2 describe-instances --region eu-west-3 --instance-id i-abd123.
Then you can format the result using jq.
aws ec2 describe-instances --region us-east-1 --instance-id i-abc123 | jq -r '.Reservations[] | .OwnerId as $OwnerId | ( .Instances[] | { "ARN": "arn:aws:ec2:\(.Placement.AvailabilityZone[:-1]):\($OwnerId):instance/\(.InstanceId)"} )' | jq -s .
The output should look like this:
[
{
"ARN": "arn:aws:ec2:us-east-1:***:instance/i-abc123"
}
]

Create a TXT record in AWS by using the aws cli

When I execute this:
aws route53 list-hosted-zones
I have the following response:
{
"HostedZones": [
{
"Id": "/hostedzone/Z209RXXXXXXE2L",
"Name": "ultrasist.net.",
"CallerReference": "ECXXXX62-EXXA-8XX9-8XX5-8E52XXXXXX81",
"Config": {
"Comment": "Corporate zone",
"PrivateZone": false
},
"ResourceRecordSetCount": 46
}
]
}
So, I am pretty sure that I have the domain "ultrasist.net". Now, if I do this:
aws \
lightsail \
create-domain-entry \
--region 'us-east-1' \
--domain-name 'ultrasist.net' \
--domain-entry '{"name":"_test-txt.ultrasist.net","target":"\"xyz\"", "isAlias":false,"type":"TXT"}'
I got this response:
An error occurred (NotFoundException) when calling the CreateDomainEntry operation: The Domain does not exist: ultrasist.net
However, as you can see, the domain DOES exist. So, my question is pretty obvious: Why Do I get this message if the domain of course is there?

What is the --rest-api-id and --resource-id and where do I find them?

I want to run this command: https://docs.aws.amazon.com/cli/latest/reference/apigateway/test-invoke-method.html
It requires these two fields, but I can't find any docs and where these are:
aws apigateway test-invoke-method --rest-api-id 1234123412 --resource-id avl5sg8fw8 --http-method GET --path-with-query-string '/'
My API gateway endpoint looks like this:
https://abc123.execute-api.us-east-1.amazonaws.com/MyStage/
I only see on unique identifier there - but this command seems to require two IDs. Where do I find them in the API Gateway console?
Your rest-api-id is the identifier before 'execute-api' in your endpoint URL.
In your example URL:
https://abc123.execute-api.us-east-1.amazonaws.com/MyStage/
The rest-api-id is abc123
The resource ID can be obtained with the CLI using the get-resources call and the rest-api-id:
> aws apigateway get-resources --rest-api-id abc123
{
"items": [
{
"id": "xxxx1",
"parentId": "xxxx0",
"pathPart": "foo",
"path": "/foo",
"resourceMethods": {
"GET": {}
}
},
{
"id": "xxxx0",
"path": "/"
}
]}
Each one of the records in the items attribute is a resource, and its id attribute is a resource ID you can use in your test-invoke-method in conjunction with a method that is associated with the resource.
Both values are visible at the top of the console when you select one of your endpoints/resources:
Here is a portion of a bash script that you could use, assuming you have jq installed to parse the json and only one API. It gets the first API ID from the items array, then looks up that Resource ID, then invokes the API using POST:
API_ID=`aws apigateway get-rest-apis | jq -r ".items[0].id"`
RESOURCE_ID=`aws apigateway get-resources --rest-api-id $API_ID | jq -r ".items[].id"`
aws apigateway test-invoke-method --rest-api-id $API_ID --resource-id $RESOURCE_ID --http-method POST