Am I correct in assuming that it is NOT possible to put AWS managed elasticsearch (opensearch) - Kibana - behind ALB ? I would like to configure ALB so it authenticates with OKTA SSO oidc before redirecting request to Kibana (AWS managed elasticsearch).
What are the alternatives ? I see some people mentioning using lambda as proxy - putting lambda behind ALB, and then let lambda redirect request to the elasticsearch. I dont know how this could be done - did anyone had similar experiences before ? Any recommended reading regarding that?
Thank you
It is possible to configure an application load balancer to authenticate the user using Cognito service and forward requests to any application available in private subnets.
You need to create a listener rule with 2 actions:
authenticate-cognito action to redirect user to your SSO provider login page (a cognito user pool must be configured);
forward action to your target group with the application.
See an example of terraform aws_lb_listener_rule definition:
resource "aws_lb_listener_rule" "listener_rule" {
listener_arn = your_alb_listener_443_arn
action {
type = "authenticate-cognito"
authenticate_cognito {
user_pool_arn = your_cognito_user_pool.cognito_user_pool.arn
user_pool_client_id = your_user_pool_client_id
user_pool_domain = your_cognito_user_pool_domain
}
}
action {
type = "forward"
target_group_arn = your_lb_target_group_arn
}
condition {
host_header {
values = [
"your_domain" # resolves ALB endpoint
]
}
}
lifecycle {
create_before_destroy = true
}
}
As Patrick and Leo mentioned in comments, AWS OpenSearch provides fine-grained access control and has embedded SSO authentication mechanisms that lets you use your existing identity provider:
SAML authentication
AWS Cognito authentication
It works very well if your cluster is publicly available
However, documentation does not bring the light how it works when a cluster provisioned in VPC, in private subnets).
Refer to this question.
I have the following
let lambdaFunction = <some lambda construct here>
lambdaFunction.grantInvoke(props.invocationRole);
And that grants hydra the ability to invoke that lambda, but what if I had an api endpoint method?
let endpoint = resource.addMethod(method['method'], lambdaIntegration, {
authorizationType: AuthorizationType.IAM,
requestValidator: props.requestValidator,
requestModels: {'application/json': props.validationModel}
})
Is it possible to grant the role that I have access to just this endpoint access or do I have to do it on the api gateway resource level?
I have no experience with API Gateway, however it looks like it does support access control using the specificity of a particular endpoint.
Reference Docs:
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-control-access-using-iam-policies-to-invoke-api.html
Using the CDK, you can apply a policy with the same structure as the examples in the docs.
I am trying to run Terraform in a private VPC using the AWS provider. Terraform supports overriding various AWS endpoints and so I have been using VPC Endpoints to expose relevant services in my VPC and overriding those endpoints in my provider configuration.
Unfortunately IAM is not supported by VPC Endpoints - I need this API to create some resource for example AWS roles, so how can I use this API in my private VPC from Terraform?
$ terraform -v
Terraform v0.11.13
+ provider.aws v2.17.0
Cloudformation supports VPC endpoints. So you can make a Cloudformation template that creates the IAM resources. Then apply that cloudformation stack via terraform. Then do another Terraform module that refers to those resources as data sources.
Given the constraints of your problem, it is only possible to solve this indirectly.
Docs: Terraform Docs for Cloudformation
Docs for AWS Cloudformation VPC Endpoints
I have a Lambda function in my VPC, and I want to access S3 bucket.
I have setup the S3 VPC endpoint correctly, I think, because I created an EC2 instance in the same subnet and security group as the Lambda function. When I ran a copy of Lambda function code on the EC2 instance, it can correctly showed the S3 file content.
But when I run the code in Lambda, it failed. So, I want to know what is the difference between "run in EC2" and "run in Lambda"? Why did it fail when I ran it in Lambda?
Here is my Lambda function code:
import boto3
s3 = boto3.client('s3', region_name='ap-northeast-1')
def lambda_handler(event, context):
bucket = '*xxxxxx*'
key = 's3-upload.json'
try:
response = s3.get_object(Bucket=bucket, Key=key)
print('--------------------------------------')
print(response)
print('--------------------------------------')
body = response['Body'].read()
print(body)
print('--------------------------------------')
print("CONTENT TYPE: " + response['ContentType'])
except Exception as e:
print('Error getting object.')
print(e)
raise e
If you want to allow an AWS Lambda to access Amazon S3, use one of these methods:
Do not associate the function to a VPC. Access is then automatic.
If the function is attached to a public subnet in the VPC, associate an Elastic IP to the Lambda function's ENI that appears in the VPC (Not recommended)
If the function is attached to a private subnet in the VPC, launch a NAT Gateway in the public subnet and update Route Tables. Traffic will flow to the Internet via the NAT Gateway.
Add an Amazon S3 VPC Endpoint in the VPC and update Route Tables. Traffic will flow through that instead of the Internet Gateway.
Even though they're in the same VPC, EC2 and Lambda are still different environments within AWS. Being able to run your code in one and not the other implies that your code is fine and works, so it's likely to be a configuration issue with AWS.
Have you checked the service/execution role that the lambda is using?
You need to ensure that the IAM role that it's using is allowed the correct level of S3 access.
This documentation on execution roles for lambda might provide a useful jumping off point: https://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html#lambda-intro-execution-role
An IAM policy like this would give whatever execution role you use read-only access to all your S3 buckets, and happens to be one of the AWS managed policies.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "*"
}
]
}
Thanks everyone! I found the reason.
My Lambda have two subnets, private_sn_1 and private_sn_2,
private_sn_1 have correctly set the vpc endpoint route table,
but the private_sn_2 set a wrong route table,
and my ec2 created in private_sn_1 so it can access the vpc endpoint.
In normal, Lambda will run randomly in private_sn_1 or private_sn_2,
but in my case it always run in private_sn_2(I don't know why),
so when I fixed the private_sn_2 route table,
everything is right.
In addition to all said above, it is also possible that VPC Endpoint policy can be prohibitive and not allowing traffic to/from S3 through. Make sure you allow traffic through endpoint by using "Full access" policy.
Edit: here's related bit of documentation:
Your policy must contain a Principal element. For gateway endpoints
only, you cannot limit the principal to a specific IAM role or user.
Specify "*" to grant access to all IAM roles and users. Additionally,
for gateway endpoints only, if you specify the principal in the format
"AWS":"AWS-account-ID" or "AWS":"arn:aws:iam::AWS-account-ID:root",
access is granted to the AWS account root user only, and not all IAM
users and roles for the account.
So for S3 endpoints to work you need to specify '*' as a principal in general case
I have a group of aws api gateways which I am trying to restrict access to them through only one internal lambda function. I created a role, added the AmazonAPIGatewayInvokeFullAccess policy to it and assigned this role to the lambda function. However, I am still not able to access the APIs from the lambda and getting 403s. Thanks for your help.
Under IAM you need to add trust relationship with apigateway.amazonaws.com