I am using a Lambda function which uses boto library to perform actions over EMR such as describe-cluster. I would like to know which ports are being used to restrict outbound Lambda security group for this communication. I do not find this information in AWS documentation.
All AWS endpoints are HTTPS.
See AWS Regions and Endpoints - Amazon EMR
Related
I have jenkins agents on aws (all in a particular vpc) running terraform but it needs to configure SQS queues with resource policies that only allow access based on particular fixed ip & vpc endpoints ... I've tried allowing the vpc for the jenkins slaves using aws:SourceVpc but they don't seem to use the vpc to configure aws, rather ,I think they use the wider internet to do the config.
How would I go about achieving this other than say adding an allow for the jenkins principal on the resources?
Some clarification - yes there is a VPCE configured for SQS which currently work for connections coming from a peering connection via private dns allowed using aws:SourceVpce. But jenkins/terraform runs on a VPC that obviously isn't using the VPCE to configure SQS because it fails with a 403, do i have to somehow tell terraform to configure via vpce for the sqs resources? or something else?
I have a Python application deployed on EKS (Elastic Kubernetes Service). This application saves large files inside an S3 bucket using the AWS SDK for Python (boto3). Both the EKS cluster and the S3 bucket are in the same region.
My question is, how is communication between the two services (EKS and S3) handled by default?
Do both services communicate directly and internally through the Amazon network, or do they communicate externally via the Internet?
If they communicate via the internet, is there a step by step guide on how to establish a direct internal connection between both services?
how is communication between the two services (EKS and S3) handled by default?
By default the network topology of your EKS offers route to the public AWS S3 endpoints.
Do both services communicate directly and internally through the Amazon network, or do they communicate externally via the Internet?
Your cluster needs to have network access to the said public AWS S3 endpoints. Example, worker nodes running in public subnet or the use of NAT gateway in private subnet.
...is there a step by step guide on how to establish a direct internal connection between both services?
You create VPC endpoints for S3 in the VPC that your EKS runs to ensure network communication with S3 stay within AWS network. VPC endpoints for S3 support both interface and gateway type. Try this article to learn about the basic of S3 endpoints, you can use the same method to create endpoints in the VPC where your EKS runs. Request to S3 from your pods will then use the endpoint to reach out to S3 within AWS network.
You can add S3 access to your EKS node IAM role, this link shows you how to add ECR registry access to EKS node IAM role, but it is the same for S3.
The other way is to make environment variables available in your container, see this link, though I would recommend the first way.
Google failed me again or may be I wasnt too clear in my question.
Is there an easy way or rather how do we determine what services are VPC bound and what services are non-vpc ?
For example - EC2, RDS require a VPC setup
Lambda, S3 are publicly available services and doesn't need a VPC setup.
The basic services that require an Amazon VPC are all related to Amazon EC2 instances, such as:
Amazon RDS
Amazon EMR
Amazon Redshift
Amazon Elasticsearch
AWS Elastic Beanstalk
etc
These resources run "on top" of Amazon EC2 and therefore connect to a VPC.
There are also other services that use a VPC, but you would only use them if you are using some of the above services, such as:
Elastic Load Balancer
NAT Gateway
So, if you wish to run "completely non-vpc", then avoid services that are "deployed". It means you would use AWS Lambda for compute, probably DynamoDB for database, Amazon S3 for object storage, etc. This is otherwise referred to as going "serverless".
I have a lambda in a vpc in an aws account and I have another lambda in a vpc in another aws account. How to make them be able to communicate with each other ?
You can use the appropriate AWS SDK to invoke a Lambda (even if it is in a VPC). The Lambda making the call though, will need internet access (NAT Gateway).
The better way to do it (IMO) would be to link them via SNS. So here are some relevant links:
Using Amazon SNS for System-to-System Messaging with an AWS Lambda Function as a Subscriber
Invoke Lambda using SNS from Outside Account
You can do VPC peering from one AWS account to another AWS account, setup ACL's and have the Lambda's communicate across the AWS accounts.
You can peer the vpcs (keep in mind IP address ranges cannot overlap if you do this) or expose the lambdas using api gateway and get them to invoke each other using http requests.
Is there a way to use AWS CLI to call different services such as SQS, EC2, SNS from EC2 linux instance?
The EC2 instance from where the AWS CLI command are invoked does not have access to internet. It is in private subnet. It is not using internet gateway or NAT.
Thanks,
Not possible. The CLI has to access the API endpoints for all the services you mentioned. For that the CLI needs internet access. Only service it can access without internet is the internal metadata server.
AWS Regions and Endpoints
VPC endpoints create a private connection between your VPC and an AWS service. However, currently the only supported service is S3 and none of the services listed in your question.
Currently, we support endpoints for connections with Amazon S3 only.
We'll add support for other AWS services later. Endpoints are
supported within the same region only.