We have 2 environments Dev and Prod both are in different VPC.
Also, we are using MSK Managed Kafka with EKS Fargate Cluster(no nodes) successfully running.
Now we moved from Managed Kafka to MSK serverless environment.
We attached both VPCs and the same Subnet and SG what we used in order to connect MSK Managed Kafka to EKS Cluster.
We are able to connect EC2 instance to MSK serverless from both VPCs,
but we try to connect any EKS cluster we are facing timeout issue or not able to connect with MSK even though same MSK Managed configurations we are using.
How to connect MSK Serverless to EKS Fargate Cluster.
Related
I have hosted my K8s cluster on AWS EC2. I want to use AWS EBS Self-provisioning into my k8s cluster for PVC.so I came across EBS-CSI but I can't find the document that can help me to install into my self hosted cluster.
https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html
I have AWS EKS cluster with Fargate profile.
The cluster is in private subnets, and in VPC, which is not public.
I can use AWS cli from local machine, and I'd like to connect to Fargate instance from local machine.
I have a use case to replicate Kafka topics and data from one Kafka cluster in AWS to Confluent Kafka deployed on AWS. The issue is, my Kafka in AWS is deployed in a VPC, say VPC-1 that dosent allow for VPC peering with confluent cloud. I need to use a Load balancer/Proxy service deployed in another VPC say VPC-2 that is peered with VPC-1 and Confluent Kafka cluster VPC (VPC-3). Also, VPC-3 is peered with VPC-1. What would be a idea Load balancer setup to point to AWS VPC-1 so that that data could be then replicated to Confluent cloud kafka? Below is how the VPC's are peered and I need something to run on VPC-2 and forward data from Confluent Kafka to Kafka in VPC-1. The data in source Kafka is few hundred GB per day.
I did not quite understand the configuring of VPC "CIDR block" while creating fargate cluster. Based on the link https://www.datadoghq.com/blog/aws-fargate-metrics/, there is a fleet that runs outside my VPC that has the infrastructure to run my fargate tasks.
What I dont understand if I configure a dedicated VPC for my fargate cluster. How does it connect with dedicated AWS managed infrastructure for fargate.
I did not find any documentation with some explaination.
After googling for sometime, found this https://thenewstack.io/aws-fargate-through-the-lens-of-kubernetes/
The author states the VPC configured during fargate cluster creation acts as proxy and requests are fwded to EC2 instance running in VPC owned/managed by AWS. Configuring VPC serves the purpose of controlling the IP range of ENI attached to containers. This is based on my observation, need something more to back it up.
Does ECS with EC2 machines can use service discovery?
The AWS documentation says all about Fargate,
have anyone done it with ALB, EC2 ECS cluster, service discovery.
ECS uses AWS Cloud Map for service discovery. You can use the AWS Cloud Map for other services as well:
You can also use AWS Cloud Map to register and locate any cloud resources, such as Amazon EC2 instances, Amazon DynamoDB tables, Amazon S3 buckets, Amazon Simple Queue Service (Amazon SQS) queues, or APIs deployed on top of Amazon API Gateway, among others.