I am trying to configure MSK connect in AWS and the below is the configuration.
INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
[Worker-02003b81ffe0ee9c3] [2022-06-02 14:26:40,955] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
[Worker-02003b81ffe0ee9c3] org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1654180000954, tries=1, nextAllowedTryMs=1654180001055) timed out at 1654180000955 after 1 attempt(s)
As per https://aws.amazon.com/premiumsupport/knowledge-center/msk-connector-connect-errors/ I have opened all traffic for the MSK connector to be able to reach the msk cluster, yet I notice timeout errors.
The connector and the cluster are both in same subnets and uses same security group ID. I ma able to telnet to the broker from a VM in the same subnet.
Note: I have plaintext enabled and no authentication. I have also given proper IAM permission and role attached. This is verified.
Adding the solution in case if it helps someone.
https://docs.aws.amazon.com/msk/latest/developerguide/mkc-tutorial-setup.html
I had to create a vpc endpoint as mentioned in the above doc and also associate the subnet route tables that my kafka uses.
Additionally also make sure your SG's have correct inbound and outbound rules
Related
I am a beginner at AWS services. I am getting this Error:
"ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr.us-east-1.amazonaws.com/: dial tcp 52..*.105:443: i/o timeout"
Some details that might help you help me:
Setting up "Run task" in 2 private subnets. (should it be one?)
In the "Run task" , I am using the same security group associated with my VPC endpoints
In the Service , I am using the same security group associated with my VPC endpoints
I am using a "Network Only" cluster.
I am using a task definition of type "Fargate"
For the task definition container, I am leaving port mappings blank
For Task execution role in task definition I have: ecsTaskExecutionRole
I am leaving "Mesh integration"/"proxy configuration"/" FireLens integration" unchecked
Auto-assign public IP: "Disabled"
My goal is to run an instance of my app in a private subnet and connect it to AWS resources through PrivateLink. I DO NOT intend on setting up an internet gateway/NAT device/AWS direct connection/VPN. I am almost sure I am missing something. Thorough explanations will be highly appreciated. Thank you.
I am new to AWS Services and I have angular dotnet core project deployed to AWS EB. I am also used Amazon RDS SQL Server Database and AWS Cognito for autherization.
My project was work fine, but after some times I got a message from Amazon Web Services that says
TCP and UDP traffic on port 3343, directionally in-bound and out-bound, need to be allowed in Network ACLs of your VPC as well as in the Security Group that is attached to your Amazon RDS for SQL Server Multi-AZ instance(s).
I enable TCP and UDP traffic on port 3343 in-bound and out-bound as
in-bound
out-bound
but It didn't work. please help me, thank you.
Hi I just found the right answer from aws
Hello,
Thank you for reaching out to AWS Premium Support.
From your case notes, I understand that you have received an advisory
email which recommends opening of port 3343 to avoid unexpected
failures in the WSFC service for RDS SQL Server Multi-AZ instances. In
relation to this, you would like to know if self-reference security
group as the source will work. Please correct me if I misunderstood.
To answer your query , yes Indeed, setting the source for the security
group rules as the security group itself , should do the trick here.
This would allow all resources associated with the security group to
communicate with other associated resources.
Further , If you have NOT changed any default rules in ACLs, then no
actions are needed for ACLs .
This means you can open the port 3343 to the same Security group (the sg has inbound role from itself on that port once for TCP and another for UDP)
and if your ACLs are default, no need to do anything else.
for the Security group outbound, you don't need to do anything because it already allows all traffic.
I'm looking for whitelisting Ip addresses to secure an internet facing transfer server via terraform but unfortunately terraform AWS provider still doesn't support adding new security group to vpc endpoint via terraform aws transfer resource.
I tried to update server using aws cli command but getting an error "An error occurred (InvalidRequestException) when calling the UpdateServer operation: Changing Security Group is not supported"
Any suggestion?
Assuming they are using EndpointType=VPC, the UpdateServer command does not support updating SecurityGroups.
Attaching a Security Group can either be done at server creation time using CreateServer or use EC2's ModifyVPCEndpoint API to update Security Group once the server has been created.
Refer to the documentation here: EndpointDetails - AWS Transfer Family (Under SecurityGroupIds)
(Console) In order to modify the Security Group of an AWS Transfer server once created do the following:
Go to the VPC service
Go to "Endpoints"
Click on the Endpoint that has "transfer" on the Server Name field
Click on the "Security Groups" Tab
Click on "Edit Security Groups"
Trying to Migrate RDS mysql to Redshift, When connecting the AWS RedShift Database in Target Connection it Throws the Error:
Test Endpoint failed: Application-Status: 1020912, Application-Message: IN/A, Application-Detailed-Message: N/A
Please help to resolve...
I have resolved my issue by adding few ingress/egress rules weren't defined on the security groups which are attached to DMS and Redshift.
Check this link for further information: AWS DMS endpoint connection to Redshift not working
For me, the private IP of redshift in the server name of the endpoint worked. I don't know why but it's working now.
AWS - Data Migration Service Endpoint Redshift error (Test failed)
The Leader private IP of redshift in the server name of the endpoint will work.
In your DMS (Data Migration Service) configuration where you enter the Server Name value, avoid the Redshift cluster DNS Name or the Public IP address. Instead, try using the private IP of the Redshift Leader node. It worked for me after I changed to the private IP of leader node.
Ive went through the whole start-up tutorial and connect to the tinkerpop3 server remotely from an EC2 that is in the same VPC and get the error
gremlin> g.addV('person').property(id, '1').property('name', 'marko')
Host did not respond in a timely fashion - check the server status and submit ag ain.
Type ':help' or ':h' for help.
Display stack trace? [yN]
any reason this might be happening?
Let's try a couple of things to get you started with debugging the issue here:
Have you tried hitting the /status endpoint? If this endpoint is working, then there is a problem with the console configuration. If it isn't, then there is an issue with the connectivity of the EC2 instance to the DB.
Can you ensure that the EC2 instance has been launched with the same security group for which you gave inbound access to port 8182 on the DB (during step#8 in the setting up instructions?
Please ensure that your cluster and instance status is "available" as observed from the Neptune console.
The recommended way to manage such connections is 2 have 2 security groups:
client - A security group that you attach to all clients, like Lambdas, EC2 instances etc. The default outbound rule gives you outbound access to every resource in the VPC. You can tighten that if you'd like.
db - A security group that you should attach to your Neptune cluster. In this security group, edit hte inbound rules, and explicitly add a TCP rule that allows inbound connections to your database port (8182 is the default port).
You can attach the db security group to your cluster either during creation or by modifying existing clusters.