I am trying to setup a Jenkins agent running on a Mac Mini that will connect to the Master instance in AWS using JNLP. I have created a Network Load Balancer that listens on port 50000 and forwards the traffic to the Master instance. The security group attached to the Master instance allows for traffic from the public IP of the mac mini. On the node configuration I specified the tunnel connection with the DNS name of the NLB.
While trying to connect I receive an IO Exception that the Operation timed out. When I run TCP Dump on the master instance I can see traffic coming from the mac mini to the Master instance.
If I connect the mac mini with a vpn connection I can connect skipping the NLB, so I believe something is wrong with how I setup the NLB. Is there a way to change the connection timeout to increase it, or set the TCP keepalive interfal?
Related
I create ECS service and it runs 1 ecs instance and I can see the instance is registered as a target of the load balancer.
Now I trigger a Auto Scaling Group (by just incrementing desired instance count) to launch a new instance.
The instance is launched and added to the ECS cluster. (I can see it on ECS instances tab)
But the instance is not added to the ALB target. (I expect to see 2 instances in the following image, but I only see 1)
I can edit AutoScalingGroup 's target group like the following
Then I see the following .
But the health check fails. It seems the 80 port is not reachable.
Although I have port 80 open for public in the security group for the instance. (Also, instance created from ecs service uses dynamic port mapping but instance created by ALS does not)
So AutoScalingGroup can launch new instance but my load balancer never gives traffic to the new instance.
I did try https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-unhealthy-checks-ecs/?nc1=h_ls and it shows I can connect to port 80 from host to the docker container by something like curl -v http://${IPADDR}/health.
So it must be the case that there's something wrong with host port 80 (load balancer can't connect to it).
But it is also the case the security group setting is not wrong, because the working instance and this non working instance is using the same SG.
Edit
Because I used dynamic mapping, my webserver is running on some random port.
As you can see the instance started by ecs service has registered itself to target group with random port.
However instance started by ALB has registered itself to target group with port 80.
The instance will not be added to the target group if it's not healthy. So you need to fix the health check first.
From your first instance, your mapped port is 32769 so I assume if this is the same target group and if it is the same application then the port in new instance should be 32769.
When you curl the IP endpoint curl -I -v http://${IPADDR}/health. is the HTTP status code was 200, if it is 200 then it should be healthy if it's not 200 then update the backend http-status code or you can update health check HTTP status code.
I assume that you are also running ECS in both instances, so ECS create target group against each ECS services, are you running some mix services that you need target group in AS group? if you are running dynamic port then remove the health check path to traffic port.
Now if we look the offical possible causes for 502 bad Gateway
Dynamic port mapping is a feature of container instance in Amazon Elastic Container Service (Amazon ECS)
Dynamic port mapping with an Application Load Balancer makes it easier
to run multiple tasks on the same Amazon ECS service on an Amazon ECS
cluster.
With the Classic Load Balancer, you must statically map port numbers
on a container instance. The Classic Load Balancer does not allow you
to run multiple copies of a task on the same instance because the
ports conflict. An Application Load Balancer uses dynamic port mapping
so that you can run multiple tasks from a single service on the same
container instance.
Your created target group will not work with dynamic port, you have to bind the target group with ECS services.
dynamic-port-mapping-ecs
HTTP 502: Bad Gateway
Possible causes:
The load balancer received a TCP RST from the target when attempting to establish a connection.
The load balancer received an unexpected response from the target, such as "ICMP Destination unreachable (Host unreachable)", when attempting to establish a connection. Check whether traffic is allowed from the load balancer subnets to the targets on the target port.
The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.
The target response is malformed or contains HTTP headers that are not valid.
The load balancer encountered an SSL handshake error or SSL handshake timeout (10 seconds) when connecting to a target.
The deregistration delay period elapsed for a request being handled by a target that was deregistered. Increase the delay period so that lengthy operations can complete.
http-502-issues
It seems you know the root cause, which is that port 80 is failing the health check and thats why it is never added to ALB. Here is what you can try
First, check that your service is listening on port 80 on the new host. You can use command like netcat
nv -v localhost 80
Once you know that the service is listening, the recommended way to allow your ALB to connect to your host is to add a Security group inbound rule for your instance to allow traffic from your ALB security group on port 80
I am trying to send the data I receive to my computer into AWS; I use a socket to test for communication. How can I configure the EC2 and make the data received in AWS?
I have set the inbound rule with the port which is configured in the server and it starts working
I'm having an issue where ELB gives me a timeout when connecting to a service on the same EC2 instance.
I have an ECS cluster with two EC2 instances (launched through the ECS wizard). I'm currently running two services: a RabbitMQ queue, and two Celery workers. I put an internal ELB network load balancer in front of the RabbitMQ container.
The celery worker on the other EC2 instance can connect without issues, but the worker that's on the same host as the RabbitMQ container can't connect:
[2018-01-24 12:00:55,128: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**#rabbitmq-abcdefghijklmnop.elb.eu-central-1.amazonaws.com:5672//: timed out.
I've checked the flow logs for the VPC, and all packages are accepted (.157 being the EC2 instance, .136 the ELB):
A network load balancer presents the connection to the server as though it came from the client machine's IP address. Replies are magically mangled back to the correct address/port pairs by the network infrastructure.
But when the server tries to reply, it replies to that source address... and in your configuration, that source address is that same machine... which didn't try to connect to itself, it tried to connect to a different machine... so the forward path and return path source and destination address/port pairs don't correlate correctly and the connection times out.
This appears to be a limitation in Network Load Balancer. Any similarly-designed layer 3 balancer would have the same limitation.
See also https://forums.aws.amazon.com/thread.jspa?messageID=805583󄫏
I installed memcached on AWS EC2 Ubuntu, and I can connect it by telnet in the server:
telnet localhost 11211
But how can I connect it from other machine? I know the interval ip is 172.31.17.208, but when I try to connect it from another EC2 by:
telnet 172.31.17.208 11211
the response is
Could not open connection to the host, on port 11211: connect failed.
You will need a Public IP/ Elastic IP if you want to access your Memcache from outside of the AWS.
Your internal IP will work within the VPC and not outside of your VPC. I am guessing the another instance that you are trying to access is not in the same VPC. Try pinging your Memcache server from another instance and check if it is resolved using internal IP.
Edit:
Apart from this, you might need to check your security group and make sure the ports are open for incoming connection.
I created a new Ubuntu T2 Micro instance on EC2.
Created a new Elastic IP and selected "EIP used in: VPC"
Associated the address to my new EC2 Ubuntu instance.
I now have a Private IP and a Public/Elastic IP. No Public DNS.
My security group has SSH port 22 and HTTP port 80 open.
I can connect to the instance just fine through SSH using the Public IP, but when I try to browse to the Public IP through the browser it says connection refused. I can't ping it either.
I'm out of ideas.
Amazon EC2 HTTP connection refused, but HTTP port is open
That's already proven by the fact that you got 'connection refused'. If the port wasn't open it would have been 'connect timeout'.
'Connection refused' means that the service you tried to connect to wasn't listening at that port. Possibly it isn't started at all, or even installed.
You need to install a web server on your machine, such as Apache or Nginx. This is not provided by default in EC2.
go to security groups --> edit inbound rules --> add rule (add a custom TCP port 8888 with 0.0.0.0/0 ).
There are two major things that can happen to your web server.
Connection refused :- Which means there is no service running (http/JBOSS/nginx) on your server which is available to accept connections on port specified (which is 80 in this case)
Connection timed out :- Would mean server is not able to process any incoming connection hitting it at port 80. Once you fix the security group and your NACL (if you don't have a default one), then you need to re-check to see if it's service which is giving out the error, not to forget that the error response will change.