I am attempting to run a simple driver to write some data to an Accumulo 1.5 instance running on AWS that is using a single node cluster managed by CDH 4.7 . The client successfully connects to zookeeper but then fails with the following message:
2015-06-26 12:12:13 WARN ServerClient:163 - Failed to find an available server in the list of servers: [172.31.13.210:10011:9997 (120000)]
I tried applying the solution listed
here
, but this has not resolved the issue. The IP that is set for the master/slave is the internal AWS IP for the server.
Other than the warning message, I have not been able to find anything else in the Accumulo logs that indicate what is preventing connection to the master server. Any suggestions on where to look next?
--EDIT--
It looks like zookeeper is returning connectors to the remote client that contain references to the internal IP of the AWS server. The remote client cannot use these connectors because it does not know about the internal IP. When I changed the internal IPs in the thrift connector objects to the public IP, the connection works fine. In essence I can't figure out how to get zookeeper to return public IPs and not AWS internal ones for remote clients
172.31.13.210:10011:9997
This looks really strange. This should be an IP/hostname and a port. It looks like you have two ports somehow..
Did you list ports in the slaves file in ACCUMULO_CONF_DIR? This file should only contain the hostname/IP. If you want to change the port that a TabletServer listens on, you need to change tserver.port.client.
Related
I have configured a Hashicorp Vault server on a EC2 instance. When trying to use postman to test transit secret engine API I keep getting a error connection refused on postman, I went full ape mode and opened all ports on the security group inbound rule and it didn't work, I attached an elastic IP to the instance and didnt work either, im just trying with a simple GET and I just keep getting the same connectionrefused error.
When I use cUrl on the ssh connected session i have no issues though. The specified hosted adress is 127.0.0.1:8200, in postman I replaced that localhost with the public adress of the instance that i obviously censored in the screencap, in the headers theres the token needed to access vault, for simplicity I was just using the root token.
Postman screecap if it helps
#Emilio Marchant
I have faced similar issue (not with postman, but with telnet), Let's try to understand problem here.
The issue is with 127.0.0.1 IP. This is loopback IP and When you (or your computer) call an IP address, you are usually trying to contact another computer on the internet. However, if you call the IP address 127.0.0.1 then you are communicating with the localhost – in principle, with your own computer.
Reference link : https://www.ionos.com/digitalguide/server/know-how/localhost/
What you can try is below.
Start vault dev server with --dev-listen-address parameter.
Eg:
vault server -dev -dev-listen-address="123.456.789.1:8200"
in above command replace '123.456.789.1:8200' with '<your ec2 instance private IP : 8200'>
Next set VAULT_ADDR and VAULT_TOKEN parameter as below
export VAULT_ADDR='http://123.456.789.1:8200'
export VAULT_TOKEN='*****************'
Again replace 'http://123.456.789.1:8200' with 'http://[Your ec2 instance private IP]:8200'
For Vault_token : you should get a root token in console, when you start vault server , use that token
Now try to connect from postman or using curl command. It should work.
Reference question and solution :
How to connect to remote hashicorp vault server
The notable thing here is that the response is "connection refused". This error means that the connection is getting established and it found that there are no processes running on that port. This error means that there is no issue with firewall. A firewall will cause the connection to either drop (reject) or timeout (ignore), but won't give "Econnrefused".
The most likely issue is that the vault server process is not bound to the correct network interface. There must be a configuration in hashicorp-vault to setup the IP on which to bind. Most servers, by default, bind only on loopback address which is accessible only from 127.0.0.1. You need to bind it to "all" network interfaces by changing that to 0.0.0.0. I am not aware of the specific configuration option of hashicorp vault, but there has to be something to this effect.
Possible security issue:
Note that some servers expect you to run it behind a reverse proxy so that you can setup SSL (https) and other authentication if needed. Applications like vault servers should not be publicly accessible on http without SSL.
I have a compiled Go project that I want to deploy to an AWS EC2 instance. I just simply upload the application and run ./application on the remote server.
In the terminal, the application is running and says he's listening to localhost:3000.
I've already added the 3000 port to the security group.
However, when I tried to access it in my browser using <public-ip>:3000, it always shows connection refused, whether I've run the application or not.
I tried to run the app locally, it does work.
So is it because I deploy it incorrectly?
It is a bit difficult to help you because of no code being shared.
Some reasons why you got connection refused:
Your application is listening only localhost:3000
EC2 security group does not expose port 3000
How to fix:
Most applications are defining the host address on a config file or env variables. If you have access to change it, change it from localhost:3000 to 0.0.0.0:3000 to accepts connection from all IP or to your_ec2_public_ip:3000
If host address is hardcoded and you have access to code, change the code per above
If you don't have access to config or code to change the host address, then add a reverse proxy to route the incoming call to localhost:3000. This is a good link about using Nginx as reverse proxy https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/
Ensure EC2 Security Group allowing inbound connection for the designated port, in this case, is 3000 if you manage to route the incoming to your_ip:3000
I am trying to run NiFi on an AWS machine and access the web GUI on my local computer.
I have followed guides such as: https://community.hortonworks.com/articles/47778/hdf-installation-on-ec2.html but whenever I type in the DNS:8080/nifi into my web browser I get a "connection refused" or timed out message.
I have created an AWS Red Hat machine, installed NiFi + java, and edited the nifi.properties file such that it is now:
# Site to Site properties
nifi.remote.input.host=ec2-34-224-216-146.compute-1.amazonaws.com
nifi.remote.input.secure=false
nifi.remote.input.socket.port=
I have tried leaving the port number blank, as well as other numbers such as: nifi.remote.input.socket.port=8082
but neither work when I enter
ec2-34-224-216-146.compute-1.amazonaws.com:8080/nifi into my browser.
I have also tried adding the domain to my local computer's /etc/hosts file in the form of the Public DNS as well as IPv4. I have also configured the security group on AWS such that I have a "Custom TCP Rule" with the port range 8081, 8082, etc. for the respective ports I have attempted.
I am not sure what I am doing wrong or if I am missing a step. Any help is appreciated.
The properties you are configuring are for site-to-site connections and are not related to the UI. These would be used if another NiFi or MiNiFi was making a site-to-site connection to your NiFi instance.
To control the UI you should be configuring:
nifi.web.http.host=
nifi.web.http.port=8080
nifi.web.https.host=
nifi.web.https.port=
I have tried to build as much diagnostics into my Kafka connection setup as possible, but it still leads to mystery problems. In particular, the first thing I do is use the Kafka Admin Client to get the clusterId, because if this operation fails, nothing else is likely to succeed.
def getKafkaClusterId(describeClusterResult: DescribeClusterResult): Try[String] = {
try {
val clusterId = describeClusterResult.clusterId().get(futureTimeout.length / 2, futureTimeout.unit)
Success(clusterId)
} catch {
case cause: Exception =>
Failure(cause)
}
}
In testing this usually works, and everything is fine. It generally only fails when the endpoint is not reachable somehow. It fails because the future times out, so I have no other diagnostics to go by. To test these problems, I usually telnet to the endpoint, for example
$ telnet blah 9094
Trying blah...
Connected to blah.
Escape character is '^]'.
Connection closed by foreign host.
Generally if I can telnet to a Kafka broker, I can connect to Kafka from my server. So my questions are:
What does it mean if I can reach the Kafka brokers via telnet, but I cannot connect via the Kafka Admin Client
What other diagnostic techniques are there to troubleshoot Kafka broker connection problems?
In this particular case, I am running Kafka on AWS, via a Docker Swarm, and trying to figure out why my server cannot connect successfully. I can see in the broker logs when I try to telnet in, so I know the brokers are reachable. But when my server tries to connect to any of 3 brokers, the logs are completely silent.
This is a good article that explains the steps that happens when you first connect to a Kafka broker
https://community.hortonworks.com/articles/72429/how-kafka-producer-work-internally.html
If you can telnet to the bootstrap server then it is listening for client connections and requests.
However clients don't know which real brokers are the leaders for each of the partitions of a topic so the first request they always send to a bootstrap server is a metadata request to get a full list of all the topic metadata. The client uses the metadata response from the bootstrap server to know where it can then make new connections to each of Kafka brokers with the active leaders for each topic partition of the topic you are trying to produce to.
That is where your misconfigured broker problem comes into play. When you misconfigure the advertised.listener port the results of the first metadata request are redirecting the client to connect to unreachable IP addresses or hostnames. It's that second connection that is timing out, not the first one on the port you are telnet'ing into.
Another way to think of it is that you have to configure a Kafka server to work properly as both a bootstrap server and a regular pub/sub message broker since it provides both services to clients. Yours are configured correctly as a pub/sub server but incorrectly as a bootstrap server because the internal and external ip addresses are different in AWS (also in docker containers or behind a NAT or a proxy).
It might seem counter intuitive in small clusters where your bootstrap servers are often the same brokers that the client is eventually connecting to but it is actually a very helpful architectural design that allow kafka to scale and to failover seamlessly without needing to provide a static list of 20 or more brokers on your bootstrap server list, or maintain extra load balancers and health checks to know onto which broker to redirect the client requests.
If you do not configure listeners and advertised.listeners correctly, basically Kafka just does not listen. Even though telnet is listening on the ports you've configured, the Kafka Client Library silently fails.
I consider this a defect in the Kafka design which leads to unnecessary confusion.
Sharing Anand Immannavar's answer from another question:
Along with ADVERTISED_HOST_NAME, You need to add ADVERTISED_LISTENERS to container environment.
ADVERTISED_LISTENERS - Broker will register this value in zookeeper and when the external world wants to connect to your Kafka Cluster they can connect over the network which you provide in ADVERTISED_LISTENERS property.
example:
environment:
- ADVERTISED_HOST_NAME=<Host IP>
- ADVERTISED_LISTENERS=PLAINTEXT://<Host IP>:9092
I have set up a a micro EC2 instance on AWS. Currently, I am using the free tier in Oregon. There are two problems which I am facing.
When I try to SSH the instance using the public DNS, it says host does not exist but when I try conencting it using the public IP, it connects to it. What setting is needed to use the public DNS ?
I have opened the SSH client using the IP address. I want to set up my application which needs Node.js and MongoDB. I installed Node.js using this
Next I installed MongoDB using this
Then I connected to my instance using Filezilla and uploaded my code to it. I then start my node application which uses socket.io.
When I try to connect to socket.io server using web browser, I get a message which says connection refused "error 111". I have opened TCP port 80 in instance's security groups. In iptables, I have forwarded port 80 to 8080, but still it does not work. I have also checked that the firewall is disabled in ec2. Kindly help me to resolve this issue.
Did you check if all of the necessary ports are open on Amazon Security Policy?
What you can do is to allow all traffic on Amazon Security Policy for test and see if the connection goes well or not.
You might also check if you need access DB from outside. In that case, you also have to open the mongodb port and setup mongodb correctly as well.
Other tools that might useful to test firewall and connection issue will be tcpdump and syslog file
For the dns issue, did you try to nslookup on that name and see if the IP shown matches your server IP?
As Amazon gives a long DNS hostname for the server, I always use my own domain name. It's much easier.
example : ec2.domainname.com, which points to the Amazon IP address
Hope that help.
My problem is resolved now..
For the DNS issue, earlier I needed proxy to access internet, so I guess the DNS name was not getting resolved. When I tried using proxy free internet, I was able to ssh using public DNS.
And regarding connection to socket.io, I used port 8080 instead of 80 and used "sudo node main.js" to run my node file. Now I am able to connect to the socket.io server and MongoDB.
Another thing which I want to ask is that would running the node file with sudo rights create some security issue ?
Thanks for the answer! That also worked for me. I had the same problem trying to connect through sockets (http://myipaddress:3000) to a node.js server, i tried opening ports on the actual ec2 instance and disabling the firewall through SSH but nothing worked. Had to go to Security Groups on the ec2 console and open a new inbound tcp rule enabling that port