Cloudera manager while installation , get HostName - python-2.7

As a student we were asked to implement a cloudier cluster using Vm’s , i tried to install cloudier cluster using 3 Vm’s under ubuntu 14.01 ,
My steps so far :
I used ssh passwordless connection between all nodes , (tested )
Configure /etc/hosts to all my vm’s
installed cloudera CDH ,
the problem occurs when i try to install every thing on each node this thing occurs .enter image description here
but whats make me so unconfident about this its it identify perfectly my hosts names and address in the pre-installation
[enter image description here][2]
enter image description here
but when i try to install it fails and give me the error mentioned ,
so i tried to see if python dont recognize my hostnames so i went on some codes to test and every thing worked perfectly :
>>> import socket
>>> socket.gethostbyname("master1")
'10.211.55.13'
>>> socket.gethostbyname("slave1")
'10.211.55.11'
>>> socket.gethostbyname("slave2")
'10.211.55.12'
>>>
my /etc/hosts file its similar to all hosts
127.0.0.1 localhost
#127.0.1.1 ubuntu
10.211.55.13 master1.bdsas.com master1
10.211.55.11 slave1.bdsas.com slave1
10.211.55.12 slave2.bdsas.com slave2
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

CDH does not support IPv6 (see this link: https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_ig_req_supported_versions.html) and search for IPv6. Turn off IPv6 in your network configuration and see if that resolves the issue.

Related

Unable to connect to server: PgAdmin 4

I have installed pgadmin on a new windows laptop and when I try to create a new server, it says:
When I try to run my django app in pycharm it is giving me the same error
could not connect to server: Connection refused (0x0000274D/10061)
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
How to solve this ?
In case someone is running the pgadmin-4 in docker, and not able to connect to postgres container, like me.
The solution is to first find the ip at which the docker image is running.
Step-1, make sure the postgres container is running.
Step-2 write command- PS C:\docker> docker ps
Should result as below or similar,
Step3- in order to find the ip address running the postgres use part of container ID and analyze like below command
PS C:\docker> docker inspect fc834
Note: Here I have only used part of container id that is fc834..
This should result the following or similar,
Step4-
Use this ip address in the connection as below with your correct username and password
You may need to installing PostgreSQL Server first.
You can verify if the folder is created in the below folder,
C:\Program Files\PostgreSQL
You can configurate your newly created server to run on localhost and port 5432.
First select the “Connection” tab in the “Create-Server” window. Then, configure the connection as follows:
Enter your server’s IP address in the “Hostname/ Address” field. Default is localhost.
Specify the “Port” as “5432”.
Enter the name of the database in the “Database Maintenance” field.
Enter your username as postgres and password (use the same password you used when previously configuring the server to accept remote connections) for the database.
Click “Save” to apply the configuration.
NOTE You first have to install PostgreSQL on your machine and run it or run it with docker.
I had the same issue. But in my case I had installed pgadmin in version 9. But also installed version 12 at the same time.
When I now uninstalled version 9, the port was already set in the config of version 12 and not given free.
So my solution was to change the port of version 12 in the postgresql.conf file. Or even simplier, change the port in the server creation from 5432 to 5433. Now you are able to create a server again.
You should uninstall Postgres and pgAdmin from your PC. Then install postgres, note that you have the option of installing pgAdmin together with Postgres, so you don't have to download pgAdmin separately. Allow the installation to complete then restart you PC. Hopefully you should be able to create your server/database
I was running postgress and pgadmin both using docker container.
sudo docker ps
sudo docker inspect <postgress_container_id>
Output:
"Networks": {
"work_file_default": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"postgres",
"578a7a1050d1"
],
"NetworkID": "49dbe9d7280b55e36afc4308469c1b55e051d7eea8f1c03f08728e652cf22b5b",
"EndpointID": "c30a642c5a0f2970147c9734cadfbe1e8d7c29fcba8a83a628b7c2b3db114716",
"Gateway": "172.18.0.1",
**"IPAddress": "172.18.0.4",**
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:12:00:04",
"DriverOpts": null
}
Instead of localhost put the IP obtained from above command (172.18.0.4)
In my case was I got both pgadmin and postgresql services running in separate docker containers and I was trying to connect to localhost(127.0.0.1), which is cause of unable to connect to server error.
Note: 5438 port on my computer (host machine) was mapped to 5432 port of postgresql container.
so practically there are two solutions (if you have these services in separate containers and you have mapped postgresql port to your host machine ):
1-find out your local IP (mine is 192.168.1.106) and put it in the Host field.
2- you can put two containers(pgadmin and postgres) in one network (docker network)
and instead of your local IP, put postgres container IP in the Host field.
-Another tip that may help: what I've recently find out was if you are linux user and have ufw enabled, you should allow the port.
e.g. on my computer postgres is running on 5438 port, so I performed below command (so I could connect from pgadmin container to 5438 port of host wich postgres is running)
ufw allow 5438
Execute the container with the data Eg:
docker run --name postgresdb -e POSTGRES_USER=username -e POSTGRES_PASSWORD=password -e POSTGRES_DB=mydb -p 5432:5432 --restart always -d postgres
Then in the PGAdmin client in the Host Name/Address use:
host.docker.internal
Image Conn PGAdmin
I was trying to install PostgreSQL and pgAdmin with an installer that is given here https://www.postgresql.org/download/windows/. This installer includes the PostgreSQL server, pgAdmin;
I was facing an error while starting pgAdmin: "The pgAdmin 4 server could not be contacted". I tried different solutions but did not work.
Then I uninstalled both of them, deleted the temp folder C:\Users\%USERNAME%\AppData\Roaming\pgAdmin and delete those ones too %temp%.
Then I installed the pgAdmin separately from this link https://www.pgadmin.org/download/
and it works. If you need to connect it with your local server I think you should install the PostgreSQL server first and then pgAdmin separately.
I faced the same problem. So I uninstalled pgAdmin through control panel. after that deleted the folder where pgAdmin was located. Then I went to this link and installed pgadmin whole package from there and now it works fine.
I was getting this error when I was running pgadmin in a docker container on my machine, which meant that localhost:5432 was not accessible.
I worked around this by using the native version of pgadmin.
If you are running PostgreSQL in a docker container, set the host name in pgAdmin to postgres not the mapped address or localhost.
press win key+R then Search for services.msc A window will open in that find postgresql-x64-13 right click, in that tab click start option For me its works perfectly.
Check out this stackoverflow link
unable to connect to server for Postgres
how I solved this problem in ubuntu 22.04
I didn't have a password set in Postgres, that's why the error occurred 'unable to connect server 127.0.0.0 port 5432
Open the terminal in ubuntu and enter this command;
sudo -u postgres psql
Run the statement to add new password. ALTER USER Postgres PASSWORD 'AddNewPasswordHere'; in '' you should enter your new password
Example:
1)sudo -u postgres psql
2)ALTER USER postgres PASSWORD 'mynewpassword'
3)sudo service postgresql restart
4)Then you can create a server in pgadmin
If you already tried with “127.0.0.1” and it didn’t work then use “localhost”
After two years i think this would be of good help to so many people.
You don't have to uninstall postgresql or PGADMIN from your system.
What you need to do i input the username and password for a particular user created on postgresql into the server input box.
And that all you need.
i hope this helps anyone

Host key verification failed in google compute engine based mpich cluster

TLDR:
I have 2 google compute engine instances, I've installed mpich on both.
When I try to run a sample I get Host key verification failed.
Detailed version:
I've followed this tutorial in order to get this task done: http://mpitutorial.com/tutorials/running-an-mpi-cluster-within-a-lan/.
I have 2 google compute engine vms with ubuntu 14.04 (the google cloud account is a trial one, btw). I've downloaded this version of mpich on both instances: http://www.mpich.org/static/downloads/3.3rc1
/mpich-3.3rc1.tar.gz and I installed it using these steps:
./configure --disable-fortran
sudo make
sudo make install
This is the way the /etc/hosts file looks on the master-node:
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
169.254.169.254 metadata.google.internal metadata
10.128.0.3 client
10.128.0.2 master
10.128.0.2 linux1.us-central1-c.c.ultimate-triode-161918.internal linux
1 # Added by Google
169.254.169.254 metadata.google.internal # Added by Google
And this is the way the /etc/hosts file looks on the client-node:
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
169.254.169.254 metadata.google.internal metadata
10.128.0.2 master
10.128.0.3 client
10.128.0.3 linux2.us-central1-c.c.ultimate-triode-161918.internal linux
2 # Added by Google
169.254.169.254 metadata.google.internal # Added by Google
The rest of the steps involved adding an user named mpiuser on both nodes and configuring passwordless ssh authentication between the nodes. And configuring a cloud shared directory between nodes.
The configuration worked till this point. I've downloaded this file https://raw.githubusercontent.com/pmodels/mpich/master/examples/cpi.c to /home/mpiuser/cloud/mpi_sample.c, compiled it this way:
mpicc -o mpi_sample mpi_sample.c
and issued this command on the master node while logged in as the mpiuser:
mpirun -np 2 -hosts client,master ./mpi_sample
and I got this error:
Host key verification failed.
What's wrong? I've tried to troubleshoot this problem over more than 2 days but I can't get a valid solution.
Add
package-lock.json
in ".gcloudignore file".
And deploy it again.
It turned out that my password less ssh wasn't configured properly. I've created 2 new instances and did the following things to get a working password less and thus get a working version of that sample. The following steps were execute on an ubuntu server 18.04.
First, by default, instances on google cloud have PasswordAuthentication setting turned off. In the client server do:
sudo vim /etc/ssh/sshd_config
and change PasswordAuthentication no to PasswordAuthentication yes. Then
sudo systemctl restart ssh
Generate a ssh key from the master server with:
ssh-keygen -t rsa -b 4096 -C "user.mail#server.com"
Copy the generated ssh key from the master server to the client
ssh-copy-id client
Now you get a fully functional password less ssh from master to client. However mpich still failed.
The additional steps that I did was to copy the public key to the ~/.ssh/authorized_keys file, both on master and client. So execute this command from both servers:
sudo cat .ssh/id_rsa.pub >> .ssh/authorized_keys
Then make sure the /etc/ssh/sshd_config files from both the client and server have the following configurations:
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no
Restart the ssh service from both client and master
sudo systemctl restart ssh
And that's it, mpich works smoothly now.

how could i connect to Ambari from hortonworks through url "sandbox.hortonworks.com:8080"?

i got installed Hortonworks data platform (HDP 2.4) on oracle virtual box, when i trying to use storm streaming analytics it uses to connect ambari by url
sandbox.hortonworks.com:8080
instead of
127.0.0.1:8080
loop back address and i got an error message over browser
Error : This site can’t be reached
so tell me that how could i assign
sandbox.hortonworks.com:8080
domain name to loop back address from virtual machine OS terminal.
Just add this line
127.0.0.1 localhost sandbox.hortonworks.com
to-
If you're on a windows machine, this file
%systemroot%\system32\drivers\etc\hosts
If you're on a Mac, then, this file
/etc/hosts
This will need to be done for your host machine(the one on which you installed VirtualBox) and not inside the VM.
If you are using bridged mode or host-only-adapter, replace the 127.0.0.1 with your VM's IP, so it should look like
192.168.56.101 localhost sandbox.hortonworks.com
That should help you use the FQDN.

Access django server on virtual Machine

I know there is a lot of similar question but I haven't be able to make it work.
I have a VM (VirtualBox) running CentOS 7
I set up 2 adapters, a NAT and a host-only adapter. This way I have internet output on the VM and I can enter via ssh using a ip (192.168.56.102) from the host
Now, I'm trying to reach a django server running on the guest from the host
I've tried running at 0.0.0.0:8000 and 192.168.56.102:8000 but I always got
$ curl 192.168.56.2:8000
curl: (7) Failed connect to 192.168.56.2:8000; No route to host
I've read some solutions using a bridge instead of NAT, but I need to keep the output to internet and the access via ssh
Finally I opened the port 8000 on the guest firewall and its working now.
For CentOS 7, i followed those steps centos 7 - open firewall port
firewall-cmd --zone=public --add-port=8000/tcp --permanent
firewall-cmd --reload

Connecting to EC2 Django development Server

I am new to EC2 and web development. Currently I have a Linux EC2 instance running, and have installed Django. I am creating a test project before I start on my real project and tried running a Django test server.
This is my output in the shell:
python manage.py runserver ec2-###-##-##-##.compute-1.amazonaws.com:8000
Validating models...
0 errors found
Django version 1.3, using settings 'testsite.settings'
Development server is running at http://ec2-###-##-##-##.compute-1.amazonaws.com:8000/
Quit the server with CONTROL-C.
To test that it is wroking I have tried visiting: ec2-###-##-##-##.compute-1.amazonaws.com:8000 but I always get a "Cannot connect" message from my browser.
Whenever I do this lcoally on my computer however I do successfully get to the DJango development home page at 127.0.0.1:8000. Could someone help me figure out what I am doing wrong / might be missing when I am doing this on my EC2 instance as opposed to my own laptop?
Using an ec-2 instance with Ubuntu, I found that specifying 0.0.0.0:8000 worked:
$python manage.py runserver 0.0.0.0:8000
Of course 8000 does need to be opened for TCP in your security group settings.
You probably don't have port 8000 open on the firewall. Check which security group your instance is running (probably "default") and check the rules it is running. You will probably find that port 8000 is not listed.
1) You need to make sure port 8000 is added as a Custom TCP Rule into your Security Group list of inbound ports
2) Odds are that the IP that you see listed on your AWS Console, which is associated to your instance is a PUBLIC IP OR a PUBLIC Domain Name(i.e. ec2-###-##-##-##.compute-1.amazonaws.com or 174.101.122.132) that Amazon assigns.
2.1) If it is a public IP, then your instance has no way of knowing what the Public IP assigned to it is, rather it will only know the its assigned Local IP.
2.2) To get your Local IP on a Linux System, type:
$ ifconfig
Then look at the eth0 Data and you'll see an IP next to "inet addr" of the format xxx.xxx.xxx.xxx (e.g. 10.10.12.135) This is your Local IP
3) To successfully runserver you can do one of the following two:
$ python manage.py runserver <LOCAL IP>:8000
or
$ python manage.py runserver 0.0.0.0:8000
** Option Two also works great as Ernest Ezis mentioned in his answer.
EDIT : From The Django Book : "The IP address 0.0.0.0 tells the server to listen on any network interface"
** My theory of Public IP could be wrong, since I'm not sure how Amazon assigns IPs. I'd appreciate being corrected.
I was having the same problem. But I was running RHEL on EC2. Besides from adding a rule to security group, I had to manually add a port to firewalld.
firewall-cmd --permanent --add-port=8000/tcp
firewall-cmd --reload
That worked for me! (Although no idea why I had to do that)
Yes, if you use quick launch EC2 option, you should add new HTTP rule (just as it appears on the list) to run a development server.
Adding a security group with the inbound rules as follows usually does the trick unless you have something else misconfigured. The port range specifies which port you want to allow incoming traffic on.
HTTP access would need 80
HTTP access over port 8000 would need 8000
SSH to server would need 22
HTTPS would need 443