parallel python not discovering worker nodes - python-2.7

I am trying to implement a basic pp example as shown below (SCRIPT1).
import pp
ppservers = ("*",) # autodiscovery mode on!
# create the job server
job_server = pp.Server(ppservers=ppservers)
for computer, cpu_count in job_server.get_active_nodes().iteritems():
print "Found {} with CPU count {}!".format(computer, cpu_count)
Environment Details
OS : windows10
python version : 2.7 (on machine 1 and machine 2)
machine1 IP : 192.168.1.4
machine2 IP : 192.168.56.1
Steps followed till now :
machine1 and machine2 are connected in same network using wifi
SCRIPT1 is placed on machine1 (machine1 is job server)
installed pp in machine2(worker node) and started server on machine2 (port 60000) using
python ppserver.py -a
triggered SCRIPT1 on machine1, machine 1 output is as shown below
Found local with CPU count 8!
But output at step4 is expected to also detect machine 2 and show output as
Found 192.168.56.1:60000 with CPU count 2!
Found local with CPU count 4!
A newbie to parrallel computing, how do i debug this ?
Appreciate any kinda pointers for this.

Firewall and virtualBox was the issue.
Steps followed to get to the fix:
ping xxx.xxx.x.x. (machine1 IP)
run ppserver on same machine1 and see if jobserver can list the same machine1 as a worker node
if 1 and 2 is the issue, it is mostly caused by firewall, in this case add an inbound rule for the required port and continue.

Related

Packet gen DPDK master lcore cannot be used for a port error

I am facing the issue of master lcore cannot be used for a port while trying to run the default configuration of run.py in packet gen dpdk. When i run the same command using sudo, it gives the error that the packet gen is not available. I have searched all over the internet and did not came across anyone facing such error. kindly guide me about this error. I am attaching the screenshot of the error.Master lcore not available for port error
default configuration file part 1
default configuration file part 2
One can not pass the master core as RX or TX core for any port for pktgen. Since this will be running periodic handler for CLI and other services.
Please change the core to port mapping [0:1].0-3 to [1].0. This will make port 0 to use core 1 to run RX and TX.

Unable to register AWS host to Ambari server

While registering a host to the cluster of Ambari-server, I am getting the following error.
"Host checks were skipped on 1 hosts that failed to register."
I'm trying to install HDP 2.5 version on the instance of AWS.
I have tried to follow the documentation of Hortonworks.
https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-installation/content/set_the_hostname.html
I have added public ip address and public hostname to /etc/hosts file and change the name of host in /etc/hostname file on the server and on the host. Rebooted both, hostname got changed. Then I have stop iptables by
sudo service iptables stop
After doing everything, the host registration is still failing. Kindly help. I am stuck.
Background
From my experience with Ambari (Hortonworks) you have to explicitly setup your Hadoop nodes in each other's /etc/hosts file with the actual name/IPs that the Hadoop services will bind to. NOTE: hostnames should also be FQDN - fully qualified domain names.
For example if you're setting up the hosts as:
node01.mydom.com (10.0.0.2)
node02.mydom.com (10.0.0.3)
node03.mydom.com (10.0.0.4)
These entries should be in all 3 server's /etc/hosts and these should be the names used when referencing them within Ambari's installation/setup wizards.
If you do not pay special attention to this detail, Ambari's server will fail to find/manage any of the other node's that you're telling it to manage.
hostname of ambari-agents
The other item to look at is that the ambari-agent's and what hostnames they think they're going as.
$ ps -eaf|grep ambari_agent
root 3282 1 0 Jul30 ? 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start --expected-hostname=node01.mydom.com
root 3290 3282 1 Jul30 ? 08:24:29 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py start --expected-hostname=node01.mydom.com
Debugging further
In the screen where you're attempting to register the other nodes as agents, there's a full log of what's happening and you can typically get the commands from this area and attempt to run them directly. I've done this on a number of occasions. The commands will often be python ... commands which you can then copy/paste from the logs and run on the Ambari server where you're attempting to run the install.

Connecting two peers on different machines in a private network

I am using geth to create a Private blockchain in my system which is connected to my office network.
I want to add another peer(different PC) to my blockchain network which is connected to the same network.
How can I do that??
Any suggestions on, creating an app which indeed does solve the above requirement.
The key change that would be needed is to start geth nodes on those machines using rpc mode
For example start geth node on Machine 1 and Machine 2 using the commands:
Machine 1 : geth --datadir "C:\ethereum\geth\data\01" --networkid 1234 --rpc --rpcport "8545" --rpcaddr "127.198.216.197" --rpccorsdomain "*"
Machine 2 : geth --datadir "C:\ethereum\geth\data\01" --networkid 1234 --rpc --rpcport "8545" --rpcaddr "127.198.216.198" --rpccorsdomain "*"
Now add Machine 2 as a peer to Machine 1 if you want to do it manually or use a Bootnode and start Geth using Bootnode. You can refer to the link if you want to use Boot nodes: https://github.com/ethereum/go-ethereum/wiki/Setting-up-private-network-or-local-cluster
Then you should be able to attach your console using the command: geth attach http://127.198.216.197:8545
One caveat: You need to open respective ports if they are blocked by your office network
When running the client daemon you can add the 'connect' parameter to specify the IP:Port of the computer you want to try to connect to when starting.
bitcoind --daemon -connect=127.0.0.1:9090
This usually has to be done when the ip's of nodes hard coded into your blockchain codebase have not been updated to the IP's where you have nodes running. Thus peers.dat does not know where to look for a connection to the blockchain.
You can also specify these in the config file with the addnode parameter. e.g.
server=1
rpcport=9090
rpcuser=bitcoinrpc
rpcpassword=3QtnxrB7P5y4EpBdad1MkCeB2RHmArvcarw7udgXsAce
addnode=10.0.0.2:8333

H2O + HDFS (Cloudera)

We have a Cloudera cluster up and running with an h2o instance although it appears to be running off h2o.jar (which as I understand it--please correct me if incorrect) is the stand-alone h2o. I can connect, but it will not load any files from our HDFS. (all of this i can see via 'ps' on edge node.
So I started an instance with h2odriver.jar
java -jar /path/to/h2odriver.jar -nodes 2 -mapperXmx 5g -output /my/hdfs/dir
I get several output/callback addresses:
[Possible callback IP address: 10.96.243.46:33728]
[Possible callback IP address: 127.0.0.1]
Using mapper->driver callback IP address and port: 10.96.243.46:33728
So I fire up python and try and connect (same thing happens if I use 10.96.243.46):
>>>h2o.connection(ip='127.0.0.1', port='33728')
and get
'Connecting to H2O server at http://127.0.0.1:33728..... failed.
H2OConnectionError: COuld not estalich link to the H2O cloud http://127.0.0.1:33728 after 5 retries
...
Failed to establish a new connection:[Errno 111] Connection refused',))`
Thing is on my screen with the H2O jar/java job I can see:
`MapperToDriverMessage: Read invalid type (G) from socket, ignoring...
MapperToDriverMessage: read: Unknown Type `
I cannot figure out how to launch h2o in cluster mode and have it access our hdfs system or even connect. I can connect to the h2o.jar version, but that sees no hdfs (it can see the filesystem of the edgenode). What is the proper way to launch H2O so that it can see the attached HDFS system (We are running Cloudera 5.7 in a enterprise environment, Python is 3.6, H2O is 3.10.0.6 and I know we have a ton of firewalls/security-- i beleive we are setup through LDAP
You are correct that h2o.jar is meant to be the standalone version of H2O which is not meant for connecting to HDFS.
Using the appropriate h2odriver.jar for your particular hadoop distribution is the way to go.
The correct beginner instructions can be found here:
go to http://www.h2o.ai/download/
choose H2O "Latest Stable Release"
choose tab "Install on Hadoop"
It says to run the following command:
hadoop jar h2odriver.jar -nodes 1 -mapperXmx 6g -output hdfsOutputDirName
[ Note this is "hadoop jar", not "java -jar" as written in the question. ]
You should see output like this:
Determining driver host interface for mapper->driver callback...
[Possible callback IP address: 172.16.2.181]
[Possible callback IP address: 127.0.0.1]
...
Waiting for H2O cluster to come up...
H2O node 172.16.2.188:54321 requested flatfile
Sending flatfiles to nodes...
[Sending flatfile to node 172.16.2.188:54321]
H2O node 172.16.2.188:54321 reports H2O cluster size 1
H2O cluster (1 nodes) is up
(Note: Use the -disown option to exit the driver after cluster formation)
Open H2O Flow in your web browser: http://172.16.2.188:54321
(Press Ctrl-C to kill the cluster)
Blocking until the H2O cluster shuts down...
Then point your web browser to the place where it says to "Open H2O Flow in your web browser".
(The other addresses in the output are diagnostics, and not for end users.)
In this case, the python connection command would be:
h2o.connect(ip = '172.16.2.188', port = 54321)
I recommend going to Flow in a web browser, start importing a file by typing in "hdfs://", and seeing if autocompletion works. If it does, your HDFS connection is working.

Getting acces to host VM in a VirtualBox with Puppet

I have an application running in a VirtualBox VM that for testing purposes needs to connect to the host machine. The VM is started with Vagrant and managed by Puppet.
What is the best way to set-up this connection? For example, om my host machine the app runs on port 9200. So from my VM I'd like to go to myhostmachine:9200.
Currently I'm thinking of hacking in a small command that adds hostvm as a entry to /etc/hosts using a simple command like this to figure out my host ip (which is the same as the default route).
/sbin/ip -4 route list 0/0 | grep -m 1 default | awk '/default/ { print $3 }'
And just let Puppet run that every time using the exec functionality. However, I get the feeling there has to be a better way.
The guest OS is Ubuntu 12.04 and the Host is OS-X.
Thanks!
As far as I know, at the moment Vagrant always sets up a natted interface to connect to virtualbox, so I think that the ip of your host machine will always be the 10.0.2.2 address you mentioned. I reckon a puppet host declaration might be easier to manage than running that command each time.
host { 'myhostmachine':
ip => '10.0.2.2',
}
The puppet resource reference for hosts has all the other params you can set too.
I was then able to access the host using myhostmachine:9200