docker swarm - Leader(Manager) Node service endpoint and load balancing - web-services

docker swarm - Leader(Manager) Node service endpoint and load balancing.
Have two Queries related to Web service application deployment in docker swarm.
We have exposed the service endpoint to our customer as :
http://server1:8080/Application/Service
where server1 is our Leader(Manager) in our swarm cluster.This acts as our load balancer link also.
But what happens to the service endpoint, when server1(Leader) goes down.
As per Swarm, out of the other two more managers, one will be selected as Leader.
Let us assume that server2 becomes the Leader.
But does that mean that the previous server1 service endpoint will not work and it needs to get changed to:
http://server2:8080/Application/Service and we need to shared this new URL to our consumers?
If the URL gets changed, then its really very challenging.If not how does the previous server1 endpoint work,eventhough the server1 is down as a leader.?
In our current 10 nodes Swarm cluster, eventhough the service is deployed on all the 10 Servers, the service is working via only the Leader server endpoint:
http://server1:8080/Application/Service
where server1 is our Leader(Manager) in our swarm cluster.This acts as our load balancer link and shared with consumers.
All the individual endpoint on all the other servers doesn't work. Is this expected behavior in Swarm, that all traffic will go via the leader endpoint?
Leader(Manager):server1 -> Works:
http://server1:8080/Application/Service
Other Managers: server2 and server3 -> Don't work:
http://server2:8080/Application/Service
http://server3:8080/Application/Service
Workers: All other 7 servers/nodes -> Don't work:
http://server4:8080/Application/Service
http://server5:8080/Application/Service
.....................
The docker swarm details are provided below:
Docker version: 1.12.1
Swarm Structure:
- Leader(Manager):server1
- Other Managers: server2 and server3
- Workers: All other 7 servers/nodes
Application/Service endpoint:
http://server1:8080/Application/Service
The above endpoint has been shared with our clients, so act as the load balancing endpoint.
$ sudo docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0415czstge3vibxh55kwnywyd server6 Ready Active
2keplduit5vaycpwzv9419gh7 server4 Ready Active
2r5e2ye9jhgko29s5bv7rolxq server3 Ready Active Reachable
5btrbs5qkrlr50uiip6n0y260 server9 Ready Active
7aqpnf79tv7aj1j5gqsmqph7x server10 Ready Active
856fyn6rdv9ypfz8o2jdsuj7p server2 Ready Active Reachable
a1gcuucxuuupg9gleu9miz7uk server5 Ready Active
a2uyjjhh7phm3wei2e1ydsc4o server7 Ready Active
bm7ztqyrbt7noak6lerfmcs3j * server1 Ready Active Leader
dwto8iizy8li46b7u6v9e4qk1 server8 Ready Active

Related

docker swarm - Performance Bottle Neck

While doing Performance testing in Docker Swarm cluster, the Transactions Per Second(TPS) is not going beyond 400 TPS and the response time is gradually increasing.
While doing Performance testing on one server the TPS is around 200.
So with 10 nodes cluster, it should atleast go beyond 1500 TPS.
But the TPS is not going beyond 400. It seems the Leader is not able to handle more than 400 requests and distribute to the other 9 nodes in the cluster.
Any information on this will be really helpful.Is there any configurations that needs to be done in the Swarm cluster, that will increase the TPS
The docker swarm details are provided below:
Docker version: 1.12.1
Swarm Structure: - Leader(Manager):server1 - Other Managers: server2 and server3 - Workers: All other 7 servers/nodes
Application/Service endpoint:
http://server1:8080/Application/Service The above endpoint has been shared with our clients, so act as the load balancing endpoint.
The application is a webservice deployed in Tomcat 8 using docker.
Swarm Cluster
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0415czstge3vibxh55kwnywyd server6 Ready Active
2keplduit5vaycpwzv9419gh7 server4 Ready Active
2r5e2ye9jhgko29s5bv7rolxq server3 Ready Active Reachable
5btrbs5qkrlr50uiip6n0y260 server9 Ready Active
7aqpnf79tv7aj1j5gqsmqph7x server10 Ready Active
856fyn6rdv9ypfz8o2jdsuj7p server2 Ready Active Reachable
a1gcuucxuuupg9gleu9miz7uk server5 Ready Active
a2uyjjhh7phm3wei2e1ydsc4o server7 Ready Active
bm7ztqyrbt7noak6lerfmcs3j * server1 Ready Active Leader
dwto8iizy8li46b7u6v9e4qk1 server8 Ready Active

Unable to access REST service deployed in docker swarm in AWS

I used the cloud formation template provided by Docker for AWS setup & prerequisites to set up a docker swarm.
I created a REST service using Tibco BusinessWorks Container Edition and deployed it into the swarm by creating a docker service.
docker service create --name aka-swarm-demo --publish 8087:8085 akamatibco/docker_swarm_demo:part1
The service starts successfully but the CloudWatch logs show the below exception:
I have tried passing the JVM environment variable in the Dockerfile as :
ENV JAVA_OPTS= "-Dbw.rest.docApi.port=7778"
but it doesn't help.
The interesting fact is at the end the log says:
com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300006: Started BW Application [SFDemo:1.0]
So I tried to access the application using CURL -
curl -X GET --header 'Accept: application/json' 'URL of AWS load balancer : port which I exposed while creating the service/resource URI'
But I am getting the below message:
The REST service works fine when I do docker run.
I have checked the Security Groups of the manager and load-balancer. The load-balancer has inbound open to all traffic and for the manager I opened HTTP connections.
I am not able to figure out if anything I have missed. Can anyone please help ?
As mentioned in Deploy services to swarm, if you read along, you will find the following:
PUBLISH A SERVICE’S PORTS DIRECTLY ON THE SWARM NODE
Using the routing mesh may not be the right choice for your application if you need to make routing decisions based on application state or you need total control of the process for routing requests to your service’s tasks. To publish a service’s port directly on the node where it is running, use the mode=host option to the --publish flag.
Note: If you publish a service’s ports directly on the swarm node using mode=host and also set published= this creates an implicit limitation that you can only run one task for that service on a given swarm node. In addition, if you use mode=host and you do not use the --mode=global flag on docker service create, it will be difficult to know which nodes are running the service in order to route work to them.
Publishing ports for services works different than for regular containers. The problem was; the image does not expose the port after running service create --publish and hence the swarm routing layer cannot reach the REST service. To resolve this use mode = host.
So I used the below command to create a service:
docker service create --name tuesday --publish mode=host,target=8085,published=8087 akamatibco/docker_swarm_demo:part1
Which eventually removed the exception.
Also make sure to configure the firewall settings of your load balancer so as to allow communications through desired protocols in order to access your applications deployed inside the container.
For my case it was HTTP protocol, enabling port 8087 on load balancer which served the purpose.

consul agent join - i/o timeout

I have 3 Consul Servers I have created within AWS. They were created with Terraform and are joined as part of a cluster.
There is a security group created as part of that Terraform which allowed inbound TCP/UDP on 8300, 8301, 8302, 8400, 8500.
I have installed the consul agent on a new Ubuntu 16.04 instance.
I collect the private IP of one of the Consul servers and try to join it from the client:
consul agent -join 172.1.1.1:8301 -data-dir /tmp/consul
Result:
==> Starting Consul agent...
==> Joining cluster...
==> 1 error(s) occurred:
* Failed to join 172.1.1.1: dial tcp 172.1.1.1:8301: i/o timeout
I can't see what is missing here that is stopping the client from joining.
Not enough data in the question. What do you mean you collected the private IP, was it the server's private IP assigned by the subnet, or is the IP you listed actually a "TaggedAddresses" from the consul itself, which is created if you are not running consul on the host network. So clearly, you need to share some of your consul server configuration too.
Secondly, if it the server's private IP only, please make sure that there is no issue in the NACL or ephemeral ports. You will find more information on the following link from amazon's official documentation:
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html#VPC_ACLs_Ephemeral_Ports

Zookeeper in AWS VPC - This ZooKeeper instance is not currently serving requests

I'm trying to setup a Mesos cluster in AWS VPC.
I've setup 3 Ubuntu machines with Mesos according to this tutorial.
The problem is inside the web UI I see a message No master is currently leading ... and when I ssh to any of the 3 master machines and run echo srvr | nc localhost 2182 I get This ZooKeeper instance is not currently serving requests on all of them.
I also can't ping between the 3 servers.

WSO2 Kuberentes AWS deployment

Here is the issue I am encountering.
I am trying to deploy the WSO2 API Manager which is open source.
Can find the documenation on how to do this here:
https://github.com/wso2/kubernetes-artifacts/tree/master/wso2am
Dockerfiles:
https://github.com/wso2/dockerfiles/tree/master/wso2am
What I did was take the build the docker images which is required for kuberenetes.
I than take these docker images and deploy them to EC2 Container Service.
I than update the wso2 kuberenetes spec files (controllers) to use the image I pushed to EC2 Container Service.
I then go into kubernetes:
kubernetes-artifacts/wso2am and run "./deploy -d"
It than runs the wait for launch script but it just keeps looping and never "finds" that it is up.
root#aw-kubernetes:~/wso2kubernetes/kubernetes-artifacts/wso2am# ./deploy.sh -d
Deploying MySQL Governance DB Service...
service "mysql-govdb" created
Deploying MySQL Governance DB Replication Controller...
replicationcontroller "mysql-govdb" created
Deploying MySQL User DB Service...
service "mysql-userdb" created
Deploying MySQL User DB Replication Controller...
replicationcontroller "mysql-userdb" created
Deploying APIM database Service...
service "mysql-apim-db" created
Deploying APIM database Replication Controller...
replicationcontroller "mysql-apim-db" created
Deploying wso2am api-key-manager Service...
You have exposed your service on an external port on all nodes in your
cluster. If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:32013,tcp:32014,tcp:32015) to serve traffic.
See http://releases.k8s.io/release-1.3/docs/user-guide/services-firewalls.md for more details.
service "wso2am-api-key-manager" created
Deploying wso2am api-store Service...
You have exposed your service on an external port on all nodes in your
cluster. If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:32018,tcp:32019) to serve traffic.
See http://releases.k8s.io/release-1.3/docs/user-guide/services-firewalls.md for more details.
service "wso2am-api-store" created
Deploying wso2am api-publisher Service...
You have exposed your service on an external port on all nodes in your
cluster. If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:32016,tcp:32017) to serve traffic.
See http://releases.k8s.io/release-1.3/docs/user-guide/services-firewalls.md for more details.
service "wso2am-api-publisher" created
Deploying wso2am gateway-manager Service...
You have exposed your service on an external port on all nodes in your
cluster. If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:32005,tcp:32006,tcp:32007,tcp:32008) to serve traffic.
See http://releases.k8s.io/release-1.3/docs/user-guide/services-firewalls.md for more details.
service "wso2am-gateway-manager" created
Deploying wso2am api-key-manager Replication Controller...
replicationcontroller "wso2am-api-key-manager" created
Waiting wso2am to launch on http://172.20.0.30:32013
.......
I tried to comment out the "/wait-until-server-starts.sh" script and have it just start everything. But still not able to access the API Manager.
Could really use some insight on this as I am completely stuck.
I have tried everything I can think of.
If anyone on the WSO2 team or that has done this could help out it would really be appreciated.
My theory right now is maybe this was never tested deploying this to AWS but only to a local setup? but I could be wrong.
Any help would be greatly appreciated!
EDIT:
Adding some outputs from kubectl logs etc while it is in the loop waiting for server to come up I see these things:
root#aw-kubernetes:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-apim-db-b6b0u 1/1 Running 0 11m
mysql-govdb-0b0ud 1/1 Running 0 11m
mysql-userdb-fimc6 1/1 Running 0 11m
wso2am-api-key-manager-0pse8 1/1 Running 0 11m
Also doing a kubectl logs shows that everything started properly:
[2016-07-21 18:46:59,049] INFO - StartupFinalizerServiceComponent Server : WSO2 API Manager-1.10.0
[2016-07-21 18:46:59,049] INFO - StartupFinalizerServiceComponent WSO2 Carbon started in 34 sec
[2016-07-21 18:46:59,262] INFO - CarbonUIServiceComponent Mgt Console URL : https://wso2am-api-key-manager:32014/carbon/
[2016-07-21 18:46:59,262] INFO - CarbonUIServiceComponent API Publisher Default Context : http://wso2am-api-key-manager:32014/publisher
[2016-07-21 18:46:59,263] INFO - CarbonUIServiceComponent API Store Default Context : http://wso2am-api-key-manager:32014/store
#Alex This was an issue in WSO2 Kubernetes Artifacts v1.0.0 release. We have fixed this in the master branch [1].
The problem was that the deployment process was trying to verify WSO2 API-M server sockets using private IP addresses of the Kubernetes nodes. We updated the scripts to use the public/external IP address if they are available via the Kubernetes CLI. For this to work, you may need to setup Kubernetes on AWS according to [2].
[1] https://github.com/wso2/kubernetes-artifacts/commit/53cc6979965ebed8800b803bb3454f3b758b8c05
[2] http://kubernetes.io/docs/getting-started-guides/aws/