I have integrated Prometheus with my flask application and my application is running in kubernates pods(2 pods).
I have attached same volume for both pods. When i hit "/metrics" endpoint i'm getting single pod data at a time.
is there any way to get the two pods data.
Related
We want to communicate between the two pods that are deployed on AWS kops Kubernetes.
Pods are deployed with spring boot microservices. Trying to call an endpoint of one microservice from the other microservice that is deployed on the pod.
But while accessing the endpoint we are getting access denied error,
[reactor-http-epoll-3] INFO WebClientExceptionHandler:29 - [pod1] Handle 4xx error.
[reactor-http-epoll-3] ERROR WebClientExceptionHandler:32 - {"timestamp":1644985441996,"status":403,"error":"Forbidden","message":"Access Denied","path":"/pod2 endpoint"}
We have created ServiceAccount for the authentication for communicating between pods using delault namespace.
i am running a minimal stateful database service on GKE. single node cluster. i've setup a database as a stateful set on a single pod as of now. the database has exposed a management console on a particular port along with the mandatory database port. i am attempting to do two things.
expose management port over a global HTTP(S) load balancer
expose database port outside of GKE to be consumed by the likes of Cloud Functions or App Engine Applications.
My stateful set is running fine and i can see from the container logs that the database is properly booted up and is listening on required ports.
i am attempting to setup a standalone NEG (ref: https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg) using a simple ClusterIP service.
the cluster service comes up fine and i can see it using
kubectl get service service-name
but i dont see the NEG setup as such... the following command returns nothing
$ gcloud compute network-endpoint-groups list
Listed 0 items.
my pod exposes the port 8080 my service maps 51000 to 8080 and i have provided the neg annotation
cloud.google.com/neg: '{"exposed_ports": {"51000":{}}'
I dont see any errors as such but neither do i see a NEG created/listed.
Any suggestions on how i would go about debugging this.
As a followup question...
when exposing NEG over global load balancer, how do i enforce authn?
im ok with either of service account roles or oauth/openid.
would i be able to expose multiple ports using a single NEG? for
e.g. if i wanted to expose one port to my global load balancer and
another to local services, is this possible with a single NEG or
should i expose each port using a dedicated ClusterIP service?
where can i find documentation/specification for google kubernetes
annotations. i tried to expose two ports on the neg using the
following annotation syntax. is that even supported/meaningful?
cloud.google.com/neg: '{"exposed_ports": {"51000":{},"51010":{}}'
Thanks in advance!
In order to create the service that is backed by a network endpoint group, you need to be working on a GKE Cluster that is VPC Native:
https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg#before_you_begin
When you create a new cluster, this option is disabled by default and you must enable it upon creation. You can confirm if your cluster is VPC Native going to your Cluster details in GKE. It should appear like this:
VPC-native (alias IP) Enabled
If the cluster is not VPC Native, you won’t be able to use this feature as described on their restrictions:
https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips#restrictions
In case you have VPC Native enabled, make sure as well that the pods have the same labels “purpose:” and “topic:” to make sure they are members of the service:
kubectl get pods --show-labels
You can also create multi-port services as it is described on Kubernetes documentation:
https://kubernetes.io/docs/concepts/services-networking/service/#multi-port-services
I am trying to containerize my angular+java app in Kubernetes cluster. I have a frontend deployment and a backend deployment in my k8 cluster. My database is in AWS{RDS}. But i am confused that what API-URL should i give in my Frontend code so that it can get connected to my backend app in k8 cluster.
For e.g :-
In local system i use something like {localhost:8080/api/customers} in my Frontend code but what should i change it to at the time of deploying in Kubernetes cluster.
I have a Kubernetes cluster setup with 1 master and 2 slave nodes, I created a deployment of my backend app and exposed it through Cluster Ip, and than i gave this cluster ip and port in my frontend application.
After that i pushed the image to docker hub and than created a k8 deployment for it, but still its not working.
My main ask is what URL and Port should i mention in my Frontend application target URL so that it can find hit my java APIs.
The front end angular application is running inside the browser of a user. This is outside of the kubernetes Cluster and you therefore can not use the kubernetes Service Name as api endpoint.
You need to make the spring boot api accessible from outside of kubernetes, usually using an ingress or load balancer. You use this external ip or host name as api url in the angular application.
if your two applications run in the same kubernetes cluster so you would have to call your backend service like this: svcname:port for example
http://login:8080/login
This assuming the pods for your frontend are on the same Kubernetes namespace. If they are on a different namespace you would call something like this:
http://login.<namespace>.svc.cluster.local:5555/login
Exposing my back-end service to a Load Balancer, and than using that Load Balancer endpoint in my Front-end application worked for me.
I am running JupyterHub application on a kubernetes cluster (specifically, managed kubernetes on aws, EKS). Each JupyterHub user has their own pod, when they spin up their JupyterHub notebook server.
I need to be able to monitor the HTTP requests that are being made from their notebook server.
Is there any way for me to enable this type of logging? And if so, how could I consume these logs?
With Istio service mesh you will be able to trace all incoming/outgoing HTTP requests within your JupyterHub pod.
Alternatively, you may use Zipkin - a distributed tracing system
I used the cloud formation template provided by Docker for AWS setup & prerequisites to set up a docker swarm.
I created a REST service using Tibco BusinessWorks Container Edition and deployed it into the swarm by creating a docker service.
docker service create --name aka-swarm-demo --publish 8087:8085 akamatibco/docker_swarm_demo:part1
The service starts successfully but the CloudWatch logs show the below exception:
I have tried passing the JVM environment variable in the Dockerfile as :
ENV JAVA_OPTS= "-Dbw.rest.docApi.port=7778"
but it doesn't help.
The interesting fact is at the end the log says:
com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300006: Started BW Application [SFDemo:1.0]
So I tried to access the application using CURL -
curl -X GET --header 'Accept: application/json' 'URL of AWS load balancer : port which I exposed while creating the service/resource URI'
But I am getting the below message:
The REST service works fine when I do docker run.
I have checked the Security Groups of the manager and load-balancer. The load-balancer has inbound open to all traffic and for the manager I opened HTTP connections.
I am not able to figure out if anything I have missed. Can anyone please help ?
As mentioned in Deploy services to swarm, if you read along, you will find the following:
PUBLISH A SERVICE’S PORTS DIRECTLY ON THE SWARM NODE
Using the routing mesh may not be the right choice for your application if you need to make routing decisions based on application state or you need total control of the process for routing requests to your service’s tasks. To publish a service’s port directly on the node where it is running, use the mode=host option to the --publish flag.
Note: If you publish a service’s ports directly on the swarm node using mode=host and also set published= this creates an implicit limitation that you can only run one task for that service on a given swarm node. In addition, if you use mode=host and you do not use the --mode=global flag on docker service create, it will be difficult to know which nodes are running the service in order to route work to them.
Publishing ports for services works different than for regular containers. The problem was; the image does not expose the port after running service create --publish and hence the swarm routing layer cannot reach the REST service. To resolve this use mode = host.
So I used the below command to create a service:
docker service create --name tuesday --publish mode=host,target=8085,published=8087 akamatibco/docker_swarm_demo:part1
Which eventually removed the exception.
Also make sure to configure the firewall settings of your load balancer so as to allow communications through desired protocols in order to access your applications deployed inside the container.
For my case it was HTTP protocol, enabling port 8087 on load balancer which served the purpose.