I am running JupyterHub application on a kubernetes cluster (specifically, managed kubernetes on aws, EKS). Each JupyterHub user has their own pod, when they spin up their JupyterHub notebook server.
I need to be able to monitor the HTTP requests that are being made from their notebook server.
Is there any way for me to enable this type of logging? And if so, how could I consume these logs?
With Istio service mesh you will be able to trace all incoming/outgoing HTTP requests within your JupyterHub pod.
Alternatively, you may use Zipkin - a distributed tracing system
Related
I have a Node API deployed to Google CloudRun and it is responsible for managing external servers (clean, new Amazon EC2 Linux VM's), including through SSH and SFTP. SSH and SFTP actually work eventually but the connections take 2-5 MINUTES to initiate. Sometimes they timeout with handshake timeout errors.
The same service running on my laptop, connecting to the same external servers, has no issues and the connections are as fast as any normal SSH connection.
The deployment on CloudRun is pretty standard. I'm running it with a service account that permits access to secrets, etc. Plenty of memory allocated.
I have a VPC Connector set up, and have routed all traffic through the VPC connector, as per the instructions here: https://cloud.google.com/run/docs/configuring/static-outbound-ip
I also tried setting UseDNS no in the /etc/ssh/sshd_config file on the EC2 as per some suggestions online re: slow SSH logins, but that has not make a difference.
I have rebuilt and redeployed the project a few dozen times and all tests are on brand new EC2 instances.
I am attempting these connections using open source wrappers on the Node ssh2 library, node-ssh and ssh2-sftp-client.
Ideas?
Cloud Run works only until you have a HTTP request active.
You proably don't have an active request during this on Cloud Run, as outside of the active request the CPU is throttled.
Best for this pipeline is Cloud Workflows and regular Compute Engine instances.
You can setup a Workflow to start a Compute Engine for this task, and stop once it finished doing the steps.
I am the author of article: Run shell commands and orchestrate Compute Engine VMs with Cloud Workflows it will guide you how to setup.
Executing the Workflow can be triggered by Cloud Scheduler or by HTTP ping.
Using AWS to run kubernetes cluster which installed by kops.
Using alb-ingress-controller to realize application load balancer(ALB) on AWS.
Deployed a websocket application into the kubernetes cluster. When try to access the application from ALB DNS, because of it's using load balancer, so sometimes can't catch the response. It's went to the target group one by one.
Maybe the default proxy-mode of kube-proxy is iptables, so thinking about change iptable to userspace, here is a related question:
What does userspace mode means in kube-proxy's proxy mode?
But is it the right way? To make sure the websocket application can run correctly on AWS using ALB?
If it's the right way, then how to change kube-proxy's proxy-mode to userspace?
Here is another article about running socket.io application on kubernetes:
Running Socket.IO Applications on Kubernetes
But it's using ELB. And the method seems a little complex.
Finding a good way.
I used the cloud formation template provided by Docker for AWS setup & prerequisites to set up a docker swarm.
I created a REST service using Tibco BusinessWorks Container Edition and deployed it into the swarm by creating a docker service.
docker service create --name aka-swarm-demo --publish 8087:8085 akamatibco/docker_swarm_demo:part1
The service starts successfully but the CloudWatch logs show the below exception:
I have tried passing the JVM environment variable in the Dockerfile as :
ENV JAVA_OPTS= "-Dbw.rest.docApi.port=7778"
but it doesn't help.
The interesting fact is at the end the log says:
com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300006: Started BW Application [SFDemo:1.0]
So I tried to access the application using CURL -
curl -X GET --header 'Accept: application/json' 'URL of AWS load balancer : port which I exposed while creating the service/resource URI'
But I am getting the below message:
The REST service works fine when I do docker run.
I have checked the Security Groups of the manager and load-balancer. The load-balancer has inbound open to all traffic and for the manager I opened HTTP connections.
I am not able to figure out if anything I have missed. Can anyone please help ?
As mentioned in Deploy services to swarm, if you read along, you will find the following:
PUBLISH A SERVICE’S PORTS DIRECTLY ON THE SWARM NODE
Using the routing mesh may not be the right choice for your application if you need to make routing decisions based on application state or you need total control of the process for routing requests to your service’s tasks. To publish a service’s port directly on the node where it is running, use the mode=host option to the --publish flag.
Note: If you publish a service’s ports directly on the swarm node using mode=host and also set published= this creates an implicit limitation that you can only run one task for that service on a given swarm node. In addition, if you use mode=host and you do not use the --mode=global flag on docker service create, it will be difficult to know which nodes are running the service in order to route work to them.
Publishing ports for services works different than for regular containers. The problem was; the image does not expose the port after running service create --publish and hence the swarm routing layer cannot reach the REST service. To resolve this use mode = host.
So I used the below command to create a service:
docker service create --name tuesday --publish mode=host,target=8085,published=8087 akamatibco/docker_swarm_demo:part1
Which eventually removed the exception.
Also make sure to configure the firewall settings of your load balancer so as to allow communications through desired protocols in order to access your applications deployed inside the container.
For my case it was HTTP protocol, enabling port 8087 on load balancer which served the purpose.
I have an app with several containers running just fine using kubernetes on AWS however now I need to port this to a AWS Dedicated Host VPC where the cluster has previously been created NOT using Kubernetes so I am not able to execute kube-up.sh or its kops equivalent
Is it possible to orchestrate my containers using kubernetes on a pre-existing cluster ? ( IE. have kubernetes probe the parent AWS cluster and treat it as if it created it )
Of course until this linkage is made between my calls to kubectl and the parent AWS Dedicated Host VPC it has no Kubernetes context and just times out :
kubectl create -f /my/app/goodie.yaml
Unable to connect to the server: dial tcp 34.199.89.247:443: i/o timeout
Possible alternative would be to call kube-up.sh or kops and demand the new cluster live inside a specified AWS Dedicated Host ... alas its not apparent Kubernetes has this flexibility ... yet !
Yes, definitely. kubectl is just a client application and it can connect to any kubernetes cluster and orchestrate it.
If you get i/o timeout, you most likely have connectivity issues and some firewall/proxy in place. Did you try to just access the kubernetes API through curl or telnet?
I'm investigating a move to Kubernetes (coming from AWS ECS). But I haven't solved the local development issue when depending on internal services.
Let me elaborate:
When developing and testing microservices, before they are deployed as a Kubernetes Service I want to be able to talk to other, internal Kubernetes Services. As there are > 20 microservices I have a Kubernetes cluster running latest development versions. I can't run a MiniKube.
example:
I'm developing an user-service which needs access to the email service. The Email service is already on Kubernetes and is an internal service.
So before the user-service is deployed I want to be able to talk to the internal email service for dev/testing. I can't make use of K8S nice service discovery env vars.
As we currently already have a VPN up to restrict DEV env to testers/development only, could I use this VPN to provide access to the Kubernetes-Service IP-addresses? I do have Kubernetes DEV-env on the same VPC as the VPN is in.
If you deploy your internal services as type NodePort, then you can access them over your VPN via that nodePort. NodePorts can be dynamically allocated or you can customize them to be 'static' where they are known by you up front.
When developing an app on your local machine, you can access the dependent service by that NodePort.
As an alternative, you can use port-forwarding from kubectl (https://kubernetes.io/docs/user-guide/connecting-to-applications-port-forward/) to forward a pod to your local machine. (Note: This only handles traffic to a pod not a service).
Telepresence (http://telepresence.io) is designed for this scenario, though it presumes developers have kubectl access to the staging/dev cluster.