kubernetes pod talking to a localhost port - kubectl

I have a celery instance running inside a pod in local kubernetes cluster whereas the redis server/broker it connects to is started on my localhost:6379 without kubernetes . How can i get my k8 pod to talk to locally deployed redis?

You can create a Headless Service and an Endpoint with statically defined IP address of the node where the redis server is running.
I've created an example to illustrate you how it works.
First, I created a Headless Service and an Endpoint.
NOTE: Endpoint has the IP address of the node where redis server is running:
# example.yml
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: default
spec:
clusterIP: None
ports:
- name: redis
port: 6379
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: redis
namespace: default
subsets:
- addresses:
- ip: 10.156.0.58 # your node's IP address
ports:
- port: 6379
name: redis
protocol: TCP
After creating above resources, we are able to resolve the redis service name to the IP address:
# kubectl get svc,ep redis
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/redis ClusterIP None <none> 6379/TCP 28m
NAME ENDPOINTS AGE
endpoints/redis 10.156.0.58:6379 28m
# kubectl run dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 -it --rm
/ # nslookup redis
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: redis.default.svc.cluster.local
Address: 10.156.0.58
Additionally, if your redis server is only listening on localhost, you need to modify the iptables rules. To configure port forwarding from port 6379 (default redis port) to localhost you can use:
NOTE: Instead of 10.156.0.58 use the IP address of the node where your redis server is running.
# iptables -t nat -A PREROUTING -p tcp -d 10.156.0.58 --dport 6379 -j DNAT --to-destination 127.0.0.1:6379
As you can see, it is easier if redis is listening not only on the localhost, as we don't have to modify the iptables rules then.
Finally, let's see if we can connect from Pod to the redis server on the host machine:
# kubectl exec -it redis-client -- bash
root#redis-client:/# redis-cli -h redis
redis:6379> SET key1 "value1"
OK

Related

accessing kubernetes service from local host

I created a single node cluster. There is a nodeport service
kubectl get all --namespace default
service/backend-org-1-substra-backend-server NodePort 10.43.81.5 <none> 8000:30068/TCP 4d23h
The node ip is
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k3d-k3s-default-server-0 Ready control-plane,master 5d v1.24.4+k3s1 172.18.0.2 <none> K3s dev 5.15.0-1028-aws containerd://1.6.6-k3s1
From the same host, but not inside the cluster, I can ping the 172.18.0.2 ip. Since the backend-org-1-substra-backend-server is a nodeport, shouldn't I be able to access it by
curl 172.18.0.2:30068? I get
curl: (7) Failed to connect to 172.18.0.2 port 30068 after 0 ms: Connection refused
additional information:
$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
$ kubectl get nodes -o yaml
...
addresses:
- address: 172.24.0.2
type: InternalIP
- address: k3d-k3s-default-server-0
type: Hostname
allocatable:
$ kubectl describe svc backend-org-1-substra-backend-server
Name: backend-org-1-substra-backend-server
Namespace: org-1
Labels: app.kubernetes.io/instance=backend-org-1
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=substra-backend-server
app.kubernetes.io/part-of=substra-backend
app.kubernetes.io/version=0.34.1
helm.sh/chart=substra-backend-22.3.1
skaffold.dev/run-id=394a8d19-bbc8-4a3b-b04e-08e0fff40681
Annotations: meta.helm.sh/release-name: backend-org-1
meta.helm.sh/release-namespace: org-1
Selector: app.kubernetes.io/instance=backend-org-1,app.kubernetes.io/name=substra-backend-server
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.68.217
IPs: 10.43.68.217
Port: http 8000/TCP
TargetPort: http/TCP
NodePort: http 31960/TCP
Endpoints: <none>
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Here, I noticed the endpoints shows . which worries me.
I followed the doc at https://docs.substra.org/en/stable/contributing/getting-started.html
It's a lot to ask someone to replicate the whole thing.
My point is AFAIK, the nodeport service allows callers from outside the cluster to call pods inside the cluster. But neither the cluster ip nor the node ip allows me to curl that service.
I found that it was due to a faulty installation. Now wget to the load balancer ip and port does get a connection.

Kubernetes can't port-forward externalName service

Im create service with type external name:
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: dev
spec:
externalName: google.com
ports:
- port: 80
protocol: TCP
targetPort: 80
sessionAffinity: None
type: ExternalName
By off K8s docs add new endpoint:
apiVersion: v1
kind: Endpoints
metadata:
name: my-service
namespace: dev
subsets:
- addresses:
- ip: 172.217.20.206
ports:
- port: 80
protocol: TCP
And trying forward it to my localhost:
kubectl port-forward -n dev svc/my-service 8080:80
and got the error:
error: cannot attach to *v1.Service: invalid service 'my-service':
Service is defined without a selector
AFAIU, I did all steps by off docs, where I missed ? Or K8s not provide ability port-forward externalName in general?
kubectl port-forward only actually forwards a local connection to a single specific pod. While it looks like you can port-forward to other things, these are just means of picking a pod. If you run kubectl port-forward service/foo 12345:80, it actually looks at the pods selected by that Service, remaps the service's port 80 to the corresponding pod port, and forwards to that specific pod.
In your case, this means you can't port-forward to an ExternalName service, because there isn't a pod behind it, and kubectl port-forward only actually forwards to pods.
There are a couple of other implications (or demonstrations) of this. Start a normal Deployment running some service with 3 replicas, with a normal Service in front of it. Port-forward to either the Deployment or the Service, and run a load test; you will see only one pod receive all the traffic. Delete that specific pod, and the port-forward will shut down.
If you want to connect to an ExternalName service, or otherwise do any of the more interesting things services do, you need to make the connection originate from inside the cluster. You could kubectl run a temporary pod as an example:
kubectl run curl-test --rm --image=curlimages/curl --generator=run-pod/v1 -- \
http://my-service.dev.svc.cluster.local

Expose internal IP so it can be accessed from internet

I just deployed nginx on a K8S Node in a cluster, the master and worker communicate using internal IP address.
I can curl http://worker_ip:8080 (nginx) from internal network, but how to make it can be accessed from external/internet network?
Or should I use public IP as my node host?
update the service type to NodePort. grab the nodePort that is assigned to the service.
you should be able to access nginx using host:nodeport
see below for reference
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
type: NodePort
ports:
- port: 8080
targetPort: 80
protocol: TCP
name: http
- port: 443
protocol: TCP
name: https
selector:
run: my-nginx

Kubernetes Cluster-IP service not working as expected

Ok, so currently I've got kubernetes master up and running on AWS EC2 instance, and a single worker running on my laptop:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 34d v1.9.2
worker Ready <none> 20d v1.9.2
I have created a Deployment using the following configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hostnames
labels:
app: hostnames-deployment
spec:
selector:
matchLabels:
app: hostnames
replicas: 1
template:
metadata:
labels:
app: hostnames
spec:
containers:
- name: hostnames
image: k8s.gcr.io/serve_hostname
ports:
- containerPort: 9376
protocol: TCP
The deployment is running:
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hostnames 1 1 1 1 1m
A single pod has been created on the worker node:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hostnames-86b6bcdfbc-v8s8l 1/1 Running 0 2m
From the worker node, I can curl the pod and get the information:
$ curl 10.244.8.5:9376
hostnames-86b6bcdfbc-v8s8l
I have created a service using the following configuration:
kind: Service
apiVersion: v1
metadata:
name: hostnames-service
spec:
selector:
app: hostnames
ports:
- port: 80
targetPort: 9376
The service is up and running:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hostnames-service ClusterIP 10.97.21.18 <none> 80/TCP 1m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 34d
As I understand, the service should expose the pod cluster-wide and I should be able to use the service IP to get the information pod is serving from any node on the cluster.
If I curl the service from the worker node it works just as expected:
$ curl 10.97.21.18:80
hostnames-86b6bcdfbc-v8s8l
But if I try to curl the service from the master node located on the AWS EC2 instance, the request hangs and gets timed out eventually:
$ curl -v 10.97.21.18:80
* Rebuilt URL to: 10.97.21.18:80/
* Trying 10.97.21.18...
* connect to 10.97.21.18 port 80 failed: Connection timed out
* Failed to connect to 10.97.21.18 port 80: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to 10.97.21.18 port 80: Connection timed out
Why can't the request from the master node reach the pod on the worker node by using the Cluster-IP service?
I have read quite a bit of articles regarding kubernetes networking and the official kubernetes services documentation and couldn't find a solution.
Depends of which mode you using it working different in details, but conceptually same.
You trying to connect to 2 different types of addresses - the pod IP address, which is accessible from the node, and the virtual IP address, which is accessible from pods in the Kubernetes cluster.
IP address of the service is not an IP address on some pod or any other subject, that is a virtual address which mapped to pods IP address based on rules you define in service and it managed by kube-proxy daemon, which is a part of Kubernetes.
That address specially desired for communication inside a cluster for make able to access the pods behind a service without caring about how much replicas of pod you have and where it actually working, because service IP is static, unlike pod's IP.
So, service IP address desired to be available from other pod, not from nodes.
You can read in official documentation about how the Service Virtual IPs works.
kube-proxy is responsible for setting up the IPTables rules (by default) that route cluster IPs. The Service's cluster IP should be routable from anywhere running kube-proxy. My first guess would be that kube-proxy is not running on the master.

Why doesn't my pod respond to requests on the exposed port?

I've just launched a fairly basic cluster based on the CoreOS kube-aws scripts.
https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html
I've activated the registry add-on, and I have it correctly proxying to my local box so I can push images to the cluster on localhost:5000. I also have the proxy pod correctly loaded on each node so that localhost:5000 will also pull images from that registry.
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry
Then I dockerized a fairly simple Sinatra app to run on my cluster and pushed it to the registry. I also prepared a ReplicationController definition and Service definition to run the app. The images pulled and started no problem, I can use kubectl to get the startup logs from each pod that belongs to the replication group.
My problem is that when I curl the public ELB endpoint for my service, it just hangs.
Things I've tried:
I got the public IP for one of the nodes running my pod and attempted to curl it at the NodePort described in the service description, same thing.
I SSH'd into that node and attempted curl localhost:3000, same result.
Also SSH'd into that node, I attempted to curl <pod-ip>:3000, same result.
ps shows the Puma process running and listening on port 3000.
docker ps on the node shows that the app container is not forwarding any ports to the host. Is that maybe the problem?
The requests must be routing correctly because hitting those IPs at any other port results in a connection refused rather than hanging.
The Dockerfile for my app is fairly straightforward:
FROM ruby:2.2.4-onbuild
RUN apt-get update -qq && apt-get install -y \
libpq-dev \
postgresql-client
RUN mkdir -p /app
WORKDIR /app
COPY . /app
EXPOSE 3000
ENTRYPOINT ['ruby', '/app/bin/entrypoint.rb']
Where entrypoint.rb will start a Puma server listening on port 3000.
My replication group is defined like so:
apiVersion: v1
kind: ReplicationController
metadata:
name: web-controller
namespace: app
spec:
replicas: 2
selector:
app: web
template:
metadata:
labels:
app: web
spec:
volumes:
- name: secrets
secret:
secretName: secrets
containers:
- name: app
image: localhost:5000/app:v2
resources:
limits:
cpu: 100m
memory: 50Mi
env:
- name: DATABASE_NAME
value: app_production
- name: DATABASE_URL
value: postgresql://some.postgres.aws.com:5432
- name: ENV
value: production
- name: REDIS_URL
value: redis://some.redis.aws.com:6379
volumeMounts:
- name: secrets
mountPath: "/etc/secrets"
readOnly: true
command: ['/app/bin/entrypoint.rb', 'web']
ports:
- containerPort: 3000
And here is my service:
apiVersion: v1
kind: Service
metadata:
name: web-service
spec:
ports:
- port: 80
targetPort: 3000
protocol: TCP
selector:
app: web
type: LoadBalancer
Output of kubectl describe service web-service:
Name: web-service
Namespace: app
Labels: <none>
Selector: app=web
Type: LoadBalancer
IP: 10.3.0.204
LoadBalancer Ingress: some.elb.aws.com
Port: <unnamed> 80/TCP
NodePort: <unnamed> 32062/TCP
Endpoints: 10.2.47.3:3000,10.2.73.3:3000
Session Affinity: None
No events.
docker ps on one of the nodes shows that the app container is not forwarding any ports to the host. Is that maybe the problem?
Edit to add entrypoint.rb and Procfile
entrypoint.rb:
#!/usr/bin/env ruby
db_user_file = '/etc/secrets/database_user'
db_password_file = '/etc/secrets/database_password'
ENV['DATABASE_USER'] = File.read(db_user_file) if File.exists?(db_user_file)
ENV['DATABASE_PASSWORD'] = File.read(db_password_file) if File.exists?(db_password_file)
exec("bundle exec foreman start #{ARGV[0]}")
Procfile:
web: PORT=3000 bundle exec puma
message_worker: bundle exec sidekiq -q messages -c 1 -r ./config/environment.rb
email_worker: bundle exec sidekiq -q emails -c 1 -r
There was nothing wrong with my Kubernetes set up. It turns out that the app was failing to start because the connection to the DB was timing out due to some unrelated networking issue.
For anyone curious: don't launch anything external to Kubernetes in the 10.x.x.x IP range (e.g. RDS, Elasticache, etc). Long story short, Kubernetes currently has an IPTables masquerade rule hardcoded that messes up communication with anything in that range that isn't part of the cluster. See the details here.
What I ended up doing was creating a separate VPC for my data stores on a different IP range and peering it with my Kubernetes VPC.