Unable to access exposed k8s service on AWS on the internet - amazon-web-services

I have an image called hello-node on my AWS ECR (described at bottom of post). If I run it locally and go to localhost:8080 I see "hello world."
On my node on AWS I run:
kubectl run hello-node --image=xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/hello-node:v1 --port=8080 (account number blocked out)
then:
kubectl expose deployment hello-node --type="LoadBalancer"
and if I run:
kubectl describe service hello-node
I see (info edited)
Name: hello-node
Namespace: default
Labels: run=hello-node
Annotations: <none>
Selector: run=hello-node
Type: LoadBalancer
IP: xx.xx.xx.xx
LoadBalancer Ingress: xxxxxxxxx-xxxxxx.us-east-1.elb.amazonaws.com
Port: <unset> 8080/TCP
NodePort: <unset> 32059/TCP
Endpoints: xx.xx.xx.xx:8080
Session Affinity: None
I am trying to access my hello-node service online. I want to go to some web address or IP address in my browser and see 'hello world.'
I have tried going to the above-listed IP, LoadBalancer Ingress, and Endpoints, but those sites do not load in my browser. How can I access this service on the internet? I will provide any more info if required.
NOTE: if I do kubectl port-forward hello-node-621867969-ztxnr 8080 then I can access it on localhost:8080 on my machine, so something is working at least.
hello-node service:
Source here under 'Create an app, package it in a container and publish to a Docker registry' section
server.js:
var http = require('http');
var handleRequest = function(request, response) {
response.writeHead(200);
response.end("Hello World!");
}
var www = http.createServer(handleRequest);
www.listen(8080);
Dockerfile:
FROM node:6.9.2
EXPOSE 8080
COPY server.js .
CMD node server.js
run docker build -t hello-node:v1 .
run docker tag hello-node:v1 AWS_ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/hello-node:v1
run docker push AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/hello-node:v1 (put in my account and region info, and I verified that it is uploaded to my ECR)
Then I ran the two kubectl commands I listed in the start of this post.

I can access it through <loadbalanceringress>:<port>. I honestly can't remember if I had tried that or not (I tried a lot of address:Port combinations.

Related

How to expose kafka brokers ports externally in Kubernetes on AWS private/public VPC

I cannot access Kafka brokers externally (from a public IP address).
I am using https://github.com/Yolean/kubernetes-kafka
It has a very good guide, but I believe their built in method of exposing ports publicly does not work since I am running this cluster privately in a private/public VPC on AWS.
I believe their built in method of outside access simply exposes host ports on private subnet address (Is this correct?)
I know I can set up a load balancer per broker and alias a domain to each load balancer. But then I'm incurring extra costs on load balancers.
I have been looking at ingress resources, have successfully setup an nginx controller that would communicate to different services based on the url path to the host domain.
However with nginx I would receive a 503 Service Temporarily Unavailable with a curl to the url (would get success on echoserver url). So I quickly realised that http requests aren't making sense here. Not to the brokers anyway?
I'm now stuck on learning nginx and a succesful way of proxying the requests.
Is there a specific proxy protocol I should use?
This could also be incorrect server.properties config.
When using nginx I would have the ingress resource connect to the outside-${BROKER_ID} services (I changed the first one to a clusterIP service, others stayed as NodePort). To me this is external DNS mapping to internal IPs. So I would think that the default listeners setting on the Kafka server.properties are OK for this?. Otherwise should the listener become the domain aliased to the load balancer? I had tried the domain with the URL path as an advertised listener, but that didn't make any sense to me, and resulted in crash loops!
For anyone wanting to look at configs, I'm currently running with the default (kinda, 5 pzoos, no ezoos [they were always stuck as pending]) version of:
https://github.com/Yolean/kubernetes-kafka
This can be setup very quickly on an existing cluster (For AWS):
git clone https://github.com/Yolean/kubernetes-kafka
cd kubernetes-kafka
(AWS) rm configure/!(aws*)
kubectl apply -f configure
kubectl apply -f 00-namespace*
kubectl apply -f rbac*
kubectl apply -f zookeeper
kubectl apply -f kafka
kubectl config set-context $(kubectl config current-context) --namespace=kafka
I am running this version of nginx
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/service-l4.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/patch-configmap-l4.yaml
The echoserver came from:
https://github.com/kubernetes/kops/tree/master/addons/ingress-nginx
Specifc lines used:
kubectl run echoheaders --image=k8s.gcr.io/echoserver:1.4 --replicas=1 --port=8080
kubectl expose deployment echoheaders --port=80 --target-port=8080 --name=echoheaders-x
kubectl expose deployment echoheaders --port=80 --target-port=8080 --name=echoheaders-y
Here is my ingress resource for nginx:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: echomap
# annotations:
# nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: brokers.my-domain.com
http:
paths:
- path: /broker0
backend:
serviceName: outside-0
servicePort: 31100
- path: /broker1
backend:
serviceName: outside-1
servicePort: 31101
- path: /broker2
backend:
serviceName: outside-2
servicePort: 31102
- path: /bar
backend:
serviceName: echoheaders-y
servicePort: 80
- path: /foo
backend:
serviceName: echoheaders-x
servicePort: 80
EDIT:
I focused on getting external access through load balancers, and somewhat succeeded. Problems can be found here https://serverfault.com/questions/949367/can-connect-to-kafka-but-cannot-consume
Pretty sure nginx isn't going to work as ingress? I can't figure out how a http request becomes a TCP request.
Moving on to internal kafka streams apps now. Will come back to this when having a separate cluster for kafka streams becomes more necessary.

Kubernetes Cluster-IP service not working as expected

Ok, so currently I've got kubernetes master up and running on AWS EC2 instance, and a single worker running on my laptop:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 34d v1.9.2
worker Ready <none> 20d v1.9.2
I have created a Deployment using the following configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hostnames
labels:
app: hostnames-deployment
spec:
selector:
matchLabels:
app: hostnames
replicas: 1
template:
metadata:
labels:
app: hostnames
spec:
containers:
- name: hostnames
image: k8s.gcr.io/serve_hostname
ports:
- containerPort: 9376
protocol: TCP
The deployment is running:
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hostnames 1 1 1 1 1m
A single pod has been created on the worker node:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hostnames-86b6bcdfbc-v8s8l 1/1 Running 0 2m
From the worker node, I can curl the pod and get the information:
$ curl 10.244.8.5:9376
hostnames-86b6bcdfbc-v8s8l
I have created a service using the following configuration:
kind: Service
apiVersion: v1
metadata:
name: hostnames-service
spec:
selector:
app: hostnames
ports:
- port: 80
targetPort: 9376
The service is up and running:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hostnames-service ClusterIP 10.97.21.18 <none> 80/TCP 1m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 34d
As I understand, the service should expose the pod cluster-wide and I should be able to use the service IP to get the information pod is serving from any node on the cluster.
If I curl the service from the worker node it works just as expected:
$ curl 10.97.21.18:80
hostnames-86b6bcdfbc-v8s8l
But if I try to curl the service from the master node located on the AWS EC2 instance, the request hangs and gets timed out eventually:
$ curl -v 10.97.21.18:80
* Rebuilt URL to: 10.97.21.18:80/
* Trying 10.97.21.18...
* connect to 10.97.21.18 port 80 failed: Connection timed out
* Failed to connect to 10.97.21.18 port 80: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to 10.97.21.18 port 80: Connection timed out
Why can't the request from the master node reach the pod on the worker node by using the Cluster-IP service?
I have read quite a bit of articles regarding kubernetes networking and the official kubernetes services documentation and couldn't find a solution.
Depends of which mode you using it working different in details, but conceptually same.
You trying to connect to 2 different types of addresses - the pod IP address, which is accessible from the node, and the virtual IP address, which is accessible from pods in the Kubernetes cluster.
IP address of the service is not an IP address on some pod or any other subject, that is a virtual address which mapped to pods IP address based on rules you define in service and it managed by kube-proxy daemon, which is a part of Kubernetes.
That address specially desired for communication inside a cluster for make able to access the pods behind a service without caring about how much replicas of pod you have and where it actually working, because service IP is static, unlike pod's IP.
So, service IP address desired to be available from other pod, not from nodes.
You can read in official documentation about how the Service Virtual IPs works.
kube-proxy is responsible for setting up the IPTables rules (by default) that route cluster IPs. The Service's cluster IP should be routable from anywhere running kube-proxy. My first guess would be that kube-proxy is not running on the master.

How to configure an AWS Elastic IP to point to an OpenShift Origin running pod?

We have set up OpenShift Origin on AWS using this handy guide. Our eventual
hope is to have some pods running REST or similar services that we can access
for development purposes. Thus, we don't need DNS or anything like that at this
point, just a public IP with open ports that points to one of our running pods.
Our first proof of concept is trying to get a jenkins (or even just httpd!) pod
that's running inside OpenShift to be exposed via an allocated Elastic IP.
I'm not a network engineer by any stretch, but I was able to successuflly get
an Elastic IP connected to one of my OpenShift "worker" instances, which I
tested by sshing to the public IP allocated to the Elastic IP. At this point
we're struggling to figure out how to make a pod visible that allocated Elastic IP,
owever. We've tried a kubernetes LoadBalancer service, a kubernetes Ingress,
and configuring an AWS Network Load Balancer, all without being able to
successfully connect to 18.2XX.YYY.ZZZ:8080 (my public IP).
The most promising success was using oc port-forward seemed to get at least part way
through, but frustratingly hangs without returning:
$ oc port-forward --loglevel=7 jenkins-2-c1hq2 8080 -n my-project
I0222 19:20:47.708145 73184 loader.go:354] Config loaded from file /home/username/.kube/config
I0222 19:20:47.708979 73184 round_trippers.go:383] GET https://ec2-18-2AA-BBB-CCC.us-east-2.compute.amazonaws.com:8443/api/v1/namespaces/my-project/pods/jenkins-2-c1hq2
....
I0222 19:20:47.758306 73184 round_trippers.go:390] Request Headers:
I0222 19:20:47.758311 73184 round_trippers.go:393] X-Stream-Protocol-Version: portforward.k8s.io
I0222 19:20:47.758316 73184 round_trippers.go:393] User-Agent: oc/v1.6.1+5115d708d7 (linux/amd64) kubernetes/fff65cf
I0222 19:20:47.758321 73184 round_trippers.go:393] Authorization: Bearer Pqg7xP_sawaeqB2ub17MyuWyFnwdFZC5Ny1f122iKh8
I0222 19:20:47.800941 73184 round_trippers.go:408] Response Status: 101 Switching Protocols in 42 milliseconds
I0222 19:20:47.800963 73184 round_trippers.go:408] Response Status: 101 Switching Protocols in 42 milliseconds
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
( oc port-forward hangs at this point and never returns)
We've found a lot of information about how to get this working under GKE, but
nothing that's really helpful for getting this working for OpenShift Origin on
AWS. Any ideas?
Update:
So we realized that sysdig.com's blog post on deploying OpenShift Origin on AWS was missing some key AWS setup information, so based on OpenShift Origin's Configuring AWS page, we set the following env variables and re-ran the ansible playbook:
$ export AWS_ACCESS_KEY_ID='AKIASTUFF'
$ export AWS_SECRET_ACCESS_KEY='STUFF'
$ export ec2_vpc_subnet='my_vpc_subnet'
$ ansible-playbook -c paramiko -i hosts openshift-ansible/playbooks/byo/config.yml --key-file ~/.ssh/my-aws-stack
I think this gets us closer, but creating a load-balancer service now gives us an always-pending IP:
$ oc get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins-lb 172.30.XX.YYY <pending> 8080:31338/TCP 12h
The section on AWS Applying Configuration Changes seems to imply I need to use AWS Instance IDs rather than hostnames to identify my nodes, but I tried this and OpenShift Origin fails to start if I use that method. Still at a loss.
It may not satisfy the "Elastic IP" part but how about using AWS cloud provider ELB to expose the IP/port to the pod via a service to the pod with LoadBalancer option?
Make sure to configure the AWS cloud provider for the cluster (References)
Create a svc to the pod(s) with type LoadBalancer.
For instance to expose a Dashboard via AWS ELB.
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: LoadBalancer <-----
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
Then the svc will be exposed as an ELB and the pod can be accessed via the ELB public DNS name a53e5811bf08011e7bae306bb783bb15-953748093.us-west-1.elb.amazonaws.com.
$ kubectl (oc) get svc kubernetes-dashboard -n kube-system -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes-dashboard LoadBalancer 10.100.96.203 a53e5811bf08011e7bae306bb783bb15-953748093.us-west-1.elb.amazonaws.com 443:31636/TCP 16m k8s-app=kubernetes-dashboard
References
K8S AWS Cloud Provider Notes
Reference Architecture OpenShift Container Platform on Amazon Web Services
DEPLOYING OPENSHIFT CONTAINER PLATFORM 3.5 ON AMAZON WEB SERVICES
Configuring for AWS
Check this guide out: https://github.com/dwmkerr/terraform-aws-openshift
It's got some significant advantages vs. the one you referring to in your post. Additionally, it has a clear terraform spec that you can modify and reset to using an Elastic IP (haven't tried myself but should work).
Another way to "lock" your access to the installation is to re-code the assignment of the Public URL to the master instance in the terraform script, e.g., to a domain that you own (the default script sets it to an external IP-based value with "xip.io" added - works great for testing), then set up a basic ALB that forwards https 443 and 8443 to the master instance that the install creates (you can do it manually after the install is completed, also need a second dummy Subnet; dummy-up the healthcheck as well) and link the ALB to your domain via Route53. You can even use free Route53 wildcard certs with this approach.

container engine ingress not working

I have a simple container in google container registry which basically does a few things and executes a binary which is a go based server, here are the contents of the DockerFile:
FROM debian:stable
WORKDIR /workspace/
COPY key.json .
COPY bin/user-creds.
EXPOSE 1108
ENV GOOGLE_APPLICATION_CREDENTIALS /workspace/key.json
RUN apt-get update \
&& apt-get install -y ca-certificates \
&& chmod +x user-creds
CMD ["./user-creds"]
this container has been tested locally and works perfectly. So using the google cloud shell I ran this container:
kubectl run user-creds --image=eu.gcr.io/GCLOUD_PROJECT/user-creds:COMMIT_SHA --port=1108
Then like it says on the doc, i exposed it on a nodeport
kubectl expose deployment user-creds --target-port=1108 --type=NodePort
Then I created an ingress with a path to the sevice:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: INGRESS_NAME
annotations:
kubernetes.io/ingress.global-static-ip-name: IP_NAME
spec:
rules:
- http:
paths:
- path: /user/creds/*
backend:
serviceName: user-creds
servicePort: 1108
then i created the ingress:
kubectl create -f INGRESS_NAME.yaml
the ingress was created and i waited some time, here is the details of the ingress:
NAME HOSTS ADDRESS PORTS AGE
INGRESS_NAME * IP_ADDRESS 80 38m
but when i go the the actual url with the path I get a 502 error:
When I go to any other path I get the default backend 404 error but when i visit the specific /user/creds/ path i get the 502 error.
To check if it is something wrong with the cluster or my specific container, port or something else, I tried exposing the container as a LoadBalancer and it works perfectly, the Command:
kubectl expose deployment user-creds --target-port=1108 --port=80 --type=LoadBalancer
service details:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP INT_IP_ADDRESS <none> 443/TCP 1h
user-creds LoadBalancer INT_IP_ADDRESS IP_ADDRESS 80:31618/TCP 1m
result: 200 with the correst response body.
Been stuck on this for time now, tried the ingress with no paths just the user-creds as the backend but still has the same error.
Any help or suggestion would be appreciated, thanks :)
Finally figured it out, it was to do with the health check. The health check visits / and expects a 200, if it doesn't get it then it marks the backend as unhealthy and returns 502 for every requests sent to it. My problem was that I was using the / endpoint which would've normally returned a 400 if its being called with no specific request parameters.
It was really a human error on my side, it even specifically said that in the docs: https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer#remarks
Another thing to consider is that the ingress returns all the the paths before the route so the the server needs to literally listen for /user/creds/ in my case.

Why doesn't my pod respond to requests on the exposed port?

I've just launched a fairly basic cluster based on the CoreOS kube-aws scripts.
https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html
I've activated the registry add-on, and I have it correctly proxying to my local box so I can push images to the cluster on localhost:5000. I also have the proxy pod correctly loaded on each node so that localhost:5000 will also pull images from that registry.
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry
Then I dockerized a fairly simple Sinatra app to run on my cluster and pushed it to the registry. I also prepared a ReplicationController definition and Service definition to run the app. The images pulled and started no problem, I can use kubectl to get the startup logs from each pod that belongs to the replication group.
My problem is that when I curl the public ELB endpoint for my service, it just hangs.
Things I've tried:
I got the public IP for one of the nodes running my pod and attempted to curl it at the NodePort described in the service description, same thing.
I SSH'd into that node and attempted curl localhost:3000, same result.
Also SSH'd into that node, I attempted to curl <pod-ip>:3000, same result.
ps shows the Puma process running and listening on port 3000.
docker ps on the node shows that the app container is not forwarding any ports to the host. Is that maybe the problem?
The requests must be routing correctly because hitting those IPs at any other port results in a connection refused rather than hanging.
The Dockerfile for my app is fairly straightforward:
FROM ruby:2.2.4-onbuild
RUN apt-get update -qq && apt-get install -y \
libpq-dev \
postgresql-client
RUN mkdir -p /app
WORKDIR /app
COPY . /app
EXPOSE 3000
ENTRYPOINT ['ruby', '/app/bin/entrypoint.rb']
Where entrypoint.rb will start a Puma server listening on port 3000.
My replication group is defined like so:
apiVersion: v1
kind: ReplicationController
metadata:
name: web-controller
namespace: app
spec:
replicas: 2
selector:
app: web
template:
metadata:
labels:
app: web
spec:
volumes:
- name: secrets
secret:
secretName: secrets
containers:
- name: app
image: localhost:5000/app:v2
resources:
limits:
cpu: 100m
memory: 50Mi
env:
- name: DATABASE_NAME
value: app_production
- name: DATABASE_URL
value: postgresql://some.postgres.aws.com:5432
- name: ENV
value: production
- name: REDIS_URL
value: redis://some.redis.aws.com:6379
volumeMounts:
- name: secrets
mountPath: "/etc/secrets"
readOnly: true
command: ['/app/bin/entrypoint.rb', 'web']
ports:
- containerPort: 3000
And here is my service:
apiVersion: v1
kind: Service
metadata:
name: web-service
spec:
ports:
- port: 80
targetPort: 3000
protocol: TCP
selector:
app: web
type: LoadBalancer
Output of kubectl describe service web-service:
Name: web-service
Namespace: app
Labels: <none>
Selector: app=web
Type: LoadBalancer
IP: 10.3.0.204
LoadBalancer Ingress: some.elb.aws.com
Port: <unnamed> 80/TCP
NodePort: <unnamed> 32062/TCP
Endpoints: 10.2.47.3:3000,10.2.73.3:3000
Session Affinity: None
No events.
docker ps on one of the nodes shows that the app container is not forwarding any ports to the host. Is that maybe the problem?
Edit to add entrypoint.rb and Procfile
entrypoint.rb:
#!/usr/bin/env ruby
db_user_file = '/etc/secrets/database_user'
db_password_file = '/etc/secrets/database_password'
ENV['DATABASE_USER'] = File.read(db_user_file) if File.exists?(db_user_file)
ENV['DATABASE_PASSWORD'] = File.read(db_password_file) if File.exists?(db_password_file)
exec("bundle exec foreman start #{ARGV[0]}")
Procfile:
web: PORT=3000 bundle exec puma
message_worker: bundle exec sidekiq -q messages -c 1 -r ./config/environment.rb
email_worker: bundle exec sidekiq -q emails -c 1 -r
There was nothing wrong with my Kubernetes set up. It turns out that the app was failing to start because the connection to the DB was timing out due to some unrelated networking issue.
For anyone curious: don't launch anything external to Kubernetes in the 10.x.x.x IP range (e.g. RDS, Elasticache, etc). Long story short, Kubernetes currently has an IPTables masquerade rule hardcoded that messes up communication with anything in that range that isn't part of the cluster. See the details here.
What I ended up doing was creating a separate VPC for my data stores on a different IP range and peering it with my Kubernetes VPC.