Kubernetes manifest apiserver, no forwarding? - amazon-web-services

Question

I am working on building a kubernetes cluster on AWS using terraform, by reverse engineering the kube-aws script here:
https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html
However when it is created, the kube-apiserver pod does not forward 443 to the host, so the api cannot be reached (it does forward 8080 to 127.0.0.1)
The manifest in question:
apiVersion: v1
kind: Pod
metadata:
name: kube-apiserver
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-apiserver
image: gcr.io/google_containers/hyperkube:${K8S_VER}
command:
- /hyperkube
- apiserver
- --bind-address=0.0.0.0
- --etcd_servers=${ETCD_ENDPOINTS}
- --allow-privileged=true
- --service-cluster-ip-range=${SERVICE_IP_RANGE}
- --secure_port=443
- --advertise-address=${ADVERTISE_IP}
- --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota
- --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
- --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --client-ca-file=/etc/kubernetes/ssl/ca.pem
- --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --cloud-provider=aws
ports:
- containerPort: 443
hostPort: 443
name: https
- containerPort: 8080
hostPort: 8080
name: local
volumeMounts:
- mountPath: /etc/kubernetes/ssl
name: ssl-certs-kubernetes
readOnly: true
- mountPath: /etc/ssl/certs
name: ssl-certs-host
readOnly: true
volumes:
- hostPath:
path: /etc/kubernetes/ssl
name: ssl-certs-kubernetes
- hostPath:
path: /usr/share/ca-certificates
name: ssl-certs-host
Some output:
ip-10-0-0-50 core # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
47d36516ada9 gcr.io/google_containers/hyperkube:v1.0.7 "/hyperkube apiserve 18 minutes ago Up 18 minutes k8s_kube-apiserver.daa12bc1_kube-apiserver-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_0ff7c6642d467da6eec9af9d96af0622_b88e9ada
48f85774ff5c gcr.io/google_containers/hyperkube:v1.0.7 "/hyperkube schedule 38 minutes ago Up 38 minutes k8s_kube-scheduler.cca58e1_kube-scheduler-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_8aa2dd5e26e716aa54d97e2691e100e0_d6865ecb
1242789081a9 gcr.io/google_containers/hyperkube:v1.0.7 "/hyperkube controll 38 minutes ago Up 38 minutes k8s_kube-controller-manager.9ddfd2a0_kube-controller-manager-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_66bae8c21c0937cc285af054be236103_16b6bfb9
2ebafb2a3413 gcr.io/google_containers/hyperkube:v1.0.7 "/hyperkube proxy -- 38 minutes ago Up 38 minutes k8s_kube-proxy.de5c3084_kube-proxy-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_e6965a2424ca55206c44b02ad95f479e_dacdc559
ade9cd54f391 gcr.io/google_containers/pause:0.8.0 "/pause" 38 minutes ago Up 38 minutes k8s_POD.e4cc795_kube-scheduler-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_8aa2dd5e26e716aa54d97e2691e100e0_b72b8dba
78633207462f gcr.io/google_containers/pause:0.8.0 "/pause" 38 minutes ago Up 38 minutes k8s_POD.e4cc795_kube-controller-manager-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_66bae8c21c0937cc285af054be236103_71057c93
b97643a86f51 gcr.io/google_containers/podmaster:1.1 "/podmaster --etcd-s 39 minutes ago Up 39 minutes k8s_controller-manager-elector.663462cc_kube-podmaster-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_8e57c3cada4c03fae8d01352505c25e5_0bb98126
0859c891679e gcr.io/google_containers/podmaster:1.1 "/podmaster --etcd-s 39 minutes ago Up 39 minutes k8s_scheduler-elector.468957a0_kube-podmaster-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_8e57c3cada4c03fae8d01352505c25e5_fe401f47
e948e718f3d8 gcr.io/google_containers/pause:0.8.0 "/pause" 39 minutes ago Up 39 minutes k8s_POD.e4cc795_kube-apiserver-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_0ff7c6642d467da6eec9af9d96af0622_774d1393
eac6b18c0900 gcr.io/google_containers/pause:0.8.0 "/pause" 39 minutes ago Up 39 minutes k8s_POD.e4cc795_kube-podmaster-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_8e57c3cada4c03fae8d01352505c25e5_949f1945
6411aed07d40 gcr.io/google_containers/pause:0.8.0 "/pause" 39 minutes ago Up 39 minutes k8s_POD.e4cc795_kube-proxy-ip-10-0-0-50.eu-west-1.compute.internal_kube-system_e6965a2424ca55206c44b02ad95f479e_160a3b0f
ip-10-0-0-50 core # netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 1818/hyperkube
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 7966/hyperkube
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 1335/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 1800/hyperkube
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN 1820/hyperkube
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 610/systemd-resolve
tcp6 0 0 :::10255 :::* LISTEN 1335/kubelet
tcp6 0 0 :::22 :::* LISTEN 1/systemd
tcp6 0 0 :::55447 :::* LISTEN 1800/hyperkube
tcp6 0 0 :::42274 :::* LISTEN 1800/hyperkube
tcp6 0 0 :::10250 :::* LISTEN 1335/kubelet
tcp6 0 0 :::5355 :::* LISTEN 610/systemd-resolve
udp 0 0 10.0.0.50:68 0.0.0.0:* 576/systemd-network
udp 0 0 0.0.0.0:8285 0.0.0.0:* 1456/flanneld
udp 0 0 0.0.0.0:5355 0.0.0.0:* 610/systemd-resolve
udp6 0 0 :::5355 :::* 610/systemd-resolve
udp6 0 0 :::52627 :::* 1800/
ip-10-0-0-50 core # docker logs 47d36516ada9
I1127 23:47:15.421827 1 aws.go:489] Zone not specified in configuration file; querying AWS metadata service
I1127 23:47:15.523047 1 aws.go:595] AWS cloud filtering on tags: map[KubernetesCluster:kubernetes]
I1127 23:47:15.692595 1 master.go:273] Node port range unspecified. Defaulting to 30000-32767.
[restful] 2015/11/27 23:47:15 log.go:30: [restful/swagger] listing is available at https://10.0.0.50:443/swaggerapi/
[restful] 2015/11/27 23:47:15 log.go:30: [restful/swagger] https://10.0.0.50:443/swaggerui/ is mapped to folder /swagger-ui/
E1127 23:47:15.718842 1 reflector.go:136] Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused
E1127 23:47:15.719005 1 reflector.go:136] Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token: dial tcp 127.0.0.1:8080: connection refused
E1127 23:47:15.719150 1 reflector.go:136] Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused
E1127 23:47:15.719307 1 reflector.go:136] Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused
E1127 23:47:15.719457 1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E1127 23:47:15.719506 1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
I1127 23:47:15.767717 1 server.go:441] Serving securely on 0.0.0.0:443
I1127 23:47:15.767796 1 server.go:483] Serving insecurely on 127.0.0.1:8080

Answer 1

So it immediately occurred to me to check the certificates that I was using after posting this (Rubberduck ftw.)
Turns out I was merely passing the wrong file to the tls-cert-file= argument.
After correcting it to the right one , everything started working straight away!