Get openstack helm running in a vm behind a proxy - virtualbox

Question

I have been working on getting a helm distribution running on a vm for some time. I have found conflicting arguments about it being possible or not due to virtualization layers, but here it goes.
I am working with a virtualbox ubuntu 16.04 with 16gb ram
following the guide at https://docs.openstack.org/openstack-helm/latest/install/developer/requirements-and-host-config.html
I am also working behind a proxy and I have added all the relevant proxy settings in accordance with https://docs.openstack.org/openstack-helm/latest/troubleshooting/proxy.html
When I run the command make dev-deploy setup-host it passes and has the following output:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f6859822cc9a gcr.io/google_containers/kube-apiserver-amd64 "kube-apiserver --se…" 17 minutes ago Up 17 minutes k8s_kube-apiserver_kube-apiserver-ram_kube-system_9b1ce48429c89e5c30202699d11086af_2
c18ec1473790 gcr.io/google_containers/kube-scheduler-amd64 "kube-scheduler --ad…" 17 minutes ago Up 17 minutes k8s_kube-scheduler_kube-scheduler-ram_kube-system_65a679e8f744d3d257f72713d3790c3b_2
2b631d5fb2a3 gcr.io/google_containers/etcd-amd64 "etcd --listen-clien…" 17 minutes ago Up 17 minutes k8s_etcd_etcd-ram_kube-system_7278f85057e8bf5cb81c9f96d3b25320_2
5a8b005119e3 gcr.io/google_containers/kube-controller-manager-amd64 "kube-controller-man…" 17 minutes ago Up 17 minutes k8s_kube-controller-manager_kube-controller-manager-ram_kube-system_2150e730dce733115de72022e9130f4c_2
a1917416a6e1 gcr.io/google_containers/pause-amd64:3.0 "/pause" 17 minutes ago Up 17 minutes k8s_POD_kube-controller-manager-ram_kube-system_2150e730dce733115de72022e9130f4c_2
caa82c2f2e24 gcr.io/google_containers/pause-amd64:3.0 "/pause" 17 minutes ago Up 17 minutes k8s_POD_kube-scheduler-ram_kube-system_65a679e8f744d3d257f72713d3790c3b_2
183a2a436c5e gcr.io/google_containers/pause-amd64:3.0 "/pause" 17 minutes ago Up 17 minutes k8s_POD_etcd-ram_kube-system_7278f85057e8bf5cb81c9f96d3b25320_3
a3fbddd01f99 gcr.io/google_containers/pause-amd64:3.0 "/pause" 17 minutes ago Up 17 minutes k8s_POD_kube-apiserver-ram_kube-system_9b1ce48429c89e5c30202699d11086af_2
But when I run the coomand make dev-deploy k8s if fails with:
TASK [deploy-kubeadm-aio-master : deploying kubernetes on master node] *********************************************
TASK [deploy-kubeadm-aio-common : performing deploy-kube action] ***************************************************
fatal: [local]: FAILED! => {"changed": false, "msg": "+ '[' xdeploy-kube == xdeploy-kubelet ']'\n+ '[' xdeploy-kube == xdeploy-kube ']'\n+ '[' x '!=' x ']'\n+ '[' xdocker0 '!=' x ']'\n++ echo '{' '\"my_container_name\":' '\"kubeadm-deploy-kube\",' '\"user\":' '{' '\"uid\":' 1000, '\"gid\":' 1000, '\"home\":' '\"/home/nicolas\"' '},' '\"cluster\":' '{' '\"cni\":' '\"calico\"' '},' '\"kubelet\":' '{' '\"container_runtime\":' '\"docker\",' '\"net_support_linuxbridge\":' true, '\"pv_support_nfs\":' true, '\"pv_support_ceph\":' true '},' '\"helm\":' '{' '\"tiller_image\":' '\"gcr.io/kubernetes-helm/tiller:v2.7.2\"' '},' '\"k8s\":' '{' '\"kubernetesVersion\":' '\"v1.9.3\",' '\"imageRepository\":' '\"gcr.io/google_containers\",' '\"certificatesDir\":' '\"/etc/kubernetes/pki\",' '\"selfHosted\":' '\"false\",' '\"api\":' '{' '\"bindPort\":' 6443 '},' '\"networking\":' '{' '\"dnsDomain\":' '\"cluster.local\",' '\"podSubnet\":' '\"192.168.0.0/16\",' '\"serviceSubnet\":' '\"10.96.0.0/12\"' '}' '}' '}'\n++ jq '.k8s.api += {\"advertiseAddressDevice\": \"docker0\"}'\n+ PLAYBOOK_VARS='{\n \"my_container_name\": \"kubeadm-deploy-kube\",\n \"user\": {\n \"uid\": 1000,\n \"gid\": 1000,\n \"home\": \"/home/nicolas\"\n },\n \"cluster\": {\n \"cni\": \"calico\"\n },\n \"kubelet\": {\n \"container_runtime\": \"docker\",\n \"net_support_linuxbridge\": true,\n \"pv_support_nfs\": true,\n \"pv_support_ceph\": true\n },\n \"helm\": {\n \"tiller_image\": \"gcr.io/kubernetes-helm/tiller:v2.7.2\"\n },\n \"k8s\": {\n \"kubernetesVersion\": \"v1.9.3\",\n \"imageRepository\": \"gcr.io/google_containers\",\n \"certificatesDir\": \"/etc/kubernetes/pki\",\n \"selfHosted\": \"false\",\n \"api\": {\n \"bindPort\": 6443,\n \"advertiseAddressDevice\": \"docker0\"\n },\n \"networking\": {\n \"dnsDomain\": \"cluster.local\",\n \"podSubnet\": \"192.168.0.0/16\",\n \"serviceSubnet\": \"10.96.0.0/12\"\n }\n }\n}'\n+ exec ansible-playbook /opt/playbooks/kubeadm-aio-deploy-master.yaml --inventory=/opt/playbooks/inventory.ini --inventory=/opt/playbooks/vars.yaml '--extra-vars={\n \"my_container_name\": \"kubeadm-deploy-kube\",\n \"user\": {\n \"uid\": 1000,\n \"gid\": 1000,\n \"home\": \"/home/nicolas\"\n },\n \"cluster\": {\n \"cni\": \"calico\"\n },\n \"kubelet\": {\n \"container_runtime\": \"docker\",\n \"net_support_linuxbridge\": true,\n \"pv_support_nfs\": true,\n \"pv_support_ceph\": true\n },\n \"helm\": {\n \"tiller_image\": \"gcr.io/kubernetes-helm/tiller:v2.7.2\"\n },\n \"k8s\": {\n \"kubernetesVersion\": \"v1.9.3\",\n \"imageRepository\": \"gcr.io/google_containers\",\n \"certificatesDir\": \"/etc/kubernetes/pki\",\n \"selfHosted\": \"false\",\n \"api\": {\n \"bindPort\": 6443,\n \"advertiseAddressDevice\": \"docker0\"\n },\n \"networking\": {\n \"dnsDomain\": \"cluster.local\",\n \"podSubnet\": \"192.168.0.0/16\",\n \"serviceSubnet\": \"10.96.0.0/12\"\n }\n }\n}'\n\nPLAY [all]
*********************************************************************\n\nTASK [Gathering Facts] *********************************************************\nok:
[/mnt/rootfs]\n\nTASK [deploy-kubeadm-master : storing node hostname] ***************************\nok: [/mnt/rootfs]\n\nTASK [deploy-kubeadm-master : setup directorys on host] ************************\nok: [/mnt/rootfs] => (item=/etc/kubernetes)\nchanged: [/mnt/rootfs] => (item=/etc/kubernetes/pki)\n\nTASK [deploy-kubeadm-master : generating initial admin token] ******************\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : storing initial admin token] *********************\nok: [/mnt/rootfs]\n\nTASK [deploy-kubeadm-master : kubelet | copying config to host] ****************\nchanged: [/mnt/rootfs]\n\nTASK [deploy-kubeadm-master : master | deploy | certs | ca] ********************\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | certs | apiserver] *************\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | certs | apiserver-kubelet-client] ***\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | certs | sa] ********************\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | certs | front-proxy-ca] ********\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | certs | front-proxy-client] ****\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | kubeconfig | admin] ************\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | kubeconfig | kubelet] **********\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | kubeconfig | controller-manager] ***\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | kubeconfig | scheduler] ********\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : generating etcd static manifest] *****************\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | controlplane | apiserver] ******\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | controlplane | controller-manager] ***\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : master | deploy | controlplane | scheduler] ******\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : wait for kube api] *******************************\nFAILED - RETRYING: wait for kube api (120 retries left).\nchanged: [/mnt/rootfs -> 127.0.0.1]\n\nTASK [deploy-kubeadm-master : wait for node to come online] ********************\nFAILED
and then:
TASK [deploy-kubeadm-aio-common : dumping logs for deploy-kube action] *********************************************
ok: [local] => {
"out.stdout_lines": [
"",
"PLAY [all] *********************************************************************",
"",
"TASK [Gathering Facts] *********************************************************",
"ok: [/mnt/rootfs]",
"",
"TASK [deploy-kubeadm-master : storing node hostname] ***************************",
"ok: [/mnt/rootfs]",
"",
"TASK [deploy-kubeadm-master : setup directorys on host] ************************",
"ok: [/mnt/rootfs] => (item=/etc/kubernetes)",
"changed: [/mnt/rootfs] => (item=/etc/kubernetes/pki)",
"",
"TASK [deploy-kubeadm-master : generating initial admin token] ******************",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : storing initial admin token] *********************",
"ok: [/mnt/rootfs]",
"",
"TASK [deploy-kubeadm-master : kubelet | copying config to host] ****************",
"changed: [/mnt/rootfs]",
"",
"TASK [deploy-kubeadm-master : master | deploy | certs | ca] ********************",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | certs | apiserver] *************",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | certs | apiserver-kubelet-client] ***",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | certs | sa] ********************",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | certs | front-proxy-ca] ********",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | certs | front-proxy-client] ****",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | kubeconfig | admin] ************",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | kubeconfig | kubelet] **********",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | kubeconfig | controller-manager] ***",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | kubeconfig | scheduler] ********",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : generating etcd static manifest] *****************",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | controlplane | apiserver] ******",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | controlplane | controller-manager] ***",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : master | deploy | controlplane | scheduler] ******",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : wait for kube api] *******************************",
"FAILED - RETRYING: wait for kube api (120 retries left).",
"changed: [/mnt/rootfs -> 127.0.0.1]",
"",
"TASK [deploy-kubeadm-master : wait for node to come online] ********************",
"FAILED - RETRYING: wait for node to come online (120 retries left).",
"FAILED - RETRYING: wait for node to come online (119 retries left).",
...
"FAILED - RETRYING: wait for node to come online (2 retries left).",
"FAILED - RETRYING: wait for node to come online (1 retries left).",
"fatal: [/mnt/rootfs -> 127.0.0.1]: FAILED! => {\"attempts\": 120, \"changed\": true, \"cmd\": \"kubectl get node \\\"Ram\\\" --no-headers | gawk '{ print $2 }' | grep -q '\\\\(^Ready\\\\)\\\\|\\\\(^NotReady\\\\)'\", \"delta\": \"0:00:01.188128\", \"end\": \"2018-04-05 17:06:51.647344\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2018-04-05 17:06:50.459216\", \"stderr\": \"Error from server (NotFound): nodes \\\"Ram\\\" not found\", \"stderr_lines\": [\"Error from server (NotFound): nodes \\\"Ram\\\" not found\"], \"stdout\": \"\", \"stdout_lines\": []}",
"\tto retry, use: --limit #/opt/playbooks/kubeadm-aio-deploy-master.retry",
"",
"PLAY RECAP *********************************************************************",
"/mnt/rootfs : ok=21 changed=18 unreachable=0 failed=1 "
]
}
TASK [deploy-kubeadm-aio-common : exiting if deploy-kube action failed] ********************************************
fatal: [local]: FAILED! => {"changed": false, "cmd": "exit 1", "msg": "[Errno 2] No such file or directory", "rc": 2}
Any pointers would be greatly appreciated

Answer 1

I had the same issue in this step of the Ansible playbook when deploying an AIO openstack with Helm on a VM.
In this step a number of docker containers are created. (You can verify this with "docker ps" in another terminal. It takes some time)
ubuntu#xxx:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
07ee2d56038a ac5f7ee9ae7e "/tiller" 44 seconds ago Up 43 seconds k8s_tiller_tiller-deploy-7df7fbdd9c-dwzs8_kube-system_28952c16-1d8d-11e9-9ac4-fa163e98e40b_0
e49ef4eaee2c k8s.gcr.io/pause-amd64:3.1 "/pause" 46 seconds ago Up 45 seconds k8s_POD_tiller-deploy-7df7fbdd9c-dwzs8_kube-system_28952c16-1d8d-11e9-9ac4-fa163e98e40b_0
282cbd3089a6 fed89e8b4248 "/sidecar --v=2 --lo…" About a minute ago Up About a minute k8s_sidecar_kube-dns-7485c77cc-wr88r_kube-system_1c2a3761-1d8d-11e9-9ac4-fa163e98e40b_0
b93e2ab03f6c 459944ce8cc4 "/dnsmasq-nanny -v=2…" About a minute ago Up About a minute k8s_dnsmasq_kube-dns-7485c77cc-wr88r_kube-system_1c2a3761-1d8d-11e9-9ac4-fa163e98e40b_0
4148856fcdd4 512cd7425a73 "/kube-dns --domain=…" About a minute ago Up About a minute k8s_kubedns_kube-dns-7485c77cc-wr88r_kube-system_1c2a3761-1d8d-11e9-9ac4-fa163e98e40b_0
14b8cd3cd089 k8s.gcr.io/pause-amd64:3.1 "/pause" About a minute ago Up About a minute k8s_POD_kube-dns-7485c77cc-wr88r_kube-system_1c2a3761-1d8d-11e9-9ac4-fa163e98e40b_0
3bab5bea7b6e 6f1a824b2c81 "/usr/bin/kube-contr…" About a minute ago Up About a minute k8s_calico-kube-controllers_calico-kube-controllers-6bccb8d477-qbk55_kube-system_089abbd3-1d8d-11e9-9ac4-fa163e98e40b_0
f6916bba8ccb k8s.gcr.io/pause-amd64:3.1 "/pause" About a minute ago Up About a minute k8s_POD_calico-kube-controllers-6bccb8d477-qbk55_kube-system_089abbd3-1d8d-11e9-9ac4-fa163e98e40b_0
80cfd3f75475 a89b45f36d5e "start_runit" About a minute ago Up About a minute k8s_calico-node_calico-node-d5n8s_kube-system_0890d2d2-1d8d-11e9-9ac4-fa163e98e40b_0
9b8c7f4e9655 58c02f00d03b "/usr/local/bin/etcd…" About a minute ago Up About a minute k8s_calico-etcd_calico-etcd-d658p_kube-system_088f6c7e-1d8d-11e9-9ac4-fa163e98e40b_0
d78105adb690 k8s.gcr.io/pause-amd64:3.1 "/pause" About a minute ago Up About a minute k8s_POD_calico-node-d5n8s_kube-system_0890d2d2-1d8d-11e9-9ac4-fa163e98e40b_0
7a58c798a05a k8s.gcr.io/pause-amd64:3.1 "/pause" About a minute ago Up About a minute k8s_POD_calico-etcd-d658p_kube-system_088f6c7e-1d8d-11e9-9ac4-fa163e98e40b_0
5a8b67e4f12d 3eb53757e3db "/usr/local/bin/kube…" About a minute ago Up About a minute k8s_kube-proxy_kube-proxy-fc557_kube-system_01792fcc-1d8d-11e9-9ac4-fa163e98e40b_0
0cf582074588 k8s.gcr.io/pause-amd64:3.1 "/pause" About a minute ago Up About a minute k8s_POD_kube-proxy-fc557_kube-system_01792fcc-1d8d-11e9-9ac4-fa163e98e40b_0
c51f0e9ab095 a8706603e59d "kube-scheduler --ad…" 2 minutes ago Up 2 minutes k8s_kube-scheduler_kube-scheduler-openstack---helm_kube-system_4cfd2774e591ce0cf177c635d6ca6850_0
c1bdb76ab9cf 52920ad46f5b "etcd --listen-clien…" 2 minutes ago Up 2 minutes k8s_etcd_etcd-openstack---helm_kube-system_1a6fe3b688c5f1cbaa41d4e4e0dc702b_0
1d39d3d9cb39 a0e5065bdee0 "kube-controller-man…" 2 minutes ago Up 2 minutes k8s_kube-controller-manager_kube-controller-manager-openstack---helm_kube-system_7f18c2dbce121d9cbf120054643f96cf_0
e6cf7228793e f8187c0f74c8 "kube-apiserver --fe…" 2 minutes ago Up 2 minutes k8s_kube-apiserver_kube-apiserver-openstack---helm_kube-system_ef303a0f16f6d02a535b073b1ce77bd1_0
16ca266396fc 18be53808701 "dnsmasq --keep-in-f…" 2 minutes ago Up 2 minutes k8s_osh-dns-redirector_osh-dns-redirector-openstack---helm_kube-system_b4dcd4cd9d071cf5c953824f3630b532_0
ff4850d1148d k8s.gcr.io/pause-amd64:3.1 "/pause" 2 minutes ago Up 2 minutes k8s_POD_kube-scheduler-openstack---helm_kube-system_4cfd2774e591ce0cf177c635d6ca6850_0
093826e65165 k8s.gcr.io/pause-amd64:3.1 "/pause" 2 minutes ago Up 2 minutes k8s_POD_kube-controller-manager-openstack---helm_kube-system_7f18c2dbce121d9cbf120054643f96cf_0
498e4e03c00b k8s.gcr.io/pause-amd64:3.1 "/pause" 2 minutes ago Up 2 minutes k8s_POD_kube-apiserver-openstack---helm_kube-system_ef303a0f16f6d02a535b073b1ce77bd1_0
d31b3720f9d2 k8s.gcr.io/pause-amd64:3.1 "/pause" 2 minutes ago Up 2 minutes k8s_POD_etcd-openstack---helm_kube-system_1a6fe3b688c5f1cbaa41d4e4e0dc702b_0
fb5c6c8adec0 k8s.gcr.io/pause-amd64:3.1 "/pause" 2 minutes ago Up 2 minutes k8s_POD_osh-dns-redirector-openstack---helm_kube-system_b4dcd4cd9d071cf5c953824f3630b532_0
When I re-tried to deploy in another VM with more resources the playbook could finnish without errors.
TASK [deploy-kubeadm-aio-common : performing deploy-kubelet action] ************************************************************************************************************************************
changed: [local]
TASK [deploy-kubeadm-aio-common : removing container for deploy-kubelet action] ************************************************************************************************************************
changed: [local]
TASK [deploy-kubeadm-aio-master : deploying kubernetes on master node] *********************************************************************************************************************************
TASK [deploy-kubeadm-aio-common : performing deploy-kube action] ***************************************************************************************************************************************
changed: [local]
TASK [deploy-kubeadm-aio-common : removing container for deploy-kube action] ***************************************************************************************************************************
changed: [local]
[WARNING]: Could not match supplied host pattern, ignoring: nodes
PLAY [nodes] *******************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY RECAP *********************************************************************************************************************************************************************************************
local : ok=18 changed=13 unreachable=0 failed=0
I'm not 100% sure but it could be a resource issue. Maybe the containers aren't created fast enough on your VM and therefore the script returns these failures:
"FAILED - RETRYING: wait for node to come online (1 retries left).",
As you can see in my output, I have more containers running.
Working VM specs:
8 VCPUS
16GB RAM
60GB disk

Answer 2

I have the same error.
Problem is simple: your hostname doesn't match to this one that Ansible want to run a job.
E.g.:
root#k8s1:~# hostname
k8s1
So in /etc/hosts, it add:
XXX.XXX.XXX.XXX k8s1.XXX.test k8s1
Solution:
echo "k8s1.XXX.test" > /etc/hostname
hostname "k8s1.XXX.test"
reboot