How to deploy dask-kubernetes adaptive cluster onto aws kubernetes instance - amazon-web-services

Question

I am attempting to deploy an adaptive dask kubernetes cluster to my aws K8s instance (I want to use the kubeControl interface found here). It is unclear to me where and how I execute this code such that it is active on my existing cluster. In addition to this, I want to have an ingress rule such that another ec2 instance I have can connect to the cluster and execute code within an aws VPC to maintain security and network performance.
So far I have managed to get a functional k8s cluster running with dask and jupyterhub running on it. I am using the sample helm chart found here which reference the docker image here. I can see this image does not even install dask-kubernetes. With that being said, I am able to connect to this cluster from my other ec2 instance using the exposed AWS dns server and execute custom code but this is not the kubernetes native dask cluster.
I have worked on modifying the deploy yaml for kubernetes but it is unclear to me what I would need to change to have it use the proper kubernetes cluster/schedulers. I do know I need to modify the docker image I am using to have in install dask-kubernetes, but this still does not help me. Below is the sample helm deploy chart I am using
---
# nameOverride: dask
# fullnameOverride: dask
scheduler:
name: scheduler
image:
repository: "daskdev/dask"
tag: 2.3.0
pullPolicy: IfNotPresent
# See https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
pullSecrets:
# - name: regcred
replicas: 1
# serviceType: "ClusterIP"
# serviceType: "NodePort"
serviceType: "LoadBalancer"
servicePort: 8786
resources: {}
# limits:
# cpu: 1.8
# memory: 6G
# requests:
# cpu: 1.8
# memory: 6G
tolerations: []
nodeSelector: {}
affinity: {}
webUI:
name: webui
servicePort: 80
worker:
name: worker
image:
repository: "daskdev/dask"
tag: 2.3.0
pullPolicy: IfNotPresent
# dask_worker: "dask-cuda-worker"
dask_worker: "dask-worker"
pullSecrets:
# - name: regcred
replicas: 3
aptPackages: >-
default_resources: # overwritten by resource limits if they exist
cpu: 1
memory: "4GiB"
env:
# - name: EXTRA_CONDA_PACKAGES
# value: numba xarray -c conda-forge
# - name: EXTRA_PIP_PACKAGES
# value: s3fs dask-ml --upgrade
resources: {}
# limits:
# cpu: 1
# memory: 3G
# nvidia.com/gpu: 1
# requests:
# cpu: 1
# memory: 3G
# nvidia.com/gpu: 1
tolerations: []
nodeSelector: {}
affinity: {}
jupyter:
name: jupyter
enabled: true
image:
repository: "daskdev/dask-notebook"
tag: 2.3.0
pullPolicy: IfNotPresent
pullSecrets:
# - name: regcred
replicas: 1
# serviceType: "ClusterIP"
# serviceType: "NodePort"
serviceType: "LoadBalancer"
servicePort: 80
# This hash corresponds to the password 'dask'
password: 'sha1:aae8550c0a44:9507d45e087d5ee481a5ce9f4f16f37a0867318c'
env:
# - name: EXTRA_CONDA_PACKAGES
# value: "numba xarray -c conda-forge"
# - name: EXTRA_PIP_PACKAGES
# value: "s3fs dask-ml --upgrade"
resources: {}
# limits:
# cpu: 2
# memory: 6G
# requests:
# cpu: 2
# memory: 6G
tolerations: []
nodeSelector: {}
affinity: {}

Answer 1

To run a Dask cluster on Kubernetes there are three recommended approaches. Each of these approaches require you to have an existing Kubernetes cluster and credentials correctly configured (kubectl works locally).
Dask Helm Chart
You can deploy a standalone Dask cluster using the Dask helm chart.
helm repo add dask https://helm.dask.org/
helm repo update
helm install --name my-release dask/dask
Note that this is not an adaptive cluster but you can scale it by modifying the size of the deployment via kubectl.
kubectl scale deployment dask-worker --replicas=10
Helm Chart Documentation
Python dask-kubernetes API
You can also use dask-kubernetes which is a Python library for creating ad-hoc clusters on the fly.
pip install dask-kubernetes
from dask_kubernetes import KubeCluster
cluster = KubeCluster()
cluster.scale(10) # specify number of nodes explicitly
cluster.adapt(minimum=1, maximum=100) # or dynamically scale based on current workload
This will create a Dask cluster from scratch and will tear it down when the cluster object is garbage collected (most likely on exit).
dask-kubernetes Documentation
Dask Gateway
Dask Gateway provides a secure, multi-tenant server for managing Dask clusters.
To get started on Kubernetes you need to create a Helm configuration file (config.yaml) with a gateway proxy token.
gateway:
proxyToken: "<RANDOM TOKEN>"
Hint: You can generate a suitable token with openssl rand -hex 32.
Then install the chart.
helm repo add dask-gateway https://dask.org/dask-gateway-helm-repo/
helm repo update
helm install --values config.yaml my-release dask-gateway/dask-gateway
Dask Gateway Documentation