Confused about GCP Dataproc sofwareConfig values - google-cloud-platform

Question

I'm attempting to modify Airflow's dataproc operator to include Anaconda and Jupyter to the cluster.
I'm overriding DataprocClusterCreateOperator to include optionalComponents.
After reading Google docs I understand that I need to pass an enum;
Every time I try to run this task I encounter invalid value errors, or TypeError: Object of type 'EnumMeta' is not JSON serializable
I'd really appreciate it if someone can tell me how to correctly pass in this field.
cluster_data = {
'projectId': self.project_id,
'clusterName': self.cluster_name,
'config': {
'gceClusterConfig': {
},
'masterConfig': {
'numInstances': self.num_masters,
'machineTypeUri': master_type_uri,
'diskConfig': {
'bootDiskType': self.master_disk_type,
'bootDiskSizeGb': self.master_disk_size
}
},
'workerConfig': {
'numInstances': self.num_workers,
'machineTypeUri': worker_type_uri,
'diskConfig': {
'bootDiskType': self.worker_disk_type,
'bootDiskSizeGb': self.worker_disk_size
}
},
'secondaryWorkerConfig': {},
'softwareConfig': {
# I've tried the following:
'optionalComponents': 'ANACONDA,JUPYTER'
#from google.cloud.dataproc_v1 import enums
'optionalComponents': [enums.Component.ANACONDA.value]
},
}
}

Answer 1

You want to use a JSON list there ['ANACONDA', 'JUPYTER'].
As general guidance for figuring out how to structure things, you can create a cluster with gcloud and then run:
gcloud dataproc clusters describe my-cluster --format json
That --format json is the key. The result should be directly copy-pastable.