Cloud Foundry with 3 AZ without shared storage - cloud-foundry

Question

I am trying to install Cloud Foundry on three vSphere clusters (one node each) using Bosh without having shared storage. The nodes use their own local storages.
If I use scale-to-one-az.yml file, I can install CF and it works. If I deploy to three AZ, I am getting an error:
Error: CPI error 'Bosh::Clouds::CloudError' with message 'No valid placement found for VM compute and storage requirement' in 'create_vm' CPI method (CPI request ID: 'cpi-717770')
Can I deploy Cloud Foundry with 3 AZ without shared storage?
Additional information:
Cloud Foundry manifest_version: v13.23.0
Bosh version: version 6.3.1-c44e8a1d-2020-07-09T21:08:12Z
bosh cloud-config output:
azs:
- cloud_properties:
datacenters:
- clusters:
- AZ01:
datastore_pattern: node1-datastore
persistent_datastore_pattern: node1-datastore
resource_pool: Pool01
name: DC
name: z1
- cloud_properties:
datacenters:
- clusters:
- AZ02:
datastore_pattern: node2-datastore
persistent_datastore_pattern: node2-datastore
resource_pool: Pool02
name: DC
name: z2
- cloud_properties:
datacenters:
- clusters:
- AZ03:
datastore_pattern: node3-datastore
persistent_datastore_pattern: node3-datastore
resource_pool: Pool03
name: DC
name: z3
compilation:
az: z1
network: default
reuse_compilation_vms: true
vm_type: small-highmem
workers: 6
disk_types:
- disk_size: 5120
name: 5GB
- disk_size: 10240
name: 10GB
- disk_size: 102400
name: 100GB
- disk_size: 1024000
name: 1000GB
networks:
- name: default
subnets:
- az: z1
cloud_properties:
name: LS-Cloud01
dns:
- 8.8.8.8
gateway: x.x.10.1
range: x.x.10.0/24
reserved:
- x.x.10.1-x.x.10.49
static:
- x.x.10.50 - x.x.10.60
- az: z2
cloud_properties:
name: LS-Cloud02
dns:
- 8.8.8.8
gateway: x.x.40.1
range: x.x.40.0/24
reserved:
- x.x.40.1 - x.x.40.49
static:
- x.x.40.50 - x.x.40.60
- az: z3
cloud_properties:
name: LS-Cloud03
dns:
- 8.8.8.8
gateway: x.x.50.1
range: x.x.50.0/24
reserved:
- x.x.50.1 - x.x.50.49
static:
- x.x.50.50 - x.x.50.60
vm_extensions:
- name: cf-router-network-properties
- name: cf-tcp-router-network-properties
- name: diego-ssh-proxy-network-properties
- cloud_properties:
disk: 51200
name: 50GB_ephemeral_disk
- cloud_properties:
disk: 102400
name: 100GB_ephemeral_disk
vm_types:
- cloud_properties:
cpu: 2
disk: 10240
ram: 4096
name: minimal
- cloud_properties:
cpu: 6
disk: 10240
ram: 16384
name: small
- cloud_properties:
cpu: 6
disk: 10240
ram: 65536
name: small-highmem
Succeeded

Answer 1

yes, it can be done.
Storage configuration (node1-datastore, node2-datastore, node3-datastore) and information about the resource pools (Pool01, Pool02, Pool03) and AZ (AZ01, AZ02, AZ03)need to be provided to bosh. My cpi.yml (https://github.com/cloudfoundry/bosh-deployment/blob/master/vsphere/cpi.yml) can be found below:
- name: cpi
path: /releases/-
type: replace
value:
name: bosh-vsphere-cpi
sha1: 25c53531bf9efeb86d093c2acded62f638e12f0f
url: https://bosh.io/d/github.com/cloudfoundry/bosh-vsphere-cpi-release?v=54.1.0
version: 54.1.0
- name: stemcell
path: /resource_pools/name=vms/stemcell?
type: replace
value:
sha1: 025432cad0600ee0c05d185bc64b88d250c65de4
url: https://bosh-core-stemcells.s3-accelerate.amazonaws.com/621.85/bosh-stemcell-621.85-vsphere-esxi-ubuntu-xenial-go_agent.tgz
- path: /resource_pools/name=vms/cloud_properties?
type: replace
value:
cpu: 2
disk: 40000
ram: 4096
- path: /networks/name=default/subnets/0/cloud_properties?
type: replace
value:
name: ((network_name))
- path: /instance_groups/name=bosh/jobs/-
type: replace
value:
name: vsphere_cpi
release: bosh-vsphere-cpi
- path: /instance_groups/name=bosh/properties/director/cpi_job?
type: replace
value: vsphere_cpi
- path: /cloud_provider/template?
type: replace
value:
name: vsphere_cpi
release: bosh-vsphere-cpi
- path: /instance_groups/name=bosh/properties/vcenter?
type: replace
value:
address: ((vcenter_ip))
datacenters:
- clusters:
- ((vcenter_cluster)): {}
- AZ01:
resource_pool: Pool01
- AZ02:
resource_pool: Pool02
- AZ03:
resource_pool: Pool03
datastore_pattern: ((vcenter_ds))
disk_path: ((vcenter_disks))
name: ((vcenter_dc))
persistent_datastore_pattern: ((vcenter_ds))
template_folder: ((vcenter_templates))
vm_folder: ((vcenter_vms))
password: ((vcenter_password))
user: ((vcenter_user))
- path: /cloud_provider/properties/vcenter?
type: replace
value:
address: ((vcenter_ip))
datacenters:
- clusters:
- ((vcenter_cluster)): {}
- AZ01:
resource_pool: Pool01
- AZ02:
resource_pool: Pool02
- AZ03:
resource_pool: Pool03
datastore_pattern: ((vcenter_ds))
disk_path: ((vcenter_disks))
name: ((vcenter_dc))
persistent_datastore_pattern: ((vcenter_ds))
template_folder: ((vcenter_templates))
vm_folder: ((vcenter_vms))
password: ((vcenter_password))
user: ((vcenter_user))
where vcenter_ds: node1-datastore|node2-datastore|node3-datastore
bosh cloud-config looks following:
azs:
- cloud_properties:
datacenters:
- clusters:
- AZ01:
resource_pool: Pool01
name: DC
name: z1
- cloud_properties:
datacenters:
- clusters:
- AZ02:
resource_pool: Pool02
name: DC
name: z2
- cloud_properties:
datacenters:
- clusters:
- AZ03:
resource_pool: Pool03
name: DC
name: z3