Kubernetes PetSet - PersistentVolumeClaim "ProvisioningFailed" on AWS - amazon-web-services

Question

I’m in the process of migrating ElasticSearch to a Kubernete’s PetSet but there’s a problem when provisioning the PersistentVolumeClaim.
I have two volumes in AWS “vol-84094e30” “vol-87e59f33”. Both are 5 GiB, 100/3000 IOPS, us-west-2a Availability Zone and are volume type gp2.
I have a StorageClass definition for the volume type:
{
"kind": "StorageClass",
"apiVersion": "apps/v1alpha1",
"metadata": {
"name": "fast"
},
"provisioner": "kubernetes.io/aws-ebs",
"parameters": {
"type": "gp2",
"zone": "us-west-2a"
}
}
… two PersistentVolume definitions:
{
"kind": "PersistentVolume",
"apiVersion": "v1",
"metadata": {
"name": "es-persistent-vol",
"labels": {
"type": "amazonEBS"
}
},
"spec": {
"capacity": {
"storage": "5Gi"
},
"accessModes": [
"ReadWriteOnce"
],
"awsElasticBlockStore": {
"volumeID": "vol-87e59f33",
"fsType": "ext4"
}
}
}
{
"kind": "PersistentVolume",
"apiVersion": "v1",
"metadata": {
"name": "es-persistent-vol2",
"labels": {
"type": "amazonEBS"
}
},
"spec": {
"capacity": {
"storage": "5Gi"
},
"accessModes": [
"ReadWriteOnce"
],
"awsElasticBlockStore": {
"volumeID": "vol-84094e30",
"fsType": "ext4"
}
}
}
… an Elastic Search service:
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"name": "elasticsearch-logging",
"namespace": "kube-system",
"labels": {
"k8s-app": "elasticsearch-logging",
"kubernetes.io/name": "Elasticsearch"
}
},
"spec": {
"ports": [
{
"port": 9200,
"protocol": "TCP",
"targetPort": "db"
}
],
"selector": {
"k8s-app": "elasticsearch-logging"
}
}
}
… and an Elastic Search PetSet:
{
"apiVersion": "apps/v1alpha1",
"kind": "PetSet",
"metadata": {
"name": "elasticsearch-logging-v1",
"namespace": "kube-system",
"labels": {
"k8s-app": "elasticsearch-logging",
"version": "v1",
"kubernetes.io/cluster-service": "true"
}
},
"spec": {
"serviceName": "elasticsearch-logging-v1",
"replicas": 2,
"template": {
"metadata": {
"annotations": {
"pod.beta.kubernetes.io/initialized": "true"
},
"labels": {
"app": "elasticsearch-data"
}
},
"spec": {
"containers": [
{
"image": "gcr.io/google_containers/elasticsearch:v2.4.1",
"name": "elasticsearch-logging",
"resources": {
"limits": {
"cpu": "1000m"
},
"requests": {
"cpu": "100m"
}
},
"ports": [
{
"containerPort": 9200,
"name": "db",
"protocol": "TCP"
},
{
"containerPort": 9300,
"name": "transport",
"protocol": "TCP"
}
],
"volumeMounts": [
{
"name": "es-persistent-storage",
"mountPath": "/data"
}
]
}
]
}
},
"volumeClaimTemplates": [
{
"metadata": {
"name": "es-persistent-storage",
"annotations": {
"volume.beta.kubernetes.io/storage-class": "fast"
},
"labels": {
"type": "amazonEBS"
}
},
"spec": {
"accessModes": [
"ReadWriteOnce"
],
"resources": {
"requests": {
"storage": "5Gi"
}
}
}
}
]
}
}
When I create all these the PersistentStorageVolume (defined in PetSet) is unable to provision the volumes (ProvisioningFailed - no volume plugin matched). I think it may be AWS specific, I’ve looked on various forums for (what I thought were) the same issue, but were not applicable to AWS.
Any help is much appreciated.
Here are the kubectl describe outputs:
$ kubectl describe pv
Name: es-persistent-vol
Labels: type=amazonEBS
Status: Available
Claim:
Reclaim Policy: Retain
Access Modes: RWO
Capacity: 5Gi
Message:
Source:
Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)
VolumeID: vol-87e59f33
FSType: ext4
Partition: 0
ReadOnly: false
No events.
Name: es-persistent-vol2
Labels: type=amazonEBS
Status: Available
Claim:
Reclaim Policy: Retain
Access Modes: RWO
Capacity: 5Gi
Message:
Source:
Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)
VolumeID: vol-84094e30
FSType: ext4
Partition: 0
ReadOnly: false
No events.
$ kubectl describe pvc
Name: es-persistent-storage-elasticsearch-logging-v1-0
Namespace: kube-system
Status: Pending
Volume:
Labels: app=elasticsearch-data
Capacity:
Access Modes:
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
18s 1s 3 {persistentvolume-controller } Warning ProvisioningFailed no volume plugin matched
Name: es-persistent-storage-elasticsearch-logging-v1-1
Namespace: kube-system
Status: Pending
Volume:
Labels: app=elasticsearch-data
Capacity:
Access Modes:
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
19s 2s 3 {persistentvolume-controller } Warning ProvisioningFailed no volume plugin matched
$ kubectl describe petset
Name: elasticsearch-logging-v1
Namespace: kube-system
Image(s): gcr.io/google_containers/elasticsearch:v2.4.1
Selector: app=elasticsearch-data
Labels: k8s-app=elasticsearch-logging,kubernetes.io/cluster-service=true,version=v1
Replicas: 2 current / 2 desired
Annotations: <none>
CreationTimestamp: Wed, 02 Nov 2016 13:06:23 +0000
Pods Status: 0 Running / 1 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 {petset } Normal SuccessfulCreate pvc: es-persistent-storage-elasticsearch-logging-v1-0
1m 1m 1 {petset } Normal SuccessfulCreate pet: elasticsearch-logging-v1-0
1m 1m 1 {petset } Normal SuccessfulCreate pvc: es-persistent-storage-elasticsearch-logging-v1-1
$ kubectl describe service el
Name: elasticsearch-logging
Namespace: kube-system
Labels: k8s-app=elasticsearch-logging
kubernetes.io/name=Elasticsearch
Selector: k8s-app=elasticsearch-logging
Type: ClusterIP
IP: 192.168.157.15
Port: <unset> 9200/TCP
Endpoints: <none>
Session Affinity: None
No events.