Difficulty using Kubernetes Persistent Volume Claims with Amazon EBS - amazon-web-services

Question

I'm attempting to follow the instructions at https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry to add a private docker registry to Kubernetes, but the pod created by the rc isn't able to mount the persistent volume claim.
First I'm creating a volume on EBS like so:
aws ec2 create-volume --region us-west-1 --availability-zone us-west-1a --size 32 --volume-type gp2
(us-west-1a is also the availability zone that all of my kube minions are running in.)
Then I create a persistent volume like so:
kind: PersistentVolume
apiVersion: v1
metadata:
name: kube-system-kube-registry-pv
labels:
kubernetes.io/cluster-service: "true"
spec:
capacity:
storage: 30Gi
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
volumeID: vol-XXXXXXXX
fsType: ext4
And a claim on the persistent volume like so:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: kube-registry-pvc
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
The replication controller is specified like so:
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
namespace: kube-system
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-registry
version: v0
template:
metadata:
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: registry
image: registry:2
resources:
limits:
cpu: 100m
memory: 100Mi
env:
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
volumeMounts:
- name: image-store
mountPath: /var/lib/registry
ports:
- containerPort: 5000
name: registry
protocol: TCP
volumes:
- name: image-store
persistentVolumeClaim:
claimName: kube-registry-pvc
When I create the rc, It successfully starts a pod, but the pod is unable to mount the volume:
$ kubectl describe po kube-registry --namespace=kube-system
...
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
1m 1m 1 {scheduler } Scheduled Successfully assigned kube-registry-v0-3jobf to XXXXXXXXXXXXXXX.us-west-1.compute.internal
22s 22s 1 {kubelet XXXXXXXXXXXXXXX.us-west-1.compute.internal} FailedMount Unable to mount volumes for pod "kube-registry-v0-3jobf_kube-system": Timeout waiting for volume state
22s 22s 1 {kubelet XXXXXXXXXXXXXXX.us-west-1.compute.internal} FailedSync Error syncing pod, skipping: Timeout waiting for volume state
I'm able to successfully mount EBS volumes if I don't use persistent volumes and persistent volume claims. The following works without error, for example:
apiVersion: v1
kind: Pod
metadata:
name: test-ebs
spec:
containers:
- image: gcr.io/google_containers/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-ebs
name: test-volume
volumes:
- name: test-volume
awsElasticBlockStore:
volumeID: vol-XXXXXXXX
fsType: ext4
My two questions are:
Does anyone know what might be going wrong and how to fix it?
In general, where can I look for more details on errors like these? I haven't been able to find more detailed log messages anywhere, and "Unable to mount volumes...Timeout waiting for volume state" isn't terribly helpful.

Answer 1

I think I was likely running into https://github.com/kubernetes/kubernetes/issues/15073 . (If I create a new EBS volume, I first get a different failure, and then after the pod has been killed if I try to re-create the rc I get the failure I mentioned in my question.)
Also, for anyone else wondering where to look for logs, /var/log/syslog and /var/log/containers/XXX on the kubelet was where I ended up having to look.