Multiple Volume mounts in EKS pod - amazon-web-services

I running AWS EKS and want 1 of the container to share multiple mounts to the same.
I created 1 EFS , 2 PV and 2 PVC
kind: Pod
metadata:
name: app1
spec:
containers:
- name: app1
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date -u) >> /data/out1.txt; sleep 5; done"]
volumeMounts:
- name: data
mountPath: /data
- name: docket
mountPath: /docket
volumes:
- name: data
persistentVolumeClaim:
claimName: efs-data-claim
- name: docket
persistentVolumeClaim:
claimName: efs-docket-claim
And these are my PV / PVCs
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-data-pv
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
csi:
driver: efs.csi.aws.com
volumeHandle: fs-XXXXX
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: efs-data-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-sc
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-docket-pv
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
csi:
driver: efs.csi.aws.com
volumeHandle: fs-XXXXX
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: efs-docket-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-sc
resources:
requests:
storage: 5Gi
---
.. When i deploy the pod, I always get the following error
But if I go with only 1 PVC for both the mounts it is working fine.. Could anyone please let me know what is happening..

There is a 1-to-1 mapping between PhysicalVolumes and EFS storage.
If you use two different EFS with volumeHandle: fs-XXXXX and volumeHandle: fs-YYYYY it will work.
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: "logpv"
spec:
capacity:
storage: "2Gi"
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
csi:
driver: efs.csi.aws.com
volumeHandle: "fs-28147e70"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: "qmpv"
spec:
capacity:
storage: "2Gi"
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
csi:
driver: efs.csi.aws.com
volumeHandle: "fs-13c3ac4b"
---

Related

0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available

I am deploying a jenkins on one master one node Kubernetes cluster, iam getting error when i try to do Dynamic Volume Provisioning. not sure what went wrong. please help.
my storageclass file
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
volumeBindingMode: Immediate
my PVC file
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pvc
labels:
type: amazonEBS
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 40Gi
storageClassName: standard
volumeMode: Filesystem
Deployment file
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
spec:
selector:
matchLabels:
app: jcasc
replicas: 1
template:
metadata:
labels:
app: jcasc
spec:
volumes:
- name: jenkins-pvc
persistentVolumeClaim:
claimName: jenkins-pvc
containers:
- name: jenkins
image: jenkins:latest
ports:
- containerPort: 8080
volumeMounts:
- name: jenkins-pvc
mountPath: "/var/jenkins_home"
Find this troubleshooting doc for Fixing Pod Has Unbound Immediate PersistentVolumeClaims Error and also
For dynamic provisioning you can see this doc.
For pv doc

PV & PVC with EKS Cluster

After kubectl apply -f pvc.yaml the below yaml file, I can able to find the mount path /var/local/pvctest inside the container that has been created. But, the host path /var/local/pvctest in the worker node is not created.
I'm new to PV & PVC with EKS and any help to fix this issue is much appreciated!
kind: Deployment
apiVersion: apps/v1
metadata:
name: pvctest
labels:
alias: pvctest
spec:
selector:
matchLabels:
alias: pvctest
replicas: 1
template:
metadata:
labels:
alias: pvctest
spec:
containers:
- name: pvctest
image: neo4j
ports:
- containerPort: 7474
- containerPort: 7687
volumeMounts:
- name: testpv
mountPath: /var/local/pvctest
volumes:
- name: testpv
persistentVolumeClaim:
claimName: pvctest-claim
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: pvtest
labels:
type: local
spec:
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /var/local/pvctest
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvctest-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
PersistentVolume with hostPath requires the directory on the host to be pre-created. If you want the directory to be created automatically for you:
...
containers:
- name: pvctest
image: neo4j
...
volumeMounts:
- name: testpv
mountPath: /var/local/pvctest
volumes:
- name: testpv
hostPath:
path: /data
type: DirectoryOrCreate
PV/PVC is actually optonal for hostPath.

Kubernetes pod failing with Invalid Volume Zone mismatch

I have a jenkins service deployed in EKS v 1.16 using helm chart. The PV and PVC had been accidentally deleted so I have recreated the PV and PVC as follows:
Pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-vol
spec:
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
fsType: ext4
volumeID: aws://us-east-2b/vol-xxxxxxxx
capacity:
storage: 120Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: jenkins-ci
namespace: ci
persistentVolumeReclaimPolicy: Retain
storageClassName: gp2
volumeMode: Filesystem
status:
phase: Bound
PVC.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-ci
namespace: ci
spec:
storageClassName: gp2
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 120Gi
volumeMode: Filesystem
volumeName: jenkins-vol
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 120Gi
phase: Bound
kubectl describe sc gp2
Name: gp2
IsDefaultClass: Yes
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2","namespace":""},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner: kubernetes.io/aws-ebs
Parameters: fsType=ext4,type=gp2
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
The issue I'm facing is that the pod is not running when its scheduled on a node in a different availability zone than the EBS volume? How can I fix this
Add a nodeSelector to your deployment file, which will match it to a node in the needed availability zone (in your case us-east-2b):
nodeSelector:
topology.kubernetes.io/zone: us-east-2b
Add following labels to the PersistentVolume.
labels:
failure-domain.beta.kubernetes.io/region: us-east-2b
failure-domain.beta.kubernetes.io/zone: us-east-2b
example:
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.beta.kubernetes.io/gid: "1000"
labels:
failure-domain.beta.kubernetes.io/region: us-east-2b
failure-domain.beta.kubernetes.io/zone: us-east-2b
name: test-pv-1
spec:
accessModes:
- ReadWriteOnce
csi:
driver: ebs.csi.aws.com
fsType: xfs
volumeHandle: vol-0d075fdaa123cd0e
capacity:
storage: 100Gi
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
With the above labels the pod will automatically run in the same AZ where the volume is.

AWS EBS Volume with kubernetes issue

Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-2a/vol-011d7bb42da888b82 /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv
Output: Running scope as unit run-20000.scope.
mount: /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-2a/vol-011d7bb42da888b82 does not exist.
Warning FailedAttachVolume 7s (x6 over 23s) attachdetach-controller AttachVolume.NewAttacher failed for volume "pv" : Failed to get AWS Cloud Provider. GetCloudProvider returned <nil> instead
Warning FailedMount 7s kubelet, ip-172-31-3-191.us-east-2.compute.internal MountVolume.SetUp failed for volume "pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-2a/vol-011d7bb42da888b82 /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv
Output: Running scope as unit run-20058.scope.
mount: /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-2a/vol-011d7bb42da888b82 does not exist.
I have Kubernetes cluster running in same availability zone where EBS volumes is available
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2-retain
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
volumeBindingMode: Immediate
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: asvignesh
name: _PVC_
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: gp2-retain
volumeMode: Filesystem
volumeName: _PV_
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: _PV_
spec:
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
fsType: xfs
volumeID: aws://us-east-1a/vol-xxxxxxxxx
capacity:
storage: 10Gi
persistentVolumeReclaimPolicy: Retain
storageClassName: gp2-retain
volumeMode: Filesystem
---
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
app: asvignesh
spec:
ports:
- port: 3306
targetPort: 3306
selector:
app: asvignesh
tier: mysql
clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
labels:
app: asvignesh
spec:
selector:
matchLabels:
app: asvignesh
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: asvignesh
tier: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: _PVC_
Are you running a cluster on managed K8s or bare metal?
Because
On the Kubernetes side of the house, you’ll need to make sure that the
--cloud-provider=aws command-line flag is present for the API server, controller manager, and every Kubelet in the cluster.
Document to refer : https://blog.scottlowe.org/2018/09/28/setting-up-the-kubernetes-aws-cloud-provider/
Example YAML
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
Ref : https://faun.pub/mysql-pod-with-persistent-ebs-volume-in-eks-150af369ff94

How to remove sub path folder of pvc when call delete pods, ... with client-go k8s

I using k8s/client-go library to control and develop my application (https://github.com/kubernetes/client-go).
I have an issue when use sub-path of persistent volume claim.
Example, I'm having two pod and mount data of each container to 2 subpath ORG1/DIR1 and ORG2/DIR2 on persistent volume claim (efs file), detail in blow:
apiVersion: v1
kind: Pod
metadata:
name: my-lamp-site
spec:
containers:
- name: mysql
image: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: "rootpasswd"
volumeMounts:
- mountPath: /var/lib/mysql
name: site-data
subPath: ORG1/DIR1
- name: php
image: php:7.0-apache
volumeMounts:
- mountPath: /var/www/html
name: site-data
subPath: ORG1/DIR2
volumes:
- name: site-data
persistentVolumeClaim:
claimName: hpc-vinhha-test
And when I call to delete this pod, currently k8s only delete pod, and core lib not delete data of pod on persistent volume claim. So, data of PVC will become garbage and become bigger and bigger.
I want to delele all data in sub path ORG1/DIR1 and ORG1/DIR2 when pods deleted.
This is file yaml of pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"efs-claim","namespace":"default"},"spec":{"accessModes":["ReadWriteMany"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"efs-sc"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2020-07-10T04:02:51Z"
finalizers:
- kubernetes.io/pvc-protection
name: efs-claim
namespace: default
resourceVersion: "887409"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/efs-claim
uid: ab66c2f7-744c-4d6f-a508-2bc90f0b1897
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: efs-sc
volumeMode: Filesystem
volumeName: efs-pv-shared
status:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
phase: Bound
So, can you help me with this problem. Because I'm a newbie in k8s and aws-efs. So, I don't have much experience about it :(
Thank so much.