How to remove sub path folder of pvc when call delete pods, ... with client-go k8s - amazon-web-services

I using k8s/client-go library to control and develop my application (https://github.com/kubernetes/client-go).
I have an issue when use sub-path of persistent volume claim.
Example, I'm having two pod and mount data of each container to 2 subpath ORG1/DIR1 and ORG2/DIR2 on persistent volume claim (efs file), detail in blow:
apiVersion: v1
kind: Pod
metadata:
name: my-lamp-site
spec:
containers:
- name: mysql
image: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: "rootpasswd"
volumeMounts:
- mountPath: /var/lib/mysql
name: site-data
subPath: ORG1/DIR1
- name: php
image: php:7.0-apache
volumeMounts:
- mountPath: /var/www/html
name: site-data
subPath: ORG1/DIR2
volumes:
- name: site-data
persistentVolumeClaim:
claimName: hpc-vinhha-test
And when I call to delete this pod, currently k8s only delete pod, and core lib not delete data of pod on persistent volume claim. So, data of PVC will become garbage and become bigger and bigger.
I want to delele all data in sub path ORG1/DIR1 and ORG1/DIR2 when pods deleted.
This is file yaml of pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"efs-claim","namespace":"default"},"spec":{"accessModes":["ReadWriteMany"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"efs-sc"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2020-07-10T04:02:51Z"
finalizers:
- kubernetes.io/pvc-protection
name: efs-claim
namespace: default
resourceVersion: "887409"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/efs-claim
uid: ab66c2f7-744c-4d6f-a508-2bc90f0b1897
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: efs-sc
volumeMode: Filesystem
volumeName: efs-pv-shared
status:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
phase: Bound
So, can you help me with this problem. Because I'm a newbie in k8s and aws-efs. So, I don't have much experience about it :(
Thank so much.

Related

How to share the same folder in EFS between deployments in EKS

I have an EKS cluster with multiple deployments (microservices). I would like all of them to write logs to same folder in an EFS mount even between restarts/scaling etc. Currently it creates a folder with the persistent volume id, which breaks our requirement. Is it possible to always mount the same folder even when the persistent volume is recreated. Can it always to point to the same folder in EFS?
It currently creates folders like this:
"/logs/pvc-2exxxcxs4-0xx7-4e11-813a-65xxxxxxxx/"
Instead, I would like it to be just "/logs" or a fixed path without any dependency on pvc id/name.
Below are the current yamls:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
........
volumeMounts:
- name: dev-logs-efs
mountPath: /logs
volumes:
- name: dev-logs-efs
persistentVolumeClaim:
claimName: dev-logs-efs-pvc
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: dev-logs-efs-sc
provisioner: efs.csi.aws.com
parameters:
provisioningMode: efs-ap
fileSystemId: fs-xxxxxxxxxxx
basePath: "/logs"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dev-logs-efs-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: dev-logs-efs-sc
resources:
requests:
storage: 5Gi

0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available

I am deploying a jenkins on one master one node Kubernetes cluster, iam getting error when i try to do Dynamic Volume Provisioning. not sure what went wrong. please help.
my storageclass file
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
volumeBindingMode: Immediate
my PVC file
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pvc
labels:
type: amazonEBS
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 40Gi
storageClassName: standard
volumeMode: Filesystem
Deployment file
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
spec:
selector:
matchLabels:
app: jcasc
replicas: 1
template:
metadata:
labels:
app: jcasc
spec:
volumes:
- name: jenkins-pvc
persistentVolumeClaim:
claimName: jenkins-pvc
containers:
- name: jenkins
image: jenkins:latest
ports:
- containerPort: 8080
volumeMounts:
- name: jenkins-pvc
mountPath: "/var/jenkins_home"
Find this troubleshooting doc for Fixing Pod Has Unbound Immediate PersistentVolumeClaims Error and also
For dynamic provisioning you can see this doc.
For pv doc

PV & PVC with EKS Cluster

After kubectl apply -f pvc.yaml the below yaml file, I can able to find the mount path /var/local/pvctest inside the container that has been created. But, the host path /var/local/pvctest in the worker node is not created.
I'm new to PV & PVC with EKS and any help to fix this issue is much appreciated!
kind: Deployment
apiVersion: apps/v1
metadata:
name: pvctest
labels:
alias: pvctest
spec:
selector:
matchLabels:
alias: pvctest
replicas: 1
template:
metadata:
labels:
alias: pvctest
spec:
containers:
- name: pvctest
image: neo4j
ports:
- containerPort: 7474
- containerPort: 7687
volumeMounts:
- name: testpv
mountPath: /var/local/pvctest
volumes:
- name: testpv
persistentVolumeClaim:
claimName: pvctest-claim
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: pvtest
labels:
type: local
spec:
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /var/local/pvctest
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvctest-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
PersistentVolume with hostPath requires the directory on the host to be pre-created. If you want the directory to be created automatically for you:
...
containers:
- name: pvctest
image: neo4j
...
volumeMounts:
- name: testpv
mountPath: /var/local/pvctest
volumes:
- name: testpv
hostPath:
path: /data
type: DirectoryOrCreate
PV/PVC is actually optonal for hostPath.

AWS EBS Volume with kubernetes issue

Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-2a/vol-011d7bb42da888b82 /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv
Output: Running scope as unit run-20000.scope.
mount: /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-2a/vol-011d7bb42da888b82 does not exist.
Warning FailedAttachVolume 7s (x6 over 23s) attachdetach-controller AttachVolume.NewAttacher failed for volume "pv" : Failed to get AWS Cloud Provider. GetCloudProvider returned <nil> instead
Warning FailedMount 7s kubelet, ip-172-31-3-191.us-east-2.compute.internal MountVolume.SetUp failed for volume "pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-2a/vol-011d7bb42da888b82 /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv
Output: Running scope as unit run-20058.scope.
mount: /var/lib/kubelet/pods/2e47e8b4-4755-46d6-9bc4-461ea02a6cb9/volumes/kubernetes.io~aws-ebs/pv: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-2a/vol-011d7bb42da888b82 does not exist.
I have Kubernetes cluster running in same availability zone where EBS volumes is available
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2-retain
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
volumeBindingMode: Immediate
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: asvignesh
name: _PVC_
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: gp2-retain
volumeMode: Filesystem
volumeName: _PV_
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: _PV_
spec:
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
fsType: xfs
volumeID: aws://us-east-1a/vol-xxxxxxxxx
capacity:
storage: 10Gi
persistentVolumeReclaimPolicy: Retain
storageClassName: gp2-retain
volumeMode: Filesystem
---
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
app: asvignesh
spec:
ports:
- port: 3306
targetPort: 3306
selector:
app: asvignesh
tier: mysql
clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
labels:
app: asvignesh
spec:
selector:
matchLabels:
app: asvignesh
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: asvignesh
tier: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: _PVC_
Are you running a cluster on managed K8s or bare metal?
Because
On the Kubernetes side of the house, you’ll need to make sure that the
--cloud-provider=aws command-line flag is present for the API server, controller manager, and every Kubelet in the cluster.
Document to refer : https://blog.scottlowe.org/2018/09/28/setting-up-the-kubernetes-aws-cloud-provider/
Example YAML
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
Ref : https://faun.pub/mysql-pod-with-persistent-ebs-volume-in-eks-150af369ff94

Google Cloud, Kubernetes and Volumes

I'm new to GCE and K8s and I'm trying to figure out my first deployment, but I get an error with my volumes:
Failed to attach volume "pv0001" on node "xxxxx" with: GCE persistent disk not found: diskName="pd-disk-1" zone="europe-west1-b"
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "xxx". list of unattached/unmounted volumes=[registrator-claim0]
This is my storage yaml:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv0001
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
gcePersistentDisk:
fsType: ext4
pdName: pd-disk-1
This is my Claim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: registrator-claim0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
status: {}
This is my Deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: consul
spec:
replicas: 1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
service: consul
spec:
restartPolicy: Always
containers:
- name: consul
image: eu.gcr.io/xxxx/consul
ports:
- containerPort: 8300
protocol: TCP
- containerPort: 8400
protocol: TCP
- containerPort: 8500
protocol: TCP
- containerPort: 53
protocol: UDP
env:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
args:
- -server
- -bootstrap
- -advertise=$(MY_POD_IP)
- name: registrator
args:
- -internal
- -ip=192.168.99.101
- consul://localhost:8500
image: eu.gcr.io/xxxx/registrator
volumeMounts:
- mountPath: /tmp/docker.sock
name: registrator-claim0
volumes:
- name: registrator-claim0
persistentVolumeClaim:
claimName: registrator-claim0
status: {}
What am I doing wrong? Figuring out K8s and GCE isn't that easy. These errors are not exactly helping. Hope someone can help me.
you've to create the actual storage before you define the PV, this can be done with sth like:
# make sure you're in the right zone
$ gcloud config set compute/europe-west1-b
# create the disk
$ gcloud compute disks create --size 10GB pd-disk-1
Once thats available you can create the PV and the PVC