EKS aws-ebs-csi-driver failed to mount volume - amazon-web-services

Question

We are using EKS varsion v1.17.17-eks-087e67
With installed aws-ebs-csi-driver components versions:
aws-ebs-csi-driver:v1.1.3
csi-provisioner:v2.1.1
csi-attacher:v3.1.0
csi-snapshotter:v3.0.3
csi-resizer:v1.0.0
When we create PVC driver could not mount volume. As I can see, AWS volume continuously creating and deleting (from cloud trail):
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAV5QH66QYOM4FMMPFI:1631165222580844502",
"arn": "arn:aws:sts::XXXXXXXXXX:assumed-role/EKSEBSCSIServiceRole-cluster01-eks-external-sandbox/XXXXXXXXXXXXXXXXXXXXXXXX",
"accountId": "XXXXXXXXXX",
"accessKeyId": "ASIAV5QH66QYFCKRZG43",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAV5QH66QYOM4FMMPFI",
"arn": "arn:aws:iam::XXXXXXXXXX:role/eks/EKSEBSCSIServiceRole-cluster01-eks-external-sandbox",
"accountId": "XXXXXXXXXX",
"userName": "EKSEBSCSIServiceRole-cluster01-eks-external-sandbox"
},
"webIdFederationData": {
"federatedProvider": "arn:aws:iam::XXXXXXXXXX:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXX",
"attributes": {}
},
"attributes": {
"creationDate": "2021-09-09T05:27:03Z",
"mfaAuthenticated": "false"
}
}
},
"eventTime": "2021-09-09T06:11:12Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "CreateVolume",
"awsRegion": "eu-central-1",
"sourceIPAddress": "18.157.68.62",
"userAgent": "aws-sdk-go/1.35.37 (go1.15.6; linux; amd64) exec-env/aws-ebs-csi-driver-v1.1.3",
"requestParameters": {
"size": "8",
"zone": "eu-central-1a",
"volumeType": "gp2",
"encrypted": true,
"tagSpecificationSet": {
"items": [
{
"resourceType": "volume",
"tags": [
{
"key": "ebs.csi.aws.com/cluster",
"value": "true"
},
{
"key": "CSIVolumeName",
"value": "pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2"
},
{
"key": "kubernetes.io/created-for/pv/name",
"value": "pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2"
},
{
"key": "kubernetes.io/created-for/pvc/name",
"value": "data-postgres-postgresql-0"
},
{
"key": "kubernetes.io/created-for/pvc/namespace",
"value": "default"
}
]
}
]
}
},
"responseElements": {
"requestId": "5404a63c-a8d6-4bfa-b18f-ce1fba1060ee",
"volumeId": "vol-032b5c6671123cc35",
"size": "8",
"zone": "eu-central-1a",
"status": "creating",
"createTime": 1631167872000,
"volumeType": "gp2",
"iops": 100,
"encrypted": true,
"masterEncryptionKeyId": "arn:aws:kms:eu-central-1:XXXXXXXXXX:key/ef3b2237-00c3-4fd0-b556-91cda7f7db95",
"tagSet": {
"items": [
{
"key": "ebs.csi.aws.com/cluster",
"value": "true"
},
{
"key": "CSIVolumeName",
"value": "pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2"
},
{
"key": "kubernetes.io/created-for/pv/name",
"value": "pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2"
},
{
"key": "kubernetes.io/created-for/pvc/name",
"value": "data-postgres-postgresql-0"
},
{
"key": "kubernetes.io/created-for/pvc/namespace",
"value": "default"
}
]
},
"multiAttachEnabled": false
},
"requestID": "5404a63c-a8d6-4bfa-b18f-ce1fba1060ee",
"eventID": "0941702c-119c-45fb-8c9e-6ef8918db6da",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "XXXXXXXXXX",
"eventCategory": "Management"
}
"eventTime": "2021-09-09T06:11:15Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "DeleteVolume",
"awsRegion": "eu-central-1",
"sourceIPAddress": "x.x.x.x",
"userAgent": "aws-sdk-go/1.35.37 (go1.15.6; linux; amd64) exec-env/aws-ebs-csi-driver-v1.1.3",
"errorCode": "Client.InvalidVolume.NotFound",
"errorMessage": "The volume 'vol-032b5c6671123cc35' does not exist.",
"requestParameters": {
"volumeId": "vol-032b5c6671123cc35"
},
"responseElements": null,
"requestID": "3cf2ce00-5845-436b-8470-3e1918dd24af",
"eventID": "e5fbd13c-fc72-4cc1-9468-2a928d52a186",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "XXXXXXXXXX",
"eventCategory": "Management"
}
But eventually provisioner could not find this volume
0909 06:11:12.088851 1 controller.go:1332] provision "default/data-postgres-postgresql-0" class "ebs-default": started
I0909 06:11:12.089028 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"data-postgres-postgresql-0", UID:"27fa1e04-c99d-48d2-9efa-0633ee3669d2", APIVersion:"v1", ResourceVersion:"145344106", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/data-postgres-postgresql-0"
I0909 06:11:15.565942 1 controller.go:1099] Final error received, removing PVC 27fa1e04-c99d-48d2-9efa-0633ee3669d2 from claims in progress
W0909 06:11:15.565962 1 controller.go:958] Retrying syncing claim "27fa1e04-c99d-48d2-9efa-0633ee3669d2", failure 18
E0909 06:11:15.565981 1 controller.go:981] error syncing claim "27fa1e04-c99d-48d2-9efa-0633ee3669d2": failed to provision volume with StorageClass "ebs-default": rpc error: code = Internal desc = Could not create volume "pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-032b5c6671123cc35' does not exist.
status code: 400, request id: a396c26c-71c6-4c88-8f2f-ebb3aa492447
I0909 06:11:15.566164 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"data-postgres-postgresql-0", UID:"27fa1e04-c99d-48d2-9efa-0633ee3669d2", APIVersion:"v1", ResourceVersion:"145344106", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-default": rpc error: code = Internal desc = Could not create volume "pvc-27fa1e04-c99d-48d2-9efa-0633ee3669d2": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-032b5c6671123cc35' does not exist.
status code: 400, request id: a396c26c-71c6-4c88-8f2f-ebb3aa492447
Here is the policy from AWS Role for annotated CA:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteSnapshot",
"ec2:DeleteTags",
"ec2:DeleteVolume",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInstances",
"ec2:DescribeSnapshots",
"ec2:DescribeTags",
"ec2:DescribeVolumes",
"ec2:DescribeVolumesModifications",
"ec2:DetachVolume",
"ec2:ModifyVolume"
],
"Resource": "*"
}
]
}
Here is StorageClass:
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
name: ebs-default
parameters:
csi.storage.k8s.io/fstype: ext4
encrypted: "true"
type: gp2
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
~
We are running workers in eu-central-1 region in 3 AZ

Answer 1

I wrote a document. Please do let me know if this helps.
Use this GitHub Page Link!
Follow it from Step 2 - https://github.com/parjun8840/ekscsidriver/blob/main/README.md