Velero - not able to restore PVC - amazon-web-services

Question

While trying to restore the EBS volumes from the snapshot it returns status as lost. we are using AWS KMS CMK keys with policy having kms* permission. The backup operation went fine.. the restore operation is able to restore all k8s resources expect the PVC.
k get pvc -n nginx-example
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nginx-logs Lost pvc-bda55207-a1e5-11ea-b7e6-02b82f6b7f4e 0 gp2-encrypt 4m22s
k get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-bda55207-a1e5-11ea-b7e6-02b82f6b7f4e 1Gi RWO Retain Released nginx-example/nginx-logs gp2-encrypt 33m
We noticed the UID of PV and PVC are not matching after the PVC is restored.
The service account used by velero pod has below policy as
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ec2:DeleteSnapshot",
"kms:Decrypt",
"ec2:CreateTags",
"kms:GenerateDataKeyWithoutPlaintext",
"s3:ListBucket",
"kms:GenerateDataKeyPairWithoutPlaintext",
"ec2:DescribeSnapshots",
"kms:GenerateDataKeyPair",
"kms:ReEncryptFrom",
"ec2:CreateVolume",
"s3:ListMultipartUploadParts",
"s3:PutObject",
"s3:GetObject",
"s3:AbortMultipartUpload",
"ec2:DescribeVolumes",
"ec2:CreateSnapshot",
"kms:GenerateDataKey",
"kms:ReEncryptTo",
"s3:DeleteObject"
],
"Resource": "*"
}
]
}
we are using the below yaml to define storageclass and PVC
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp2-encrypt
parameters:
type: gp2
encrypted: "true"
fsType: ext4
kmsKeyId: arn:aws:kms:us-east-XXXXXX
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-logs
namespace: nginx-example
labels:
app: nginx
spec:
storageClassName: gp2-encrypt
accessModes:
- ReadWriteOnce
resources:
requests:
storage: [50Mi]
Below are logs from velero pods..
> time="2020-05-29T19:59:04Z" level=info msg="Starting restore of backup
> cluster-addons/nginx-backup-5" logSource="pkg/restore/restore.go:394"
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T19:59:04Z" level=info msg="Restoring cluster level
> resource 'persistentvolumes'" logSource="pkg/restore/restore.go:779"
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T19:59:04Z" level=info msg="Getting client for /v1,
> Kind=PersistentVolume" logSource="pkg/restore/restore.go:821"
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Restoring resource
> 'persistentvolumeclaims' into namespace 'nginx-example'"
> logSource="pkg/restore/restore.go:777"
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Getting client for /v1,
> Kind=PersistentVolumeClaim" logSource="pkg/restore/restore.go:821"
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Executing item action for
> persistentvolumeclaims" logSource="pkg/restore/restore.go:1030"
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Executing
> AddPVFromPVCAction" cmd=/velero
> logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Adding PV
> pvc-bda55207-a1e5-11ea-b7e6-02b82f6b7f4e as an additional item to
> restore" cmd=/velero
> logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Skipping
> persistentvolumes/pvc-bda55207-a1e5-11ea-b7e6-02b82f6b7f4e because
> it's already been restored." logSource="pkg/restore/restore.go:910"
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Executing item action for
> persistentvolumeclaims" logSource="pkg/restore/restore.go:1030"
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Executing
> ChangeStorageClassAction" cmd=/velero
> logSource="pkg/restore/change_storageclass_action.go:63"
> pluginName=velero restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Attempting to restore
> PersistentVolumeClaim: nginx-logs"
> logSource="pkg/restore/restore.go:1136"
> restore=cluster-addons/nginx-backup-5-20200529155858
> time="2020-05-29T20:09:04Z" level=info msg="Done executing
> ChangeStorageClassAction" cmd=/velero
> logSource="pkg/restore/change_storageclass_action.go:74"
> pluginName=velero restore=cluster-addons/nginx-backup-5-20200529155858
>
> The cloudtrail does not have much information. Would you please let us
> know any additional. settings needed here?