I am very new to Kubernetes. My task is to move the existing application from Kubernetes to EKS. I am using CDK EKS Blueprints to create the cluster in AWS and have AWS secret manager to create the Kubernetes secret. I followed the same steps as given in here https://aws-quickstart.github.io/cdk-eks-blueprints/addons/secrets-store/
As mentioned on the above page I got the service account, a role in the service account to access the secret and the secret created.
Though I have a volume block, mount path for the secret and used env variables to refer the secret, I am not able to get my pod up and running. Instead it complains that the key is not found in the secret.
The reason may be because when I try to create a secret manually using the create command the Kubernetes create the secret as below.
enter image description here
But when the Kubernetes secret is created by EKS blueprints by lookingup the existing AWS secret like
secretProvider: new blueprints.LookupSecretsManagerSecretByName('test-aws-secret'),
it is creating as an encoded object.
enter image description here
Now I am not sure how to reference the nested object in the yaml. I tried many iterations, something like enter image description here. But no luck. Any help is much appreciated.
Thanks.
The value of the key field should be key1:
- name: key1-value
valueFrom:
secretKeyRef:
name: secret-test
key: key1
Including data/secret-test/ before the key name is unnecessary because Kubernetes already knows the secret name from the name field and knows to look for keys under the data field of secrets.
See Secrets for more information.
Related
I have a service account which I am trying to use across multiple pods installed in the same namespace.
One of the pods is created by Airflow KubernetesPodOperator.
The other is created via Helm through Kubernetes deployment.
In the Airflow deployment, I see the IAM role being assigned and DynamoDB tables are created, listed etc however in the second helm chart deployment (or) in a test pod (created as shown here), I keep getting AccessDenied error for CreateTable in DynamoDB.
I can see the AWS Role ARN being assigned to the service account and the service account being applied to the pod and the corresponding token file also being created, but I see AccessDenied exception.
arn:aws:sts::1234567890:assumed-role/MyCustomRole/aws-sdk-java-1636152310195 is not authorized to perform: dynamodb:CreateTable on resource
ServiceAccount
Name: mypipeline-service-account
Namespace: abc-qa-daemons
Labels: app.kubernetes.io/managed-by=Helm
chart=abc-pipeline-main.651
heritage=Helm
release=ab-qa-pipeline
tier=mypipeline
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/MyCustomRole
meta.helm.sh/release-name: ab-qa-pipeline
meta.helm.sh/release-namespace: abc-qa-daemons
Image pull secrets: <none>
Mountable secrets: mypipeline-service-account-token-6gm5b
Tokens: mypipeline-service-account-token-6gm5b
P.S: Both the client code created using KubernetesPodOperator and through Helm chart deployment is same i.e. same docker image. Other attributes like nodeSelector, tolerations etc, volume mounts are also same.
The describe pod output for both of them is similar with just some name and label changes.
The KubernetesPodOperator pod has QoS class as Burstable while the Helm chart ones is BestEffort.
Why do I get AccessDenied in Helm deployment but not in KubernetesPodOperator? How to debug this issue?
Whenever we get an AccessDenied exception, there can be two possible reasons:
You have assigned the wrong role
The assigned role doesn't have necessary permissions
In my case, latter is the issue. The permissions assigned to particular role can be sophisticated i.e. they can be more granular.
For example, in my case, the DynamoDB tables which the role can create/describe is limited to only those that are starting with a specific prefix but not all the DynamoDB tables.
So, it is always advisable to check the IAM role permissions whenever
you get this error.
As stated in the question, be sure to check the service account using the awscli image.
Keep in mind that, there is a credential provider chain used in AWS SDKs which determines the credentials to be used by the application. In most cases, the DefaultAWSCredentialsProviderChain is used and its order is given below. Ensure that the SDK is picking up the intended provider (in our case it is WebIdentityTokenCredentialsProvider)
super(new EnvironmentVariableCredentialsProvider(),
new SystemPropertiesCredentialsProvider(),
new ProfileCredentialsProvider(),
WebIdentityTokenCredentialsProvider.create(),
new EC2ContainerCredentialsProviderWrapper());
Additionally, you might also want to set the AWS SDK classes to DEBUG mode in your logger to see which credentials provider is being picked up and why.
To check if the service account is applied to a pod, describe it and check if the AWS environment variables are set to it like AWS_REGION, AWS_DEFAULT_REGION, AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE.
If not, then check your service account if it has the AWS annotation eks.amazonaws.com/role-arn by describing that service account.
ON GCP,I need to use 2 GCP project; One is for web-application, the other is for storing secrets for web-application ( which structure comes from google's repository
As written in README, I'll store secrets using GCP Secret Manager
This project is allocated for GCP Secret Manager for secrets shared by the organization.
procedure I'm planning
prj-secret : create secrets in secrets-manager
prj-application : read secret using kubernetes-external-secrets
in prj-application I want to use workload identity , because I don't want to use as serviceaccountkey doc saying
What I did
create cluser with -workload-pool=project-id.svc.id.goog option
helm install kubernetes-external-secrets
[skip] kubectl create namespace k8s-namespace ( because I install kubernetes-external-secrets on default name space)
[skip] kubectl create serviceaccount --namespace k8s-namespace ksa-name ( because I use default serviceaccount with exist by default when creating GKE)
create google-service-account with module "workload-identity
module "workload-identity" {
source = "github.com/terraform-google-modules/terraform-google-kubernetes-engine//modules/workload-identity"
use_existing_k8s_sa = true
cluster_name = var.cluster_name
location = var.cluter_locaton
k8s_sa_name = "external-secrets-kubernetes-external-secrets"
name = "external-secrets-kubernetes"
roles = ["roles/secretmanager.admin","roles/secretmanager.secretAccessor"]
project_id = var.project_id #it is prj-aplication's project_id
}
kubernetes_serviceaccount called external-secrets-kubernetes-external-secrets was already created when installing kubernetes-external-secrets with helm. and it bind k8s_sa_name &' external-secrets-kubernetes#my-project-id.iam.gserviceaccount.com, which has ["roles/secretmanager.admin","roles/secretmanager.secretAccessor"].
create externalsecret and apply
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
name: external-key-test
spec:
backendType: gcpSecretsManager
projectId: my-domain
data:
- key: key-test
name: password
result
I got permission problem
ERROR, 7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/project-id/secrets/external-key-test/versions/latest' (or it may not exist).
I already checked that, if I prj-secret and prj-application is same project, it worked.
So what I thought is,
kubernetes serviceaccount (in prj-secret) & google serviceaccount (in prj-application) cannot bind correctly.
I wonder if someone know
workload-identity works only in same project or not
if it is, how can I get secret data from different project
Thank you.
You have an issue in your role binding I think. When you say this:
kubernetes_serviceaccount called external-secrets-kubernetes-external-secrets was already created when installing kubernetes-external-secrets with helm. and it bind k8s_sa_name &' external-secrets-kubernetes#my-project-id.iam.gserviceaccount.com, which has ["roles/secretmanager.admin","roles/secretmanager.secretAccessor"].
It's unclear.
external-secrets-kubernetes#my-project-id.iam.gserviceaccount.com, is created on which project? I guess in prj-application, but not clear.
I take the assumption (with the name and the link with the cluster) that the service account is created in the prj-application. you grant the role "roles/secretmanager.admin","roles/secretmanager.secretAccessor" on which resource?
On the IAM page of the prj-application?
On the IAM page of the prj-secret?
On the secretId of the secret in the prj-secret?
If you did the 1st one, it's the wrong binding, the service account can only access to the secret of the prj-application, and not these of prj-secret.
Note, if you only need to access the secret, don't grand the admin role, only the accessor is required.
I currently have the following problem. I am creating a Template in which I specify a ServiceAccount adn a RoleBinding. Openshift Creates a Token on its own and stores it in a secret with the name [service-account-name]-[a-z,1-9{5}]. Now I want to pass that secret on to an env Variable (as it will be consumed by another config in that container that can process env variables)
Now you can easily use env variables like
env:
- name: something
valueFrom:
secretKeyRef:
name: someKey
key: someValue
But now I've got the problem, that there is a secret, but I don't know the exact name as part of it is random. Now my question is
Is there a way to use the contents of a secret of a serviceaccount in a template?
You can check your secrets by running
kubectl get secret and then view more by running kubectl describe secret mysecret You will need to decode it to view it (I do not have experience with OpenShift). You can also use them as Environment Variables as explained here.
As for ServiceAccount and the token you can use it inside a container as specified in the OpenShift documentation
A file containing an API token for a pod’s service account is
automatically mounted at
/var/run/secrets/kubernetes.io/serviceaccount/token.
I think you could add commands from the documentation to the Pod Template into command: section similar to this example. Also you can find more about using secrets here.
I'm trying to use the aws-sdk-go in my application. It's running on EC2 instance. Now in the Configuring Credentials of the doc,https://docs.aws.amazon.com/sdk-for-go/api/, it says it will look in
*Environment Credentials - Set of environment variables that are useful when sub processes are created for specific roles.
* Shared Credentials file (~/.aws/credentials) - This file stores your credentials based on a profile name and is useful for local development.
*EC2 Instance Role Credentials - Use EC2 Instance Role to assign credentials to application running on an EC2 instance. This removes the need to manage credential files in production.`
Wouldn't the best order be the reverse order? But my main question is do I need to ask the instance if it has a role and then use that to set up the credentials if it has a role? This is where I'm not sure of what I need to do and how.
I did try a simple test of creating a empty config with essentially only setting the region and running it on the instance with the role and it seems to have "worked" but in this case, I am not sure if I need to explicitly set the role or not.
awsSDK.Config{
Region: awsSDK.String(a.region),
MaxRetries: awsSDK.Int(maxRetries),
HTTPClient: http.DefaultClient,
}
I just want to confirm is this the proper way of doing it or not. My thinking is I need to do something like the following
role = use sdk call to get role on machine
set awsSDK.Config { Credentials: credentials form of role,
...
}
issue service command with returned client.
Any more docs/pointers would be great!
I have never used the go SDK, but the AWS SDKs I used automatically use the EC2 instance role if credentials are not found from any other source.
Here's an AWS blog post explaining the approach AWS SDKs follow when fetching credentials: https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/. In particular, see this:
If you use code like this, the SDKs look for the credentials in this
order:
In environment variables. (Not the .NET SDK, as noted earlier.)
In the central credentials file (~/.aws/credentials or
%USERPROFILE%.awscredentials).
In an existing default, SDK-specific
configuration file, if one exists. This would be the case if you had
been using the SDK before these changes were made.
For the .NET SDK, in the SDK Store, if it exists.
If the code is running on an EC2
instance, via an IAM role for Amazon EC2. In that case, the code gets
temporary security credentials from the instance metadata service; the
credentials have the permissions derived from the role that is
associated with the instance.
In my apps, when I need to connect to AWS resources, I tend to use an access key and secret key that have specific predefined IAM roles. Assuming I have those two, the code I use to create a session is:
awsCredentials := credentials.NewStaticCredentials(awsAccessKeyID, awsSecretAccessKey, "")
awsSession = session.Must(session.NewSession(&aws.Config{
Credentials: awsCredentials,
Region: aws.String(awsRegion),
}))
When I use this, the two keys are usually specified as either environment variables (if I deploy to a docker container).
A complete example: https://github.com/retgits/flogo-components/blob/master/activity/amazons3/activity.go
I have been using kops to build the kubernetes cluster which is really easy-to-go tool, however i am unable to find a way-out on how to change the admin password which is auto-generated while the cluster is being created.
As it is currently not possible to modify or delete + create secrets of type "Secret" with the CLI you have to modify them directly in the kops s3 bucket.
They are stored /clustername/secrets/ and contain the secret as a base64 encoded string. To change the secret base64 encode it with:
echo -n 'MY_SECRET' | base64
and replace it in the "Data" field of the file. Verifiy your change with get secrets and perform a rolling update of the cluster
Seen in Managing secrets Kops’s documentation: Workaround for changing secrets with type "Secret"