I created two secrets one with
--data-file=-
and one without above flags,
So first was created as followes
echo -n "Demo" | gcloud secrets create First-password --data-file=-
Second was created as
echo -n "mySuperSecert" | gcloud secrets create xyz-password
Now when i try to retrieve the xyz-password , it reports
ERROR: (gcloud.secrets.versions.access) NOT_FOUND: Secret
[projects/ProjectNumber/secrets/XYZ-password] not found or has no
versions.
A screenshot is attached below, actual project number and variable name is hidden as i used closed one's name in example, so in screenshot it says XYZ.
How to access this secret now or delete it, It shows up when 'gcloud secrets list' but when actually try to get value it fails, and the way to reproduce the issue is not to specify
--data-file=-
You have 2 types of resources: the secret and the versions
In the first case, the gcloud CLI conveniently create the secret and the first version with the value your provided (data-file)
In the second one, the gcloud CLI only created the secret, not the version. Therefore, if you try to access the secret value, there is no value! It's totally file.
You can delete the secret if you want, but you won't be able to get the mySuperSecert value, because it has not been stored.
Related
I have script written in bash where I create a key with a certain name.
#!/bin/bash
project_id="y"
secret_id="x"
secret_value="test"
gcloud config set project "$project_id"
gcloud secrets create "$secret_id" --replication-policy="automatic"
I want to be able to also directly add the secret-value to my secret, so that I do not have to go into my GCP account and set it manually (which would defeat the purpose). I have seen that it is possible to attach files through the following command, however there does not seem to be a similar command for a secret value.
--data-file="/path/to/file.txt"
From https://cloud.google.com/sdk/gcloud/reference/secrets/create#--data-file:
--data-file=PATH
File path from which to read secret data. Set this to "-" to read the secret data from stdin.
So set --data-file to - and pass the value over stdin. Note, if you use echo use -n to avoid adding a newline.
echo -n $secret_value | gcloud secrets create ... --data-file=-
This seems straightforward to do that passing the Service Account key file (generated from the GCP console) by specifying the file location in the application.properties file. However, I tried all the following options:
1. spring.cloud.gcp.credentials.location=file:/home/my_user_id/mp6key.json
2. spring.cloud.gcp.credentials.location=file:src/main/resources/mp6key.json
3. spring.cloud.gcp.credentials.location=file:./main/resources/mp6key.json
4. spring.cloud.gcp.credentials.location=file:/src/main/resources/mp6key.json
It all ended up with the same error:
java.io.FileNotFoundException: /home/my_user_id/mp6key.json (No such file or directory)
Could anyone advise where I should put the key file and then how should I specify the path to the file properly?
The same programs run successfully in Ecplise with messages published and subscribed using the Pub/Sub processing from GCP (using the Project Id/Service Account key generated in GCP), but now stuck with the above issue after deployed to run on GCP.
As mentioned in the official documentation, the credentials file can be obtained from a number of different locations such as the file system, classpath, URL, etc.
for example, if the service account key file is stored in the classpath as src/main/resources/key.json, pass the following property
spring.cloud.gcp.credentials.location=classpath:key.json
if the key file is stored somewhere else in your local file system, use the file prefix in the property value
spring.cloud.gcp.credentials.location=file:<path to key file>
My line looks like this:
spring.cloud.gcp.credentials.location=file:src/main/resources/[my_json_file]
And this works.
The following also works if I put it in the root of the project directory:
spring.cloud.gcp.credentials.location=file:./[my_json_file]
Have you tried to follow this quickstart? Please, try to follow it thoughtfully and explain if you get any error finishing the quickstart.
Anyway, before running your Java script, try running on the console the following (please modify with the exact path where you store your key):
export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/mp6key.json"
How are you authenticating your credentials in your Java script?
My answer is easy: if you run you code on GCP, you don't have to use service account key file. Problem eliminated, problem solved!
More seriously, have a look on service identity. I don't know what is your current service (Compute? Function? Cloud Run?). Anyway, you can attach any service account on GCP components. Then, when you code, simply use the default credential. Automatically the component identity is loaded. No key to manage, no key to store securely, no key to rotate!
If you provide more detail on your target platform, I could provide your some guidance to achieve this.
Keep in mind that the service account key file are designed to be used by automatic apps (w/o user account involved) hosted outside GCP (on prem, other Cloud Provider, a CI/CD, Apigee,...)
UPDATE
When you use your personal account, you can also use the default credential.
Install gcloud SDK on your computer
Use the command gcloud auth application-default login
Follow the instructions
Enjoy!
If it doesn't work, get the <path> displayed after the login command and set this value in the environment variable named GOOGLE_APPLICATION_CREDENTIALS.
If you definitively want to use service account key file (which are a security issue for the previous reason, but...), you can use it locally
Either set the json key file path into the GOOGLE_APPLICATION_CREDENTIALS environment variable
Or run this command gcloud auth activate-service-account --key-file=<path to your json key file>
Provided your file is in the resources folder try
file://mp6key.json
using file:// instead of file:/ works for me at least
Im working on creating the CodeBuild which is integrated with SonarQube, So I pass values and sonar credentials directly in my Buildspec.yaml
Instead of Hardcoding directly, I tried to retrieve using the below command from SecretManager as it is mentioned in the below link. But it is not getting the correct values. it throws an error.
Command : '{{resolve:secretsmanager:MyRDSSecret:SecretString:username}}'
Link: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager
Error [ERROR] SonarQube server [{{resolve:secretsmanager:arn:aws:secretsmanager:us-east-1:********:secret:**********:SecretString:SonarURL}}] can not be reached
How I used echo '{{resolve:secretsmanager:arn:aws:secretsmanager:us-east-1:***:secret:**************:SecretString:*******}}'
Note: All the * inside my commard are the secretname and secreturl
CodeBuild just launched this today - https://aws.amazon.com/about-aws/whats-new/2019/11/aws-codebuild-adds-support-for-aws-secrets-manager/
If you wish to retrieve secrets in your buildspec file, I would recommend to use Systems Manager Parameter Store which is natively integrated with CodeBuild. Systems Manager is a service in itself, search it from the AWS Console homepage, then Paramater Store is in the bottom left of the Systems Manager Console page.
Lets assume you want to include Access Key and Secret Key in buildspec.yml file:
- Create AccessKey/SecretKey pair for a IAM User
- Save the above keys in an SSM parameter store as secure string (e.g. '/CodeBuild/AWS_ACCESS_KEY_ID' and '/CodeBuild/AWS_SECRET_ACCESS_KEY')
- Export the two values in your build environment using the following buildspec directive(s):
version: 0.2
env:
parameter-store:
AWS_ACCESS_KEY_ID_PARAM: /CodeBuild/AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY_PARAM: /CodeBuild/AWS_SECRET_ACCESS_KEY
phases:
build:
commands:
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_PARAM
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_PARAM
# Your Ansible commands below
- ansible-playbook -i hosts ec2-key.yml
[1] Build Specification Reference for CodeBuild - Build Spec Syntax - https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
The dynamic reference syntax you are trying to use only works with the Cloud Formation (CFN) service. In some cases, CFN restricts where these dynamic references to secrets will expand. Specifically, they do not expand in places where the secrets might be visible in the console, such as in EC2 metadata.
If you are trying to setup Code Build via CFN, this may be what you are seeing. However, as shariqmaws mentioned, you can use parameter store and either store your secret there or use parameter store as a pass through to secrets manager (in case you want to use secrets manager to rotate your secrets or for other reasons).
version: 0.2
env:
parameter-store:
AWS_ACCESS_KEY_ID : /terraform-cicd/AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY : /terraform-cicd/AWS_SECRET_ACCESS_KEY
AWS_CODECOMMIT_SSH_ID : /terraform-cicd/AWS_CODECOMMIT_SSH_ID
secrets-manager:
AWS_CODECOMMIT_SSH_PRIVATE: /terraform-cicd/AWS_CODECOMMIT_SSH_PRIVATE
I am facing access credentials error when i ran copy Command in S3.
my copy command is :
copy part from 's3://lntanbusamplebucket/load/part-csv.tbl'
credentials 'aws_access_key_id=D93vB$;yYq'
csv;
error message is:
error: Invalid credentials. Must be of the format: credentials 'aws_iam_role=...' or 'aws_access_key_id=...;aws_secret_access_key=...[;token=...]'
'aws_access_key_id=?;
aws_secret_access_key=?''
Could you please can any one explain what is aws_access_key_id and aws_secret_access_key ?
where we can see this?
Thanks in advance.
Mani
The access key you're using looks more like a secret key, they usually look something like "AKIAXXXXXXXXXXX".
Also, don't post them openly in StackOverflow questions. If someone gets a hold of a set of access keys, they can access your AWS environment.
Access Key & Secret Key are the most basic form of credentials / authentication used in AWS. One is useless without the other, so if you've lost one of the two, you'll need to regenerate a set of keys.
To do this, go into the AWS console, go to the IAM services (Identity and Access Management) and go into users. Here, select the user that you're currently using (probably yourself) and go to the Security Credentials tab.
Here, under Access keys, you can see which sets of keys are currently active for this user. You can only have 2 sets active at one time, so if there's already 2 sets present, delete one and create a new pair. You can download the new pair as a file called "credentials.csv" and this will contain your user, access key and secret key.
I am using the AWS command line tool and am having issue with copying the parameter group to a different region
Error:
An error occurred (DBParameterGroupNotFound) when calling the CopyDBParameterGroup operation: DB ParameterGroup not found, not allowed to do cross region copy.
The command is:
>aws rds copy-db-parameter-group --source-db-parameter-group-identifier arn:aws:rds:ap-southeast-1:MyActID:pg:source-para-group --target-db-parameter-group-identifier dest-para-group --target-db-parameter-group-description dest-para-group-description
I also tried with:
>aws rds copy-db-parameter-group --source-db-parameter-group-identifier arn:aws:rds:ap-southeast-1:MyActID:pg:source-para-group --target-db-parameter-group-identifier arn:aws:rds:ap-south-1:MyActID:pg:dest-para-group --target-db-parameter-group-description arn:aws:rds:ap-south-1:MyActID:pg:dest-para-group-description
Please let help me if anyone else come across the similar issue?
According to the copy-db-parameter-group, the tricky part :
If the source DB parameter group is in a different region than the copy, specify a valid DB parameter group ARN, for example arn:aws:rds:us-west-2:123456789012:pg:special-parameters .
Apparently it is a "from - to" copy option. Try using the TARGET region AWS credential when making the copy, then specify the source ARN.