I am trying to use private_key for some GCP service nodejs client libraries, e.g. #google-cloud/pubsub, #google-cloud/trace-agent
I got private_key from service account credential json file like this:
I am trying to use it as an environment variable for cloud function.
.env.yaml:
And use it like this:
// ...
credentials: {
private_key: envs.private_key,
client_email: envs.client_email
},
projectId: envs.X_GOOGLE_GCLOUD_PROJECT
But got an error:
Error: error:0906D06C:PEM routines:PEM_read_bio:no start line
I check stackdriver logs, here is the private_key environment variable I got:
My guess is the format of private_key is not correct. It's probably caused by the newline symbol \n. So, what's the correct format when using private_key like this?
Setting the key in the .env.yaml file is not a good idea. Indeed, you will be able to commit it to git, maybe in a public repo, and you will set it in plain text as environment variable of your function.
It will be better if you set the file in a bucket, and load it in the runtime. BTW you will keep no secret in the project files.
Another solution is to encrypt with kms the key and decrypt it at runtime. This time you still have the secret in your project files, but encrypted.
But, what do you need another service account? This one on the function is not enough?
GCLOUD_KEY='{"private_key_id":"XXX", "private_key":"YYY",
"client_email":"ZZZ#ZZZ.COM", "client_id":"ABC123",
"type":"service_account"}'
Related
I am using SQLWorkbench to connect to AWS Athena and SQLWorkbench Variables section to specify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. This works. However, when I have to connect to accounts, which require AWS_SESSION_TOKEN, the connection fails. I can connect by modifying credentials file, but that's inconvenient. Is there a better way?
I received an answer from AWS support, and at this point, according to them, it appears that the driver does not support AWS_SESSION_TOKEN parameter.
Answering the question, which appeared on the thread, if you have to use session token, it appears that the only way is to modify your aws credentials file. This can be done either by adding a section or modifying default. Here is an example of a connection string for the former, where simba_session is a profile in credentials:
jdbc:awsathena://AwsRegion=us-west-2;AwsCredentialsProviderClass=com.simba.athena.amazonaws.auth.profile.ProfileCredentialsProvider;AwsCredentialsProviderArguments=simba_session;
If you don't need to use session token, you can specify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY by pressing the Variables button and adding the keys/values. In this case, the connection string can look like this:
jdbc:awsathena://AwsRegion=us-west-2;AwsCredentialsProviderClass=com.simba.athena.amazonaws.auth.DefaultAWSCredentialsProviderChain;
Also note that you can add S3OutputLocation (if needed) and a Workgroup (if needed) by pressing Extended Properties button and adding keys/values, rather than doing it in the connection string.
This seems straightforward to do that passing the Service Account key file (generated from the GCP console) by specifying the file location in the application.properties file. However, I tried all the following options:
1. spring.cloud.gcp.credentials.location=file:/home/my_user_id/mp6key.json
2. spring.cloud.gcp.credentials.location=file:src/main/resources/mp6key.json
3. spring.cloud.gcp.credentials.location=file:./main/resources/mp6key.json
4. spring.cloud.gcp.credentials.location=file:/src/main/resources/mp6key.json
It all ended up with the same error:
java.io.FileNotFoundException: /home/my_user_id/mp6key.json (No such file or directory)
Could anyone advise where I should put the key file and then how should I specify the path to the file properly?
The same programs run successfully in Ecplise with messages published and subscribed using the Pub/Sub processing from GCP (using the Project Id/Service Account key generated in GCP), but now stuck with the above issue after deployed to run on GCP.
As mentioned in the official documentation, the credentials file can be obtained from a number of different locations such as the file system, classpath, URL, etc.
for example, if the service account key file is stored in the classpath as src/main/resources/key.json, pass the following property
spring.cloud.gcp.credentials.location=classpath:key.json
if the key file is stored somewhere else in your local file system, use the file prefix in the property value
spring.cloud.gcp.credentials.location=file:<path to key file>
My line looks like this:
spring.cloud.gcp.credentials.location=file:src/main/resources/[my_json_file]
And this works.
The following also works if I put it in the root of the project directory:
spring.cloud.gcp.credentials.location=file:./[my_json_file]
Have you tried to follow this quickstart? Please, try to follow it thoughtfully and explain if you get any error finishing the quickstart.
Anyway, before running your Java script, try running on the console the following (please modify with the exact path where you store your key):
export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/mp6key.json"
How are you authenticating your credentials in your Java script?
My answer is easy: if you run you code on GCP, you don't have to use service account key file. Problem eliminated, problem solved!
More seriously, have a look on service identity. I don't know what is your current service (Compute? Function? Cloud Run?). Anyway, you can attach any service account on GCP components. Then, when you code, simply use the default credential. Automatically the component identity is loaded. No key to manage, no key to store securely, no key to rotate!
If you provide more detail on your target platform, I could provide your some guidance to achieve this.
Keep in mind that the service account key file are designed to be used by automatic apps (w/o user account involved) hosted outside GCP (on prem, other Cloud Provider, a CI/CD, Apigee,...)
UPDATE
When you use your personal account, you can also use the default credential.
Install gcloud SDK on your computer
Use the command gcloud auth application-default login
Follow the instructions
Enjoy!
If it doesn't work, get the <path> displayed after the login command and set this value in the environment variable named GOOGLE_APPLICATION_CREDENTIALS.
If you definitively want to use service account key file (which are a security issue for the previous reason, but...), you can use it locally
Either set the json key file path into the GOOGLE_APPLICATION_CREDENTIALS environment variable
Or run this command gcloud auth activate-service-account --key-file=<path to your json key file>
Provided your file is in the resources folder try
file://mp6key.json
using file:// instead of file:/ works for me at least
I am having a hard time setting up the credentials for AWS S3 usage via aws-php-sdk within Media Temple.
I continue to receive the error: Cannot read credentials from /.aws/credentials
I tried to follow the guide to install the AWS CLI via https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-linux.html. I then used the following to set the credentials via https://docs.aws.amazon.com/cli/latest/userguide/tutorial-ec2-ubuntu.html#configure-cli
... But I get that error still.
I then had a chat with Media Temple support, who created the .aws/credentials file in root, but then the error message changed to:
Warning: is_readable(): open_basedir restriction in effect. File(/.aws/credentials) is not within the allowed path(s)
MT advised me to not change the basedir settings. They also advised me to simply change where the credentials are read from if possible.
Anyone successfully use AWS credentials on MT?
Trying to do this with the AWS CLI via SSH was like beating my head against a brick wall on Media Temple.
I then tried to set the credentials via environment variables, but that was a no-go.
I then got the idea to put the credentials file within a directory that PHP could access. However, I had to set the location where aws-php-sdk would look for it. I found the environment variable within some documentation and tried to set the variable via php's setenv() function. No dice.
I then searched the aws-php-sdk for the initial error I was seeing, backtracked until I could find where the credentials file location was being set. Turns out the documentation was wrong and the correct environment variable name was HOME.
In the end, all that was needed was to set HOME prior to using AWS. Easy enough, but should have been 100x easier to figure out. Something along these lines:
// Set environment variable for credentials location
putenv('HOME=../');
// Set bucket name
$this->bucket = $bucket;
// Create an S3Client
$this->s3Client = new Aws\S3\S3Client([
'profile' => $this->profile,
'version' => $this->version,
'region' => $this->region
]);
Im facing some sort of credentials issue when trying to connect to my dynamoDB on aws. Locally it all works fine and I can connect using env variables for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION and then
dynamoConnection = boto3.resource('dynamodb', endpoint_url='http://localhost:8000')
When changing to live creds in the env variables and setting the endpoint_url to the dynamoDB on aws this fails with:
"botocore.exceptions.ClientError: An error occurred (InvalidSignatureException) when calling the Query operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."
The creds are valid as they are used in a different app which talks to the same dynamoDB. Ive also tried not using env variables but rather directly in the method but the error persisted. Furthermore, to avoid any issues with trailing spaces Ive even used the credentials directly in the code. Im using Python v3.4.4.
Is there maybe a header that also should be set that Im not aware of? Any hints would be apprecihated.
EDIT
Ive now also created new credentials (to make sure there are only alphanumerical signs) but still no dice.
You shouldn't use the endpoint_url when you are connecting to the real DynamoDB service. That's really only for connecting to local services or non-standard endpoints. Instead, just specify the region you want:
dynamoConnection = boto3.resource('dynamodb', region_name='us-west-2')
It sign that your time zone is different. Maybe you can check your:
1. Time zone
2. Time settings.
If there are some automatic settings, you should fix your time settings.
"sudo hwclock --hctosys" should do the trick.
Just wanted to point out that accessing DynamoDB from a C# environment (using AWS.NET SDK) I ran into this error and the way I solved it was to create a new pair of AWS access/secret keys.
Worked immediately after I changed those keys in the code.
I have setup my pc with python and connections to AWS. This has been successfully tested using the s3_sample.py file, I had to create an IAM user account with the credentials in a file which worked fine for S3 buckets.
My next task was to create an mqtt bridge and put some data in a stream in kinesis using the awslab - awslabs/mqtt-kinesis-bridge.
This seems to be all ok except I get an error when I run the bridge.py. The error is:
Could not find ACTIVE stream:my_first_stream error:Stream my_first_stream under account 673480824415 not found.
Strangely this is not the account I use in the .boto file that is suggested to be set up for this bridge, which are the same credentials I used for the S3 bucket
[Credentials]
aws_access_key_id = AA1122BB
aws_secret_access_key = LlcKb61LTglis
It would seem to me that the bridge.py has a hardcoded account but I can not see it and i can't see where it is pointing to the .boto file for credentials.
Thanks in Advance
So the issue of not finding the Active stream for the account is resolved by:
ensure you are hooked into the US-EAST-1 data centre as this is the default data centre for bridge.py
create your stream, you will only need 1 shard
The next problem stems from the specific version of MQTT and the python library paho-mqtt I installed. The bridge application was written with the API of MQTT 1.2.1 using paho-mqtt 0.4.91 in mind.
The new version which is available for download on their website has a different way of interacting with the paho-mqtt library which passes an additional "flags" object to the on_connect callback. This generates the error I was experiencing, since its not expecting the 5th argument.
You should be able to fix it by making the following change to bridge.py
Line 104 currently looks like this:
def on_connect(self, mqttc, userdata, msg):
Simply add flags, after userdata, so that the callback function looks like this:
def on_connect(self, mqttc, userdata,flags, msg):
This should resolve the issue of the final error of the incorrect number of arguments being passed.
Hope this helps others, thank for the efforts.
When you call python SDK for aws service, there is a line to import the boto modules for aws services in bridge.py.
import boto
The setting is pointing to the .boto for credentials and defined defaultly in boto.
Here is the explanation Boto Config :
Details
A boto config file is a text file formatted like an .ini configuration file that specifies values for options that control the behavior of the boto library. In Unix/Linux systems, on startup, the boto library looks for configuration files in the following locations and in the following order:
/etc/boto.cfg - for site-wide settings that all users on this machine will use
~/.boto - for user-specific settings
~/.aws/credentials - for credentials shared between SDKs
Of course, you can set the environment directly,
export AWS_ACCESS_KEY_ID="Your AWS Access Key ID"
export AWS_SECRET_ACCESS_KEY="Your AWS Secret Access Key"