How do we create a AWS service client (e.g. EC2, Autoscaling) without using a session, and instead with directly using the sahred credentials, like in boto3.
Using session like this works:
sess := session.New(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: credentials.NewSharedCredentials("", profile),
})
svc := ec2.New(sess)
However, this does not work:
svc := ec2.New(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: credentials.NewSharedCredentials("", profile),
})
Error:
cannot use aws.Config literal (type *aws.Config) as type
client.ConfigProvider in argument to ec2.New: *aws.Config does not
implement client.ConfigProvider (missing ClientConfig method)
How to directly create a client with Go AWS SDK without session?
The SDK needed to avoid a circular dependency, and to do this it used an abstraction called session.Session. However, V2 gets rid of this abstraction by flattening some of the packages :)
Related
I have an application instance running in EKS with the following variables set:
declare -x AWS_DEFAULT_REGION="us-west-2"
declare -x AWS_REGION="us-west-2"
declare -x AWS_ROLE_ARN="xxxxx"
declare -x AWS_WEB_IDENTITY_TOKEN_FILE="/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
As I understand there is a default Java SDK authorization chain that contains com.amazonaws.auth.WebIdentityTokenCredentialsProvider which builds com.amazonaws.services.securitytoken.AWSSecurityTokenService under the hood.
But I can't realize how this circular dependency is solved? I mean you need to specify credentials during creation of AWSSecurityTokenService but credentials create service itself.
I have practical requirements to do that, I want to customize endpoint in sts client but can't since circular dependency.
AWSSecurityTokenServiceClientBuilder.standard()
.withCredentials(new STSAssumeRoleWithWebIdentitySessionCredentialsProvider.Builder(
"arn",
"session",
"tokenfile")
.withStsClient(xxxx)
.build())
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://localhost:4566", null))
.build()
It was easy. It's just done with anonymous auth (https://github.com/aws/aws-sdk-java/blob/1.11.792/aws-java-sdk-sts/src/main/java/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java#L122-L125)
return AWSSecurityTokenServiceClientBuilder.standard()
.withClientConfiguration(clientConfiguration)
.withCredentials(new AWSStaticCredentialsProvider(new AnonymousAWSCredentials()))
.build();
I am writing a client to get the continuous messages from RabbitMQ and pushing to AWS SQS service. But I'm not sure about the session expiry, If the session expires do we need to recreate the session or AWS SDK handles it automatically?
log.Printf("PPU Message Broker: Pushing messages to SQS")
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
//Credentials: credentials.NewSharedCredentials("", "sqs_user"),
})
_, err = sess.Config.Credentials.Get()
if err != nil {
log.Fatalf("PPU Message Broker: Credentails Failed")
}
svc := sqs.New(sess)
result, err := svc.SendMessage(&sqs.SendMessageInput{
MessageBody: aws.String(string(data)),
MessageGroupId: aws.String("TestGroup"),
QueueUrl: &qURL,
})
There is a default configuration for session expiration, but you can specify yours:
In addition to NewSession, you can create sessions using
NewSessionWithOptions. This function allows you to control and
override how the session will be created through code, instead of
being driven by environment variables only.
Use NewSessionWithOptions when you want to provide the config profile
Inside of the Options object, there is an attribute for changing the default expiration time, by default is 15 minutes:
// When the SDK's shared config is configured to assume a role this
option
// may be provided to set the expiry duration of the STS credentials.
// Defaults to 15 minutes if not set as documented in the
// stscreds.AssumeRoleProvider.
AssumeRoleDuration time.Duration
https://docs.aws.amazon.com/sdk-for-go/api/aws/session/
Updated answer:
Just noticed the question is about AWS SDK for Go.
From the AWS SDK for Go documentation: https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#Credentials.Get
func (*Credentials) Get
func (c *Credentials) Get() (Value, error)
Get returns the credentials value, or error if the credentials Value failed to be retrieved.
Will return the cached credentials Value if it has not expired. If the credentials Value has expired the Provider's Retrieve() will be called to refresh the credentials.
If Credentials.Expire() was called the credentials Value will be force expired, and the next call to Get() will cause them to be refreshed.
Original answer:
From AWS Javascript SDK documentation at https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Credentials.html#get-property
get(callback) ⇒ void
Gets the existing credentials, refreshing them if they are not yet loaded or have expired. Users should call this method before using refresh(), as this will not attempt to reload credentials when they are already loaded into the object.
Do I have to Define Session and S3 as a package variable or Create instance in every api request.sample code as following.thanks a lot.
var AWSSession *session.Session
var S3Client *s3.S3
func InitAws() error {
log.Info("InitAws")
AWSSession, err := session.NewSession(&aws.Config{
Region: aws.String("XXXX"),
Credentials: credentials.NewStaticCredentials( "XX","XX",""), //just for test
})
if err != nil {
return err
}
log.Info("InitAws S3 Client")
S3Client = s3.New(AWSSession)
log.Info("InitAws end")
return nil
}
The package documentation says:
Sessions are safe to use concurrently as long as the Session is not being modified. Sessions should be cached when possible, because creating a new Session will load all configuration values from the environment, and config files each time the Session is created. Sharing the Session value across all of your service clients will ensure the configuration is loaded the fewest number of times possible.
Do create a single session and reuse it.
Whether you should store that single session in a package-level variable or pass it through function arguments is a separate question with opinionated answers.
When using google cloud translation API I dont want to have to use a generated keyfile (https://cloud.google.com/translate/docs/basic/setup-basic?hl=de#node.js). We use docker containers deployed to some random host. I cannot add the keyfile to my source code to be compiled into the docker container for obvious security reasons and I dont want to have to copy a keyfile to every host to which the container is deployed (or might be deployed!)
Usually APIs are fine with a Token that I can set using my container management environment variables which I can then aply to all instances of the container when I have to scale it or switch hosts, etc. Does google offer that kind of setup? I'd be fine using REST requests, no need for any sdk.
The only alternative seems to me, adding the keyfile json as environment variable in our gitlab and then building the file into the container.
Or is there any other way of using the google translate API with just a token and no keyfile?
Google's SDK's can implicitly use the default service account (https://cloud.google.com/docs/authentication/production).
EDIT: This might solve your problem: https://github.com/googleapis/google-api-go-client/issues/185
Also: https://godoc.org/golang.org/x/oauth2/google#CredentialsFromJSON
Here's the code example:
json := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS_JSON") // `{"type": "service_account", "project_id": "my-project", ...}`
ctx := context.Background()
jwtConfig, err := google.JWTConfigFromJSON([]byte(json), datastore.ScopeDatastore)
if err != nil {
...
}
ts := jwtConfig.TokenSource(ctx)
datastoreClient, err := datastore.NewClient(ctx, projectID, option.WithTokenSource(ts))
EDIT2:
Also check https://github.com/googleapis/google-auth-library-nodejs#loading-credentials-from-environment-variables
Loading credentials from environment variables
Instead of loading credentials from a key file, you can also provide them using an environment variable and the GoogleAuth.fromJSON() method. This is particularly convenient for systems that deploy directly from source control (Heroku, App Engine, etc).
Start by exporting your credentials:
$ export CREDS='{
"type": "service_account",
"project_id": "your-project-id",
"private_key_id": "your-private-key-id",
"private_key": "your-private-key",
"client_email": "your-client-email",
"client_id": "your-client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "your-cert-url"
}'
Now you can create a new client from the credentials:
const {auth} = require('google-auth-library');
// load the environment variable with our keys
const keysEnvVar = process.env['CREDS'];
if (!keysEnvVar) {
throw new Error('The $CREDS environment variable was not found!');
}
const keys = JSON.parse(keysEnvVar);
async function main() {
// load the JWT or UserRefreshClient from the keys
const client = auth.fromJSON(keys);
client.scopes = ['https://www.googleapis.com/auth/cloud-platform'];
const url = `https://dns.googleapis.com/dns/v1/projects/${keys.project_id}`;
const res = await client.request({url});
console.log(res.data);
}
main().catch(console.error);
I was previously using the all-in-one aws-sdk npm module (https://www.npmjs.com/package/aws-sdk) to invoke an AWS Lambda function, and for that the following code had been working well:
//Some code to get "credentials"
...
const AWS = require('aws-sdk');
const lambda = new AWS.Lambda({
accessKeyId: credentials.accessKeyId,
secretAccessKey: credentials.secretAccessKey,
region: Config.REGION
});
lambda.invoke(pullParams, (err, data) =>
//I would do something with data
);
...
Now, taking a cue from https://github.com/aws/aws-sdk-js-v3, I wish to use to modularised #aws-sdk/client-lambda-node, since it is the only class that I need in my project. Thus, I have changed my code (as suggested here: https://github.com/aws/aws-sdk-js-v3/tree/master/packages/client-lambda-node#usage) like so:
import * as AWS from "#aws-sdk/client-lambda-node/Lambda";
/*
I believe there is a typo in the form of
"
import * as AWS from "#aws-sdk/#aws-sdk/client-lambda-node/Lambda";
"
at the original page
*/
...
//Some code to get the same "credentials" as above
const lambda = new AWS.Lambda({
accessKeyId: credentials.accessKeyId,
secretAccessKey: credentials.secretAccessKey,
region: Config.REGION
});
lambda.invokeAsync(pullParams, (err, data) =>
//I want to do something with err / data
);
...
For what its worth, this is inside a ReactJS app (though I'm sure thats not relevant). Trying the above code with version 0.1.0-preview.5 inside a browser (where it worked earlier) perpetually gives me
http://169.254.169.254/latest/meta-data/iam/security-credentials/ net::ERR_CONNECTION_TIMED_OUT
Error: Unable to connect to instance metadata service (I guess related to (1))
Is the library unstable for use, or am I doing something wrong
You have to pass your credentials using key credentials.
Like:
const lambda = new AWS.Lambda({
credentials: {
accessKeyId: credentials.accessKeyId,
secretAccessKey: credentials.secretAccessKey,
},
region: Config.REGION
});
Or:
const lambda = new AWS.Lambda({
credentials,
region: Config.REGION
});
Yes, version 3 of the SDK is still in beta preview and they've stated that breaking changes are to be expected:
While the SDK is in preview, you may encounter bugs.
To answer your question, yes, it is absolutely unstable for production at the present time. Your specific issue is quite common and I was unable to get to the bottom of it either. My production work all uses v2 still.