AWS by default provides retries support on its service calls, which is usually set to a max of 3 attempts.
Can we configure the retry object to set retry attempts to 5?
you can define a custom retry strategy for the SDK to use:
func main() {
sess := session.Must(
session.NewSession(&aws.Config{
// Use a custom retryer to provide custom retry rules.
Retryer: CustomRetryer{
DefaultRetryer: client.DefaultRetryer{
NumMaxRetries: client.DefaultRetryerMaxNumRetries,
}},
// Use the SDK's SharedCredentialsProvider directly instead of the
// SDK's default credential chain. This ensures that the
// application can call Config.Credentials.Expire. This is counter
// to the SDK's default credentials chain, which will never reread
// the shared credentials file.
Credentials: credentials.NewCredentials(&credentials.SharedCredentialsProvider{
Filename: defaults.SharedCredentialsFilename(),
Profile: "default",
}),
Region: aws.String(endpoints.UsWest2RegionID),
}),
)
// Add a request handler to the AfterRetry handler stack that is used by the
// SDK to be executed after the SDK has determined if it will retry.
// This handler forces the SDK's Credentials to be expired, and next call to
// Credentials.Get will attempt to refresh the credentials.
sess.Handlers.AfterRetry.PushBack(func(req *request.Request) {
if aerr, ok := req.Error.(awserr.RequestFailure); ok && aerr != nil {
if aerr.Code() == "InvalidClaimException" {
// Force the credentials to expire based on error code. Next
// call to Credentials.Get will attempt to refresh credentials.
req.Config.Credentials.Expire()
}
}
})
See the sample code here
Yes, AWS provides support to configure their retry and timeouts features. Here are two ways to increase the max number of retries to 5 in AWS Golang SDK v2:
Configure the retry logic on the AWS Config object cfg and it can be used with various AWS service clients using NewFromConfig function
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRetryer(func() aws.Retryer {
return retry.AddWithMaxAttempts(retry.NewStandard(), 5)
}))
client := s3.NewFromConfig(cfg)
Configure the retry logic only for a specific AWS service client
customRetry := retry.NewStandard(func(o *retry.StandardOptions) {
o.MaxAttempts = 5
})
sqsClient := sqs.NewFromConfig(creds,
func(o *sqs.Options) {
o.Retryer = customRetry
},
)
More info can be found at https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/retries-timeouts/ and https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/retry#hdr-Standard
Related
I am writing a client to get the continuous messages from RabbitMQ and pushing to AWS SQS service. But I'm not sure about the session expiry, If the session expires do we need to recreate the session or AWS SDK handles it automatically?
log.Printf("PPU Message Broker: Pushing messages to SQS")
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
//Credentials: credentials.NewSharedCredentials("", "sqs_user"),
})
_, err = sess.Config.Credentials.Get()
if err != nil {
log.Fatalf("PPU Message Broker: Credentails Failed")
}
svc := sqs.New(sess)
result, err := svc.SendMessage(&sqs.SendMessageInput{
MessageBody: aws.String(string(data)),
MessageGroupId: aws.String("TestGroup"),
QueueUrl: &qURL,
})
There is a default configuration for session expiration, but you can specify yours:
In addition to NewSession, you can create sessions using
NewSessionWithOptions. This function allows you to control and
override how the session will be created through code, instead of
being driven by environment variables only.
Use NewSessionWithOptions when you want to provide the config profile
Inside of the Options object, there is an attribute for changing the default expiration time, by default is 15 minutes:
// When the SDK's shared config is configured to assume a role this
option
// may be provided to set the expiry duration of the STS credentials.
// Defaults to 15 minutes if not set as documented in the
// stscreds.AssumeRoleProvider.
AssumeRoleDuration time.Duration
https://docs.aws.amazon.com/sdk-for-go/api/aws/session/
Updated answer:
Just noticed the question is about AWS SDK for Go.
From the AWS SDK for Go documentation: https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#Credentials.Get
func (*Credentials) Get
func (c *Credentials) Get() (Value, error)
Get returns the credentials value, or error if the credentials Value failed to be retrieved.
Will return the cached credentials Value if it has not expired. If the credentials Value has expired the Provider's Retrieve() will be called to refresh the credentials.
If Credentials.Expire() was called the credentials Value will be force expired, and the next call to Get() will cause them to be refreshed.
Original answer:
From AWS Javascript SDK documentation at https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Credentials.html#get-property
get(callback) ⇒ void
Gets the existing credentials, refreshing them if they are not yet loaded or have expired. Users should call this method before using refresh(), as this will not attempt to reload credentials when they are already loaded into the object.
Do I have to Define Session and S3 as a package variable or Create instance in every api request.sample code as following.thanks a lot.
var AWSSession *session.Session
var S3Client *s3.S3
func InitAws() error {
log.Info("InitAws")
AWSSession, err := session.NewSession(&aws.Config{
Region: aws.String("XXXX"),
Credentials: credentials.NewStaticCredentials( "XX","XX",""), //just for test
})
if err != nil {
return err
}
log.Info("InitAws S3 Client")
S3Client = s3.New(AWSSession)
log.Info("InitAws end")
return nil
}
The package documentation says:
Sessions are safe to use concurrently as long as the Session is not being modified. Sessions should be cached when possible, because creating a new Session will load all configuration values from the environment, and config files each time the Session is created. Sharing the Session value across all of your service clients will ensure the configuration is loaded the fewest number of times possible.
Do create a single session and reuse it.
Whether you should store that single session in a package-level variable or pass it through function arguments is a separate question with opinionated answers.
I have a project on Google Cloud Platform and I have created a Service Account with some roles. I want to validate the quota for the project I'm using, with Go API. I would like to get an example of using the appropriate Go client library for this purpose.
After investigating in the Official Google APis Go Client Library, I found the code for Service Usage.
You can use the following code in order to call the Service Usage API with Go Client Library:
package main
import (
"fmt"
"io/ioutil"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/api/serviceusage/v1beta1"
)
func main() {
// Download the Private Key from your IAM & Admin and add
// the Service Account private key path
serviceAccountFile := "credentials.json"
// Read the private key
serviceAccountJSON, err := ioutil.ReadFile(serviceAccountFile)
if err != nil {
// Handle error
}
// Add the scopes
config, err := google.JWTConfigFromJSON(serviceAccountJSON, serviceusage.CloudPlatformScope, serviceusage.CloudPlatformReadOnlyScope)
// Add the service account.
config.Subject = "serviceaccount#domain.com"
srv, err := serviceusage.New(config.Client(context.Background()))
if err != nil {
// Handle error
}
// REQUEST: https://cloud.google.com/service-usage/docs/reference/rest/v1beta1/services.consumerQuotaMetrics.limits.consumerOverrides/list
resp, err := srv.Services.ConsumerQuotaMetrics.Limits.ConsumerOverrides.List("projects/<PROJECT-NUMBER>/services/compute.googleapis.com/consumerQuotaMetrics/compute.googleapis.com%2Fcpus/limits/%2Fproject%2Fregion").Do()
if err != nil {
// Handle error
}
// Handle your response
fmt.Println(resp)
}
Please make sure that the credentials.json is in the same directory.
This line specifies the response body. Keep in mind that this is in Beta.
How do we create a AWS service client (e.g. EC2, Autoscaling) without using a session, and instead with directly using the sahred credentials, like in boto3.
Using session like this works:
sess := session.New(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: credentials.NewSharedCredentials("", profile),
})
svc := ec2.New(sess)
However, this does not work:
svc := ec2.New(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: credentials.NewSharedCredentials("", profile),
})
Error:
cannot use aws.Config literal (type *aws.Config) as type
client.ConfigProvider in argument to ec2.New: *aws.Config does not
implement client.ConfigProvider (missing ClientConfig method)
How to directly create a client with Go AWS SDK without session?
The SDK needed to avoid a circular dependency, and to do this it used an abstraction called session.Session. However, V2 gets rid of this abstraction by flattening some of the packages :)
How can i send an email/trigger a lambda function when a new user registers?
Under "edit identity pool" i only found a sync trigger.
If i understand correctly: This one is triggered every time a user syncs his data...
Is there any way to trigger a lambda function only for the "initial" sync or when a certain dataset is created for the user?
Edit:
To be more specific: I do create the user via lambdas using the JS SDK. I use developer authentication with my own oauth2 flow. I don't know how to distinguish between a user granting access e.g. via Google the first time from someone doing this the second time. The json with the access code seams the same to me... Maybe I am mistaken.
Also using the getOpenIdTokenForDeveloperIdentity call I don't know how to distinguish between an ID that is new to cognito from one cognito already knows.
Edit 2:
To be even more precise:
I am building on this project: https://github.com/laardee/serverless-authentication-boilerplate/blob/master/authentication/lib/storage/usersStorage.js
here is how i do save the User to cognito at the moment.
I do run this code for first time users as well as nth time users. My problem is that i dont know how to distinguish...
const saveCognito = (profile) => new Promise((resolve, reject) => {
if (profile) {
cognitoidentity.getOpenIdTokenForDeveloperIdentity({
IdentityPoolId: process.env.COGNITO_IDENTITY_POOL_ID,
Logins: {
// profile.userId = encrypted id of the e.g. google oauth2 id
[process.env.COGNITO_PROVIDER_NAME]: profile.userId
}
}, (err, dat) => {
if (err) {
reject(err);
} else {
var list_params = {
DatasetName: 'user-data', /* dataset name */
IdentityId: dat.IdentityId, /* cognito id */
IdentityPoolId: process.env.COGNITO_IDENTITY_POOL_ID
};
cognitosync.listRecords(list_params, function(err, data) {
if (err) {
reject(err); // an error occurred
} else {
var RecordPatches = //[Parts of the i want to write to the user]
// SyncSessionToken is returned by the cognitosync.listRecords call
list_params["SyncSessionToken"] = data.SyncSessionToken;
list_params["RecordPatches"] = RecordPatches;
cognitosync.updateRecords(list_params, function(err, update_data) {
if (err){
reject(err);
} else {
resolve();
}
});
}
});
}
});
} else {
reject('Invalid profile');
}
});
So this is something which is not currently supported in Cognito out of the box. You are correct in saying that the only built in Cognito Event that will trigger a Lambda Function is the "Sync Trigger" Event. This Sync event is fired every time that a Cognito IdentityId Synchronizes some of their data to the Cognito Sync cloud data store.
This event is unrelated to the creation of a new IdentityId by Cognito Federated Identity.
You could in theory:
Run a list-identities call on the IdentityPool, before the user logs
in.
Login the user. Check whether the IdentityId which has been given to the user is present in the list you retrieved before they logged
in. This would tell you whether or not the identity that they were
given existed before this login.
Based on this information you could
make a decision whether or not to programatically call the Lambda
Function from your application.
The setup of the above would be complex, as for security reasons you would need to maintain this service, server-side. The list-identities call requires AWS credentials to call. And I doubt you'd want to include permissions for that call in your IAM policy for unauthenticated users.
Aside from the above there is not much you can do at the moment.
In order to do this, you would need to setup a DynamoDB table (or some similar low latency datastore) where you could maintain the state of the IdentityId list, and then query this service/store whenever you login a user to compare new logins to the pre-existing list.
If this is critical to your use case I would suggest heading over to AWS Support, and create a case where you can log this as a feature request.
https://aws.amazon.com/premiumsupport/