I have a project on Google Cloud Platform and I have created a Service Account with some roles. I want to validate the quota for the project I'm using, with Go API. I would like to get an example of using the appropriate Go client library for this purpose.
After investigating in the Official Google APis Go Client Library, I found the code for Service Usage.
You can use the following code in order to call the Service Usage API with Go Client Library:
package main
import (
"fmt"
"io/ioutil"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/api/serviceusage/v1beta1"
)
func main() {
// Download the Private Key from your IAM & Admin and add
// the Service Account private key path
serviceAccountFile := "credentials.json"
// Read the private key
serviceAccountJSON, err := ioutil.ReadFile(serviceAccountFile)
if err != nil {
// Handle error
}
// Add the scopes
config, err := google.JWTConfigFromJSON(serviceAccountJSON, serviceusage.CloudPlatformScope, serviceusage.CloudPlatformReadOnlyScope)
// Add the service account.
config.Subject = "serviceaccount#domain.com"
srv, err := serviceusage.New(config.Client(context.Background()))
if err != nil {
// Handle error
}
// REQUEST: https://cloud.google.com/service-usage/docs/reference/rest/v1beta1/services.consumerQuotaMetrics.limits.consumerOverrides/list
resp, err := srv.Services.ConsumerQuotaMetrics.Limits.ConsumerOverrides.List("projects/<PROJECT-NUMBER>/services/compute.googleapis.com/consumerQuotaMetrics/compute.googleapis.com%2Fcpus/limits/%2Fproject%2Fregion").Do()
if err != nil {
// Handle error
}
// Handle your response
fmt.Println(resp)
}
Please make sure that the credentials.json is in the same directory.
This line specifies the response body. Keep in mind that this is in Beta.
Related
AWS by default provides retries support on its service calls, which is usually set to a max of 3 attempts.
Can we configure the retry object to set retry attempts to 5?
you can define a custom retry strategy for the SDK to use:
func main() {
sess := session.Must(
session.NewSession(&aws.Config{
// Use a custom retryer to provide custom retry rules.
Retryer: CustomRetryer{
DefaultRetryer: client.DefaultRetryer{
NumMaxRetries: client.DefaultRetryerMaxNumRetries,
}},
// Use the SDK's SharedCredentialsProvider directly instead of the
// SDK's default credential chain. This ensures that the
// application can call Config.Credentials.Expire. This is counter
// to the SDK's default credentials chain, which will never reread
// the shared credentials file.
Credentials: credentials.NewCredentials(&credentials.SharedCredentialsProvider{
Filename: defaults.SharedCredentialsFilename(),
Profile: "default",
}),
Region: aws.String(endpoints.UsWest2RegionID),
}),
)
// Add a request handler to the AfterRetry handler stack that is used by the
// SDK to be executed after the SDK has determined if it will retry.
// This handler forces the SDK's Credentials to be expired, and next call to
// Credentials.Get will attempt to refresh the credentials.
sess.Handlers.AfterRetry.PushBack(func(req *request.Request) {
if aerr, ok := req.Error.(awserr.RequestFailure); ok && aerr != nil {
if aerr.Code() == "InvalidClaimException" {
// Force the credentials to expire based on error code. Next
// call to Credentials.Get will attempt to refresh credentials.
req.Config.Credentials.Expire()
}
}
})
See the sample code here
Yes, AWS provides support to configure their retry and timeouts features. Here are two ways to increase the max number of retries to 5 in AWS Golang SDK v2:
Configure the retry logic on the AWS Config object cfg and it can be used with various AWS service clients using NewFromConfig function
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRetryer(func() aws.Retryer {
return retry.AddWithMaxAttempts(retry.NewStandard(), 5)
}))
client := s3.NewFromConfig(cfg)
Configure the retry logic only for a specific AWS service client
customRetry := retry.NewStandard(func(o *retry.StandardOptions) {
o.MaxAttempts = 5
})
sqsClient := sqs.NewFromConfig(creds,
func(o *sqs.Options) {
o.Retryer = customRetry
},
)
More info can be found at https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/retries-timeouts/ and https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/retry#hdr-Standard
My question is the same as this other question: How to run AWS SDK with credentials from variables? but I am using SDK version 2 which no longer uses Session (if I understand correctly).
So, I am creating a new client, and I have the credentials as variables. I need to use the IAM service. Here is the function:
func getIAMClient(ctx context.Context) (*iam.Client, error) {
cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion("no-region"))
if err != nil {
return nil, errors.Wrap(err)
}
cfg.HTTPClient, err = getHTTPClient(ctx)
if err != nil {
return nil, err
}
return iam.NewFromConfig(cfg), nil
}
Different users will use the app at the same time, so I can't just use ENV files, but I haven't been able to find a documentation page explaining how to pass these credentials to my Client. Any support will be appreciated!
This can be achieved with the StaticCredentialsProvider as described in section "Static Credentials" of the AWS SDK for Go V2 documentation:
cfg, err := config.LoadDefaultConfig(ctx, config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("AKID", "SECRET_KEY", "TOKEN")))
To init configs from runtime variables, it's fine to use credentials.NewStaticCredentialsProvider:
staticProvider := credentials.NewStaticCredentialsProvider(
accessKey,
secretKey,
sessionToken,
)
cfg, err := config.LoadDefaultConfig(
context.Background(),
config.WithCredentialsProvider(staticProvider),
)
if err != nil {
return nil, err
}
client := iam.New(cfg)
However the AWS SDK documentation correctly reminds you that:
Do not embed credentials inside an application. Use this method only for testing purposes.
This is because typically code snippets that use static credentials pass hardcoded strings, which obviously is a security issue. In your case, you are attempting to pass runtime variables, so as long as those are not checked in with your application sources, you should be fine.
For the general use case, i.e. environment variables, you can use external.LoadDefaultAWSConfig, which automatically looks for, in this order:
Environment Variables
Shared Configuration and Shared Credentials files.
// import "github.com/aws/aws-sdk-go-v2/aws/external"
cfg, err := external.LoadDefaultAWSConfig(external.WithRegion(region))
if err != nil {
return nil, err
}
client := iam.New(cfg)
The method under the hood calls external.NewEnvConfig which tries to fetch credentials from the environment variables:
AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY
AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY
More details about the read priority of the env vars is given in EnvConfig documentation.
Do I have to Define Session and S3 as a package variable or Create instance in every api request.sample code as following.thanks a lot.
var AWSSession *session.Session
var S3Client *s3.S3
func InitAws() error {
log.Info("InitAws")
AWSSession, err := session.NewSession(&aws.Config{
Region: aws.String("XXXX"),
Credentials: credentials.NewStaticCredentials( "XX","XX",""), //just for test
})
if err != nil {
return err
}
log.Info("InitAws S3 Client")
S3Client = s3.New(AWSSession)
log.Info("InitAws end")
return nil
}
The package documentation says:
Sessions are safe to use concurrently as long as the Session is not being modified. Sessions should be cached when possible, because creating a new Session will load all configuration values from the environment, and config files each time the Session is created. Sharing the Session value across all of your service clients will ensure the configuration is loaded the fewest number of times possible.
Do create a single session and reuse it.
Whether you should store that single session in a package-level variable or pass it through function arguments is a separate question with opinionated answers.
Recently I started to get this error when testing my Google Cloud function:
textPayload: "2019/08/12 11:15:58 error publishing to the topic - rpc error: code = Unauthenticated desc = transport: compute: Received 500 `Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpubsub"
It was surprise to me. Recenty I made some changes in the function, but it seems that was not the case. I reverted to the old code which was working before and I got the same error.
I read Cloud Pub/Sub docs on errors:
INTERNAL 500 This error indicates an internal server error; it should not occur. If this error occurs, please report to cloud support. The error should be transient.
I reported it to the cloud support.
Still, I'm not sure is this Google Cloud internal error or the one induced by my code.
EDIT: Here is a pretty minimal code in Go.
package testfcn
import (
"log"
"net/http"
"cloud.google.com/go/pubsub"
)
func TestFcn(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
client, err := pubsub.NewClient(ctx, "min-fotball-staging")
if err != nil {
log.Print("error creating PubSub client - ", err)
return
}
topic := client.Topic("TestTopic")
result := topic.Publish(ctx, &pubsub.Message{
Data: []byte("TestMessage"),
})
_, err = result.Get(ctx)
if err != nil {
log.Print("error publishing to the topic - ", err)
return
}
}
And I have topic named TestTopic.
This was not a Cloud Pub/Sub client library's internal error.
Project's service-<project_number>#gcf-admin-robot.iam.gserviceaccount.com service account had not assigned Cloud Functions Service Agent role.
It was not possible to add this role through Google Cloud Console, but by using gcloud command:
gcloud projects add-iam-policy-binding <project_name> --role=roles/cloudfunctions.serviceAgent --member=serviceAccount:service-<project_number>#gcf-admin-robot.iam.gserviceaccount.com
Possibly useful links:
https://cloud.google.com/functions/docs/concepts/iam#cloud_functions_service_account
Accessing google cloud storage bucket from cloud functions throws 500 error
https://github.com/googleapis/google-cloud-go/issues/1532
Extra:
In my case, this service account had Owner role. This means Owner role does not have some permission(s) Cloud Functions Service Agent role has.
Explain to me please what is the best way to locally test the lambda function. I used sam local and this solution https://github.com/lambci/docker-lambda for testing, but for example, where I invoke one lambda from another error occurs. In general, I can't make stubs for methods since lambda runs in a container
There are a couple of options. Following two are some popular ones.
Serverless framework along with the serverless-offline
plugin.
LocalStack
This is how I test local lambda functions without Serverless frameworks, I run an HTTP post on local (quite easy setup for Go)
decouple lambda logic like so:
func HandleRequest(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
_, _ = pretty.Println("parsed:", request.Body)
return events.APIGatewayProxyResponse{Body: "response is working", StatusCode: 200}, nil
}
main function then checks if it is a local instance then run local post endpoint, else start lambda
func main() {
environment := loadEnv()
if environment == "develop" {
router.NewRouter()
select {}
} else {
lambda.Start(lambdahandler.HandleRequest)
}
}
in between you have an Adapter Pattern that converts your http request to whatever your lambda function accepts, for example:
func MapToApiGateway(w http.ResponseWriter, r *http.Request) (interface{}, error) {
request := new(EmailResponderRequest)
if err := json.NewDecoder(r.Body).Decode(request); err != nil {
return err.Error(), err
}
apiGatewayRequest := mapHttpRequestToGatewayRequest(*request)
events, err := lambdahandler.HandleRequest(nil, apiGatewayRequest)
if err != nil {
return err.Error(), err
}
return events, nil
}
For local manual testing (not unit testing) with sam cli you can specify the environment varilables file with the -n, --env-vars PATHoption and use your real resource identifiers like you would normally do within your Cloud Formation template (refer to the official documentation for more informations).
This should let you invoke other functions directly and use a real DynamoDB table and all other AWS Cloud resources.
Note: if you use VSCode you can try this helper extension.