My question is the same as this other question: How to run AWS SDK with credentials from variables? but I am using SDK version 2 which no longer uses Session (if I understand correctly).
So, I am creating a new client, and I have the credentials as variables. I need to use the IAM service. Here is the function:
func getIAMClient(ctx context.Context) (*iam.Client, error) {
cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion("no-region"))
if err != nil {
return nil, errors.Wrap(err)
}
cfg.HTTPClient, err = getHTTPClient(ctx)
if err != nil {
return nil, err
}
return iam.NewFromConfig(cfg), nil
}
Different users will use the app at the same time, so I can't just use ENV files, but I haven't been able to find a documentation page explaining how to pass these credentials to my Client. Any support will be appreciated!
This can be achieved with the StaticCredentialsProvider as described in section "Static Credentials" of the AWS SDK for Go V2 documentation:
cfg, err := config.LoadDefaultConfig(ctx, config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("AKID", "SECRET_KEY", "TOKEN")))
To init configs from runtime variables, it's fine to use credentials.NewStaticCredentialsProvider:
staticProvider := credentials.NewStaticCredentialsProvider(
accessKey,
secretKey,
sessionToken,
)
cfg, err := config.LoadDefaultConfig(
context.Background(),
config.WithCredentialsProvider(staticProvider),
)
if err != nil {
return nil, err
}
client := iam.New(cfg)
However the AWS SDK documentation correctly reminds you that:
Do not embed credentials inside an application. Use this method only for testing purposes.
This is because typically code snippets that use static credentials pass hardcoded strings, which obviously is a security issue. In your case, you are attempting to pass runtime variables, so as long as those are not checked in with your application sources, you should be fine.
For the general use case, i.e. environment variables, you can use external.LoadDefaultAWSConfig, which automatically looks for, in this order:
Environment Variables
Shared Configuration and Shared Credentials files.
// import "github.com/aws/aws-sdk-go-v2/aws/external"
cfg, err := external.LoadDefaultAWSConfig(external.WithRegion(region))
if err != nil {
return nil, err
}
client := iam.New(cfg)
The method under the hood calls external.NewEnvConfig which tries to fetch credentials from the environment variables:
AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY
AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY
More details about the read priority of the env vars is given in EnvConfig documentation.
Related
Do I have to Define Session and S3 as a package variable or Create instance in every api request.sample code as following.thanks a lot.
var AWSSession *session.Session
var S3Client *s3.S3
func InitAws() error {
log.Info("InitAws")
AWSSession, err := session.NewSession(&aws.Config{
Region: aws.String("XXXX"),
Credentials: credentials.NewStaticCredentials( "XX","XX",""), //just for test
})
if err != nil {
return err
}
log.Info("InitAws S3 Client")
S3Client = s3.New(AWSSession)
log.Info("InitAws end")
return nil
}
The package documentation says:
Sessions are safe to use concurrently as long as the Session is not being modified. Sessions should be cached when possible, because creating a new Session will load all configuration values from the environment, and config files each time the Session is created. Sharing the Session value across all of your service clients will ensure the configuration is loaded the fewest number of times possible.
Do create a single session and reuse it.
Whether you should store that single session in a package-level variable or pass it through function arguments is a separate question with opinionated answers.
I have a project on Google Cloud Platform and I have created a Service Account with some roles. I want to validate the quota for the project I'm using, with Go API. I would like to get an example of using the appropriate Go client library for this purpose.
After investigating in the Official Google APis Go Client Library, I found the code for Service Usage.
You can use the following code in order to call the Service Usage API with Go Client Library:
package main
import (
"fmt"
"io/ioutil"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/api/serviceusage/v1beta1"
)
func main() {
// Download the Private Key from your IAM & Admin and add
// the Service Account private key path
serviceAccountFile := "credentials.json"
// Read the private key
serviceAccountJSON, err := ioutil.ReadFile(serviceAccountFile)
if err != nil {
// Handle error
}
// Add the scopes
config, err := google.JWTConfigFromJSON(serviceAccountJSON, serviceusage.CloudPlatformScope, serviceusage.CloudPlatformReadOnlyScope)
// Add the service account.
config.Subject = "serviceaccount#domain.com"
srv, err := serviceusage.New(config.Client(context.Background()))
if err != nil {
// Handle error
}
// REQUEST: https://cloud.google.com/service-usage/docs/reference/rest/v1beta1/services.consumerQuotaMetrics.limits.consumerOverrides/list
resp, err := srv.Services.ConsumerQuotaMetrics.Limits.ConsumerOverrides.List("projects/<PROJECT-NUMBER>/services/compute.googleapis.com/consumerQuotaMetrics/compute.googleapis.com%2Fcpus/limits/%2Fproject%2Fregion").Do()
if err != nil {
// Handle error
}
// Handle your response
fmt.Println(resp)
}
Please make sure that the credentials.json is in the same directory.
This line specifies the response body. Keep in mind that this is in Beta.
When using google cloud translation API I dont want to have to use a generated keyfile (https://cloud.google.com/translate/docs/basic/setup-basic?hl=de#node.js). We use docker containers deployed to some random host. I cannot add the keyfile to my source code to be compiled into the docker container for obvious security reasons and I dont want to have to copy a keyfile to every host to which the container is deployed (or might be deployed!)
Usually APIs are fine with a Token that I can set using my container management environment variables which I can then aply to all instances of the container when I have to scale it or switch hosts, etc. Does google offer that kind of setup? I'd be fine using REST requests, no need for any sdk.
The only alternative seems to me, adding the keyfile json as environment variable in our gitlab and then building the file into the container.
Or is there any other way of using the google translate API with just a token and no keyfile?
Google's SDK's can implicitly use the default service account (https://cloud.google.com/docs/authentication/production).
EDIT: This might solve your problem: https://github.com/googleapis/google-api-go-client/issues/185
Also: https://godoc.org/golang.org/x/oauth2/google#CredentialsFromJSON
Here's the code example:
json := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS_JSON") // `{"type": "service_account", "project_id": "my-project", ...}`
ctx := context.Background()
jwtConfig, err := google.JWTConfigFromJSON([]byte(json), datastore.ScopeDatastore)
if err != nil {
...
}
ts := jwtConfig.TokenSource(ctx)
datastoreClient, err := datastore.NewClient(ctx, projectID, option.WithTokenSource(ts))
EDIT2:
Also check https://github.com/googleapis/google-auth-library-nodejs#loading-credentials-from-environment-variables
Loading credentials from environment variables
Instead of loading credentials from a key file, you can also provide them using an environment variable and the GoogleAuth.fromJSON() method. This is particularly convenient for systems that deploy directly from source control (Heroku, App Engine, etc).
Start by exporting your credentials:
$ export CREDS='{
"type": "service_account",
"project_id": "your-project-id",
"private_key_id": "your-private-key-id",
"private_key": "your-private-key",
"client_email": "your-client-email",
"client_id": "your-client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "your-cert-url"
}'
Now you can create a new client from the credentials:
const {auth} = require('google-auth-library');
// load the environment variable with our keys
const keysEnvVar = process.env['CREDS'];
if (!keysEnvVar) {
throw new Error('The $CREDS environment variable was not found!');
}
const keys = JSON.parse(keysEnvVar);
async function main() {
// load the JWT or UserRefreshClient from the keys
const client = auth.fromJSON(keys);
client.scopes = ['https://www.googleapis.com/auth/cloud-platform'];
const url = `https://dns.googleapis.com/dns/v1/projects/${keys.project_id}`;
const res = await client.request({url});
console.log(res.data);
}
main().catch(console.error);
Recently I started to get this error when testing my Google Cloud function:
textPayload: "2019/08/12 11:15:58 error publishing to the topic - rpc error: code = Unauthenticated desc = transport: compute: Received 500 `Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpubsub"
It was surprise to me. Recenty I made some changes in the function, but it seems that was not the case. I reverted to the old code which was working before and I got the same error.
I read Cloud Pub/Sub docs on errors:
INTERNAL 500 This error indicates an internal server error; it should not occur. If this error occurs, please report to cloud support. The error should be transient.
I reported it to the cloud support.
Still, I'm not sure is this Google Cloud internal error or the one induced by my code.
EDIT: Here is a pretty minimal code in Go.
package testfcn
import (
"log"
"net/http"
"cloud.google.com/go/pubsub"
)
func TestFcn(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
client, err := pubsub.NewClient(ctx, "min-fotball-staging")
if err != nil {
log.Print("error creating PubSub client - ", err)
return
}
topic := client.Topic("TestTopic")
result := topic.Publish(ctx, &pubsub.Message{
Data: []byte("TestMessage"),
})
_, err = result.Get(ctx)
if err != nil {
log.Print("error publishing to the topic - ", err)
return
}
}
And I have topic named TestTopic.
This was not a Cloud Pub/Sub client library's internal error.
Project's service-<project_number>#gcf-admin-robot.iam.gserviceaccount.com service account had not assigned Cloud Functions Service Agent role.
It was not possible to add this role through Google Cloud Console, but by using gcloud command:
gcloud projects add-iam-policy-binding <project_name> --role=roles/cloudfunctions.serviceAgent --member=serviceAccount:service-<project_number>#gcf-admin-robot.iam.gserviceaccount.com
Possibly useful links:
https://cloud.google.com/functions/docs/concepts/iam#cloud_functions_service_account
Accessing google cloud storage bucket from cloud functions throws 500 error
https://github.com/googleapis/google-cloud-go/issues/1532
Extra:
In my case, this service account had Owner role. This means Owner role does not have some permission(s) Cloud Functions Service Agent role has.
Explain to me please what is the best way to locally test the lambda function. I used sam local and this solution https://github.com/lambci/docker-lambda for testing, but for example, where I invoke one lambda from another error occurs. In general, I can't make stubs for methods since lambda runs in a container
There are a couple of options. Following two are some popular ones.
Serverless framework along with the serverless-offline
plugin.
LocalStack
This is how I test local lambda functions without Serverless frameworks, I run an HTTP post on local (quite easy setup for Go)
decouple lambda logic like so:
func HandleRequest(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
_, _ = pretty.Println("parsed:", request.Body)
return events.APIGatewayProxyResponse{Body: "response is working", StatusCode: 200}, nil
}
main function then checks if it is a local instance then run local post endpoint, else start lambda
func main() {
environment := loadEnv()
if environment == "develop" {
router.NewRouter()
select {}
} else {
lambda.Start(lambdahandler.HandleRequest)
}
}
in between you have an Adapter Pattern that converts your http request to whatever your lambda function accepts, for example:
func MapToApiGateway(w http.ResponseWriter, r *http.Request) (interface{}, error) {
request := new(EmailResponderRequest)
if err := json.NewDecoder(r.Body).Decode(request); err != nil {
return err.Error(), err
}
apiGatewayRequest := mapHttpRequestToGatewayRequest(*request)
events, err := lambdahandler.HandleRequest(nil, apiGatewayRequest)
if err != nil {
return err.Error(), err
}
return events, nil
}
For local manual testing (not unit testing) with sam cli you can specify the environment varilables file with the -n, --env-vars PATHoption and use your real resource identifiers like you would normally do within your Cloud Formation template (refer to the official documentation for more informations).
This should let you invoke other functions directly and use a real DynamoDB table and all other AWS Cloud resources.
Note: if you use VSCode you can try this helper extension.