When using google cloud translation API I dont want to have to use a generated keyfile (https://cloud.google.com/translate/docs/basic/setup-basic?hl=de#node.js). We use docker containers deployed to some random host. I cannot add the keyfile to my source code to be compiled into the docker container for obvious security reasons and I dont want to have to copy a keyfile to every host to which the container is deployed (or might be deployed!)
Usually APIs are fine with a Token that I can set using my container management environment variables which I can then aply to all instances of the container when I have to scale it or switch hosts, etc. Does google offer that kind of setup? I'd be fine using REST requests, no need for any sdk.
The only alternative seems to me, adding the keyfile json as environment variable in our gitlab and then building the file into the container.
Or is there any other way of using the google translate API with just a token and no keyfile?
Google's SDK's can implicitly use the default service account (https://cloud.google.com/docs/authentication/production).
EDIT: This might solve your problem: https://github.com/googleapis/google-api-go-client/issues/185
Also: https://godoc.org/golang.org/x/oauth2/google#CredentialsFromJSON
Here's the code example:
json := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS_JSON") // `{"type": "service_account", "project_id": "my-project", ...}`
ctx := context.Background()
jwtConfig, err := google.JWTConfigFromJSON([]byte(json), datastore.ScopeDatastore)
if err != nil {
...
}
ts := jwtConfig.TokenSource(ctx)
datastoreClient, err := datastore.NewClient(ctx, projectID, option.WithTokenSource(ts))
EDIT2:
Also check https://github.com/googleapis/google-auth-library-nodejs#loading-credentials-from-environment-variables
Loading credentials from environment variables
Instead of loading credentials from a key file, you can also provide them using an environment variable and the GoogleAuth.fromJSON() method. This is particularly convenient for systems that deploy directly from source control (Heroku, App Engine, etc).
Start by exporting your credentials:
$ export CREDS='{
"type": "service_account",
"project_id": "your-project-id",
"private_key_id": "your-private-key-id",
"private_key": "your-private-key",
"client_email": "your-client-email",
"client_id": "your-client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "your-cert-url"
}'
Now you can create a new client from the credentials:
const {auth} = require('google-auth-library');
// load the environment variable with our keys
const keysEnvVar = process.env['CREDS'];
if (!keysEnvVar) {
throw new Error('The $CREDS environment variable was not found!');
}
const keys = JSON.parse(keysEnvVar);
async function main() {
// load the JWT or UserRefreshClient from the keys
const client = auth.fromJSON(keys);
client.scopes = ['https://www.googleapis.com/auth/cloud-platform'];
const url = `https://dns.googleapis.com/dns/v1/projects/${keys.project_id}`;
const res = await client.request({url});
console.log(res.data);
}
main().catch(console.error);
Related
My question is the same as this other question: How to run AWS SDK with credentials from variables? but I am using SDK version 2 which no longer uses Session (if I understand correctly).
So, I am creating a new client, and I have the credentials as variables. I need to use the IAM service. Here is the function:
func getIAMClient(ctx context.Context) (*iam.Client, error) {
cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion("no-region"))
if err != nil {
return nil, errors.Wrap(err)
}
cfg.HTTPClient, err = getHTTPClient(ctx)
if err != nil {
return nil, err
}
return iam.NewFromConfig(cfg), nil
}
Different users will use the app at the same time, so I can't just use ENV files, but I haven't been able to find a documentation page explaining how to pass these credentials to my Client. Any support will be appreciated!
This can be achieved with the StaticCredentialsProvider as described in section "Static Credentials" of the AWS SDK for Go V2 documentation:
cfg, err := config.LoadDefaultConfig(ctx, config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("AKID", "SECRET_KEY", "TOKEN")))
To init configs from runtime variables, it's fine to use credentials.NewStaticCredentialsProvider:
staticProvider := credentials.NewStaticCredentialsProvider(
accessKey,
secretKey,
sessionToken,
)
cfg, err := config.LoadDefaultConfig(
context.Background(),
config.WithCredentialsProvider(staticProvider),
)
if err != nil {
return nil, err
}
client := iam.New(cfg)
However the AWS SDK documentation correctly reminds you that:
Do not embed credentials inside an application. Use this method only for testing purposes.
This is because typically code snippets that use static credentials pass hardcoded strings, which obviously is a security issue. In your case, you are attempting to pass runtime variables, so as long as those are not checked in with your application sources, you should be fine.
For the general use case, i.e. environment variables, you can use external.LoadDefaultAWSConfig, which automatically looks for, in this order:
Environment Variables
Shared Configuration and Shared Credentials files.
// import "github.com/aws/aws-sdk-go-v2/aws/external"
cfg, err := external.LoadDefaultAWSConfig(external.WithRegion(region))
if err != nil {
return nil, err
}
client := iam.New(cfg)
The method under the hood calls external.NewEnvConfig which tries to fetch credentials from the environment variables:
AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY
AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY
More details about the read priority of the env vars is given in EnvConfig documentation.
I am trying to switch profiles in code using appsettings.json for a .net core web service.
If I new up an instance of the client in code eg:
using var client = new AmazonSecretsManagerClient();
client.ListSecretsAsync(listRequest, default)
.GetAwaiter()
.GetResult();
this takes the default profile that was set up and works fine.
Instead of using the default profile, I want to control the profile from json file. I am trying to follow this:
https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-config-netcore.html
Instead of newing up the client, I am using dependency injection
public SecretsService(IAmazonSecretsManager secretsManagerClient)
{
_secretsManagerClient = secretsManagerClient;
}
Here is the config:
"AWS": {
"Profile": "default",
"Region": "us-east-2"
}
When I make a call using,
_secretsManagerClient.ListSecretsAsync(listRequest, default)
.GetAwaiter()
.GetResult();
it gives an exception
No such host is known. (secretsmanager.defaultregion.amazonaws.com:443)
What am I missing?
Check your appsettings.Development.json file. Some templates have the region set as DefaultRegion and so you have to either update it or remove it to let appsettings.json take precedence.
Do I have to Define Session and S3 as a package variable or Create instance in every api request.sample code as following.thanks a lot.
var AWSSession *session.Session
var S3Client *s3.S3
func InitAws() error {
log.Info("InitAws")
AWSSession, err := session.NewSession(&aws.Config{
Region: aws.String("XXXX"),
Credentials: credentials.NewStaticCredentials( "XX","XX",""), //just for test
})
if err != nil {
return err
}
log.Info("InitAws S3 Client")
S3Client = s3.New(AWSSession)
log.Info("InitAws end")
return nil
}
The package documentation says:
Sessions are safe to use concurrently as long as the Session is not being modified. Sessions should be cached when possible, because creating a new Session will load all configuration values from the environment, and config files each time the Session is created. Sharing the Session value across all of your service clients will ensure the configuration is loaded the fewest number of times possible.
Do create a single session and reuse it.
Whether you should store that single session in a package-level variable or pass it through function arguments is a separate question with opinionated answers.
How do we create a AWS service client (e.g. EC2, Autoscaling) without using a session, and instead with directly using the sahred credentials, like in boto3.
Using session like this works:
sess := session.New(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: credentials.NewSharedCredentials("", profile),
})
svc := ec2.New(sess)
However, this does not work:
svc := ec2.New(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: credentials.NewSharedCredentials("", profile),
})
Error:
cannot use aws.Config literal (type *aws.Config) as type
client.ConfigProvider in argument to ec2.New: *aws.Config does not
implement client.ConfigProvider (missing ClientConfig method)
How to directly create a client with Go AWS SDK without session?
The SDK needed to avoid a circular dependency, and to do this it used an abstraction called session.Session. However, V2 gets rid of this abstraction by flattening some of the packages :)
I am using IdentityServer4 with .NET Core 2.0 on AWS's ElasticBeanstalk. I have a certificate for signing tokens. What's the best way to store this certificate and retrieve it from the application? Should I just stick it with the application files? Throw it in an environment variable somehow?
Edit: just to be clear, this is a token signing certificate, not an SSL certificate.
I don't really like the term 'token signing certificate' because it sounds so benign. What you have is a private key (as part of the certificate), and everyone knows you should secure your private keys!
I wouldn't store this in your application files. If someone gets your source code, they shouldn't also get the keys to your sensitive data (if someone has your signing cert, they can generate any token they like and pretend to be any of your users).
I would consider storing the certificate in AWS parameter store. You could paste the certificate into a parameter, which can be encrypted at rest. You then lock down the parameter with an AWS policy so only admins and the application can get the cert - your naughty Devs dont need it! Your application would pull the parameter string when needed and turn it into your certificate object.
This is how I store secrets in my application. I can provide more examples/details if required.
Edit -- This was the final result from Stu's guidance
The project needs 2 AWS packages from Nuget to the project
AWSSDK.Extensions.NETCORE.Setup
AWSSDK.SimpleSystemsManagement
Create 2 parameters in the AWS SSM Parameter Store like:
A plain string named /MyApp/Staging/SigningCertificate and the value is a Base64 encoded .pfx file
An encrypted string /MyApp/Staging/SigningCertificateSecret and the value is the password to the above .pfx file
This is the relevant code:
// In Startup class
private X509Certificate2 GetSigningCertificate()
{
// Configuration is the IConfiguration built by the WebHost in my Program.cs and injected into the Startup constructor
var awsOptions = Configuration.GetAWSOptions();
var ssmClient = awsOptions.CreateServiceClient<IAmazonSimpleSystemsManagement>();
// This is blocking because this is called during synchronous startup operations of the WebHost-- Startup.ConfigureServices()
var res = ssmClient.GetParametersByPathAsync(new Amazon.SimpleSystemsManagement.Model.GetParametersByPathRequest()
{
Path = "/MyApp/Staging",
WithDecryption = true
}).GetAwaiter().GetResult();
// Decode the certificate
var base64EncodedCert = res.Parameters.Find(p => p.Name == "/MyApp/Staging/SigningCertificate")?.Value;
var certificatePassword = res.Parameters.Find(p => p.Name == "/MyApp/Staging/SigningCertificateSecret")?.Value;
byte[] decodedPfxBytes = Convert.FromBase64String(base64EncodedCert);
return new X509Certificate2(decodedPfxBytes, certificatePassword);
}
public void ConfigureServices(IServiceCollection servies)
{
// ...
var identityServerBuilder = services.AddIdentityServer();
var signingCertificate = GetSigningCertificate();
identityServerBuilder.AddSigningCredential(signingCertificate);
//...
}
Last, you may need to set an IAM role and/or policy to your EC2 instance(s) that gives access to these SSM parameters.
Edit: I have been moving my web application SSL termination from my load balancer to my elastic beanstalk instance this week. This requires storing my private key in S3. Details from AWS here: Storing Private Keys Securely in Amazon S3