AWS S3 - Golang SDK - SignatureDoesNotMatch - amazon-web-services

I'm looking to integrade an S3 bucket with an API im developing, I'm running into this error wherever I go -
SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
status code: 403
I have done the following
Installed SDK & AWS CLI, and AWS configured
Double(triple) checked spelling of key & secret key & bucket permissions
Attempted with credentials document, .env, and even hard coding the values directly
Tested with AWS CLI (THIS WORKS), so I believe I can rule out permissions, keys, as a whole.
I'm testing by trying to list buckets, here is the code taken directly from the AWS documentation-
sess := session.Must(session.NewSessionWithOptions(session.Options{ <--- DEBUGGER SET HERE
SharedConfigState: session.SharedConfigEnable,
}))
svc := s3.New(sess)
result, err := svc.ListBuckets(nil)
if err != nil { exitErrorf("Unable to list buckets, %v", err) }
for _, b := range result.Buckets {
fmt.Printf("* %s created on %s\n", aws.StringValue(b.Name), aws.TimeValue(b.CreationDate))
}
Using debugger, I can see the sessions config files as the program runs, the issue is potentially here
config -
-> credentials
-> creds
-> v
-> Access Key = ""
-> Secret Access Key = ""
-> Token = ""
-> provider
->value
-> Access Key With Value
-> Secret Access Key With Value
-> Token With Value
I personally cannot find any documentation regarding "creds" / "v", and I don't know if this is causing the issue. As I mentioned, I can use the AWS CLI to upload into the bucket, and even when I hard code my access key etc in to the Go SDK I receive this error.
Thank you for any thoughts, greatly appreciated.

I just compiled your code and its executing OK ... one of the many ways to supply credentials to your binary is to populate these env vars
export AWS_ACCESS_KEY_ID=AKfoobarT2IJEAU4
export AWS_SECRET_ACCESS_KEY=oa6oT0Xxt4foobarbambazdWFCb
export AWS_REGION=us-west-2
that is all you need when using the env var approach ( your values are available using the aws console browser )
the big picture is to create a wrapper shell script ( bash ) which contains above three lines to populate the env vars to supply credentials then in same shell script execute the golang binary ( typically you compile the golang in some preliminary process ) ... in my case I store the values of my three env vars in encrypted files which the shell script decrypts just before it calls the above export commands
sometimes its helpful to drop kick and just use the aws command line equivalent commands to get yourself into the ballpark ... from a terminal run
aws s3 ls s3://cairo-mombasa-zaire --region us-west-2
which can also use those same env vars shown above
for completeness here is your code with boilerplate added ... this runs OK and lists out the buckets
package main
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
// "github.com/aws/aws-sdk-go/service/s3/s3manager"
"fmt"
"os"
)
func exitErrorf(msg string, args ...interface{}) {
fmt.Fprintf(os.Stderr, msg+"\n", args...)
os.Exit(1)
}
func main() {
region_env_var := "AWS_REGION"
curr_region := os.Getenv(region_env_var)
if curr_region == "" {
exitErrorf("ERROR - failed to get region from env var %v", region_env_var)
}
fmt.Println("here is region ", curr_region)
// Load session from shared config
sess := session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
svc := s3.New(sess)
result, err := svc.ListBuckets(nil)
if err != nil { exitErrorf("Unable to list buckets, %v", err) }
for _, b := range result.Buckets {
fmt.Printf("* %s created on %s\n", aws.StringValue(b.Name), aws.TimeValue(b.CreationDate))
}
}

numBytes, err := downloader.Download(tempFile,
&s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(fileName),
},
)
In my case the bucket value was wrong, it is missing literal "/" at the end. Adding that fixes my problem.
Error i got - err: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
status code: 403,

If anyone else happens to have this problem,
The issue was regarding environment variables much like Scott suggest above, however it was due to lacking
export AWS_SDK_LOAD_CONFIG="true"
If this environment variable is not present, then the Golang SDK will not look for a credentials file, along with this, I instantiated environment variables for both my keys which allowed the connection to succeed.
To recap
if you're attempting to use the shared credentials folder, you must use the above noted environment variable to enable it.
If you're using environment variables, you shouldn't be affected by this problem.

Related

How to use use web identify token in aws to authenticate using go sdk

I am trying to write an example of how to use a web identity token with a container to perform EC2 operations.
The container spec contains the service account and has the necessary permission to access the token path and its namespace is a trusted entity in the role.
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/credentials/stscreds"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/aws/aws-sdk-go/service/sts"
)
func main() {
sess, _ := session.NewSession()
config := aws.NewConfig().WithRegion("us-east-1")
stsSTS := sts.New(sess)
roleARN := "arn:aws:iam::1234567:role/s2-p0o5-csi-drivers-ebs-cloud-credentials"
roleProvider := stscreds.NewWebIdentityRoleProviderWithOptions(stsSTS, roleARN, "gosession", stscreds.FetchTokenPath("/build/token"))
creds := credentials.NewCredentials(roleProvider)
credValue, _ := roleProvider.Retrieve()
fmt.Printf("credValue.AccessKeyID: %v\n", credValue.AccessKeyID)
fmt.Printf("credValue.SecretAccessKey: %v\n", credValue.SecretAccessKey)
fmt.Printf("credValue.SessionToken: %v\n", credValue.SessionToken)
config = config.WithCredentials(creds)
nodeID := "i-00843f27cfeb0beff"
svc := ec2.New(sess, config)
request := &ec2.DescribeInstancesInput{
InstanceIds: []*string{&nodeID},
}
result, _ := svc.DescribeInstances(request)
fmt.Printf("result: %v\n", result)
}
The value of result get empty. Whereas, i have exported (credValue.AccessKeyID, credValue.SecretAccessKey,credValue.SessionToken) as environment variables and aws cli is giving me output related to describing the instance.
I tried various methods like credentials.NewStaticCredentials() with the credential information, but no luck. Can some help share hint on what is going wrong and correct way of doing it.
The instance-id used should be avaiable in the region, else it returns an empty map with err == nil.

What's the correct way of loading AWS credentials from role in a Fargate task using AWS SDK Go?

I have the following snippet:
awsCredentials := credentials.NewChainCredentials(
[]credentials.Provider{
&ec2rolecreds.EC2RoleProvider{
Client: ec2metadata.New(newSession, aws.NewConfig()),
},
&credentials.SharedCredentialsProvider{},
&credentials.EnvProvider{},
})
which works fine whenever the code is running on an EC2 instance or when the access/secret key are passed through variables (used for local testing).
However, this code is failing when running on ECS+Fargate because NoCredentialProviders: no valid providers in chain. Checked the environment variables of the running container and it has the expected AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, so the credentials.EnvProvider should read it.
So, my question is, what's the correct way of reading these credentials? Because the problem I'm facing is not about lack of permissions (which would indicate an error in the policy / role), but that code is not able to get the credentials.
UPDATE
I have narrowed this to the use of ec2rolescreds.
Using this simple example:
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
"github.com/aws/aws-sdk-go/aws/ec2metadata"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
newSession, err := session.NewSession()
if err != nil {
log.Fatal(err)
}
awsCredentials := credentials.NewChainCredentials(
[]credentials.Provider{
&ec2rolecreds.EC2RoleProvider{
Client: ec2metadata.New(newSession, aws.NewConfig()),
},
&credentials.SharedCredentialsProvider{},
&credentials.EnvProvider{},
})
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: awsCredentials},
)
if err != nil {
log.Fatal(err)
}
// Create S3 service client
svc := s3.New(sess)
result, err := svc.ListBuckets(nil)
if err != nil {
log.Fatal(err)
}
fmt.Println("Buckets:")
for _, b := range result.Buckets {
fmt.Printf("* %s created on %s\n",
aws.StringValue(b.Name), aws.TimeValue(b.CreationDate))
}
}
If I remove ec2rolescreds, everything works fine both local and in ECS+Fargate.
However, if I run this code as is, I get the same error of NoCredentialProviders: no valid providers in chain
The solution is to initialize clients using sessions instead of credentials, i.e:
conf := aws.NewConfig().WithRegion("us-east-1")
sess := session.Must(session.NewSession(conf))
svc := s3.New(sess)
// others:
// svc := sqs.New(sess)
// svc := dynamodb.New(sess)
// ...
Because, as #Ay0 points, the default credential chain already includes both EnvProvider and RemoteCredProvider.
In case you still need the credentials, you can use:
creds := stscreds.NewCredentials(sess, "myRoleARN")
as the documentation points out. Notice that the policy of the role must have the sts:AssumeRole action enabled. For more information, here are the stscreds.NewCredentials(...) docs
So, a session can be configured using a Config object.
Reading through the specs of this object, it says for Credentials:
// The credentials object to use when signing requests. Defaults to a
// chain of credential providers to search for credentials in environment
// variables, shared credential file, and EC2 Instance Roles.
Credentials *credentials.Credentials
The defaults are already what my snippet was doing, so I removed all the awsCredentials block and now it's working fine everywhere. Locally, EC2, Fargate...
UPDATE
To expand the answer, the reason why removing the awsCredentials made this work is because, if you check the SDK's code, https://github.com/aws/aws-sdk-go/blob/master/aws/defaults/defaults.go#L107, the default credentials check both EnvProvider and RemoteCredProvider.
By overriding the default chain credentials, it was not able to look for credentials in RemoteCredProvider, which is the provider that handles the environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI.

Retrieve List of AWS Config Rule Names using AWS Golang SDK

AWS Config has a set of Managed Rules and I am trying to use the Golang AWS SDK to use the DescribeConfigRules API to retrieve the list of AWS Config Managed Rule Names and other details.
It seems like every request receives a response of 25 rules and a NextToken for the next set of results. What I am having trouble understanding is how do I use this NextToken to retrieve the next set of results?
Here is what I have so far.
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/configservice"
)
func main() {
//Create an aws session
sess, err := session.NewSession(&aws.Config{Region: aws.String("us-west-2"), Credentials: credentials.NewSharedCredentials("", "my-aws-profile")})
// Create a ConfigService client from just a session.
configsvc := configservice.New(sess)
rules := (*configservice.DescribeConfigRulesInput)(nil)
configrulesoutput, err := configsvc.DescribeConfigRules(rules)
if err != nil {
log.Fatal(err)
}
for _, rule := range configrulesoutput.ConfigRules {
fmt.Println("Rule: ", *rule.ConfigRuleName)
}
}
The above code successfully prints the first 25 rules received in the response. However I am not sure how to use the NextToken received in the response to get the next set of results.
Sample Response.
ConfigRules: [
{
ConfigRuleArn: "ConfigRuleARN",
ConfigRuleId: "config-rule-ppwclr",
ConfigRuleName: "cloudtrail-enabled",
ConfigRuleState: "ACTIVE",
Description: "Checks whether AWS CloudTrail is enabled in your AWS account. Optionally, you can specify which S3 bucket, SNS topic, and Amazon CloudWatch Logs ARN to use.",
InputParameters: "{}",
MaximumExecutionFrequency: "TwentyFour_Hours",
Source: {
Owner: "AWS",
SourceIdentifier: "CLOUD_TRAIL_ENABLED"
}
},
{ Rule 2 }, ....{ Rule 25}
],
NextToken: "nexttoken"
}
Code extracts the rulenames from the response and output is as below.
Rule: cloudtrail-enabled
Rule: restricted-ssh
Rule: securityhub-access-keys-rotated
Rule: securityhub-autoscaling-group-elb-healthcheck-required
Rule: securityhub-cloud-trail-cloud-watch-logs-enabled
Rule: securityhub-cloud-trail-encryption-enabled
Rule: securityhub-cloud-trail-log-file-validation-enabled
Rule: securityhub-cloudtrail-enabled
Rule: securityhub-cmk-backing-key-rotation-enabled
Rule: securityhub-codebuild-project-envvar-awscred-check
Rule: securityhub-codebuild-project-source-repo-url-check
Rule: securityhub-ebs-snapshot-public-restorable-check
Rule: securityhub-ec2-managedinstance-patch-compliance
Rule: securityhub-ec2-security-group-attached-to-eni
Rule: securityhub-eip-attached
Rule: securityhub-elasticsearch-encrypted-at-rest
Rule: securityhub-elasticsearch-in-vpc-only
Rule: securityhub-iam-password-policy-ensure-expires
Rule: securityhub-iam-password-policy-lowercase-letter-check
Rule: securityhub-iam-password-policy-minimum-length-check
Rule: securityhub-iam-password-policy-number-check
Rule: securityhub-iam-password-policy-prevent-reuse-check
Rule: securityhub-iam-password-policy-symbol-check
Rule: securityhub-iam-password-policy-uppercase-letter-check
Rule: securityhub-iam-policy-no-statements-with-admin-access
End Goal: Using golang AWS SDK, extract the AWS Config Managed Rule details and put it in an excel format using Excelize to review which AWS Config rules we want enabled.
Thanks for your help in advance.
---New based on #Adrian's comment and doc reference---
As per doc
type DescribeConfigRulesInput struct {
// The names of the AWS Config rules for which you want details. If you do not
// specify any names, AWS Config returns details for all your rules.
ConfigRuleNames []*string `type:"list"`
// The nextToken string returned on a previous page that you use to get the
// next page of results in a paginated response.
NextToken *string `type:"string"`
// contains filtered or unexported fields }
So here is what I am trying. Specifying nil should give me back all rules. nextToken is blank string for the first call.
configsvc := configservice.New(sess)
rules := (*configservice.DescribeConfigRulesInput)(nil)
nextToken := ""
rules.SetNextToken(nextToken)
getConfigRulesFunc(configsvc, rules)
//getConfigRulesFunc function
func getConfigRulesFunc(cfgsvc *configservice.ConfigService, ruleset *configservice.DescribeConfigRulesInput) {
configrulesoutput, err := cfgsvc.DescribeConfigRules(ruleset)
if err != nil {
log.Fatal(err)
}
for i, r := range configrulesoutput.ConfigRules {
fmt.Println("Rule: ", i, ""+*r.ConfigRuleName)
}
if *configrulesoutput.NextToken != "" {
ruleset := (*configservice.DescribeConfigRulesInput)(nil)
ruleset.SetNextToken(*configrulesoutput.NextToken)
getConfigRulesFunc(cfgsvc, ruleset)
}
}
Above code compiles fine but here the runtime error I believe because of nil.
configsvc type: *configservice.ConfigService
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13c7ed2]
goroutine 1 [running]:
github.com/aws/aws-sdk-go/service/configservice.(*DescribeConfigRulesInput).SetNextToken(...)
/Users/user/go/src/github.com/aws/aws-sdk-go/service/configservice/api.go:12230
main.main()
/Users/user/golang/awsgotest/awsgotest.go:26 +0x232
Ok, finally figured it out with the help of a very kind Alex Diehl via this ticket https://github.com/aws/aws-sdk-go/issues/3293 on the official aws-sdk-go repo.
I would still say the aws sdk for go definitely lacks simple examples for configservice at the least on recommended usage.
Here the code that works. This will also show how to use simple recursive function in go to use NextToken for pagination of api results that span multiple pages especially apis that do not have built in paginators.
Also note that DescribeConfigRules API does not list all AWS Managed Config Rules, only the Config rules enabled for your account.
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/configservice"
)
var i int = 0
func main() {
sess, err := session.NewSession(&aws.Config{Region: aws.String("us-west-2"), Credentials: credentials.NewSharedCredentials("", "my-profile")})
if err != nil {
log.Fatal(err)
}
//Create a ConfigService client from just a session.
configsvc := configservice.New(sess)
fmt.Printf("configsvc type: %T\n", configsvc)
rules := &configservice.DescribeConfigRulesInput{}
getConfigRulesFunc(configsvc, rules)
}
func getConfigRulesFunc(cfgsvc *configservice.ConfigService, ruleset *configservice.DescribeConfigRulesInput) {
configrulesoutput, err := cfgsvc.DescribeConfigRules(ruleset)
if err != nil {
log.Fatal(err)
}
for _, r := range configrulesoutput.ConfigRules {
fmt.Println("Rule: ", i, ""+*r.ConfigRuleName)
i = i + 1
}
if configrulesoutput.NextToken != nil {
fmt.Println("In if nexttoken is not empty")
fmt.Println("Print NextToken: ", *configrulesoutput.NextToken)
ruleset := &configservice.DescribeConfigRulesInput{}
ruleset.SetNextToken(*configrulesoutput.NextToken)
getConfigRulesFunc(cfgsvc, ruleset)
}
}
Code in Bold were the ones giving me grief on how to use the NextToken based on best practices atleast for the go sdk for aws.
FYI, you could have looked in the AWS Go guide as there is a section on pagination: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/making-requests.html#using-pagination-methods.

S3 on EC2 with IAM: Error NoCredentialProviders: no valid providers in chain. Deprecated

My application use s3 and running on EC2. The IAM is configured on the instance, so the auth happen keyless (without the access key and secret key).
I'm able to upload or download file using aws cli. However when I tried to perform download operation using aws-sdk-go, I get error below:
AccessDenied: Access Denied
status code: 403, request id: F945BDB5410E1A00, host id: m74jJ8z/AEzdkaJkWKdIqPEwPIYPZfWnLLfa5UpEwHwaBcXOuXTPY1aw/u/5HGralKg+ewAWEJA=
I followed the official guide from https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/ec2rolecreds/ and from this issue https://github.com/aws/aws-sdk-go/issues/430 but got the error above.
Below is my code:
s3UploadPath = config.GetString("upload assets to s3.bucket")
s3Config := aws.NewConfig()
s3Config.CredentialsChainVerboseErrors = aws.Bool(true)
session, err := session.NewSession(s3Config)
if err != nil {
Logger.Fatal("Error initializing s3 uploader. " + err.Error())
os.Exit(0)
}
// the upload code
uploader = s3manager.NewUploader(session)
res, err := uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(s3UploadPath),
Key: aws.String(filename),
Body: f,
})
if err != nil {
log.Fatal("error on upload. " + err.Error())
}
// then continue with the download code
Attached screenshot showing that the download and upload operations are success through aws cli
Am I doing it wrong?
You dont need to specify credentials when using IAM role on EC2 instance.
I see you are getting Access Denied which means your Go program is able to pick the EC2 profile creds but probably due to lack of permissions, its getting this error.
Reading your code, it seems you want to write object to S3. Can you make sure you have given s3:Get*, s3:List*, s3:PutObject, s3:PutObjectAcl to your IAM Role and there is no explicit Deny on S3 Bucket policy?
I managed to solve this error by doing two things.
The first one is by using stscreds.NewCredentials(session, roleArn) as the credentials during session creation.
s3Config := aws.NewConfig()
s3Config.CredentialsChainVerboseErrors = aws.Bool(true)
s3Config.WithLogLevel(aws.LogDebugWithHTTPBody)
s3Config.Region = aws.String(region)
s3Config.WithHTTPClient(&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
},
Timeout: 10 * time.Second,
})
sess, err := session.NewSession(s3Config)
if err != nil {
log.Fatal("Error initializing session. " + err.Error())
}
sess.Config.Credentials = stscreds.NewCredentials(sess, arn)
_, err = sess.Config.Credentials.Get()
if err != nil {
log.Fatal("Error getting role. " + err.Error())
}
And the 2nd thing is by defining NO_PROXY environment variable with value is 169.254.169.254. The particular IP is AWS global IP used for getting the EC2 metadata.
And since my application uses proxy to communicate with the S3 server, I need to exclude that IP.

How to create Presigned put url and use environment variable to set Bucket and Key

I'm using the following code to create a presigned put url:
svc := s3.New(nil)
req, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String("myBucket"),
Key: aws.String("myKey"),
})
str, err := req.Presign(15 * time.Minute)
log.Println("The URL is:", str, " err:", err)
But I would like to get the configuration from a environment variable:
CONFIGURATIONS={ "Bucket": "myBucket", "Key": "myKey" }
I have just two weeks of Golang, and I have mainly a background in Node.js, so, I'm sorry if this question is very basic.
To better illustrate, I'm trying to do this... but in Go:
const CONFIGURATIONS = JSON.parse(process.env.CONFIGURATIONS)
const S3 = new AWS.S3()
S3.generatePresignedUrl('putObject', CONFIGURATIONS, callback...)
Thank you very much!
If I understand your question correctly you need to retrieve the bucket and key name from env variables. This is pretty easy to do in go. The os package has a GetEnv function for that.
Assuming your env variables are called AWS_BUCKET and AWS_KEY this is the way to get the values:
bucketName := Getenv("AWS_BUCKET")
key := Getenv("AWS_KEY")
You can check the package docs here.