I'm attempting to use the docker go-sdk to push an image to AWS ECR.
This is the code I'm using to push the image.
where tag = ".dkr.ecr.us-east-1.amazonaws.com/api:mytag"
func Push(c context.Context, tag string, credentials string) error {
cli, err := client.NewClient(apiSocket, apiVersion, nil, apiHeaders)
if err != nil {
return err
}
fmt.Println(credentials)
resp, err := cli.ImagePush(c, tag, types.ImagePushOptions{
RegistryAuth: credentials,
})
if err != nil {
panic(err)
}
io.Copy(os.Stdout, resp)
resp.Close()
return nil
}
But I keep getting this response:
{"status":"The push refers to repository [<id>.dkr.ecr.us-east-1.amazonaws.com/api]"}
{"status":"Preparing","progressDetail":{},"id":"23432919a50a"}
{"status":"Preparing","progressDetail":{},"id":"9387ad10e44c"}
{"status":"Preparing","progressDetail":{},"id":"e2a4679276bf"}
{"status":"Preparing","progressDetail":{},"id":"31c5c8035e63"}
{"status":"Preparing","progressDetail":{},"id":"a73789d39a06"}
{"status":"Preparing","progressDetail":{},"id":"f36942254806"}
{"status":"Preparing","progressDetail":{},"id":"4a2596f9aa79"}
{"status":"Preparing","progressDetail":{},"id":"5cf3066ccdbc"}
{"status":"Preparing","progressDetail":{},"id":"76a1661c28fc"}
{"status":"Preparing","progressDetail":{},"id":"beefb6beb20f"}
{"status":"Preparing","progressDetail":{},"id":"df64d3292fd6"}
{"status":"Waiting","progressDetail":{},"id":"beefb6beb20f"}
{"status":"Waiting","progressDetail":{},"id":"df64d3292fd6"}
{"errorDetail":{"message":"no basic auth credentials"},"error":"no basic auth credentials"}
Any ideas?
Notes:
I've verified that the credentials string I'm passing in is a base64 encoded user:pass for the ECR registry.
I've verified that the ECR credentials I'm getting are from the same AWS Region as where im attempting to push the image.
I found out in a GitHub comment that RegistryAuth actually needs to be a base64 JSON string with username and password fields. Ugh. This is undocumented in the Docker repository.
RegistryAuth = "{ \"username\": \"myusername\", \"password\": \"mypassword\", \"email\": \"myemail\" }
Relevant GitHub comment.
It is working for me now.
Related
I'm trying to migrate from aws-sdk-go to aws-sdk-go-v2. But, I am using localstack locally to mimic some aws services such as sqs and s3. I'm not sure how to configure the new sdk to use the localstack endpoint instead of the real one.
For example, in the v1 SDK I can point it to localstack by setting the endpoint here:
session.Must(session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
Endpoint: aws.String("http://localstack:4566"),
}))
But, how do I do this in the v2 SDK? I think I need to set some param in the config but I dont see any option to specify the endpoint.
So if you go trudging through the python code, principally this, you'll see:
https://github.com/localstack/localstack/blob/25ba1de8a8841af27feab54b8d55c80ac46349e2/localstack/services/edge.py#L115
I then needed to overwrite the authorization header, when using the v2 aws golang sdk in order to add the correct structure.
I picked the structure up from running the aws cli tool and trace logging the localstack docker container:
'Authorization': 'AWS4-HMAC-SHA256 Credential=AKIAR2X5NRNSRTCOJHCI/20210827/eu-west-1/sns/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=c69672d38631752ede15d90e7047a5183ebf3707a228decf6ec26e97fdbd02aa',
In go I then needed to overwrite the http client to add that header in:
type s struct {
cl http.Client
}
func (s s) Do(r *http.Request) (*http.Response, error) {
r.Header.Add("authorization", "AWS4-HMAC-SHA256 Credential=AKIAR2X5NRNSRTCOJHCI/20210827/eu-west-1/sns/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=33fa777a3bb1241f30742419b8fab81945aa219050da6e29b34db16053661000")
return s.cl.Do(r)
}
func NewSNS(endpoint, topicARN string) (awsPubSub, error) {
cfg := aws.Config{
EndpointResolver: aws.EndpointResolverFunc(func(service, region string) (aws.Endpoint, error) {
return aws.Endpoint{
PartitionID: "aws",
URL: "http://localhost:4566",
SigningRegion: "eu-west-1",
HostnameImmutable: true,
// Source: aws.EndpointSourceCustom,
}, nil
}),
HTTPClient: s{http.Client{}},
}
....
It was very time consuming and painful and I'd love to know a better way, but this works for the time being...
It depends from the service that you use.
In order to initialize a Glue client:
cfg, err := config.LoadDefaultConfig(context.Background())
if err != nil {
panic(err)
}
glueConnection := glue.New(glue.Options{Credentials: cfg.Credentials, Region: cfg.Region})
The equivalent of your code in the SDK v2 is:
cfg, err := config.LoadDefaultConfig(
ctx,
config.WithRegion("us-east-1"),
config.WithEndpointResolverWithOptions(aws.EndpointResolverWithOptionsFunc(
func(service, region string, options ...interface{}) (aws.Endpoint, error) {
return aws.Endpoint{URL: "http://localhost:4566"}, nil
}),
),
)
Using LoadDefaultConfig you will not need to specify the region if you already set it up on the AWS config. You can read more on the AWS SDK v2 docs.
You can find the above example in the package docs.
I have the following snippet:
awsCredentials := credentials.NewChainCredentials(
[]credentials.Provider{
&ec2rolecreds.EC2RoleProvider{
Client: ec2metadata.New(newSession, aws.NewConfig()),
},
&credentials.SharedCredentialsProvider{},
&credentials.EnvProvider{},
})
which works fine whenever the code is running on an EC2 instance or when the access/secret key are passed through variables (used for local testing).
However, this code is failing when running on ECS+Fargate because NoCredentialProviders: no valid providers in chain. Checked the environment variables of the running container and it has the expected AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, so the credentials.EnvProvider should read it.
So, my question is, what's the correct way of reading these credentials? Because the problem I'm facing is not about lack of permissions (which would indicate an error in the policy / role), but that code is not able to get the credentials.
UPDATE
I have narrowed this to the use of ec2rolescreds.
Using this simple example:
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
"github.com/aws/aws-sdk-go/aws/ec2metadata"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
newSession, err := session.NewSession()
if err != nil {
log.Fatal(err)
}
awsCredentials := credentials.NewChainCredentials(
[]credentials.Provider{
&ec2rolecreds.EC2RoleProvider{
Client: ec2metadata.New(newSession, aws.NewConfig()),
},
&credentials.SharedCredentialsProvider{},
&credentials.EnvProvider{},
})
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: awsCredentials},
)
if err != nil {
log.Fatal(err)
}
// Create S3 service client
svc := s3.New(sess)
result, err := svc.ListBuckets(nil)
if err != nil {
log.Fatal(err)
}
fmt.Println("Buckets:")
for _, b := range result.Buckets {
fmt.Printf("* %s created on %s\n",
aws.StringValue(b.Name), aws.TimeValue(b.CreationDate))
}
}
If I remove ec2rolescreds, everything works fine both local and in ECS+Fargate.
However, if I run this code as is, I get the same error of NoCredentialProviders: no valid providers in chain
The solution is to initialize clients using sessions instead of credentials, i.e:
conf := aws.NewConfig().WithRegion("us-east-1")
sess := session.Must(session.NewSession(conf))
svc := s3.New(sess)
// others:
// svc := sqs.New(sess)
// svc := dynamodb.New(sess)
// ...
Because, as #Ay0 points, the default credential chain already includes both EnvProvider and RemoteCredProvider.
In case you still need the credentials, you can use:
creds := stscreds.NewCredentials(sess, "myRoleARN")
as the documentation points out. Notice that the policy of the role must have the sts:AssumeRole action enabled. For more information, here are the stscreds.NewCredentials(...) docs
So, a session can be configured using a Config object.
Reading through the specs of this object, it says for Credentials:
// The credentials object to use when signing requests. Defaults to a
// chain of credential providers to search for credentials in environment
// variables, shared credential file, and EC2 Instance Roles.
Credentials *credentials.Credentials
The defaults are already what my snippet was doing, so I removed all the awsCredentials block and now it's working fine everywhere. Locally, EC2, Fargate...
UPDATE
To expand the answer, the reason why removing the awsCredentials made this work is because, if you check the SDK's code, https://github.com/aws/aws-sdk-go/blob/master/aws/defaults/defaults.go#L107, the default credentials check both EnvProvider and RemoteCredProvider.
By overriding the default chain credentials, it was not able to look for credentials in RemoteCredProvider, which is the provider that handles the environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI.
My application use s3 and running on EC2. The IAM is configured on the instance, so the auth happen keyless (without the access key and secret key).
I'm able to upload or download file using aws cli. However when I tried to perform download operation using aws-sdk-go, I get error below:
AccessDenied: Access Denied
status code: 403, request id: F945BDB5410E1A00, host id: m74jJ8z/AEzdkaJkWKdIqPEwPIYPZfWnLLfa5UpEwHwaBcXOuXTPY1aw/u/5HGralKg+ewAWEJA=
I followed the official guide from https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/ec2rolecreds/ and from this issue https://github.com/aws/aws-sdk-go/issues/430 but got the error above.
Below is my code:
s3UploadPath = config.GetString("upload assets to s3.bucket")
s3Config := aws.NewConfig()
s3Config.CredentialsChainVerboseErrors = aws.Bool(true)
session, err := session.NewSession(s3Config)
if err != nil {
Logger.Fatal("Error initializing s3 uploader. " + err.Error())
os.Exit(0)
}
// the upload code
uploader = s3manager.NewUploader(session)
res, err := uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(s3UploadPath),
Key: aws.String(filename),
Body: f,
})
if err != nil {
log.Fatal("error on upload. " + err.Error())
}
// then continue with the download code
Attached screenshot showing that the download and upload operations are success through aws cli
Am I doing it wrong?
You dont need to specify credentials when using IAM role on EC2 instance.
I see you are getting Access Denied which means your Go program is able to pick the EC2 profile creds but probably due to lack of permissions, its getting this error.
Reading your code, it seems you want to write object to S3. Can you make sure you have given s3:Get*, s3:List*, s3:PutObject, s3:PutObjectAcl to your IAM Role and there is no explicit Deny on S3 Bucket policy?
I managed to solve this error by doing two things.
The first one is by using stscreds.NewCredentials(session, roleArn) as the credentials during session creation.
s3Config := aws.NewConfig()
s3Config.CredentialsChainVerboseErrors = aws.Bool(true)
s3Config.WithLogLevel(aws.LogDebugWithHTTPBody)
s3Config.Region = aws.String(region)
s3Config.WithHTTPClient(&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
},
Timeout: 10 * time.Second,
})
sess, err := session.NewSession(s3Config)
if err != nil {
log.Fatal("Error initializing session. " + err.Error())
}
sess.Config.Credentials = stscreds.NewCredentials(sess, arn)
_, err = sess.Config.Credentials.Get()
if err != nil {
log.Fatal("Error getting role. " + err.Error())
}
And the 2nd thing is by defining NO_PROXY environment variable with value is 169.254.169.254. The particular IP is AWS global IP used for getting the EC2 metadata.
And since my application uses proxy to communicate with the S3 server, I need to exclude that IP.
I wanted to upload an image in aws s3.
const (
S3_REGION = ""
S3_BUCKET = ""
)
func main() {
// Create a single AWS session (we can re use this if we're uploading many files)
s, err := session.NewSession(&aws.Config{Region: aws.String(S3_REGION)})
if err != nil {
log.Fatal(err)
}
// Upload
err = AddFileToS3(s, "result.csv")
if err != nil {
log.Fatal(err)
}
}
I am stuck here.
Where can I get S3_REGION as per this code standard?
Source : https://golangcode.com/uploading-a-file-to-s3/
When you log in to the AWS console you see on the top right which region you are logged in to, for example "Oregon" which refers to the "us-west-2" region.
Please refer to this table and this link.
I am working in Golang,now I am attempting to upload an image to AWS S3, but I get:
NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
My code is like this:
func firstFunction(){
//Connect to S3
AWSsession, err := ConnectAWS()
if err != nil {
fmt.Println("Error Connecting to AWS S3")
}
GetSingleMedia(AWSsession)
}
func ConnectAWS()(*session.Session, error){
//Create S3 Session
AWSsession, err := session.NewSession(&aws.Config{
Region: aws.String("us-west-2")},
)
if err != nil {
fmt.Println("Error AWS:", err.Error())
}
return AWSsession,err
}
func GetSingleMedia(...someparams,AWSsession *session.Session){
//o.Blob is correct, this is valid
data, err := ioutil.ReadAll(bytes.NewReader(o.Blob))
//Store: bytes.NewReader(o.Blob)
UploadImage(AWSsession,bytes.NewReader(o.Blob),bucket,"SomeID")
}
func UploadImage(AWSsession *session.Session,reader *bytes.Reader,bucket string, key string) (*s3manager.UploadOutput,error){
uploader := s3manager.NewUploader(AWSsession)
result, err := uploader.Upload(&s3manager.UploadInput{
Body : reader,
Bucket: aws.String(bucket),
Key : aws.String(key),
})
if err != nil {
fmt.Println("Error uploagin img: ",err.Error())
}
return result,err
}
Also, I have placed the creentials under /home/myuser/.aws/ there's a credential file, I don't get any error on creating the session, then, what could be the problem?
The error is triggered in UploadImage
EDIT:
Currently in the credentials file I have:
[default]
awsBucket = "someBucket"
awsAccessKey = "SOME_ACCESS_KEY"
awsSecretKey = "SOME_AWS_SECRET_KEY"
Sould I change any permission or something?
I would suggest you follow the guide here: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
This command will ask you for access/secret key and write them in a correct format:
aws configure
It would appear you have a wrong format of credentials file. The correct format would be something like this:
[default]
aws_access_key_id = SOME_ACCESS_KEY
aws_secret_access_key = SOME_AWS_SECRET_KEY