I am attempting to build an Amazon S3 client in GoLang but I am having trouble making API calls. I'm receiving an error that says "no such host" but I am positive the credentials I'm providing are correct.
Defining a struct to hold the client
// the Client struct holding the client itself as well as the bucket.
type S3Client struct {
S3clientObject s3.S3
bucket string
}
// Initialize the client
func CreateS3Client() S3Client{
S3clientCreate := S3Client{S3clientObject: Connect(), bucket: GetS3Bucket()}
if (!CheckBuckets(S3clientCreate)) {
exitErrorf("Bucket does not exist, try again.")
}
return S3clientCreate
}
Connecting to the bucket
func Connect() s3.S3{
// Initialize a session
sess, err := session.NewSession(&aws.Config{
Credentials: credentials.NewStaticCredentials("myCredentials", "myCreds", ""),
Endpoint: aws.String("myDomain"),
Region: aws.String("myRegion"),
},
)
if err != nil {
exitErrorf("Unable to use credentials, %v", err)
}
// Create S3 service client
svc := s3.New(sess)
return *svc
}
At this point, I am able to establish a connection and use the ListBuckets functionality to receive a list of all the buckets (like this: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#S3.ListBuckets)
When I try to call the GetObject API, it tells me it cannot find the host
// Gets an object from the bucket
func Get(client S3Client, key string) interface{} {
// golang does not support "default values" so I used a nil (same as null)
if (key == "") {
return nil
}
svc := client.S3clientObject
input := &s3.GetObjectInput{
Bucket: aws.String("myBucket"),
Key: aws.String("myPathKey"),
}
result, err := svc.GetObject(input)
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case s3.ErrCodeNoSuchKey:
fmt.Println(s3.ErrCodeNoSuchKey, aerr.Error())
case s3.ErrCodeInvalidObjectState:
fmt.Println(s3.ErrCodeInvalidObjectState, aerr.Error())
default:
fmt.Println(aerr.Error())
}
} else {
fmt.Println(err.Error())
}
}
return result
}
This returns:
dial tcp: lookup "hostname": no such host
I cannot figure out why this is happening, because I am able to successfully make a connection to the bucket, and list them out using ListBuckets, but when using another API call, it fails to find the host. Is there something wrong with my code? Is there another configuration that I forgot about?
Any help or guidance is greatly appreciated as I'm somewhat new to using GoLang and S3.
Apparently the issue was with the bucket name. All I did to resolve this was put a "/" in front of the bucket name when creating it and it worked.
Related
I am at the moment trying to set up a connection to AWS Neptune via go, but its not working. I am able to connect to AWS itself, but when I try to connect to Neptune DB it says "no successful connections could be made: dial tcp 172.31.4.48:8182: i/o timeout". I am using the Gremlingo module like in this code
package main
import (
"fmt"
"net/http"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/neptune"
"github.com/gin-gonic/gin"
gremlingo "github.com/apache/tinkerpop/gremlin-go/v3/driver"
)
func main() {
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-id1"),
Credentials: credentials.NewStaticCredentials("AWS-id key", "aws secret id key", ""),
})
if err != nil {
fmt.Println("Couldn't create new session")
return
}
neptune.New(sess)
driverRemoteConnection, err := gremlingo.NewDriverRemoteConnection("wss://database-1-instance-1.asdasdasd.us-east-1.neptune.amazonaws.com:8182/gremlin",
func(settings *gremlingo.DriverRemoteConnectionSettings) {
settings.TraversalSource = "g"
})
if err != nil {
fmt.Println(err)
return
}
//Cleanup
defer driverRemoteConnection.Close()
//Creating graph traversal
g := gremlingo.Traversal_().WithRemote(driverRemoteConnection)
// Perform traversal
results, err := g.V().Limit(2).ToList()
if err != nil {
fmt.Println(err)
return
}
// print results
for _, r := range results {
fmt.Println(r.GetString())
}
}
I wasn't quite sure what the problem was so I tried to connect to the cluster itself and as it didn't work I tried to connect to the Writer.
Thank you very much for your help.
Best regards
Amazon Neptune runs inside of a VPC and does not expose a public endpoint. Code designed to send queries must have access tho that VPC. This could be as simple as the code running on an EC2 instance in the same VPC, but there are many other ways that access to a VPC can be granted, such as Load Balancers, VPC Peering, Direct Connect, and many others.
An easy way to check if your code can access the database, is to send an HTTP request to the /status API from the same point of origin and see if it works.
I have the following snippet:
awsCredentials := credentials.NewChainCredentials(
[]credentials.Provider{
&ec2rolecreds.EC2RoleProvider{
Client: ec2metadata.New(newSession, aws.NewConfig()),
},
&credentials.SharedCredentialsProvider{},
&credentials.EnvProvider{},
})
which works fine whenever the code is running on an EC2 instance or when the access/secret key are passed through variables (used for local testing).
However, this code is failing when running on ECS+Fargate because NoCredentialProviders: no valid providers in chain. Checked the environment variables of the running container and it has the expected AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, so the credentials.EnvProvider should read it.
So, my question is, what's the correct way of reading these credentials? Because the problem I'm facing is not about lack of permissions (which would indicate an error in the policy / role), but that code is not able to get the credentials.
UPDATE
I have narrowed this to the use of ec2rolescreds.
Using this simple example:
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
"github.com/aws/aws-sdk-go/aws/ec2metadata"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
newSession, err := session.NewSession()
if err != nil {
log.Fatal(err)
}
awsCredentials := credentials.NewChainCredentials(
[]credentials.Provider{
&ec2rolecreds.EC2RoleProvider{
Client: ec2metadata.New(newSession, aws.NewConfig()),
},
&credentials.SharedCredentialsProvider{},
&credentials.EnvProvider{},
})
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: awsCredentials},
)
if err != nil {
log.Fatal(err)
}
// Create S3 service client
svc := s3.New(sess)
result, err := svc.ListBuckets(nil)
if err != nil {
log.Fatal(err)
}
fmt.Println("Buckets:")
for _, b := range result.Buckets {
fmt.Printf("* %s created on %s\n",
aws.StringValue(b.Name), aws.TimeValue(b.CreationDate))
}
}
If I remove ec2rolescreds, everything works fine both local and in ECS+Fargate.
However, if I run this code as is, I get the same error of NoCredentialProviders: no valid providers in chain
The solution is to initialize clients using sessions instead of credentials, i.e:
conf := aws.NewConfig().WithRegion("us-east-1")
sess := session.Must(session.NewSession(conf))
svc := s3.New(sess)
// others:
// svc := sqs.New(sess)
// svc := dynamodb.New(sess)
// ...
Because, as #Ay0 points, the default credential chain already includes both EnvProvider and RemoteCredProvider.
In case you still need the credentials, you can use:
creds := stscreds.NewCredentials(sess, "myRoleARN")
as the documentation points out. Notice that the policy of the role must have the sts:AssumeRole action enabled. For more information, here are the stscreds.NewCredentials(...) docs
So, a session can be configured using a Config object.
Reading through the specs of this object, it says for Credentials:
// The credentials object to use when signing requests. Defaults to a
// chain of credential providers to search for credentials in environment
// variables, shared credential file, and EC2 Instance Roles.
Credentials *credentials.Credentials
The defaults are already what my snippet was doing, so I removed all the awsCredentials block and now it's working fine everywhere. Locally, EC2, Fargate...
UPDATE
To expand the answer, the reason why removing the awsCredentials made this work is because, if you check the SDK's code, https://github.com/aws/aws-sdk-go/blob/master/aws/defaults/defaults.go#L107, the default credentials check both EnvProvider and RemoteCredProvider.
By overriding the default chain credentials, it was not able to look for credentials in RemoteCredProvider, which is the provider that handles the environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI.
Not sure what is going on, this code worked once yesterday. Now no matter what value I use, AWS is returning a error that it already exists, but that's impossible.
2020/04/17 19:10:30 error ResourceExistsException: The operation failed because the secret /gog1/RandomSiteName3 already exists.
_, err = PutParam("/gog1/RandomSiteName3", "test", true, EventGuid)
if err != nil {
log.Printf("error writing secret: %v ", err)
return
}
func PutParam(paramName string, paramValue string, encrypt bool, guid string) (output string, err error) {
svc := secretsmanager.New(AWSSession)
input := &secretsmanager.CreateSecretInput{
// ClientRequestToken: aws.String(guid),
// Description: aws.String("My test database secret created with the CLI"),
Name: aws.String(paramName),
SecretString: aws.String(paramValue),
}
fmt.Printf("putting secret key: %v", paramName)
_, err = svc.CreateSecret(input)
if err != nil {
return "", err
}
return
}
It was due to an s3 trigger firing in a loop:
NOTE: If writing to the bucket that triggers the notification, this
could cause an execution loop. For example, if the bucket triggers a
Lambda function each time an object is uploaded, and the function
uploads an object to the bucket, then the function indirectly triggers
itself. To avoid this, use two buckets, or configure the trigger to
only apply to a prefix used for incoming objects.
I am new to Go and have written a function that uses the AWS Secrets Manager to fetch a secret:
//Helper function to get secret from AWS Secret Manager
func getAWSSecrets() (secretMap map[string]string, err error) {
// Create new AWS session in order to get db info from SecretsManager
sess, err := session.NewSession()
if err != nil {
return nil, err
}
// Create a new instance of the SecretsManager client with session
svc := secretsmanager.New(sess)
//Get secret config values
req, resp := svc.GetSecretValueRequest(&secretsmanager.GetSecretValueInput{
SecretId: aws.String("my/secret/string"),
})
err = req.Send()
if err != nil {
return nil, err
}
...
}
I need to create a unit test for the function, and to do so I need to mock the AWS Secrets Manager. I discovered a Secrets Manager Interface that AWS was created to help with unit testing. In the example displayed, the AWS Secrets Manager is passed into the function being tested, making it easy to pass in the mock service. Is this the only way to successfully unit test the function? Or can the service be mocked in the function I have above?
As the comments say, make the function call a method, and take advantage of the interface AWS is providing.
I would create a service, like this one:
package service
type SecretService struct {
AwsProvider aws.SecretsManagerAPI
}
func NewSecretService() (*SecretService, err) {
// Create new AWS session in order to get db info from SecretsManager
sess, err := session.NewSession()
if err != nil {
return nil, err
}
return &SecretService{
AwsProvider: secretsmanager.New(sess),
}, nil
}
func (s *SecretService) GetAWSSecrets() {
req, resp := s.AwsProvider.GetSecretValueRequest(&secretsmanager.GetSecretValueInput{
SecretId: aws.String("my/secret/string"),
})
err = req.Send()
if err != nil {
return nil, err
}
// More magic below ...
}
That way in the tests, I could pass any mock into SecretService, like this:
func TestSecretService_GetAWSSecrets(t *testing.T) {
service := &service.SecretService{
AwsProvider: <use your mock here>
}
}
One caveat, I guess is that the mock has to implement all methods of SecretsManagerAPI, which I think it's a lot of work for this simple scenario... in any case you can create your own interface inside the service package with only the subset of methods you'll use, let's say you're only going to use GetSecretValueRequest, and CreateSecret:
package service
type SecretProvider interface {
CreateSecret(*secretsmanager.CreateSecretInput) (*secretsmanager.CreateSecretOutput, error)
GetSecretValueRequest(*secretsmanager.GetSecretValueInput) (*request.Request, *secretsmanager.GetSecretValueOutput)
}
Change the service:
type SecretService struct {
AwsProvider SecretProvider
}
Now your mock only has to implement SecretProvider interface methods only. And of course, AWS SecretsManager implicitly impleements SecretProvider.
I am working in Golang,now I am attempting to upload an image to AWS S3, but I get:
NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
My code is like this:
func firstFunction(){
//Connect to S3
AWSsession, err := ConnectAWS()
if err != nil {
fmt.Println("Error Connecting to AWS S3")
}
GetSingleMedia(AWSsession)
}
func ConnectAWS()(*session.Session, error){
//Create S3 Session
AWSsession, err := session.NewSession(&aws.Config{
Region: aws.String("us-west-2")},
)
if err != nil {
fmt.Println("Error AWS:", err.Error())
}
return AWSsession,err
}
func GetSingleMedia(...someparams,AWSsession *session.Session){
//o.Blob is correct, this is valid
data, err := ioutil.ReadAll(bytes.NewReader(o.Blob))
//Store: bytes.NewReader(o.Blob)
UploadImage(AWSsession,bytes.NewReader(o.Blob),bucket,"SomeID")
}
func UploadImage(AWSsession *session.Session,reader *bytes.Reader,bucket string, key string) (*s3manager.UploadOutput,error){
uploader := s3manager.NewUploader(AWSsession)
result, err := uploader.Upload(&s3manager.UploadInput{
Body : reader,
Bucket: aws.String(bucket),
Key : aws.String(key),
})
if err != nil {
fmt.Println("Error uploagin img: ",err.Error())
}
return result,err
}
Also, I have placed the creentials under /home/myuser/.aws/ there's a credential file, I don't get any error on creating the session, then, what could be the problem?
The error is triggered in UploadImage
EDIT:
Currently in the credentials file I have:
[default]
awsBucket = "someBucket"
awsAccessKey = "SOME_ACCESS_KEY"
awsSecretKey = "SOME_AWS_SECRET_KEY"
Sould I change any permission or something?
I would suggest you follow the guide here: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
This command will ask you for access/secret key and write them in a correct format:
aws configure
It would appear you have a wrong format of credentials file. The correct format would be something like this:
[default]
aws_access_key_id = SOME_ACCESS_KEY
aws_secret_access_key = SOME_AWS_SECRET_KEY