Not sure what is going on, this code worked once yesterday. Now no matter what value I use, AWS is returning a error that it already exists, but that's impossible.
2020/04/17 19:10:30 error ResourceExistsException: The operation failed because the secret /gog1/RandomSiteName3 already exists.
_, err = PutParam("/gog1/RandomSiteName3", "test", true, EventGuid)
if err != nil {
log.Printf("error writing secret: %v ", err)
return
}
func PutParam(paramName string, paramValue string, encrypt bool, guid string) (output string, err error) {
svc := secretsmanager.New(AWSSession)
input := &secretsmanager.CreateSecretInput{
// ClientRequestToken: aws.String(guid),
// Description: aws.String("My test database secret created with the CLI"),
Name: aws.String(paramName),
SecretString: aws.String(paramValue),
}
fmt.Printf("putting secret key: %v", paramName)
_, err = svc.CreateSecret(input)
if err != nil {
return "", err
}
return
}
It was due to an s3 trigger firing in a loop:
NOTE: If writing to the bucket that triggers the notification, this
could cause an execution loop. For example, if the bucket triggers a
Lambda function each time an object is uploaded, and the function
uploads an object to the bucket, then the function indirectly triggers
itself. To avoid this, use two buckets, or configure the trigger to
only apply to a prefix used for incoming objects.
Related
I have setup cross account reading kinesis stream, but i get security token expired error when kinesis client is reading records. I used sts assume role to assume roleA in accountA, then use roleA credentials to assume roleB, lastly return the kinesis client, so there is no refresh feature applied to it and the client will expire in 1 hr by default. I looked up the stscreds AssumeRoleProvider and the doc says it will refresh the credentials. But i have no idea on how to refresh the first credential for assumed roleA then refresh the second credential for assumed roleB. Or is it better to call the method to reinitialize the kinesis client?
Here is the code block.
cfg, err := config.LoadDefaultConfig(
context.TODO(),
config.WithRegion("us-west-2"),
)
if err != nil {
return nil, err
}
stsclient := sts.NewFromConfig(cfg)
assumingcnf, err := config.LoadDefaultConfig(
context.TODO(),
config.WithRegion("us-west-2"),
config.WithCredentialsProvider(aws.NewCredentialsCache(
stscreds.NewAssumeRoleProvider(
stsclient,
roleToAssumeArn1,
)),
),
)
if err != nil {
return nil, err
}
stsclient = sts.NewFromConfig(assumingcnf)
cnf, err := config.LoadDefaultConfig(
context.TODO(),
config.WithRegion("us-west-2"),
config.WithCredentialsProvider(aws.NewCredentialsCache(
stscreds.NewAssumeRoleProvider(
stsclient,
roleToAssumeArn2,
)),
),
)
if err != nil {
return nil, err
}
kClient := kinesis.NewFromConfig(cnf)
return kClient
You should be able to do this with the providers provided by AWS. I'm assuming you're using aws-sdk-go-v2.
This would make the resulting CredentialsProvider return the cached credentials until they expire; then it will call provider2, which uses sts2 to get new credentials for roleB, and sts2 will always first call provider1 first to get new credentials for roleA.
func createProvider(cfg aws.Config) aws.CredentialsProvider {
sts1 := sts.NewFromConfig(cfg)
provider1 := stscreds.NewAssumeRoleProvider(sts1, "roleA")
sts2 := sts.NewFromConfig(cfg, func (o *sts.Options) { o.Credentials = provider1 })
provider2 := stscreds.NewAssumeRoleProvider(sts2, "roleB")
return aws.NewCredentialsCache(provider2)
}
I am attempting to build an Amazon S3 client in GoLang but I am having trouble making API calls. I'm receiving an error that says "no such host" but I am positive the credentials I'm providing are correct.
Defining a struct to hold the client
// the Client struct holding the client itself as well as the bucket.
type S3Client struct {
S3clientObject s3.S3
bucket string
}
// Initialize the client
func CreateS3Client() S3Client{
S3clientCreate := S3Client{S3clientObject: Connect(), bucket: GetS3Bucket()}
if (!CheckBuckets(S3clientCreate)) {
exitErrorf("Bucket does not exist, try again.")
}
return S3clientCreate
}
Connecting to the bucket
func Connect() s3.S3{
// Initialize a session
sess, err := session.NewSession(&aws.Config{
Credentials: credentials.NewStaticCredentials("myCredentials", "myCreds", ""),
Endpoint: aws.String("myDomain"),
Region: aws.String("myRegion"),
},
)
if err != nil {
exitErrorf("Unable to use credentials, %v", err)
}
// Create S3 service client
svc := s3.New(sess)
return *svc
}
At this point, I am able to establish a connection and use the ListBuckets functionality to receive a list of all the buckets (like this: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#S3.ListBuckets)
When I try to call the GetObject API, it tells me it cannot find the host
// Gets an object from the bucket
func Get(client S3Client, key string) interface{} {
// golang does not support "default values" so I used a nil (same as null)
if (key == "") {
return nil
}
svc := client.S3clientObject
input := &s3.GetObjectInput{
Bucket: aws.String("myBucket"),
Key: aws.String("myPathKey"),
}
result, err := svc.GetObject(input)
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case s3.ErrCodeNoSuchKey:
fmt.Println(s3.ErrCodeNoSuchKey, aerr.Error())
case s3.ErrCodeInvalidObjectState:
fmt.Println(s3.ErrCodeInvalidObjectState, aerr.Error())
default:
fmt.Println(aerr.Error())
}
} else {
fmt.Println(err.Error())
}
}
return result
}
This returns:
dial tcp: lookup "hostname": no such host
I cannot figure out why this is happening, because I am able to successfully make a connection to the bucket, and list them out using ListBuckets, but when using another API call, it fails to find the host. Is there something wrong with my code? Is there another configuration that I forgot about?
Any help or guidance is greatly appreciated as I'm somewhat new to using GoLang and S3.
Apparently the issue was with the bucket name. All I did to resolve this was put a "/" in front of the bucket name when creating it and it worked.
I have created REST API using Go Iris framework. Now I want to deploy these API's on AWS with lambda function. I am using MySQL as database. Is it possible to deploy my Go executable file on AWS lambda or should I need to modify my code according to AWS lambda specifications? I am trying to find the solution, but not getting much information.
Here is one of my API end point.
package main
import (
"database/sql"
"github.com/kataras/iris"
"github.com/kataras/iris/middleware/logger"
"github.com/kataras/iris/middleware/recover"
)
type Reward struct {
Id int `json:"reward_id"`
LotteryID int `json:"lottery_id"`
RewardName string `json:"reward_name"`
Description string `json:"reward_description"`
Asset int `json:"reward_asset"`
AssetName string `json:"reward_asset_name"`
}
func dbConn() (db *sql.DB) {
dbDriver := "mysql"
dbUser := "xxx"
dbPass := "xxx"
dbName := "xxx"
db, err := sql.Open(xxxxxxxxx)
if err != nil {
panic(err.Error())
}
return db
}
func newApp() *iris.Application {
app := iris.New()
app.Logger().SetLevel("debug")
app.Use(recover.New())
app.Use(logger.New())
db := dbConn()
app.Get("/reward/{reward_id:int}", func(ctx iris.Context) {
id1 := ctx.Params().GetIntDefault("reward_id", 0)
stmtOut, err := db.Prepare("select id, lottery_id,reward_name,reward_description,reward_asset, reward_asset_name from rewards_table where id =?")
if err != nil {
panic(err.Error())
}
defer stmtOut.Close()
var id, lotteryId, rewardAsset int
var rewardName, rewardDescription, rewardAssetName string
err1 := stmtOut.QueryRow(id1).Scan(&id, &lotteryId, &rewardName, &rewardDescription, &rewardAsset, &rewardAssetName)
if err1 != nil {
panic(err.Error())
}
reward := Reward{
Id: id,
LotteryID: lotteryId,
RewardName: rewardName,
Description: rewardDescription,
Asset: rewardAsset,
AssetName: rewardAssetName,
}
ctx.JSON(&reward)
})
return app
}
func main() {
app := newApp()
app.Run(iris.Addr(":8080"), iris.WithoutServerError(iris.ErrServerClosed), iris.WithOptimizations)
}
I have few more API endpoints which do basic CRUD operations. I am thinking about using AWS lambda and AWS API Gateway.
should I need to modify my code according to AWS lambda specifications?
Yes. Your code for lambda will require to have a handler:
AWS Lambda function handler in Go
This is the entry point to your function.
Also it seems that your go program is a web server build on iris. If this is the case, you won't be able to use it anyway, as you can't invoke lambda from internet as you would a regular server.
Also lambda runs for max 15 minutes, thus its use as a server would be very limited.
I am new to Go and have written a function that uses the AWS Secrets Manager to fetch a secret:
//Helper function to get secret from AWS Secret Manager
func getAWSSecrets() (secretMap map[string]string, err error) {
// Create new AWS session in order to get db info from SecretsManager
sess, err := session.NewSession()
if err != nil {
return nil, err
}
// Create a new instance of the SecretsManager client with session
svc := secretsmanager.New(sess)
//Get secret config values
req, resp := svc.GetSecretValueRequest(&secretsmanager.GetSecretValueInput{
SecretId: aws.String("my/secret/string"),
})
err = req.Send()
if err != nil {
return nil, err
}
...
}
I need to create a unit test for the function, and to do so I need to mock the AWS Secrets Manager. I discovered a Secrets Manager Interface that AWS was created to help with unit testing. In the example displayed, the AWS Secrets Manager is passed into the function being tested, making it easy to pass in the mock service. Is this the only way to successfully unit test the function? Or can the service be mocked in the function I have above?
As the comments say, make the function call a method, and take advantage of the interface AWS is providing.
I would create a service, like this one:
package service
type SecretService struct {
AwsProvider aws.SecretsManagerAPI
}
func NewSecretService() (*SecretService, err) {
// Create new AWS session in order to get db info from SecretsManager
sess, err := session.NewSession()
if err != nil {
return nil, err
}
return &SecretService{
AwsProvider: secretsmanager.New(sess),
}, nil
}
func (s *SecretService) GetAWSSecrets() {
req, resp := s.AwsProvider.GetSecretValueRequest(&secretsmanager.GetSecretValueInput{
SecretId: aws.String("my/secret/string"),
})
err = req.Send()
if err != nil {
return nil, err
}
// More magic below ...
}
That way in the tests, I could pass any mock into SecretService, like this:
func TestSecretService_GetAWSSecrets(t *testing.T) {
service := &service.SecretService{
AwsProvider: <use your mock here>
}
}
One caveat, I guess is that the mock has to implement all methods of SecretsManagerAPI, which I think it's a lot of work for this simple scenario... in any case you can create your own interface inside the service package with only the subset of methods you'll use, let's say you're only going to use GetSecretValueRequest, and CreateSecret:
package service
type SecretProvider interface {
CreateSecret(*secretsmanager.CreateSecretInput) (*secretsmanager.CreateSecretOutput, error)
GetSecretValueRequest(*secretsmanager.GetSecretValueInput) (*request.Request, *secretsmanager.GetSecretValueOutput)
}
Change the service:
type SecretService struct {
AwsProvider SecretProvider
}
Now your mock only has to implement SecretProvider interface methods only. And of course, AWS SecretsManager implicitly impleements SecretProvider.
I am working in Golang,now I am attempting to upload an image to AWS S3, but I get:
NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
My code is like this:
func firstFunction(){
//Connect to S3
AWSsession, err := ConnectAWS()
if err != nil {
fmt.Println("Error Connecting to AWS S3")
}
GetSingleMedia(AWSsession)
}
func ConnectAWS()(*session.Session, error){
//Create S3 Session
AWSsession, err := session.NewSession(&aws.Config{
Region: aws.String("us-west-2")},
)
if err != nil {
fmt.Println("Error AWS:", err.Error())
}
return AWSsession,err
}
func GetSingleMedia(...someparams,AWSsession *session.Session){
//o.Blob is correct, this is valid
data, err := ioutil.ReadAll(bytes.NewReader(o.Blob))
//Store: bytes.NewReader(o.Blob)
UploadImage(AWSsession,bytes.NewReader(o.Blob),bucket,"SomeID")
}
func UploadImage(AWSsession *session.Session,reader *bytes.Reader,bucket string, key string) (*s3manager.UploadOutput,error){
uploader := s3manager.NewUploader(AWSsession)
result, err := uploader.Upload(&s3manager.UploadInput{
Body : reader,
Bucket: aws.String(bucket),
Key : aws.String(key),
})
if err != nil {
fmt.Println("Error uploagin img: ",err.Error())
}
return result,err
}
Also, I have placed the creentials under /home/myuser/.aws/ there's a credential file, I don't get any error on creating the session, then, what could be the problem?
The error is triggered in UploadImage
EDIT:
Currently in the credentials file I have:
[default]
awsBucket = "someBucket"
awsAccessKey = "SOME_ACCESS_KEY"
awsSecretKey = "SOME_AWS_SECRET_KEY"
Sould I change any permission or something?
I would suggest you follow the guide here: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
This command will ask you for access/secret key and write them in a correct format:
aws configure
It would appear you have a wrong format of credentials file. The correct format would be something like this:
[default]
aws_access_key_id = SOME_ACCESS_KEY
aws_secret_access_key = SOME_AWS_SECRET_KEY