I'm using the follow code to create a item in my dynamodb table:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/endpoints"
"github.com/aws/aws-sdk-go-v2/aws/external"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/dynamodbattribute"
)
type Record struct {
ID string
URLs []string
}
// Using the SDK's default configuration, loading additional config
// and credentials values from the environment variables, shared
// credentials, and shared configuration files
var cfg, err = external.LoadDefaultAWSConfig()
func createNewExtraction() error {
svc := dynamodb.New(cfg)
r := Record{
ID: "ABC123",
URLs: []string{
"https://example.com/first/link",
"https://example.com/second/url",
},
}
item, err := dynamodbattribute.MarshalMap(r)
if err != nil {
panic(fmt.Sprintf("failed to DynamoDB marshal Record, %v", err))
}
req := svc.PutItemRequest(&dynamodb.PutItemInput{
TableName: aws.String("test"), Item: item })
_, err = req.Send(); if err != nil {
return err
}
return nil
}
func main() {
if len(cfg.Region) > 0 {
// Set Region to us-east-1 as default.
cfg.Region = endpoints.UsEast1RegionID
}
err = createNewExtraction(); if err != nil {
panic(err.Error())
}
}
But it's returning the error:
panic: ValidationException: One or more parameter values were invalid: Missing the key id in the item
status code: 400, request id: F3VCQSGRIG5GM5PEJE7T5M4CEVVV4KQNSO5AEMVJF66Q9ASUAAJG
goroutine 1 [running]:
main.main()
/Users/user/go/src/Test2/test.go:56 +0x102
exit status 2
I already tried to declare Id, id and ID in my Record struct, but it doesn't work.
The stranger is: I got this code in the official documentation (I'm updating to work with the aws-sdk-go-v2).
Thank you in advance.
I do not know golang, but I had similar problems in nodejs.
Make sure the item you put in the table contains the 'partition key' and the sorting key, case-sensitive.
EDIT:
• It is a golang issue, the item is not built properly when the DynamoDB column names are lowercase.
• Consider redefining the Record structure (see this link for details):
type Record struct{
ID string `json:"id"`
URLs []string `json:"url"`
}
where id and url are column names in DynamoDB table.
Related
I have a Custom Resource which gets stuck in a catch22-like situation whenever a ROLLBACK occurs.
The code below is a simplified example of what my code is doing. In case of a create request, it creates a table, for delete it deletes, and in case of an update it compares the old properties with the new, and returns an error when one of the columns has a new value (column updates are not supported yet).
A problematic ROLLBACK_FAILED occurs when
[SOLVED] whenever a create request type fails (due to sql syntax error for example). In this case it will trigger a delete request for the ROLLBACK phase, but this request will fail because the table does not yet exist.
whenever a update request type fails due to a updated column value. In this case it will trigger a new update request for the ROLLBACK phase in which the event.ResourceProperties and event.OldResourceProperties are switched, which will still cause an error.
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"github.com/aws/aws-lambda-go/cfn"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/rdsdataservice"
"github.com/google/uuid"
)
func main() {
lambda.Start(cfn.LambdaWrap(handler))
}
type properties struct {
TableName string
Columns []struct {
Name string
Value string
}
}
func handler(ctx context.Context, event cfn.Event) (physicalResourceID string, data map[string]interface{}, err error) {
prid := event.PhysicalResourceID
if event.RequestType == cfn.RequestCreate {
prid = strings.ReplaceAll(uuid.New().String(), "-", "")
}
var props properties
b, _ := json.Marshal(event.ResourceProperties)
json.Unmarshal(b, &props)
rds := rdsdataservice.New(nil)
if event.RequestType == cfn.RequestCreate {
rds.ExecuteStatement(&rdsdataservice.ExecuteStatementInput{
Sql: aws.String(fmt.Sprintf("CREATE TABLE %s", props.TableName)),
})
}
if event.RequestType == cfn.RequestDelete {
rds.ExecuteStatement(&rdsdataservice.ExecuteStatementInput{
Sql: aws.String(fmt.Sprintf("CREATE TABLE %s", props.TableName)),
})
}
if event.RequestType == cfn.RequestUpdate {
var oldProps properties
b, _ := json.Marshal(event.OldResourceProperties)
json.Unmarshal(b, &oldProps)
columns := map[string]string{}
for _, column := range props.Columns {
columns[column.Name] = column.Value
}
for _, column := range oldProps.Columns {
if val, ok := columns[column.Name]; ok {
if val != column.Value {
return "", nil, fmt.Errorf("cannot change column type")
}
}
}
// Do some extra stuff here for adding/removing columns
}
return prid, nil, nil
}
I have thought of 2 possible solutions. One of them I could implement, with potential issues. But it seems to me there should be a better way, as I can't be the only one with this problem. Or I am doing something very wrong..
disable rollback for this specific resource only, in some cases (sometimes I still want a rollback)
have access to the last status, so that I can check what to do; in case of delete with last status CREATE_FAILED, don't do anything. In case of an update with last status UPDATE_FAILED, don't do anything.
The second option I could implement by using the code below. But as the number of events grow, this could become very problematic.
events, err := cloud.DescribeStackEvents(&cloudformation.DescribeStackEventsInput{
StackName: &event.StackID,
})
For anyone who runs into the same problem; I have solved this issue by looking at the stack events, and find the last resource status that is being updated. As the current status is always in the list, I ignore IN_PROGRESS values. If the first value after status IN_PROGRESS is FAILED, it means this resource could not be updated, and thus a different ROLLBACK strategie can be applied.
The corresponding function, in GO
func isFailedEvent(sess *session.Session, event cfn.Event) (bool, error) {
cloud := cloudformation.New(sess)
var isFailedResource bool
if err := cloud.DescribeStackEventsPages(&cloudformation.DescribeStackEventsInput{
StackName: aws.String(event.StackID),
}, func(out *cloudformation.DescribeStackEventsOutput, lastPage bool) bool {
for _, e := range out.StackEvents {
if *e.LogicalResourceId == event.LogicalResourceID {
if strings.HasSuffix(*e.ResourceStatus, "IN_PROGRESS") {
continue
}
if strings.HasSuffix(*e.ResourceStatus, "FAILED") && !strings.Contains(*e.ResourceStatusReason, "cancelled") {
isFailedResource = true
}
return false
}
}
return true
}); err != nil {
return false, fmt.Errorf("describe stack events: %s", err)
}
return isFailedResource, nil
}
Thanks in advance :) . I'm using the following code to get metadata from an s3 object after listing all the object in a bucket . But I don't know why it gives the error undefined: s3.HeadObject when running go run listObjects.go -bucket xxxx -prefix xxxx
I tried two solutions: giving the client as the one created from the config and from the context as in this link appears [1]. BUt both gave the same error. Can you give me any clue?
package main
import (
"context"
"flag"
"fmt"
"log"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
var (
bucketName string
objectPrefix string
objectDelimiter string
maxKeys int
)
func init() {
flag.StringVar(&bucketName, "bucket", "", "The `name` of the S3 bucket to list objects from.")
flag.StringVar(&objectPrefix, "prefix", "", "The optional `object prefix` of the S3 Object keys to list.")
flag.StringVar(&objectDelimiter, "delimiter", "",
"The optional `object key delimiter` used by S3 List objects to group object keys.")
flag.IntVar(&maxKeys, "max-keys", 0,
"The maximum number of `keys per page` to retrieve at once.")
}
// Lists all objects in a bucket using pagination
func main() {
flag.Parse()
if len(bucketName) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, bucket name required")
}
// Load the SDK's configuration from environment and shared config, and
// create the client with this.
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
log.Fatalf("failed to load SDK configuration, %v", err)
}
client := s3.NewFromConfig(cfg)
// Set the parameters based on the CLI flag inputs.
params := &s3.ListObjectsV2Input{
Bucket: &bucketName,
}
if len(objectPrefix) != 0 {
params.Prefix = &objectPrefix
}
if len(objectDelimiter) != 0 {
params.Delimiter = &objectDelimiter
}
// Create the Paginator for the ListObjectsV2 operation.
p := s3.NewListObjectsV2Paginator(client, params, func(o *s3.ListObjectsV2PaginatorOptions) {
if v := int32(maxKeys); v != 0 {
o.Limit = v
}
})
// Iterate through the S3 object pages, printing each object returned.
var i int
log.Println("Objects:")
for p.HasMorePages() {
i++
// Next Page takes a new context for each page retrieval. This is where
// you could add timeouts or deadlines.
page, err := p.NextPage(context.TODO())
if err != nil {
log.Fatalf("failed to get page %v, %v", i, err)
}
// Log the objects found
// Headobject function is called
for _, obj := range page.Contents {
input := &s3.HeadObjectInput{
Bucket: &bucketName,
Key: obj.Key,
}
result, err := &s3.HeadObject(client, input)
if err != nil {
panic(err)
}
fmt.Println("Object:", *obj.Key)
}
}
}
./listObjects.go:86:20: undefined: s3.HeadObject
1
Doing the headObject as an auxiliary method works
package main
import (
"context"
"flag"
"fmt"
"log"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
var (
bucketName string
objectPrefix string
objectDelimiter string
maxKeys int
)
func init() {
flag.StringVar(&bucketName, "bucket", "", "The `name` of the S3 bucket to list objects from.")
flag.StringVar(&objectPrefix, "prefix", "", "The optional `object prefix` of the S3 Object keys to list.")
flag.StringVar(&objectDelimiter, "delimiter", "",
"The optional `object key delimiter` used by S3 List objects to group object keys.")
flag.IntVar(&maxKeys, "max-keys", 0,
"The maximum number of `keys per page` to retrieve at once.")
}
// Lists all objects in a bucket using pagination
func main() {
flag.Parse()
if len(bucketName) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, bucket name required")
}
// Load the SDK's configuration from environment and shared config, and
// create the client with this.
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
log.Fatalf("failed to load SDK configuration, %v", err)
}
client := s3.NewFromConfig(cfg)
// Set the parameters based on the CLI flag inputs.
params := &s3.ListObjectsV2Input{
Bucket: &bucketName,
}
if len(objectPrefix) != 0 {
params.Prefix = &objectPrefix
}
if len(objectDelimiter) != 0 {
params.Delimiter = &objectDelimiter
}
// Create the Paginator for the ListObjectsV2 operation.
p := s3.NewListObjectsV2Paginator(client, params, func(o *s3.ListObjectsV2PaginatorOptions) {
if v := int32(maxKeys); v != 0 {
o.Limit = v
}
})
// Iterate through the S3 object pages, printing each object returned.
var i int
log.Println("Objects:")
for p.HasMorePages() {
i++
// Next Page takes a new context for each page retrieval. This is where
// you could add timeouts or deadlines.
page, err := p.NextPage(context.TODO())
if err != nil {
log.Fatalf("failed to get page %v, %v", i, err)
}
// Log the objects found
// Headobject function is called
for _, obj := range page.Contents {
fmt.Println("Object:", *obj.Key)
OpHeadObject(client, bucketName, *obj.Key)
}
}
}
func OpHeadObject(sess *s3.Client, bucketName, objectName string) {
input := &s3.HeadObjectInput{
Bucket: &bucketName,
Key: &objectName,
}
resp, err := sess.HeadObject(context.TODO(), input)
if err != nil {
panic(err)
}
fmt.Println(resp.StorageClass) // that you want.
}
I'm trying to create a pagination endpoint for a dynamodb table I have. But I've tried everything to get the exclusiveStartKey to be the correct type for it to work. However, everything I've tried doesn't seem to work.
example code:
func GetPaginator(tableName string, limit int32, lastEvaluatedKey string) (*dynamodb.ScanPaginator, error) {
svc, err := GetClient()
if err != nil {
logrus.Error(err)
return nil, err
}
input := &dynamodb.ScanInput{
TableName: aws.String(tableName),
Limit: aws.Int32(limit),
}
if lastEvaluatedKey != "" {
input.ExclusiveStartKey = map[string]types.AttributeValue{
"id": &types.AttributeValueMemberS{
Value: lastEvaluatedKey,
},
}
}
paginator := dynamodb.NewScanPaginator(svc, input)
return paginator, nil
}
Edit:
Okay so I'm creating a API that requires pagination. The API needs to have a query parameter where the lastEvaluatedId can be defined. I can then use the lastEvaluatedId to pass as the ExclusiveStartKey on the ScanInput. However when I do this I still received the same item from the database. I've created a test.go file and will post the code below:
package main
import (
"context"
"fmt"
"os"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
)
type PaginateID struct {
ID string `dynamodbav:"id" json:"id"`
}
func main() {
lastKey := PaginateID{ID: "ae82a99d-486e-11ec-a7a7-0242ac110002"}
key, err := attributevalue.MarshalMap(lastKey)
if err != nil {
fmt.Println(err)
return
}
cfg, err := config.LoadDefaultConfig(context.TODO(), func(o *config.LoadOptions) error {
o.Region = os.Getenv("REGION")
return nil
})
if err != nil {
fmt.Println(err)
return
}
svc := dynamodb.NewFromConfig(cfg, func(o *dynamodb.Options) {
o.EndpointResolver = dynamodb.EndpointResolverFromURL("http://localhost:8000")
})
input := &dynamodb.ScanInput{
TableName: aws.String("TABLE_NAME"),
Limit: aws.Int32(1),
ExclusiveStartKey: key,
}
paginator := dynamodb.NewScanPaginator(svc, input)
if paginator.HasMorePages() {
data, err := paginator.NextPage(context.TODO())
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data.Items[0]["id"])
fmt.Println(data.LastEvaluatedKey["id"])
}
}
When I run this test code. I get this output:
&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}
&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}
So the item that is returned is the same Id that I am passing to the ScanInput.ExclusiveStartKey. Which means it's not starting from the ExclusiveStartKey. The scan is starting from the beginning everytime.
The aws-sdk-go-v2 DynamoDB query and scan paginator constructors have a bug (see my github issue, includes the fix). They do not respect the ExclusiveStartKey param.
As an interim fix, I copied the paginator type locally and added one line in to the constructor: nextToken: params.ExclusiveStartKey.
so basically what you need to do is to get the LastEvaluatedKey and to pass it to ExclusiveStartKey
you can not use the scan paginator attributes because it's not exported attributes, therefore instead I suggest that you use the returned page by calling NextPage
in the following snippet I have an example :
func GetPaginator(ctx context.Context,tableName string, limit int32, lastEvaluatedKey map[string]types.AttributeValue) (*dynamodb.ScanOutput, error) {
svc, err := GetClient()
if err != nil {
logrus.Error(err)
return nil, err
}
input := &dynamodb.ScanInput{
TableName: aws.String(tableName),
Limit: aws.Int32(limit),
}
if len(lastEvaluatedKey) > 0 {
input.ExclusiveStartKey = lastEvaluatedKey
}
paginator := dynamodb.NewScanPaginator(svc, input)
return paginator.NextPage(), nil
}
keep in mind that paginator.NextPage(ctx) could be nil incase there is no more pages or you can use HasMorePages()
I am working with the AWS S3 SDK in GoLang, playing with uploads and downloads to various buckets. I am wondering if there is a simpler way to upload structs or objects directly to the bucket?
I have a struct representing an event:
type Event struct {
ID string
ProcessID string
TxnID string
Inputs map[string]interface{}
}
That I would like to upload into the S3 bucket. But the code that I found in the documentation only works for uploading strings.
func Save(client S3Client, T interface{}, key string) bool {
svc := client.S3clientObject
input := &s3.PutObjectInput{
Body: aws.ReadSeekCloser(strings.NewReader("testing this one")),
Bucket: aws.String(GetS3Bucket()),
Key: aws.String(GetObjectKey(T, key)),
Metadata: map[string]*string{
"metadata1": aws.String("value1"),
"metadata2": aws.String("value2"),
},
}
This is successful in uploading a basic file to the S3 bucket that when opened simply reads "testing this one". Is there a way to upload to the bucket so that it is uploading an object rather than simply just a string value??
Any help is appreciated as I am new to Go and S3.
edit
This is the code I'm using for the Get function:
func GetIt(client S3Client, T interface{}, key string) interface{} {
svc := client.S3clientObject
s3Key := GetObjectKey(T, key)
resp, err := svc.GetObject(&s3.GetObjectInput{
Bucket: aws.String(GetS3Bucket()),
Key: aws.String(s3Key),
})
if err != nil {
fmt.Println(err)
return err
}
result := json.NewDecoder(resp.Body).Decode(&T)
fmt.Println(result)
return json.NewDecoder(resp.Body).Decode(&T)
}
func main() {
client := b.CreateS3Client()
event := b.CreateEvent()
GetIt(client, event, key)
}
Encode the value as bytes and upload the bytes. Here's how to encode the value as JSON bytes:
func Save(client S3Client, value interface{}, key string) error {
p, err := json.Marshal(value)
if err != nil {
return err
}
input := &s3.PutObjectInput{
Body: aws.ReadSeekCloser(bytes.NewReader(p)),
…
}
…
}
Call Save with the value you want to upload:
value := &Event{ID: "an id", …}
err := Save(…, value, …)
if err != nil {
// handle error
}
There are many possible including including gob, xml and json, msgpack, etc. The best encoding format will depend on your application requirements.
Reverse the process when getting an object:
func GetIt(client S3Client, T interface{}, key string) error {
svc := client.S3clientObject
resp, err := svc.GetObject(&s3.GetObjectInput{
Bucket: aws.String(GetS3Bucket()),
Key: aws.String(key),
})
if err != nil {
return err
}
return json.NewDecoder(resp.Body).Decode(T)
}
Call GetIt with a pointer to the destination value:
var value model.Event
err := GetIt(client, &value, key)
if err != nil {
// handle error
}
fmt.Println(value) // prints the decoded value.
The example cited here shows that S3 allows you to upload anything that implements the io.Reader interface. The example is using the strings.NewReader syntax create a io.Reader that knows how to provide the specified string to the caller. Your job (according to AWS here) is to figure out how to adapt whatever you need to store into an io.Reader.
You can store the bytes directly JSON encoded like this
package main
import (
"bytes"
"encoding/json"
)
type Event struct {
ID string
ProcessID string
TxnID string
Inputs map[string]interface{}
}
func main() {
// To prepare the object for writing
b, err := json.Marshal(event)
if err != nil {
return
}
// pass this reader into aws.ReadSeekCloser(...)
reader := bytes.NewReader(b)
}
I have made a table Blog with DynamoDB.
I want to insert the 2 parameters title and content in it which I am fetching from an HTML form.
The parameters blogContent and blogTitle seem to be valid when I print these 2 in the console.
But when I am inserting them in the table i get the error:
One or more parameter values were invalid: Missing the key id in the item
status code: 400, request id: XXX"
type Item struct {
id int
content string
bodycontentversion string
claps int
comments int
imageid int
title string
views int
}
func awsblog() {
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
svc := dynamodb.New(sess)
item := Item{
id: 1234,
content: blogContent,
bodycontentversion: "abcd",
claps: 5,
comments: 10,
imageid: 1234,
title: blogTitle,
views: 10,
}
av, err := dynamodbattribute.MarshalMap(item)
if err != nil {
fmt.Println("Got error marshalling new item:")
fmt.Println(err.Error())
os.Exit(1)
}
tableName := "Blog"
input := &dynamodb.PutItemInput{
Item: av,
TableName: aws.String(tableName),
}
_, err = svc.PutItem(input)
if err != nil {
fmt.Println("Got error calling PutItem:")
fmt.Println(err.Error())
os.Exit(1)
}
fmt.Println("Successfully updated '")
}
Struct fields must be public in order for MarshalMap to do its magic.
Add a fmt.Printf("marshalled struct: %+v", av) to see the output of that function.
That will in turn require the DynamoDB fields to be capitalised, unless you add a json:"field-name" directive to the struct fields.
The result would be:
type Item struct {
Id int `json:"id"`
Content string `json:"content"`
Bodycontentversion string `json:"bodycontentversion"`
Claps int `json:"claps"`
Comments int `json:"comments"`
Imageid int `json:"imageid"`
Title string `json:"title"`
Views int `json:"views"`
}
I had the same error when putting an item with the boto3 Python package.
ClientError: An error occurred (ValidationException) when calling the PutItem operation: One or more parameter values were invalid: Missing the key id in the item
The problem was that I did not specify the partition key correctly when inserting. I incorrectly specified PARTITION_KEY below. I misspelled the variable name (first line below), which caused the error which was not immediately visible from the error message.
PARTTION_KEY = 'hello'
SORT_KEY = 'world'
dynamodb = boto3.client('dynamodb')
dynamodb.put_item(
TableName=TABLE,
Item={
PARTITION_KEY: {'S': 'my-part-key'},
SORT_KEY: {'S': 'my-sort-key'},
}
)