I am using GO SDK and using the DynamnoDB BatchGetItem API.
I saw this code example -
https://github.com/aws/aws-sdk-go/blob/master/service/dynamodb/examples_test.go
Is there any other code example which shows unmarshalling of the response from the BatchGetItem API?
Let me share piece of the code. The key to understand it is that when you send GetBatchItem request to dynamodb, you specify map of table names and keys for that table, so response you get is a map of tables names and matched items
placeIDs := []string { "london_123", "sanfran_15", "moscow_9" }
type Place {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
}
mapOfAttrKeys := []map[string]*dynamodb.AttributeValue{}
for _, place := range placeIDs {
mapOfAttrKeys = append(mapOfAttrKeys, map[string]*dynamodb.AttributeValue{
"id": &dynamodb.AttributeValue{
S: aws.String(place),
},
"attr": &dynamodb.AttributeValue{
S: aws.String("place"),
},
})
}
input := &dynamodb.BatchGetItemInput{
RequestItems: map[string]*dynamodb.KeysAndAttributes{
tableName: &dynamodb.KeysAndAttributes{
Keys: mapOfAttrKeys,
},
},
}
batch, err := db.BatchGetItem(input)
if err != nil {
panic(fmt.Errorf("batch load of places failed, err: %w", err))
}
for _, table := range batch.Responses {
for _, item := range table {
var place Place
err = dynamodbattribute.UnmarshalMap(item, &place)
if err != nil {
panic(fmt.Errorf("failed to unmarshall place from dynamodb response, err: %w", err))
}
places = append(places, place)
}
}
Related
I'm trying to write a lambda function which writes an array of strings to a generic collection where the Primary key is the type of a collection. It should always replace the current item (if exists, otherwise create one) with a new one.
This is my handler function
func Handler(ctx context.Context) (Response, error) {
item := lib.SkillsCollection{
Collection: lib.SkillsCollectionName,
Value: lib.Skills{
"test",
"test 2",
},
}
log.Printf("Prepare item %v", item)
sess := session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
svc := dynamodb.New(sess)
log.Printf("Created session")
av, err := dynamodbattribute.MarshalMap(item)
if err != nil {
log.Fatalf("Marshall error mapping value %s", err)
}
log.Printf("Marshalled item %v", av)
input := &dynamodb.PutItemInput{
TableName: aws.String(lib.DbCollectionsTable),
Item: av,
}
log.Println("Start scanning collections")
_, err = svc.PutItem(input)
if err != nil {
log.Fatalf("Error during put %v", err)
}
var buf bytes.Buffer
body, err := json.Marshal(map[string]interface{}{
"message": "Go Serverless v1.0! Your function executed successfully!",
})
if err != nil {
return Response{StatusCode: 404}, err
}
json.HTMLEscape(&buf, body)
resp := Response{
StatusCode: 200,
IsBase64Encoded: false,
Body: buf.String(),
Headers: map[string]string{
"Content-Type": "application/json",
},
}
return resp, nil
}
According to the logs everything is OK. No errors are thrown and I get response with status 200
These are the logs
However, after that I don't see any items in the DynamoDB table. It's empty. I can't figure out where to look for the root of the issue. At the same time I can see that there were writes to the table.
I've already rewritten the solution to follow the example in the AWS docs but no luck with that.
The struct in case it matters
type Skills []string
type SkillsCollection struct {
Collection string `dynamodbav:"collection" json:"collection"`
Value Skills `dynamodbav:"value" json:"value"`
}
And the DB setup config
CollectionsTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: collection
AttributeType: S
KeySchema:
- AttributeName: collection
KeyType: HASH
BillingMode: PAY_PER_REQUEST
TableName: ${self:custom.collectionsTableName}
You haven't demonstrated that there are no items in the table. Are you seeing that "Item count" of 0 and thinking it's timely? Notice under the Item Summary header is says the count is only updated every 6 hours. Try hitting the "Get live item count" button and I bet you'll see some items.
Why isn't the item count continuously maintained? Because it's a large distributed database which can store many trillions of items. Getting a live item count isn't always a lightweight process.
I'm trying to create a pagination endpoint for a dynamodb table I have. But I've tried everything to get the exclusiveStartKey to be the correct type for it to work. However, everything I've tried doesn't seem to work.
example code:
func GetPaginator(tableName string, limit int32, lastEvaluatedKey string) (*dynamodb.ScanPaginator, error) {
svc, err := GetClient()
if err != nil {
logrus.Error(err)
return nil, err
}
input := &dynamodb.ScanInput{
TableName: aws.String(tableName),
Limit: aws.Int32(limit),
}
if lastEvaluatedKey != "" {
input.ExclusiveStartKey = map[string]types.AttributeValue{
"id": &types.AttributeValueMemberS{
Value: lastEvaluatedKey,
},
}
}
paginator := dynamodb.NewScanPaginator(svc, input)
return paginator, nil
}
Edit:
Okay so I'm creating a API that requires pagination. The API needs to have a query parameter where the lastEvaluatedId can be defined. I can then use the lastEvaluatedId to pass as the ExclusiveStartKey on the ScanInput. However when I do this I still received the same item from the database. I've created a test.go file and will post the code below:
package main
import (
"context"
"fmt"
"os"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
)
type PaginateID struct {
ID string `dynamodbav:"id" json:"id"`
}
func main() {
lastKey := PaginateID{ID: "ae82a99d-486e-11ec-a7a7-0242ac110002"}
key, err := attributevalue.MarshalMap(lastKey)
if err != nil {
fmt.Println(err)
return
}
cfg, err := config.LoadDefaultConfig(context.TODO(), func(o *config.LoadOptions) error {
o.Region = os.Getenv("REGION")
return nil
})
if err != nil {
fmt.Println(err)
return
}
svc := dynamodb.NewFromConfig(cfg, func(o *dynamodb.Options) {
o.EndpointResolver = dynamodb.EndpointResolverFromURL("http://localhost:8000")
})
input := &dynamodb.ScanInput{
TableName: aws.String("TABLE_NAME"),
Limit: aws.Int32(1),
ExclusiveStartKey: key,
}
paginator := dynamodb.NewScanPaginator(svc, input)
if paginator.HasMorePages() {
data, err := paginator.NextPage(context.TODO())
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data.Items[0]["id"])
fmt.Println(data.LastEvaluatedKey["id"])
}
}
When I run this test code. I get this output:
&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}
&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}
So the item that is returned is the same Id that I am passing to the ScanInput.ExclusiveStartKey. Which means it's not starting from the ExclusiveStartKey. The scan is starting from the beginning everytime.
The aws-sdk-go-v2 DynamoDB query and scan paginator constructors have a bug (see my github issue, includes the fix). They do not respect the ExclusiveStartKey param.
As an interim fix, I copied the paginator type locally and added one line in to the constructor: nextToken: params.ExclusiveStartKey.
so basically what you need to do is to get the LastEvaluatedKey and to pass it to ExclusiveStartKey
you can not use the scan paginator attributes because it's not exported attributes, therefore instead I suggest that you use the returned page by calling NextPage
in the following snippet I have an example :
func GetPaginator(ctx context.Context,tableName string, limit int32, lastEvaluatedKey map[string]types.AttributeValue) (*dynamodb.ScanOutput, error) {
svc, err := GetClient()
if err != nil {
logrus.Error(err)
return nil, err
}
input := &dynamodb.ScanInput{
TableName: aws.String(tableName),
Limit: aws.Int32(limit),
}
if len(lastEvaluatedKey) > 0 {
input.ExclusiveStartKey = lastEvaluatedKey
}
paginator := dynamodb.NewScanPaginator(svc, input)
return paginator.NextPage(), nil
}
keep in mind that paginator.NextPage(ctx) could be nil incase there is no more pages or you can use HasMorePages()
I have the following IAM Policy:
{"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":"arn:aws:sts::<account>:assumed-role/custom_role/<role>"},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<account>"}}}]}
but the "AWS" portion can also be an array:
"AWS": [
"arn:aws:sts::<account>:assumed-role/custom_role/<role_1>",
"arn:aws:sts::<account>:assumed-role/custom_role/<role_2>"
]
What I need is a regex that can parse both structures and return the list of arn:aws:sts as a list of strings... how can I accomplish that using regex in Golang?
I tried to use json.Unmarshal but the object structure is different between []string and string
Edit:
I have the following snippet:
re := regexp.MustCompile(`arn:aws:sts::[a-z0-9]*:assumed-role/custom_role/[a-z0-9]-*`)
result := re.FindAll([]byte(arn), 10)
for _, res := range result {
fmt.Println(string(res))
}
>>> `arn:aws:sts::<account_id>:assumed-role/custom_role/`
Using JSON decoder
You can decode the AWS key directly into a custom type implementing the "json.Unmarshaler" interface and decode both inputs correctly.
Demo
type AWSRoles []string
func (r *AWSRoles) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err == nil {
*r = append(*r, s)
return nil
}
var ss []string
if err := json.Unmarshal(b, &ss); err == nil {
*r = ss
return nil
}
return errors.New("cannot unmarshal neither to a string nor a slice of strings")
}
type AWSPolicy struct {
Statement []struct {
Principal struct {
AWSRoles AWSRoles `json:"AWS"`
} `json:"Principal"`
} `json:"Statement"`
}
Here's a test for it
var testsAWSPolicyParsing = []struct {
name string
input []byte
wantRoles []string
}{
{
name: "unique role",
input: []byte(`{"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":"arn:aws:sts::<account>:assumed-role/custom_role/<role>"},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<account>"}}}]}`),
wantRoles: []string{"arn:aws:sts::<account>:assumed-role/custom_role/<role>"},
},
{
name: "multiple roles",
input: []byte(`{"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":["arn:aws:sts::<account>:assumed-role/custom_role/<role_1>","arn:aws:sts::<account>:assumed-role/custom_role/<role_2>"]},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<account>"}}}]}`),
wantRoles: []string{
"arn:aws:sts::<account>:assumed-role/custom_role/<role_1>",
"arn:aws:sts::<account>:assumed-role/custom_role/<role_2>",
},
},
}
func TestParseAWSPolicy(t *testing.T) {
for _, tc := range testsAWSPolicyParsing {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var p AWSPolicy
err := json.Unmarshal(tc.input, &p)
if err != nil {
t.Fatal("unexpected error parsing AWSRoles policy", err)
}
if l := len(p.Statement); l != 1 {
t.Fatalf("unexpected Statement length. want 1, got %d", l)
}
if got := p.Statement[0].Principal.AWSRoles; !reflect.DeepEqual(got, tc.wantRoles) {
t.Fatalf("roles are not the same, got %v, want %v", got, tc.wantRoles)
}
})
}
}
Using a Regex
If you still want to use a regex, this one would parse it as long as:
AWS account has only numbers [0-9]
the custom role name has only alphanumeric characters and underscores
var awsRolesRegex = regexp.MustCompile("arn:aws:sts::[a-z0-9]+:assumed-role/custom_role/[a-zA-Z0-9_]+")
Demo
I want to test AWS Lambda in local environment using DynamoDB Local and SAM CLI.
I create a simple user db table(id, name) and I'm trying to get the data.
I run "sam local start-api --env-vars test/env.json". When I access "http://localhost:3000/users/1", an error occured. Error message is below. I can't understand what this error message means. How do I fix this error?
{
"errorMessage": "InvalidParameter: 1 validation error(s) found.\n- minimum field size of 3, GetItemInput.TableName.\n",
"errorType": "ErrInvalidParams"
}
This is my code.
func handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
// Environment variables
endpoint := os.Getenv("DYNAMODB_ENDPOINT")
tableName := os.Getenv("DYNAMODB_TABLE_NAME")
// Request
id, _ := request.PathParameters["id"]
// DynamoDB
sess := session.Must(session.NewSession())
config := aws.NewConfig().WithRegion("ap-northeast-1")
if len(endpoint) > 0 {
config = config.WithEndpoint(endpoint)
}
db := dynamodb.New(sess, config)
response, err := db.GetItem(&dynamodb.GetItemInput{
TableName: aws.String(tableName),
Key: map[string]*dynamodb.AttributeValue{
"Id": {
N: aws.String(string(id)),
},
},
AttributesToGet: []*string{
aws.String("Id"),
aws.String("Name"),
},
ConsistentRead: aws.Bool(true),
ReturnConsumedCapacity: aws.String("NONE"),
})
if err != nil {
return events.APIGatewayProxyResponse{}, err
}
user := User{}
err = dynamodbattribute.Unmarshal(&dynamodb.AttributeValue{M: response.Item}, &user)
if err != nil {
return events.APIGatewayProxyResponse{}, err
}
// Json
bytes, err := json.Marshal(user)
if err != nil {
return events.APIGatewayProxyResponse{}, err
}
return events.APIGatewayProxyResponse{
Body: string(bytes),
StatusCode: 200,
}, nil
}
error message
That's been resolved.
First Problem: can't read from the environment variable
This was because the indent size deviation of template.yaml. Category "Events" and Category "Environment" had to be in line.
Second Problem: error with "Function 'UserGetFunction' timed out after 5 seconds"
This was because "localhost" expression. Rewriting "localhost" in env.json to "xxx.xxx.xxx.xxx" worked fine.("xxx.xxx.xxx.xxx" is ip address of your laptop)
I'm experimenting with AWS-SDK-GO with the DynamoDB API...
I'm trying to query the db and return a string. But I'm having some issues unmarshelling the return value....
struct
type Item struct {
slug string
destination string
}
query function
input := &dynamodb.GetItemInput{
Key: map[string]*dynamodb.AttributeValue{
"slug": {
S: aws.String(slug),
},
},
TableName: db.TableName,
}
result, err := db.dynamo.GetItem(input)
if err != nil {
return "", err
}
n := Item{}
err = dynamodbattribute.UnmarshalMap(result.Item, &n)
if err != nil {
log.Printf("Failed to unmarshal Record, %v", err)
return "", err
}
log.Printf("dump %+v", n)
log.Printf("echo %s", n.slug)
log.Printf("echo %s", n.destination)
log.Printf("orig %v", result.Item)
result
2017/10/11 14:21:34 dump {slug: destination:}
2017/10/11 14:21:34 echo
2017/10/11 14:21:34 echo
2017/10/11 14:21:34 orig map[destination:{
S: "http://example.com"
} slug:{
S: "abcde"
}]
Why is Item being returned empty?
I've tried to look everywhere but find no solution....
I am not sure whether you have tried this. I think if you can change the struct as mentioned below, it may resolve the problem.
I assumed that both slug and destination are defined/saved as String attribute in DynamoDB table.
type Item struct {
Slug string`json:"slug"`
Destination string`json:"destination"`
}
Change the print to:-
log.Printf("echo %s", n.Slug)
log.Printf("echo %s", n.Destination)
I found this issue and seems to be related, unmarsheling a specific attribute of the struct seems to do it.
https://github.com/aws/aws-sdk-go/issues/850
Example
var item Item
if err = dynamodbattribute.Unmarshal(result.Item["destination"], &item.destination); err != nil {
log.Printf("UnmarshalMap(GetItem response) err=%q", err)
}