I'm trying to write a lambda function which writes an array of strings to a generic collection where the Primary key is the type of a collection. It should always replace the current item (if exists, otherwise create one) with a new one.
This is my handler function
func Handler(ctx context.Context) (Response, error) {
item := lib.SkillsCollection{
Collection: lib.SkillsCollectionName,
Value: lib.Skills{
"test",
"test 2",
},
}
log.Printf("Prepare item %v", item)
sess := session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
svc := dynamodb.New(sess)
log.Printf("Created session")
av, err := dynamodbattribute.MarshalMap(item)
if err != nil {
log.Fatalf("Marshall error mapping value %s", err)
}
log.Printf("Marshalled item %v", av)
input := &dynamodb.PutItemInput{
TableName: aws.String(lib.DbCollectionsTable),
Item: av,
}
log.Println("Start scanning collections")
_, err = svc.PutItem(input)
if err != nil {
log.Fatalf("Error during put %v", err)
}
var buf bytes.Buffer
body, err := json.Marshal(map[string]interface{}{
"message": "Go Serverless v1.0! Your function executed successfully!",
})
if err != nil {
return Response{StatusCode: 404}, err
}
json.HTMLEscape(&buf, body)
resp := Response{
StatusCode: 200,
IsBase64Encoded: false,
Body: buf.String(),
Headers: map[string]string{
"Content-Type": "application/json",
},
}
return resp, nil
}
According to the logs everything is OK. No errors are thrown and I get response with status 200
These are the logs
However, after that I don't see any items in the DynamoDB table. It's empty. I can't figure out where to look for the root of the issue. At the same time I can see that there were writes to the table.
I've already rewritten the solution to follow the example in the AWS docs but no luck with that.
The struct in case it matters
type Skills []string
type SkillsCollection struct {
Collection string `dynamodbav:"collection" json:"collection"`
Value Skills `dynamodbav:"value" json:"value"`
}
And the DB setup config
CollectionsTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: collection
AttributeType: S
KeySchema:
- AttributeName: collection
KeyType: HASH
BillingMode: PAY_PER_REQUEST
TableName: ${self:custom.collectionsTableName}
You haven't demonstrated that there are no items in the table. Are you seeing that "Item count" of 0 and thinking it's timely? Notice under the Item Summary header is says the count is only updated every 6 hours. Try hitting the "Get live item count" button and I bet you'll see some items.
Why isn't the item count continuously maintained? Because it's a large distributed database which can store many trillions of items. Getting a live item count isn't always a lightweight process.
Related
I'm trying to create a pagination endpoint for a dynamodb table I have. But I've tried everything to get the exclusiveStartKey to be the correct type for it to work. However, everything I've tried doesn't seem to work.
example code:
func GetPaginator(tableName string, limit int32, lastEvaluatedKey string) (*dynamodb.ScanPaginator, error) {
svc, err := GetClient()
if err != nil {
logrus.Error(err)
return nil, err
}
input := &dynamodb.ScanInput{
TableName: aws.String(tableName),
Limit: aws.Int32(limit),
}
if lastEvaluatedKey != "" {
input.ExclusiveStartKey = map[string]types.AttributeValue{
"id": &types.AttributeValueMemberS{
Value: lastEvaluatedKey,
},
}
}
paginator := dynamodb.NewScanPaginator(svc, input)
return paginator, nil
}
Edit:
Okay so I'm creating a API that requires pagination. The API needs to have a query parameter where the lastEvaluatedId can be defined. I can then use the lastEvaluatedId to pass as the ExclusiveStartKey on the ScanInput. However when I do this I still received the same item from the database. I've created a test.go file and will post the code below:
package main
import (
"context"
"fmt"
"os"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
)
type PaginateID struct {
ID string `dynamodbav:"id" json:"id"`
}
func main() {
lastKey := PaginateID{ID: "ae82a99d-486e-11ec-a7a7-0242ac110002"}
key, err := attributevalue.MarshalMap(lastKey)
if err != nil {
fmt.Println(err)
return
}
cfg, err := config.LoadDefaultConfig(context.TODO(), func(o *config.LoadOptions) error {
o.Region = os.Getenv("REGION")
return nil
})
if err != nil {
fmt.Println(err)
return
}
svc := dynamodb.NewFromConfig(cfg, func(o *dynamodb.Options) {
o.EndpointResolver = dynamodb.EndpointResolverFromURL("http://localhost:8000")
})
input := &dynamodb.ScanInput{
TableName: aws.String("TABLE_NAME"),
Limit: aws.Int32(1),
ExclusiveStartKey: key,
}
paginator := dynamodb.NewScanPaginator(svc, input)
if paginator.HasMorePages() {
data, err := paginator.NextPage(context.TODO())
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data.Items[0]["id"])
fmt.Println(data.LastEvaluatedKey["id"])
}
}
When I run this test code. I get this output:
&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}
&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}
So the item that is returned is the same Id that I am passing to the ScanInput.ExclusiveStartKey. Which means it's not starting from the ExclusiveStartKey. The scan is starting from the beginning everytime.
The aws-sdk-go-v2 DynamoDB query and scan paginator constructors have a bug (see my github issue, includes the fix). They do not respect the ExclusiveStartKey param.
As an interim fix, I copied the paginator type locally and added one line in to the constructor: nextToken: params.ExclusiveStartKey.
so basically what you need to do is to get the LastEvaluatedKey and to pass it to ExclusiveStartKey
you can not use the scan paginator attributes because it's not exported attributes, therefore instead I suggest that you use the returned page by calling NextPage
in the following snippet I have an example :
func GetPaginator(ctx context.Context,tableName string, limit int32, lastEvaluatedKey map[string]types.AttributeValue) (*dynamodb.ScanOutput, error) {
svc, err := GetClient()
if err != nil {
logrus.Error(err)
return nil, err
}
input := &dynamodb.ScanInput{
TableName: aws.String(tableName),
Limit: aws.Int32(limit),
}
if len(lastEvaluatedKey) > 0 {
input.ExclusiveStartKey = lastEvaluatedKey
}
paginator := dynamodb.NewScanPaginator(svc, input)
return paginator.NextPage(), nil
}
keep in mind that paginator.NextPage(ctx) could be nil incase there is no more pages or you can use HasMorePages()
I want to test AWS Lambda in local environment using DynamoDB Local and SAM CLI.
I create a simple user db table(id, name) and I'm trying to get the data.
I run "sam local start-api --env-vars test/env.json". When I access "http://localhost:3000/users/1", an error occured. Error message is below. I can't understand what this error message means. How do I fix this error?
{
"errorMessage": "InvalidParameter: 1 validation error(s) found.\n- minimum field size of 3, GetItemInput.TableName.\n",
"errorType": "ErrInvalidParams"
}
This is my code.
func handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
// Environment variables
endpoint := os.Getenv("DYNAMODB_ENDPOINT")
tableName := os.Getenv("DYNAMODB_TABLE_NAME")
// Request
id, _ := request.PathParameters["id"]
// DynamoDB
sess := session.Must(session.NewSession())
config := aws.NewConfig().WithRegion("ap-northeast-1")
if len(endpoint) > 0 {
config = config.WithEndpoint(endpoint)
}
db := dynamodb.New(sess, config)
response, err := db.GetItem(&dynamodb.GetItemInput{
TableName: aws.String(tableName),
Key: map[string]*dynamodb.AttributeValue{
"Id": {
N: aws.String(string(id)),
},
},
AttributesToGet: []*string{
aws.String("Id"),
aws.String("Name"),
},
ConsistentRead: aws.Bool(true),
ReturnConsumedCapacity: aws.String("NONE"),
})
if err != nil {
return events.APIGatewayProxyResponse{}, err
}
user := User{}
err = dynamodbattribute.Unmarshal(&dynamodb.AttributeValue{M: response.Item}, &user)
if err != nil {
return events.APIGatewayProxyResponse{}, err
}
// Json
bytes, err := json.Marshal(user)
if err != nil {
return events.APIGatewayProxyResponse{}, err
}
return events.APIGatewayProxyResponse{
Body: string(bytes),
StatusCode: 200,
}, nil
}
error message
That's been resolved.
First Problem: can't read from the environment variable
This was because the indent size deviation of template.yaml. Category "Events" and Category "Environment" had to be in line.
Second Problem: error with "Function 'UserGetFunction' timed out after 5 seconds"
This was because "localhost" expression. Rewriting "localhost" in env.json to "xxx.xxx.xxx.xxx" worked fine.("xxx.xxx.xxx.xxx" is ip address of your laptop)
I am using GO SDK and using the DynamnoDB BatchGetItem API.
I saw this code example -
https://github.com/aws/aws-sdk-go/blob/master/service/dynamodb/examples_test.go
Is there any other code example which shows unmarshalling of the response from the BatchGetItem API?
Let me share piece of the code. The key to understand it is that when you send GetBatchItem request to dynamodb, you specify map of table names and keys for that table, so response you get is a map of tables names and matched items
placeIDs := []string { "london_123", "sanfran_15", "moscow_9" }
type Place {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
}
mapOfAttrKeys := []map[string]*dynamodb.AttributeValue{}
for _, place := range placeIDs {
mapOfAttrKeys = append(mapOfAttrKeys, map[string]*dynamodb.AttributeValue{
"id": &dynamodb.AttributeValue{
S: aws.String(place),
},
"attr": &dynamodb.AttributeValue{
S: aws.String("place"),
},
})
}
input := &dynamodb.BatchGetItemInput{
RequestItems: map[string]*dynamodb.KeysAndAttributes{
tableName: &dynamodb.KeysAndAttributes{
Keys: mapOfAttrKeys,
},
},
}
batch, err := db.BatchGetItem(input)
if err != nil {
panic(fmt.Errorf("batch load of places failed, err: %w", err))
}
for _, table := range batch.Responses {
for _, item := range table {
var place Place
err = dynamodbattribute.UnmarshalMap(item, &place)
if err != nil {
panic(fmt.Errorf("failed to unmarshall place from dynamodb response, err: %w", err))
}
places = append(places, place)
}
}
I'm using the follow code to create a item in my dynamodb table:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/endpoints"
"github.com/aws/aws-sdk-go-v2/aws/external"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/dynamodbattribute"
)
type Record struct {
ID string
URLs []string
}
// Using the SDK's default configuration, loading additional config
// and credentials values from the environment variables, shared
// credentials, and shared configuration files
var cfg, err = external.LoadDefaultAWSConfig()
func createNewExtraction() error {
svc := dynamodb.New(cfg)
r := Record{
ID: "ABC123",
URLs: []string{
"https://example.com/first/link",
"https://example.com/second/url",
},
}
item, err := dynamodbattribute.MarshalMap(r)
if err != nil {
panic(fmt.Sprintf("failed to DynamoDB marshal Record, %v", err))
}
req := svc.PutItemRequest(&dynamodb.PutItemInput{
TableName: aws.String("test"), Item: item })
_, err = req.Send(); if err != nil {
return err
}
return nil
}
func main() {
if len(cfg.Region) > 0 {
// Set Region to us-east-1 as default.
cfg.Region = endpoints.UsEast1RegionID
}
err = createNewExtraction(); if err != nil {
panic(err.Error())
}
}
But it's returning the error:
panic: ValidationException: One or more parameter values were invalid: Missing the key id in the item
status code: 400, request id: F3VCQSGRIG5GM5PEJE7T5M4CEVVV4KQNSO5AEMVJF66Q9ASUAAJG
goroutine 1 [running]:
main.main()
/Users/user/go/src/Test2/test.go:56 +0x102
exit status 2
I already tried to declare Id, id and ID in my Record struct, but it doesn't work.
The stranger is: I got this code in the official documentation (I'm updating to work with the aws-sdk-go-v2).
Thank you in advance.
I do not know golang, but I had similar problems in nodejs.
Make sure the item you put in the table contains the 'partition key' and the sorting key, case-sensitive.
EDIT:
• It is a golang issue, the item is not built properly when the DynamoDB column names are lowercase.
• Consider redefining the Record structure (see this link for details):
type Record struct{
ID string `json:"id"`
URLs []string `json:"url"`
}
where id and url are column names in DynamoDB table.
I'm experimenting with AWS-SDK-GO with the DynamoDB API...
I'm trying to query the db and return a string. But I'm having some issues unmarshelling the return value....
struct
type Item struct {
slug string
destination string
}
query function
input := &dynamodb.GetItemInput{
Key: map[string]*dynamodb.AttributeValue{
"slug": {
S: aws.String(slug),
},
},
TableName: db.TableName,
}
result, err := db.dynamo.GetItem(input)
if err != nil {
return "", err
}
n := Item{}
err = dynamodbattribute.UnmarshalMap(result.Item, &n)
if err != nil {
log.Printf("Failed to unmarshal Record, %v", err)
return "", err
}
log.Printf("dump %+v", n)
log.Printf("echo %s", n.slug)
log.Printf("echo %s", n.destination)
log.Printf("orig %v", result.Item)
result
2017/10/11 14:21:34 dump {slug: destination:}
2017/10/11 14:21:34 echo
2017/10/11 14:21:34 echo
2017/10/11 14:21:34 orig map[destination:{
S: "http://example.com"
} slug:{
S: "abcde"
}]
Why is Item being returned empty?
I've tried to look everywhere but find no solution....
I am not sure whether you have tried this. I think if you can change the struct as mentioned below, it may resolve the problem.
I assumed that both slug and destination are defined/saved as String attribute in DynamoDB table.
type Item struct {
Slug string`json:"slug"`
Destination string`json:"destination"`
}
Change the print to:-
log.Printf("echo %s", n.Slug)
log.Printf("echo %s", n.Destination)
I found this issue and seems to be related, unmarsheling a specific attribute of the struct seems to do it.
https://github.com/aws/aws-sdk-go/issues/850
Example
var item Item
if err = dynamodbattribute.Unmarshal(result.Item["destination"], &item.destination); err != nil {
log.Printf("UnmarshalMap(GetItem response) err=%q", err)
}