I'm experimenting with AWS-SDK-GO with the DynamoDB API...
I'm trying to query the db and return a string. But I'm having some issues unmarshelling the return value....
struct
type Item struct {
slug string
destination string
}
query function
input := &dynamodb.GetItemInput{
Key: map[string]*dynamodb.AttributeValue{
"slug": {
S: aws.String(slug),
},
},
TableName: db.TableName,
}
result, err := db.dynamo.GetItem(input)
if err != nil {
return "", err
}
n := Item{}
err = dynamodbattribute.UnmarshalMap(result.Item, &n)
if err != nil {
log.Printf("Failed to unmarshal Record, %v", err)
return "", err
}
log.Printf("dump %+v", n)
log.Printf("echo %s", n.slug)
log.Printf("echo %s", n.destination)
log.Printf("orig %v", result.Item)
result
2017/10/11 14:21:34 dump {slug: destination:}
2017/10/11 14:21:34 echo
2017/10/11 14:21:34 echo
2017/10/11 14:21:34 orig map[destination:{
S: "http://example.com"
} slug:{
S: "abcde"
}]
Why is Item being returned empty?
I've tried to look everywhere but find no solution....
I am not sure whether you have tried this. I think if you can change the struct as mentioned below, it may resolve the problem.
I assumed that both slug and destination are defined/saved as String attribute in DynamoDB table.
type Item struct {
Slug string`json:"slug"`
Destination string`json:"destination"`
}
Change the print to:-
log.Printf("echo %s", n.Slug)
log.Printf("echo %s", n.Destination)
I found this issue and seems to be related, unmarsheling a specific attribute of the struct seems to do it.
https://github.com/aws/aws-sdk-go/issues/850
Example
var item Item
if err = dynamodbattribute.Unmarshal(result.Item["destination"], &item.destination); err != nil {
log.Printf("UnmarshalMap(GetItem response) err=%q", err)
}
Related
I'm writing a lambda that takes in streamed data from a DynamoDB table. After parsing out the proper record, I'm trying to convert it to JSON. Currently, I'm doing this:
func LambdaHandler(ctx context.Context, request events.DynamoDBEvent) error {
// ...
// Not actual code, just for demonstration
record = request.Records[0]
data, err := events.NewMapAttribute(record.Change.NewImage).MarshalJSON()
if err != nil {
return err
}
// ...
}
The problem is that this produces a JSON payload that looks like this:
{
"M": {
"action": { "N": "0" },
"expiration": { "N":"0" },
"id": { "S": "trades|v4|2023-02-08" },
"order": { "N":"22947407" },
"price": { "N":"96.139" },
"sort_key": { "S":"22947407" },
"stop_limit": { "N":"0" },
"stop_loss": { "N":"96.7" },
"symbol": { "S":"CADJPY" },
"take_profit": { "N":"94.83" },
"type": { "N":"5" },
"type_filling": { "N":"0" },
"type_time": { "N":"0" },
"volume": { "N":"1" }
}
}
As you can see, this mimics the structure of the DynamoDB attribute value but this isn't what I want. Instead, I'm trying to generate a JSON payload that looks like this:
{
"action": 0,
"expiration": 0,
"id": "trades|v4|2023-02-08",
"order": 22947407,
"price": 96.139,
"sort_key": "22947407",
"stop_limit": 0,
"stop_loss": 96.7,
"symbol": "CADJPY",
"take_profit": 94.83,
"type": 5,
"type_filling": 0,
"type_time": 0,
"volume": 1
}
Now, I can think of a couple ways to do that: hardcoding the values from record.Change.NewImage into a map[interface{}] and then marshalling that using json.Marshal, but the type of the payload I receive could be one of several different types. I could also use reflection to do the same thing, but I'd rather not spend the time debugging reflection code. Is there functionality available from Amazon to do this? It seems like there should be but I can't find anything.
I ended up writing a function that does more or less what I need it to. This will write numeric values as strings, but otherwise will generate the JSON payload I'm looking for:
// AttributesToJSON attempts to convert a mapping of DynamoDB attribute values to a properly-formatted JSON string
func AttributesToJSON(attrs map[string]events.DynamoDBAttributeValue) ([]byte, error) {
// Attempt to map the DynamoDB attribute value mapping to a map[string]interface{}
// If this fails then return an error
keys := make([]string, 0)
mapping, err := toJSONInner(attrs, keys...)
if err != nil {
return nil, err
}
// Attempt to convert this mapping to JSON and return the result
return json.Marshal(mapping)
}
// Helper function that converts a struct to JSON field-mapping
func toJSONInner(attrs map[string]events.DynamoDBAttributeValue, keys ...string) (map[string]interface{}, error) {
jsonStr := make(map[string]interface{})
for key, attr := range attrs {
// Attempt to convert the field to a JSON mapping; if this fails then return an error
casted, err := toJSONField(attr, append(keys, key)...)
if err != nil {
return nil, err
}
// Set the field to its associated key in our mapping
jsonStr[key] = casted
}
return jsonStr, nil
}
// Helper function that converts a specific DynamoDB attribute value to its JSON value equivalent
func toJSONField(attr events.DynamoDBAttributeValue, keys ...string) (interface{}, error) {
attrType := attr.DataType()
switch attrType {
case events.DataTypeBinary:
return attr.Binary(), nil
case events.DataTypeBinarySet:
return attr.BinarySet(), nil
case events.DataTypeBoolean:
return attr.Boolean(), nil
case events.DataTypeList:
// Get the list of items from the attribute value
list := attr.List()
// Attempt to convert each item in the list to a JSON mapping
data := make([]interface{}, len(list))
for i, item := range list {
// Attempt to map the field to a JSON mapping; if this fails then return an error
casted, err := toJSONField(item, keys...)
if err != nil {
return nil, err
}
// Set the value at this index to the mapping we generated
data[i] = casted
}
// Return the list we created
return data, nil
case events.DataTypeMap:
return toJSONInner(attr.Map(), keys...)
case events.DataTypeNull:
return nil, nil
case events.DataTypeNumber:
return attr.Number(), nil
case events.DataTypeNumberSet:
return attr.NumberSet(), nil
case events.DataTypeString:
return attr.String(), nil
case events.DataTypeStringSet:
return attr.StringSet(), nil
default:
return nil, fmt.Errorf("Attribute at %s had unknown attribute type of %d",
strings.Join(keys, "."), attrType)
}
}
This code works by iterating over each key and value in the top-level mapping, and converting the value to an interface{} and then converting the result to JSON. In this case, the interface{} could be a []byte, [][]byte, string, []string, bool, interface{} or map[string]interface{} depending on the type of the attribute value.
I'm trying to create a pagination endpoint for a dynamodb table I have. But I've tried everything to get the exclusiveStartKey to be the correct type for it to work. However, everything I've tried doesn't seem to work.
example code:
func GetPaginator(tableName string, limit int32, lastEvaluatedKey string) (*dynamodb.ScanPaginator, error) {
svc, err := GetClient()
if err != nil {
logrus.Error(err)
return nil, err
}
input := &dynamodb.ScanInput{
TableName: aws.String(tableName),
Limit: aws.Int32(limit),
}
if lastEvaluatedKey != "" {
input.ExclusiveStartKey = map[string]types.AttributeValue{
"id": &types.AttributeValueMemberS{
Value: lastEvaluatedKey,
},
}
}
paginator := dynamodb.NewScanPaginator(svc, input)
return paginator, nil
}
Edit:
Okay so I'm creating a API that requires pagination. The API needs to have a query parameter where the lastEvaluatedId can be defined. I can then use the lastEvaluatedId to pass as the ExclusiveStartKey on the ScanInput. However when I do this I still received the same item from the database. I've created a test.go file and will post the code below:
package main
import (
"context"
"fmt"
"os"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
)
type PaginateID struct {
ID string `dynamodbav:"id" json:"id"`
}
func main() {
lastKey := PaginateID{ID: "ae82a99d-486e-11ec-a7a7-0242ac110002"}
key, err := attributevalue.MarshalMap(lastKey)
if err != nil {
fmt.Println(err)
return
}
cfg, err := config.LoadDefaultConfig(context.TODO(), func(o *config.LoadOptions) error {
o.Region = os.Getenv("REGION")
return nil
})
if err != nil {
fmt.Println(err)
return
}
svc := dynamodb.NewFromConfig(cfg, func(o *dynamodb.Options) {
o.EndpointResolver = dynamodb.EndpointResolverFromURL("http://localhost:8000")
})
input := &dynamodb.ScanInput{
TableName: aws.String("TABLE_NAME"),
Limit: aws.Int32(1),
ExclusiveStartKey: key,
}
paginator := dynamodb.NewScanPaginator(svc, input)
if paginator.HasMorePages() {
data, err := paginator.NextPage(context.TODO())
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data.Items[0]["id"])
fmt.Println(data.LastEvaluatedKey["id"])
}
}
When I run this test code. I get this output:
&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}
&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}
So the item that is returned is the same Id that I am passing to the ScanInput.ExclusiveStartKey. Which means it's not starting from the ExclusiveStartKey. The scan is starting from the beginning everytime.
The aws-sdk-go-v2 DynamoDB query and scan paginator constructors have a bug (see my github issue, includes the fix). They do not respect the ExclusiveStartKey param.
As an interim fix, I copied the paginator type locally and added one line in to the constructor: nextToken: params.ExclusiveStartKey.
so basically what you need to do is to get the LastEvaluatedKey and to pass it to ExclusiveStartKey
you can not use the scan paginator attributes because it's not exported attributes, therefore instead I suggest that you use the returned page by calling NextPage
in the following snippet I have an example :
func GetPaginator(ctx context.Context,tableName string, limit int32, lastEvaluatedKey map[string]types.AttributeValue) (*dynamodb.ScanOutput, error) {
svc, err := GetClient()
if err != nil {
logrus.Error(err)
return nil, err
}
input := &dynamodb.ScanInput{
TableName: aws.String(tableName),
Limit: aws.Int32(limit),
}
if len(lastEvaluatedKey) > 0 {
input.ExclusiveStartKey = lastEvaluatedKey
}
paginator := dynamodb.NewScanPaginator(svc, input)
return paginator.NextPage(), nil
}
keep in mind that paginator.NextPage(ctx) could be nil incase there is no more pages or you can use HasMorePages()
I am building a simple function that calls an API that returns a Post using GraphQL (https://github.com/machinebox/graphql). I wrapped the logic in a service that looks like this:
type Client struct {
gcl graphqlClient
}
type graphqlClient interface {
Run(ctx context.Context, req *graphql.Request, resp interface{}) error
}
func (c *Client) GetPost(id string) (*Post, error) {
req := graphql.NewRequest(`
query($id: String!) {
getPost(id: $id) {
id
title
}
}
`)
req.Var("id", id)
var resp getPostResponse
if err := c.gcl.Run(ctx, req, &resp); err != nil {
return nil, err
}
return resp.Post, nil
}
Now I'd like to add test tables for the GetPost function with a fail case when id is set to empty string which causes an error in the downstream call c.gcl.Run.
What I am struggling with is the way the gcl client can be mocked and forced to return the error (when no real API call happens).
My test so far:
package apiClient
import (
"context"
"errors"
"github.com/aws/aws-sdk-go/aws"
"github.com/google/go-cmp/cmp"
"github.com/machinebox/graphql"
"testing"
)
type graphqlClientMock struct {
graphqlClient
HasError bool
Response interface{}
}
func (g graphqlClientMock) Run(_ context.Context, _ *graphql.Request, response interface{}) error {
if g.HasError {
return errors.New("")
}
response = g.Response
return nil
}
func newTestClient(hasError bool, response interface{}) *Client {
return &Client{
gcl: graphqlClientMock{
HasError: hasError,
Response: response,
},
}
}
func TestClient_GetPost(t *testing.T) {
tt := []struct{
name string
id string
post *Post
hasError bool
response getPostResponse
}{
{
name: "empty id",
id: "",
post: nil,
hasError: true,
},
{
name: "existing post",
id: "123",
post: &Post{id: aws.String("123")},
response: getPostResponse{
Post: &Post{id: aws.String("123")},
},
},
}
for _, tc := range tt {
t.Run(tc.name, func(t *testing.T) {
client := newTestClient(tc.hasError, tc.response)
post, err := client.GetPost(tc.id)
if err != nil {
if tc.hasError == false {
t.Error("unexpected error")
}
} else {
if tc.hasError == true {
t.Error("expected error")
}
if cmp.Equal(post, &tc.post) == false {
t.Errorf("Response data do not match: %s", cmp.Diff(post, tc.post))
}
}
})
}
}
I am not sure if passing the response to the mock like this is the right way to do it. Also, I'm struggling to set the right value to the response, since an interface{} type is passed and I don't know how to convert it to the getPostResponse and set the value to Post there.
Your test cases should not go beyond the implementation. I'm specifically referring to the empty-vs-nonempty input or any kind of input really.
Let's take a look at the code you want to test:
func (c *Client) GetPost(id string) (*Post, error) {
req := graphql.NewRequest(`
query($id: String!) {
getPost(id: $id) {
id
title
}
}
`)
req.Var("id", id)
var resp getPostResponse
if err := c.gcl.Run(ctx, req, &resp); err != nil {
return nil, err
}
return resp.Post, nil
}
Nothing in the implementation above is doing anything based on the id parameter value and therefore nothing in your tests for this piece of code should really care about what input is passed in, if it is irrelevant to the implementation it should also be irrelevant to the tests.
Your GetPost has basically two code branches that are taken based on a single factor, i.e. the "nilness" of the returned err variable. This means that as far as your implementation is concerned there are only two possible outcomes, based on what err value Run returns, and therefore there should only be two test cases, a 3rd or 4th test case would be just a variation, if not an outright copy, of the first two.
Your test client is also doing some unnecessary stuff, the main one being its name, i.e. what you have there is not a mock so calling it that is not helpful. Mocks usually do a lot more than just return predefined values, they ensure that methods are called, in the expected order and with the expected arguments, etc. And actually you don't need a mock here at all so it's a good thing it isn't one.
With that in mind, here's what I would suggest you do with your test client.
type testGraphqlClient struct {
resp interface{} // non-pointer value of the desired response, or nil
err error // the error to be returned by Run, or nil
}
func (g testGraphqlClient) Run(_ context.Context, _ *graphql.Request, resp interface{}) error {
if g.err != nil {
return g.err
}
if g.resp != nil {
// use reflection to set the passed in response value
// (i haven't tested this so there may be a bug or two)
reflect.ValueOf(resp).Elem().Set(reflect.ValueOf(g.resp))
}
return nil
}
... and here are the necessary test cases, all two of them:
func TestClient_GetPost(t *testing.T) {
tests := []struct {
name string
post *Post
err error
client testGraphqlClient
}{{
name: "return error from client",
err: errors.New("bad input"),
client: testGraphqlClient{err: errors.New("bad input")},
}, {
name: "return post from client",
post: &Post{id: aws.String("123")},
client: testGraphqlClient{resp: getPostResponse{Post: &Post{id: aws.String("123")}}},
}}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
client := Client{gql: tt.client}
post, err := client.GetPost("whatever")
if !cmp.Equal(err, tt.err) {
t.Errorf("got error=%v want error=%v", err, tt.err)
}
if !cmp.Equal(post, tt.post) {
t.Errorf("got post=%v want post=%v", post, tt.post)
}
})
}
}
... there's a bit of repetition going on here, the need to spell out the post and err twice but that's a small price to pay when compared to a more sophisticated/complicated test setup that would populate the test client from the test case's expected output fields.
Addendum:
If you were to update GetPost in such a way that it checks for the empty id and returns an error before it sends a request to graphql then your initial setup would make much more sense:
func (c *Client) GetPost(id string) (*Post, error) {
if id == "" {
return nil, errors.New("empty id")
}
req := graphql.NewRequest(`
query($id: String!) {
getPost(id: $id) {
id
title
}
}
`)
req.Var("id", id)
var resp getPostResponse
if err := c.gcl.Run(ctx, req, &resp); err != nil {
return nil, err
}
return resp.Post, nil
}
... and updating the test cases accordingly:
func TestClient_GetPost(t *testing.T) {
tests := []struct {
name string
id string
post *Post
err error
client testGraphqlClient
}{{
name: "return empty id error",
id: "",
err: errors.New("empty id"),
client: testGraphqlClient{},
}, {
name: "return error from client",
id: "nonemptyid",
err: errors.New("bad input"),
client: testGraphqlClient{err: errors.New("bad input")},
}, {
name: "return post from client",
id: "nonemptyid",
post: &Post{id: aws.String("123")},
client: testGraphqlClient{resp: getPostResponse{Post: &Post{id: aws.String("123")}}},
}}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
client := Client{gql: tt.client}
post, err := client.GetPost(tt.id)
if !cmp.Equal(err, tt.err) {
t.Errorf("got error=%v want error=%v", err, tt.err)
}
if !cmp.Equal(post, tt.post) {
t.Errorf("got post=%v want post=%v", post, tt.post)
}
})
}
}
I am using GO SDK and using the DynamnoDB BatchGetItem API.
I saw this code example -
https://github.com/aws/aws-sdk-go/blob/master/service/dynamodb/examples_test.go
Is there any other code example which shows unmarshalling of the response from the BatchGetItem API?
Let me share piece of the code. The key to understand it is that when you send GetBatchItem request to dynamodb, you specify map of table names and keys for that table, so response you get is a map of tables names and matched items
placeIDs := []string { "london_123", "sanfran_15", "moscow_9" }
type Place {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
}
mapOfAttrKeys := []map[string]*dynamodb.AttributeValue{}
for _, place := range placeIDs {
mapOfAttrKeys = append(mapOfAttrKeys, map[string]*dynamodb.AttributeValue{
"id": &dynamodb.AttributeValue{
S: aws.String(place),
},
"attr": &dynamodb.AttributeValue{
S: aws.String("place"),
},
})
}
input := &dynamodb.BatchGetItemInput{
RequestItems: map[string]*dynamodb.KeysAndAttributes{
tableName: &dynamodb.KeysAndAttributes{
Keys: mapOfAttrKeys,
},
},
}
batch, err := db.BatchGetItem(input)
if err != nil {
panic(fmt.Errorf("batch load of places failed, err: %w", err))
}
for _, table := range batch.Responses {
for _, item := range table {
var place Place
err = dynamodbattribute.UnmarshalMap(item, &place)
if err != nil {
panic(fmt.Errorf("failed to unmarshall place from dynamodb response, err: %w", err))
}
places = append(places, place)
}
}
I would like to update an item under certain conditions and then I would like to know whether the item was updated when UpdateItem returns.
The documentation seems contradictory to me.
On this page: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html in the "Conditional Update" example it says "All of the item's attributes, as they appear after the update, are returned in the response."
On this page: https://godoc.org/github.com/aws/aws-sdk-go/service/dynamodb#UpdateItemOutput it says that Attributes is "A map of attribute values as they appeared before the UpdateItem operation"
I don't really want either of these. What I want is a bool that says whether or not there was an update.
This is where my brain is at now:
out, err := db.DynamoDB.UpdateItem(&dynamodb.UpdateItemInput{
TableName: tableName,
Key: map[string]*dynamodb.AttributeValue{
"KeyName": {S: aws.String(keyname)},
},
ExpressionAttributeNames: map[string]*string{
"#lock": aws.String("Lock"),
},
ExpressionAttributeValues: map[string]*string{
":now": aws.String(compfmt(time.Now())),
":promise": aws.String(compfmt(time.Now().Add(30 * time.Second))),
},
ConditionExpression: aws.String("attribute_not_exist(#lock) OR :now > #lock"),
UpdateExpression: aws.String("SET #lock = :promise"),
})
One way to do this is to check the Code on the awserr
import "github.com/aws/aws-sdk-go/aws/awserr"
func Lock()(bool, error) {
//Create value v
_, err := db.DynamoDB.UpdateItem(v)
if err != nil {
if ae, ok := err.(awserr.RequestFailure); ok && ae.Code() == "ConditionalCheckFailedException" {
return false, nil
}
return false, err
}
return true, nil
}
There are now constants to compare the errors rather than using the hardcoded string as in other answers:
result, err := svc.UpdateItem(input)
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case dynamodb.ErrCodeConditionalCheckFailedException:
fmt.Println(dynamodb.ErrCodeConditionalCheckFailedException, aerr.Error())
default:
fmt.Println(aerr.Error())
}
}
}
Turns out what I wanted to do was check the error to see if it contained the string ConditionalCheckFailedException.
func Lock() (bool, error) {
...
_, err := db.DynamoDB.UpdateItem(v)
if err != nil {
if strings.Contains(err.Error(), "ConditionalCheckFailedException") {
return false, nil
}
return false, err
}
return true, nil
}