How to parse dynamodb result to an integer? - amazon-web-services

I'm unable to unmarshall a fairly simple data structure:
"video_lite": { "id": 1573, "name": "Blade Runner (Movie)" }
Here's my code that doesn't work:
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)
type Video struct {
video_lite struct {
name string
id int
}
}
func check(err error) {
if err != nil {
panic(err.Error())
}
}
func main() {
svc := dynamodb.New(session.New())
input := &dynamodb.GetItemInput{
Key: map[string]*dynamodb.AttributeValue{"uuid": {S: aws.String("d610e853-5222-462c-9bb3-26ff5aa86e9d")}},
ProjectionExpression: aws.String("video_lite"),
TableName: aws.String("staging_video_ingestion"),
}
result, err := svc.GetItem(input)
check(err)
fmt.Printf("%+v\n", result.Item)
var t Video
err = dynamodbattribute.UnmarshalMap(result.Item, &t)
check(err)
fmt.Printf("%+v\n", t)
}
The output looks like:
map[video_lite:{
M: {
id: {
N: "1573"
},
name: {
S: "Bladerunner (Movie)"
}
}
}]
{video_lite:{name: id:0}}
Why doesn't the UnmarshalMap work?

Unmarshal instead of UnmarshalMap appears to be easier to work with.
type Video struct {
Name string `dynamodbav:"name"`
ID int `dynamodbav:"id"`
}
I threw in the dynamodbav struct tags in an act of desperation. It appears they are not needed when unmarshalling. And now I return the map value, instead of the whole map result.Item which I never figured out how to get to work with UnmarshalMap. :/
err = dynamodbattribute.Unmarshal(result.Item["video_lite"], &t)

Probably you can modify your struct to this :
type Video struct {
video_lite struct {
name struct {
N *string
S *string
}
id struct {
N *string
S *string
}
}
}

Related

Parsing ARN from IAM Policy using Regex

I have the following IAM Policy:
{"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":"arn:aws:sts::<account>:assumed-role/custom_role/<role>"},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<account>"}}}]}
but the "AWS" portion can also be an array:
"AWS": [
"arn:aws:sts::<account>:assumed-role/custom_role/<role_1>",
"arn:aws:sts::<account>:assumed-role/custom_role/<role_2>"
]
What I need is a regex that can parse both structures and return the list of arn:aws:sts as a list of strings... how can I accomplish that using regex in Golang?
I tried to use json.Unmarshal but the object structure is different between []string and string
Edit:
I have the following snippet:
re := regexp.MustCompile(`arn:aws:sts::[a-z0-9]*:assumed-role/custom_role/[a-z0-9]-*`)
result := re.FindAll([]byte(arn), 10)
for _, res := range result {
fmt.Println(string(res))
}
>>> `arn:aws:sts::<account_id>:assumed-role/custom_role/`
Using JSON decoder
You can decode the AWS key directly into a custom type implementing the "json.Unmarshaler" interface and decode both inputs correctly.
Demo
type AWSRoles []string
func (r *AWSRoles) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err == nil {
*r = append(*r, s)
return nil
}
var ss []string
if err := json.Unmarshal(b, &ss); err == nil {
*r = ss
return nil
}
return errors.New("cannot unmarshal neither to a string nor a slice of strings")
}
type AWSPolicy struct {
Statement []struct {
Principal struct {
AWSRoles AWSRoles `json:"AWS"`
} `json:"Principal"`
} `json:"Statement"`
}
Here's a test for it
var testsAWSPolicyParsing = []struct {
name string
input []byte
wantRoles []string
}{
{
name: "unique role",
input: []byte(`{"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":"arn:aws:sts::<account>:assumed-role/custom_role/<role>"},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<account>"}}}]}`),
wantRoles: []string{"arn:aws:sts::<account>:assumed-role/custom_role/<role>"},
},
{
name: "multiple roles",
input: []byte(`{"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":["arn:aws:sts::<account>:assumed-role/custom_role/<role_1>","arn:aws:sts::<account>:assumed-role/custom_role/<role_2>"]},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<account>"}}}]}`),
wantRoles: []string{
"arn:aws:sts::<account>:assumed-role/custom_role/<role_1>",
"arn:aws:sts::<account>:assumed-role/custom_role/<role_2>",
},
},
}
func TestParseAWSPolicy(t *testing.T) {
for _, tc := range testsAWSPolicyParsing {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var p AWSPolicy
err := json.Unmarshal(tc.input, &p)
if err != nil {
t.Fatal("unexpected error parsing AWSRoles policy", err)
}
if l := len(p.Statement); l != 1 {
t.Fatalf("unexpected Statement length. want 1, got %d", l)
}
if got := p.Statement[0].Principal.AWSRoles; !reflect.DeepEqual(got, tc.wantRoles) {
t.Fatalf("roles are not the same, got %v, want %v", got, tc.wantRoles)
}
})
}
}
Using a Regex
If you still want to use a regex, this one would parse it as long as:
AWS account has only numbers [0-9]
the custom role name has only alphanumeric characters and underscores
var awsRolesRegex = regexp.MustCompile("arn:aws:sts::[a-z0-9]+:assumed-role/custom_role/[a-zA-Z0-9_]+")
Demo

How to create new object from interface without knowing the explicit type in Go? [duplicate]

This question already has answers here:
How do you create a new instance of a struct from its type at run time in Go?
(5 answers)
Closed 9 months ago.
Working example files included at the end.
I have a package to assist in testing api handlers by creating test http contexts.
The issue is in the AssertJSONResponseBody. The problem is that once the concrete type and value are extracted from the interface, it contains a pointer to the original object. Any changes to the extracted object affect the original object. This then makes the following equal comparison useless because essentially they are pointing to the same value.
Here is the struct:
type TestRequest struct {
Recorder *httptest.ResponseRecorder
Context *gin.Context
t *testing.T
}
func (r *TestRequest) AssertJSONResponseBody(expectedObj interface{}, mesgAndArgs ...interface{}) bool {
outObject := reflect.Indirect(reflect.ValueOf(expectedObj)).Addr().Interface()
// Set break point at next line and compare expectedObject with outObject.
// Then, step over the line and watch the values for both objects change.
// When the decoder unmarshals the json the original object is changed
// because of the pointer in the outobject.
err := json.NewDecoder(r.Recorder.Body).Decode(outObject)
if err != nil {
return assert.Error(r.t, err)
}
return assert.Equal(r.t, expectedObj, outObject, mesgAndArgs...)
}
How do I create a new instance of the underlying type without coupling it to the original value via a pointer?
Here are the working example files.
APIHandler/main.go
package main
import (
"fmt"
"log"
"net/http"
"github.com/gin-gonic/gin"
)
const (
JSONBindingError string = "An error occurred binding the request."
EmailRequiredError string = "Email is required."
)
func main() {
router := gin.Default()
v1 := router.Group("/api/v1/contacts")
{
v1.POST("/", CreateContactHandler)
}
router.Run()
fmt.Printf("hello, world\n")
}
func CreateContactHandler(c *gin.Context) {
request := CreateContactRequest{}
err := c.Bind(&request)
if err != nil {
log.Println("ERROR:", JSONBindingError, err)
apiError := APIError{StatusCode: http.StatusBadRequest, Message: JSONBindingError}
c.JSON(http.StatusBadRequest, apiError)
return
}
if request.Contact.Email == "" {
log.Println("ERROR:", http.StatusBadRequest, EmailRequiredError)
apiError := APIError{StatusCode: http.StatusBadRequest, Message: EmailRequiredError}
c.JSON(http.StatusBadRequest, apiError)
return
}
// Successful client request
// resp := h.Client.CreateContact(request)
// c.JSON(resp.StatusCode, resp.Body)
}
type CreateContactRequest struct {
Contact Contact
}
type Contact struct {
Name string
Email string
}
type CreateContactResponse struct {
Message string
}
type APIError struct {
StatusCode int `json:"status"` // Should match the response status code
Message string `json:"message"`
}
type APIResponse struct {
StatusCode int
Body interface{}
}
APIHandler/helpers/http.go
package helpers
import (
"bytes"
"encoding/json"
"net/http/httptest"
"reflect"
"testing"
"github.com/stretchr/testify/assert"
"github.com/gin-gonic/gin"
)
// TestRequest is a struct to facilitate
// HTTP Context testing with gin handlers
type TestRequest struct {
Recorder *httptest.ResponseRecorder
Context *gin.Context
t *testing.T
}
// NewTestRequest returns a new TestRequest
func NewTestRequest(t *testing.T) *TestRequest {
rec := httptest.NewRecorder()
ctx, _ := gin.CreateTestContext(rec)
ctx.Request = httptest.NewRequest("GET", "/", nil)
return &TestRequest{
Recorder: rec,
Context: ctx,
t: t,
}
}
// SetJSONRequestBody returns a new TestRequest where the request is a post.
// Takes an interface to marshal into JSON and set as the request body.
func (r *TestRequest) SetJSONRequestBody(obj interface{}) *TestRequest {
json, err := json.Marshal(obj)
assert.NoError(r.t, err)
r.Context.Request = httptest.NewRequest("POST", "/", bytes.NewBuffer(json))
r.Context.Request.Header.Add("Content-Type", "application/json")
return r
}
// AssertStatusCode asserts that the recorded status
// is the same as the submitted status.
// The message and the args are added to the message
// when the assertion fails.
func (r *TestRequest) AssertStatusCode(expectedCode int, msgAndArgs ...interface{}) bool {
return assert.Equal(r.t, expectedCode, r.Recorder.Code, msgAndArgs...)
}
// AssertJSONResponseBody asserts that the recorded
// response body unmarshals to the given interface.
// The message and the args are added to the message
// when the assertion fails.
func (r *TestRequest) AssertJSONResponseBody(expectedObj interface{}, masgAndArgs ...interface{}) bool {
out := reflect.Indirect(reflect.ValueOf(expectedObj)).Addr().Interface()
err := json.NewDecoder(r.Recorder.Body).Decode(out)
if err != nil {
return assert.Error(r.t, err)
}
return assert.Equal(r.t, expectedObj, out, masgAndArgs...)
}
APIHandler/main_test.go
package main
import (
"APIHandler/helpers"
"net/http"
"testing"
)
func TestSingleContactCreate(t *testing.T) {
tests := []struct {
toCreate interface{}
handlerExpected APIError
clientReturn APIResponse
statusCode int
}{
// when there is a JSON binding error
{toCreate: "",
handlerExpected: APIError{StatusCode: http.StatusBadRequest, Message: EmailRequiredError},
clientReturn: APIResponse{},
statusCode: http.StatusBadRequest},
// when email is missing
{toCreate: CreateContactRequest{
Contact: Contact{
Name: "test",
}},
handlerExpected: APIError{StatusCode: http.StatusBadRequest, Message: JSONBindingError},
clientReturn: APIResponse{},
statusCode: http.StatusBadRequest},
}
// act
for i, test := range tests {
req := helpers.NewTestRequest(t)
req.SetJSONRequestBody(test.toCreate)
CreateContactHandler(req.Context)
// assert
req.AssertStatusCode(test.statusCode, "Test %d", i)
req.AssertJSONResponseBody(&test.handlerExpected, "Test %d", i)
}
}
Here's the resolution per #mkopriva:
func (r *TestRequest) AssertJSONResponseBody(expectedObj interface{}, masgAndArgs ...interface{}) bool {
elem := reflect.TypeOf(expectedObj)
theType := elem.Elem()
newInstance := reflect.New(theType)
out := newInstance.Interface()
err := json.NewDecoder(r.Recorder.Body).Decode(out)
if err != nil {
return assert.Error(r.t, err)
}
return assert.Equal(r.t, expectedObj, out, masgAndArgs...)
}

Relationships are always nil when doing dynamoattribute.UnmarshalMap

I have two structs. Sample and Test. 'Sample' has a relationship of type 'Test'. When I try to do 'dynamoattribute.UnmarshalMap', the relationship is always nil. Could you advise how to populate relationships ('Test' in this case) please?
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)
type Sample struct {
SampleId string `jsonapi:"attr,sampleId,omitempty" dynamodbav:"sample_id"`
Test *Test `jsonapi:"relation,test"`
}
type Test struct {
TestId string `jsonapi:"attr,testId,omitempty" dynamodbav:"test_id"`
}
func main() {
var m map[string]*dynamodb.AttributeValue
m = make(map[string]*dynamodb.AttributeValue)
m["sample_id"] = &dynamodb.AttributeValue{
S: aws.String("sample1"),
}
m["test_id"] = &dynamodb.AttributeValue{
S: aws.String("test"),
}
sam := Sample{}
err := dynamodbattribute.UnmarshalMap(m, &sam)
if err != nil {
fmt.Println(err)
}
fmt.Println(sam)
}
package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)
type Sample struct {
SampleId string `jsonapi:"attr,sampleId,omitempty" dynamodbav:"sample_id"`
Test *Test `jsonapi:"relation,test"`
}
type Test struct {
TestId string `jsonapi:"primary,testId" dynamodbav:"test_id"`
}
func main() {
var m map[string]*dynamodb.AttributeValue
m = make(map[string]*dynamodb.AttributeValue)
m["sample_id"] = &dynamodb.AttributeValue{
S: aws.String("sample1"),
}
var mTest map[string]*dynamodb.AttributeValue
mTest = make(map[string]*dynamodb.AttributeValue)
mTest["test_id"] = &dynamodb.AttributeValue{
S: aws.String("test1"),
}
m["test"] = &dynamodb.AttributeValue{
M: mTest,
}
sam := Sample{}
err := dynamodbattribute.UnmarshalMap(m, &sam)
if err != nil {
fmt.Println(err)
}
fmt.Println(sam)
}

How do I write unit test aws-sdk-go-v2 dynamodb implementation

I am still grasping go-interfaces and I can mock the WaitUntilTableExists func. But unable to mock PutItemRequest.
Here's my main.go snippet
func MyPutItem(d mydata, client dynamodbiface.DynamoDBAPI) error {
input := &dynamodb.PutItemInput{
....
}
req := client.PutItemRequest(input)
result, err := req.Send()
log.Println(result)
return err
}
main_test.go snippet
type mockDynamoDBClient struct {
dynamodbiface.DynamoDBAPI
}
func (m *mockDynamoDBClient) PutItemRequest(input *dynamodb.PutItemInput) dynamodb.PutItemRequest {
// Most probably this is where I need your help
}
func TestStoreInDynamoDB(t *testing.T) {
var mockClient = new(mockDynamoDBClient)
d := mydata{}
result := DynampDBPutItem(d, mockClient)
t.Log(result)
}
Taking your example, you could do your assertions directly in the mock
type mockDynamoDBClient struct {
t *testing.T
expected *dynamodb.PutItemInput
response *dynamodb.PutItemOutput
dynamodbiface.DynamoDBAPI
}
func (m *mockDynamoDBClient) PutItemRequest(input *dynamodb.PutItemInput) dynamodb.PutItemOutput {
// some kind of equality check
if !reflect.DeepEqual(m.expected, input) {
t.Errorf(...// some error message)
}
return m.response
}
The main problems with this example are:
t *testing.T, expected *dynamodb.PutItemInput and response response *dynamodb.PutItemOutput all need to be inside the struct which feels messy.
Instead you could use an anonymous function to do this:
type mockDynamoDBClient struct {
f func(input *dynmaodb.PutItemInput) *dynamodb.PutItemOutput
dynamodbiface.DynamoDBAPI
}
func (m *mockDynamoDBClient) PutItemRequest(input *dynamodb.PutItemInput) dynamodb.PutItemOutput {
return m.f(input)
}
Now in the test code you can make slightly better use of the mock struct:
m := &mockDynamoDBClient{
f: func(input *dynamodb.PutItemInput) *dynamodb.PutItemOutput {
// assertions on input
// return mock responses
}
}
EDIT based on comment:
You should also consider making your MyPutItem function dependent on the smallest interface possible. If you only need access to the PutItemRequest method then you can create your own interface for that method and use that in MyPutItem
type MyDynamoPutter interface {
func (c *DynamoDB) PutItemRequest(input *PutItemInput) PutItemRequest
}
Then in MyPutItem you can use your own interface:
func MyPutItem(d mydata, client MyDynamoPutter) error {
input := &dynamodb.PutItemInput{
....
}
req := client.PutItemRequest(input)
result, err := req.Send()
log.Println(result)
return err
}
This reduces the surface area that you need to mock!
Faking the SDK like this works:
main_test.go
type fakeDynamoDBClient struct {
dynamodbiface.DynamoDBAPI
}
func (m *fakeDynamoDBClient) GetItemRequest(input *dynamodb.GetItemInput) dynamodb.GetItemRequest {
return dynamodb.GetItemRequest{
Request: &aws.Request{
Data: &dynamodb.GetItemOutput{
Item: map[string]dynamodb.AttributeValue{
"count": dynamodb.AttributeValue{
N: aws.String("10"),
},
},
},
},
}
}
func (m *fakeDynamoDBClient) PutItemRequest(input *dynamodb.PutItemInput) dynamodb.PutItemRequest {
return dynamodb.PutItemRequest{
Request: &aws.Request{
Data: &dynamodb.PutItemOutput{},
},
}
}
func TestUpdateCount(t *testing.T) {
err := UpdateCount(10, &fakeDynamoDBClient{})
if err != nil {
t.Error("Failed to update badge count on dynamodb", err)
}
}
main.go
func UpdateCount(count int, client dynamodbiface.DynamoDBAPI) error {
...
}

How to access a struct slice element key directly in a template file

I am loading some template files and attempting to compile them with some structs I've defined.
The following example is working. I'd like to know if there is a better way of formatting my templateFile to directly access config.Servers[1].Ip1 without needing two sets of {{}}
templateFile:
{{$a := index .Servers 1 }}{{$a.Ip1}} some extra text
learn.go:
package main
import (
"html/template"
"os"
)
type Server struct {
Ip1 string
Ip2 string
}
type Configuration struct {
Servers []Server
}
func main() {
someServers := []Server{
{
Ip1: "1.1.1.1",
Ip2: "2.2.2.2",
},
{
Ip1: "3.3.3.3",
Ip2: "4.4.4.4",
},
}
config := Configuration{
Servers: someServers,
}
tmpl, err := template.ParseFiles("./templateFile")
if err != nil {
panic(err)
}
err = tmpl.Execute(os.Stdout, config)
if err != nil {
panic(err)
}
}
Please refer this:
https://golang.org/pkg/html/template/
You have to use {{}} in your HTML template, if you wish to access any Struct variable.