How to parse PutMetricData Sample Request as show below.
I want to parse all the MetricData and stores the values in a struct in golang.
https://monitoring.&api-domain;/doc/2010-08-01/
?Action=PutMetricData
&Version=2010-08-01
&Namespace=TestNamespace
&MetricData.member.1.MetricName=buffers
&MetricData.member.1.Unit=Bytes
&MetricData.member.1.Value=231434333
&MetricData.member.1.Dimensions.member.1.Name=InstanceID
&MetricData.member.1.Dimensions.member.1.Value=i-aaba32d4
&MetricData.member.1.Dimensions.member.2.Name=InstanceType
&MetricData.member.1.Dimensions.member.2.Value=m1.small
&MetricData.member.2.MetricName=latency
&MetricData.member.2.Unit=Milliseconds
&MetricData.member.2.Value=23
&MetricData.member.2.Dimensions.member.1.Name=InstanceID
&MetricData.member.2.Dimensions.member.1.Value=i-aaba32d4
&MetricData.member.2.Dimensions.member.2.Name=InstanceType
&MetricData.member.2.Dimensions.member.2.Value=m1.small**
&AUTHPARAMS
Not able to understand this is in which format and how to parse it. Any library available to generate and parse this kind of formatted message?
If you remove the newlines that is a URL. Start with url.Parse, then use the Query() function to get access to the url parameters:
func main() {
var input = `https://monitoring.&api-domain;/doc/2010-08-01/
?Action=PutMetricData
&Version=2010-08-01
&Namespace=TestNamespace
&MetricData.member.1.MetricName=buffers
&MetricData.member.1.Unit=Bytes
&MetricData.member.1.Value=231434333
&MetricData.member.1.Dimensions.member.1.Name=InstanceID
&MetricData.member.1.Dimensions.member.1.Value=i-aaba32d4
&MetricData.member.1.Dimensions.member.2.Name=InstanceType
&MetricData.member.1.Dimensions.member.2.Value=m1.small
&MetricData.member.2.MetricName=latency
&MetricData.member.2.Unit=Milliseconds
&MetricData.member.2.Value=23
&MetricData.member.2.Dimensions.member.1.Name=InstanceID
&MetricData.member.2.Dimensions.member.1.Value=i-aaba32d4
&MetricData.member.2.Dimensions.member.2.Name=InstanceType
&MetricData.member.2.Dimensions.member.2.Value=m1.small**
&AUTHPARAMS`
// possibly also needs to replace \r
input = strings.ReplaceAll(input, "\n", "")
uri, err := url.Parse(input)
if err != nil {
log.Fatal(err)
}
for key, val := range uri.Query() {
fmt.Println(key, val)
}
}
Playground
From here on out it's up to you how you want the target struct to look like.
Related
I'm having a hard time to get the current Cognito user attributes from within my lambda function, that is written in Go. I'm currently doing:
userAttributes = request.RequestContext.Authorizer["claims"]
And if I want to get the email:
userEmail = request.RequestContext.Authorizer["claims"].(map[string]interface{})["email"].(string)
I don't think this is a good way or even an acceptable way - it must have a better way to do it.
You can use 3rd party library to convert map[string]interface{} to a concrete type. Check the mitchellh/mapstructure library, it will help you to implement in a better way.
So, you could improve your code with this code :
import "github.com/mitchellh/mapstructure"
type Claims struct {
Email string
// other fields
ID int
}
func claims(r request.Request) (Claims, error) {
input := r.RequestContext.Authorizer["claims"]
output := Claims{}
err := mapstructure.Decode(input, &output)
if err != nil {
return nil, err
}
return output, nil
}
And somewhere in your handlers, you could get your claims by calling this method
func someWhere(){
userClaims, err := claims(request)
if err != nil {
// handle
}
// you can now use : userClaims.Email, userClaims.ID
}
Don't forget to change func claims request parameter type according to yours (r parameter).
I am trying to learn how to write tests for my code in order to write better code, but I just seem to have the hardest time figuring out how to actually test some code I have written. I have read so many tutorials, most of which seem to only cover functions that add two numbers or mock some database or server.
I have a simple function I wrote below that takes a text template and a CSV file as input and executes the template using the values of the CSV. I have "tested" the code by trial and error, passing files, and printing values, but I would like to learn how to write proper tests for it. I feel that learning to test my own code will help me understand and learn faster and better. Any help is appreciated.
// generateCmds generates configuration commands from a text template using
// the values from a CSV file. Multiple commands in the text template must
// be delimited by a semicolon. The first row of the CSV file is assumed to
// be the header row and the header values are used for key access in the
// text template.
func generateCmds(cmdTmpl string, filename string) ([]string, error) {
t, err := template.New("cmds").Parse(cmdTmpl)
if err != nil {
return nil, fmt.Errorf("parsing template: %v", err)
}
f, err := os.Open(filename)
if err != nil {
return nil, fmt.Errorf("reading file: %v", err)
}
defer f.Close()
records, err := csv.NewReader(f).ReadAll()
if err != nil {
return nil, fmt.Errorf("reading records: %v", err)
}
if len(records) == 0 {
return nil, errors.New("no records to process")
}
var (
b bytes.Buffer
cmds []string
keys = records[0]
vals = make(map[string]string, len(keys))
)
for _, rec := range records[1:] {
for k, v := range rec {
vals[keys[k]] = v
}
if err := t.Execute(&b, vals); err != nil {
return nil, fmt.Errorf("executing template: %v", err)
}
for _, s := range strings.Split(b.String(), ";") {
if cmd := strings.TrimSpace(s); cmd != "" {
cmds = append(cmds, cmd)
}
}
b.Reset()
}
return cmds, nil
}
Edit: Thanks for all the suggestions so far! My question was flagged as being too broad, so I have some specific questions regarding my example.
Would a test table be useful in a function like this? And, if so, would the test struct need to include the returned cmds string slice and the value of err? For example:
type tmplTest struct {
name string // test name
tmpl string // the text template
filename string // CSV file with template values
expected []string // expected configuration commands
err error // expected error
}
How do you handle errors that are supposed to be returned for specific test cases? For example, os.Open() returns an error of type *PathError if an error is encountered. How do I initialize a *PathError that is equivalent to the one returned by os.Open()? Same idea for template.Parse(), template.Execute(), etc.
Edit 2: Below is a test function I came up with. My two question from the first edit still stand.
package cmd
import (
"testing"
"strings"
"path/filepath"
)
type tmplTest struct {
name string // test name
tmpl string // text template to execute
filename string // CSV containing template text values
cmds []string // expected configuration commands
}
var tests = []tmplTest{
{"empty_error", ``, "", nil},
{"file_error", ``, "fake_file.csv", nil},
{"file_empty_error", ``, "empty.csv", nil},
{"file_fmt_error", ``, "fmt_err.csv", nil},
{"template_fmt_error", `{{ }{{`, "test_values.csv", nil},
{"template_key_error", `{{.InvalidKey}}`, "test_values.csv", nil},
}
func TestGenerateCmds(t *testing.T) {
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
cmds, err := generateCmds(tc.tmpl, filepath.Join("testdata", tc.filename))
if err != nil {
// Unexpected error. Fail the test.
if !strings.Contains(tc.name, "error") {
t.Fatal(err)
}
// TODO: Otherwise, check that the function failed at the expected point.
}
if tc.cmds == nil && cmds != nil {
t.Errorf("expected no commands; got %d", len(cmds))
}
if len(cmds) != len(tc.cmds) {
t.Errorf("expected %d commands; got %d", len(tc.cmds), len(cmds))
}
for i := range cmds {
if cmds[i] != tc.cmds[i] {
t.Errorf("expected %q; got %q", tc.cmds[i], cmds[i])
}
}
})
}
}
You basically need to have some sample files with the contents you want to test, then in your test code you can call the generateCmds function passing in the template string and the files to then verify that the results are what you expect.
It is not so much different as the examples you probably saw for simpler cases.
You can place the files under a testdata folder inside the same package (testdata is a special name that the Go tools will ignore during build).
Then you can do something like:
func TestCSVProcessing(t *testing.T) {
templateStr := `<your template here>`
testFile := "testdata/yourtestfile.csv"
result, err := generateCmds(templateStr, testFile)
if err != nil {
// fail the test here, unless you expected an error with this file
}
// compare the "result" contents with what you expected
// failing the test if it does not match
}
EDIT
About the specific questions you added later:
Would a test table be useful in a function like this? And, if so, would the test struct need to include the returned cmds string slice and the value of err?
Yes, it'd make sense to include both the expected strings to be returned as well as the expected error (if any).
How do you handle errors that are supposed to be returned for specific test cases? For example, os.Open() returns an error of type *PathError if an error is encountered. How do I initialize a *PathError that is equivalent to the one returned by os.Open()?
I don't think you'll be able to "initialize" an equivalent error for each case. Sometimes the libraries might use internal types for their errors making this impossible. Easiest would be to "initialize" a regular error with the same value returned in its Error() method, then just compare the returned error's Error() value with the expected one.
I am having some trouble when using github.com/go-validator/validator to validate regex some phone numbers with this prefix +62, 62, 0, for instance number e.g. +628112blabla, 0822blablabla, 628796blablabla.
I have try my regex on online regex tester and no issue with the regex on that. Here the regex is :
(0|\+62|062|62)[0-9]+$
But when I try with my go implement with it, the regex not working. This is my code for implement the purpose :
type ParamRequest struct {
PhoneNumber string `validate:"nonzero,regexp=(0|\+62|062|62)[0-9]+$"`
ItemCode string `validate:"nonzero"`
CallbackUrl string `validate:"nonzero"`
}
func (c *TopupAlloperatorApiController) Post() {
var v models.TopupAlloperatorApi
interf := make(map[string]interface{})
json.Unmarshal(c.Ctx.Input.RequestBody, &interf)
logs.Debug(" Json Input Request ", interf)
var phone, item, callback string
if _, a := interf["PhoneNumber"].(string); a {
phone = interf["PhoneNumber"].(string)
}
if _, b := interf["ItemCode"].(string); b {
item = interf["ItemCode"].(string)
}
if _, c := interf["CallbackUrl"].(string); c {
callback = interf["CallbackUrl"].(string)
}
ve := ParamRequest{
PhoneNumber: phone,
ItemCode: item,
CallbackUrl: callback,
}
logs.Debug(" Param Request ", ve)
err := validator.Validate(ve)
if err == nil {
//success
}else{
// not success
}
Many thanks for anything help. Thank you.
Because you are using regexp to check PhoneNumber that won't be matching if the value is empty it is better to remove nonzero from the validation.
I have checked out documentation and haven't found examples where you can use both: nonzero and regexp.
Also you need to make your regex symbol-escaped, otherwise it won't be detected by reflection. It means you should use (0|\\+62|062|62)[0-9]+$ in your code. Here is example where problem is: symbol escaping in struct tags
And also, please try to use this regexp: ^\\+{0,1}0{0,1}62[0-9]+$
I'm working on converting a pet project of mine from Python to Go just to help me get a bit familiar with the language. An issue I am currently facing is that it's escaping my forward slashes. So it will receive a string like:
/location/to/something
and it then becomes
%2flocation%2fto%2fsomething
Now, it's only doing this when it's in a link (from what I've been reading this escaping is contextual) so this is what the line in the HTML template looks like:
<tr><td>{{.FileName}}</td></tr>
If possible, how can I prevent this in either the template or the code itself?
This is what my templating function looks like (yes, I know it's hackish)
func renderTemplate(w http.ResponseWriter, tmpl string) {
t, err := template.ParseFiles(templates_dir+"base.html", templates_dir+tmpl)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if tmpl == "view.html" {
err = t.Execute(w, FileList)
} else {
err = t.Execute(w, nil)
}
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
As the value of .FullFilePath, pass a value of type template.URL instead of string, which will tell the html/template package not to escape it.
For example:
func main() {
t := template.Must(template.New("").Parse(templ))
m := map[string]interface{}{
"FileName": "something.txt",
"FileFullPath": template.URL("/location/to/something"),
}
if err := t.Execute(os.Stdout, m); err != nil {
panic(err)
}
}
const templ = `<tr><td>{{.FileName}}</td></tr>`
Output (try it on the Go Playground):
<tr><td>something.txt</td></tr>
Note that even though forward slashes / are allowed in URLs, the reason why the template package still encodes them is because it analyses the URL and sees that the value you want to include is the value of a URL parameter (file=XXX), and so it also encodes the slashes (so that everything you pass in will be part of the value of the file URL parameter).
If you plan to acquire this file path at the server side from URL parameters, then what the template package does is the correct and proper way.
But know that by doing this, you'll lose the safety that prevents code injection into URLs. If you're the one providing the values and you know they are safe, there is no problem. But if the data comes from a user input for example, never do this.
Also note that if you pass the whole URL (and not just a part of it), it will work without using template.URL (try this variant on the Go Playground):
func main() {
t := template.Must(template.New("").Parse(templ))
m := map[string]interface{}{
"FileName": "something.txt",
"FileURL": "/file?file=/location/to/something",
}
if err := t.Execute(os.Stdout, m); err != nil {
panic(err)
}
}
const templ = `<tr><td>{{.FileName}}</td></tr>`
Also note that the recommended way in my opinion would be to include the file path as part of the URL path and not as the value of a parameter, so instead you should create urls like this:
/file/location/to/something
Map your handler (which serves the file content, see this answer as an example) to the /file/ pattern, and when it is matched and your handler is called, cut off the /file/ prefix from the path r.URL.Path, and the rest will be the full file path. If you choose this, you also won't need the template.URL conversion (because the value you include is not a value of a URL parameter anymore):
func main() {
t := template.Must(template.New("").Parse(templ))
m := map[string]interface{}{
"FileName": "something.txt",
"FileFullPath": "/location/to/something",
}
if err := t.Execute(os.Stdout, m); err != nil {
panic(err)
}
}
const templ = `<tr><td>{{.FileName}}</td></tr>`
Try this on the Go Playground.
Also very important: never parse templates in your handler functions! For details see:
It takes too much time when using "template" package to generate a dynamic web page to client in golang
OK, So the solution I've found (and please post if there's a better one) is based on an answer here.
I changed the struct I was using from:
type File struct {
FullFilePath string
FileName string
}
To this:
type File struct {
FullFilePath template.HTML
FileName string
}
And moved the html into the FullFilePath name, and then placed that in template.HTML so each FullFilePath name I was generating was done like so:
file := File{template.HTML("<a href=\"/file?file=" + path + "\"</a>"), f.Name()}
And my template file line was changed to this:
<tr><td>{{.FullFilePath}}{{.FileName}}</td></tr>
Help. I can't get the right signature using the test parameters provided by Amazon and Go.
My signature hash function is as follows. I use SHA-256 and base64 encoding as per Amazon documentation.
func HashSignature(str string, secret string) string {
mac := hmac.New(sha256.New, []byte(secret))
_, err := mac.Write([]byte(str))
if err != nil { return "" }
hash := base64.StdEncoding.EncodeToString(mac.Sum(nil))
hash = url.QueryEscape(hash)
return hash
}
My signature test function is as follows. I use the canonical string below in Ruby code and it generates the correct expected signature. So the problem seems to be with the output of my HashSignature() function, but I don't see what I'm doing wrong there.
func TestAmazonSignature(t *testing.T) {
/* here is the canonical string from Amazon documentation which should yield the expected signature below
GET
webservices.amazon.com
/onca/xml
AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&AssociateTag=mytag-20&ItemId=0679722769&Operation=ItemLookup&ResponseGroup=Images%2CItemAttributes%2COffers%2CReviews&Service=AWSECommerceService&Timestamp=2014-08-18T12%3A00%3A00Z&Version=2013-08-01
*/
SECRET_KEY := "1234567890"
CANONICAL_STR := "GET\nwebservices.amazon.com\n/onca/xml\nAWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&AssociateTag=mytag-20&ItemId=0679722769&Operation=ItemLookup&ResponseGroup=Images%2CItemAttributes%2COffers%2CReviews&Service=AWSECommerceService&Timestamp=2014-08-18T12%3A00%3A00Z&Version=2013-08-01"
EXPECTED := "j7bZM0LXZ9eXeZruTqWm2DIvDYVUU3wxPPpp%2BiXxzQc%3D"
if RESULT := HashSignature(CANONICAL_STR, SECRET_KEY); RESULT != EXPECTED {
t.Errorf("\nEXPECTED:\n%v\nRESULT:\n%v", EXPECTED, RESULT)
} else { fmt.Println("TestAmazonSignature: Signature: OK") }
}
Here's a playground link with all this code.
Looks fine to me, try running:
https://play.golang.org/p/w0mQAYx2GQ
I added necessary imports and a main function