Prevent escaping forward slashes in templates - templates

I'm working on converting a pet project of mine from Python to Go just to help me get a bit familiar with the language. An issue I am currently facing is that it's escaping my forward slashes. So it will receive a string like:
/location/to/something
and it then becomes
%2flocation%2fto%2fsomething
Now, it's only doing this when it's in a link (from what I've been reading this escaping is contextual) so this is what the line in the HTML template looks like:
<tr><td>{{.FileName}}</td></tr>
If possible, how can I prevent this in either the template or the code itself?
This is what my templating function looks like (yes, I know it's hackish)
func renderTemplate(w http.ResponseWriter, tmpl string) {
t, err := template.ParseFiles(templates_dir+"base.html", templates_dir+tmpl)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if tmpl == "view.html" {
err = t.Execute(w, FileList)
} else {
err = t.Execute(w, nil)
}
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}

As the value of .FullFilePath, pass a value of type template.URL instead of string, which will tell the html/template package not to escape it.
For example:
func main() {
t := template.Must(template.New("").Parse(templ))
m := map[string]interface{}{
"FileName": "something.txt",
"FileFullPath": template.URL("/location/to/something"),
}
if err := t.Execute(os.Stdout, m); err != nil {
panic(err)
}
}
const templ = `<tr><td>{{.FileName}}</td></tr>`
Output (try it on the Go Playground):
<tr><td>something.txt</td></tr>
Note that even though forward slashes / are allowed in URLs, the reason why the template package still encodes them is because it analyses the URL and sees that the value you want to include is the value of a URL parameter (file=XXX), and so it also encodes the slashes (so that everything you pass in will be part of the value of the file URL parameter).
If you plan to acquire this file path at the server side from URL parameters, then what the template package does is the correct and proper way.
But know that by doing this, you'll lose the safety that prevents code injection into URLs. If you're the one providing the values and you know they are safe, there is no problem. But if the data comes from a user input for example, never do this.
Also note that if you pass the whole URL (and not just a part of it), it will work without using template.URL (try this variant on the Go Playground):
func main() {
t := template.Must(template.New("").Parse(templ))
m := map[string]interface{}{
"FileName": "something.txt",
"FileURL": "/file?file=/location/to/something",
}
if err := t.Execute(os.Stdout, m); err != nil {
panic(err)
}
}
const templ = `<tr><td>{{.FileName}}</td></tr>`
Also note that the recommended way in my opinion would be to include the file path as part of the URL path and not as the value of a parameter, so instead you should create urls like this:
/file/location/to/something
Map your handler (which serves the file content, see this answer as an example) to the /file/ pattern, and when it is matched and your handler is called, cut off the /file/ prefix from the path r.URL.Path, and the rest will be the full file path. If you choose this, you also won't need the template.URL conversion (because the value you include is not a value of a URL parameter anymore):
func main() {
t := template.Must(template.New("").Parse(templ))
m := map[string]interface{}{
"FileName": "something.txt",
"FileFullPath": "/location/to/something",
}
if err := t.Execute(os.Stdout, m); err != nil {
panic(err)
}
}
const templ = `<tr><td>{{.FileName}}</td></tr>`
Try this on the Go Playground.
Also very important: never parse templates in your handler functions! For details see:
It takes too much time when using "template" package to generate a dynamic web page to client in golang

OK, So the solution I've found (and please post if there's a better one) is based on an answer here.
I changed the struct I was using from:
type File struct {
FullFilePath string
FileName string
}
To this:
type File struct {
FullFilePath template.HTML
FileName string
}
And moved the html into the FullFilePath name, and then placed that in template.HTML so each FullFilePath name I was generating was done like so:
file := File{template.HTML("<a href=\"/file?file=" + path + "\"</a>"), f.Name()}
And my template file line was changed to this:
<tr><td>{{.FullFilePath}}{{.FileName}}</td></tr>

Related

Unit test function that is opening and reading a file

I am working on learning go with a simple program that is doing some file reading and am working on adding unit testing to my program. I have ran into an issue/question while doing this. I want to unit test the function below and my question is that the function takes a name of the file which is then opened and processed. During testing I do not want to actually pass it a real file. I am wondering is this something I can somehow mock so that I can just pass it a "fake" file and have it process that instead? Thanks!
func openAndReadFile(fileName string) [][]string {
file, err := os.Open(fileName)
if err != nil {
fmt.Printf("Failed to read file: %s", fileName)
}
r := csv.NewReader(file)
lines, err := r.ReadAll()
if err != nil {
log.Fatal(err)
}
return lines
}
You need to refactor your code and make more suitable for testing.
Here is how I would do it:
func openAndReadFile(fileName string) [][]string {
file, err := os.Open(fileName)
if err != nil {
fmt.Printf("Failed to open file: %s", fileName)
}
lines, err := readFile(file)
if err != nil {
fmt.Printf("Failed to read file: %s", fileName)
}
return lines
}
func readFile(reader io.Reader) ([][]string, error) {
r := csv.NewReader(reader)
lines, err := r.ReadAll()
if err != nil {
log.Fatal(err)
}
return lines, err
}
Then for testing you can simply use any data structure that implements the io.reader interface. For example, I use a bytes buffer, but you can choose a network connection:
func TestReadFile(t *testing.T) {
var buffer bytes.Buffer
buffer.WriteString("fake, csv, data")
content, err := readFile(&buffer)
if err != nil {
t.Error("Failed to read csv data")
}
fmt.Print(content)
}
The function you have shown is dominated by interactions: Interactions with the file system and interactions with the csv reader. To be sure that these interactions work nicely you will later anyway have to do some integration-testing against the file system and the csv reader. Think about which bugs you are hoping to find, and you will see that bugs are more likely on the interaction level: Is the order of file,err correct, or should it be the other way around? Is nil really the value indicating no error? Do you have to give more arguments to Open? etc.
Therefore, I would not concentrate on unit-testing this function. However, this function is a good candidate to be mocked to make unit-testing the surrounding code easier. Thus, mock openAndReadFile to unit-test the surrounding code, and test openAndReadFile using integration-testing.
I'd strongly suggest using an interface instead of the filename string like the other answers here are recommending, but if you really must do this the only way is likely with a temp file. The decision to use a string file name has locked the code into assuming something to be present on the file system and has pushed in the responsibility of file management.

How to get current cognito user from within go lambda

I'm having a hard time to get the current Cognito user attributes from within my lambda function, that is written in Go. I'm currently doing:
userAttributes = request.RequestContext.Authorizer["claims"]
And if I want to get the email:
userEmail = request.RequestContext.Authorizer["claims"].(map[string]interface{})["email"].(string)
I don't think this is a good way or even an acceptable way - it must have a better way to do it.
You can use 3rd party library to convert map[string]interface{} to a concrete type. Check the mitchellh/mapstructure library, it will help you to implement in a better way.
So, you could improve your code with this code :
import "github.com/mitchellh/mapstructure"
type Claims struct {
Email string
// other fields
ID int
}
func claims(r request.Request) (Claims, error) {
input := r.RequestContext.Authorizer["claims"]
output := Claims{}
err := mapstructure.Decode(input, &output)
if err != nil {
return nil, err
}
return output, nil
}
And somewhere in your handlers, you could get your claims by calling this method
func someWhere(){
userClaims, err := claims(request)
if err != nil {
// handle
}
// you can now use : userClaims.Email, userClaims.ID
}
Don't forget to change func claims request parameter type according to yours (r parameter).

Learning to write unit tests

I am trying to learn how to write tests for my code in order to write better code, but I just seem to have the hardest time figuring out how to actually test some code I have written. I have read so many tutorials, most of which seem to only cover functions that add two numbers or mock some database or server.
I have a simple function I wrote below that takes a text template and a CSV file as input and executes the template using the values of the CSV. I have "tested" the code by trial and error, passing files, and printing values, but I would like to learn how to write proper tests for it. I feel that learning to test my own code will help me understand and learn faster and better. Any help is appreciated.
// generateCmds generates configuration commands from a text template using
// the values from a CSV file. Multiple commands in the text template must
// be delimited by a semicolon. The first row of the CSV file is assumed to
// be the header row and the header values are used for key access in the
// text template.
func generateCmds(cmdTmpl string, filename string) ([]string, error) {
t, err := template.New("cmds").Parse(cmdTmpl)
if err != nil {
return nil, fmt.Errorf("parsing template: %v", err)
}
f, err := os.Open(filename)
if err != nil {
return nil, fmt.Errorf("reading file: %v", err)
}
defer f.Close()
records, err := csv.NewReader(f).ReadAll()
if err != nil {
return nil, fmt.Errorf("reading records: %v", err)
}
if len(records) == 0 {
return nil, errors.New("no records to process")
}
var (
b bytes.Buffer
cmds []string
keys = records[0]
vals = make(map[string]string, len(keys))
)
for _, rec := range records[1:] {
for k, v := range rec {
vals[keys[k]] = v
}
if err := t.Execute(&b, vals); err != nil {
return nil, fmt.Errorf("executing template: %v", err)
}
for _, s := range strings.Split(b.String(), ";") {
if cmd := strings.TrimSpace(s); cmd != "" {
cmds = append(cmds, cmd)
}
}
b.Reset()
}
return cmds, nil
}
Edit: Thanks for all the suggestions so far! My question was flagged as being too broad, so I have some specific questions regarding my example.
Would a test table be useful in a function like this? And, if so, would the test struct need to include the returned cmds string slice and the value of err? For example:
type tmplTest struct {
name string // test name
tmpl string // the text template
filename string // CSV file with template values
expected []string // expected configuration commands
err error // expected error
}
How do you handle errors that are supposed to be returned for specific test cases? For example, os.Open() returns an error of type *PathError if an error is encountered. How do I initialize a *PathError that is equivalent to the one returned by os.Open()? Same idea for template.Parse(), template.Execute(), etc.
Edit 2: Below is a test function I came up with. My two question from the first edit still stand.
package cmd
import (
"testing"
"strings"
"path/filepath"
)
type tmplTest struct {
name string // test name
tmpl string // text template to execute
filename string // CSV containing template text values
cmds []string // expected configuration commands
}
var tests = []tmplTest{
{"empty_error", ``, "", nil},
{"file_error", ``, "fake_file.csv", nil},
{"file_empty_error", ``, "empty.csv", nil},
{"file_fmt_error", ``, "fmt_err.csv", nil},
{"template_fmt_error", `{{ }{{`, "test_values.csv", nil},
{"template_key_error", `{{.InvalidKey}}`, "test_values.csv", nil},
}
func TestGenerateCmds(t *testing.T) {
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
cmds, err := generateCmds(tc.tmpl, filepath.Join("testdata", tc.filename))
if err != nil {
// Unexpected error. Fail the test.
if !strings.Contains(tc.name, "error") {
t.Fatal(err)
}
// TODO: Otherwise, check that the function failed at the expected point.
}
if tc.cmds == nil && cmds != nil {
t.Errorf("expected no commands; got %d", len(cmds))
}
if len(cmds) != len(tc.cmds) {
t.Errorf("expected %d commands; got %d", len(tc.cmds), len(cmds))
}
for i := range cmds {
if cmds[i] != tc.cmds[i] {
t.Errorf("expected %q; got %q", tc.cmds[i], cmds[i])
}
}
})
}
}
You basically need to have some sample files with the contents you want to test, then in your test code you can call the generateCmds function passing in the template string and the files to then verify that the results are what you expect.
It is not so much different as the examples you probably saw for simpler cases.
You can place the files under a testdata folder inside the same package (testdata is a special name that the Go tools will ignore during build).
Then you can do something like:
func TestCSVProcessing(t *testing.T) {
templateStr := `<your template here>`
testFile := "testdata/yourtestfile.csv"
result, err := generateCmds(templateStr, testFile)
if err != nil {
// fail the test here, unless you expected an error with this file
}
// compare the "result" contents with what you expected
// failing the test if it does not match
}
EDIT
About the specific questions you added later:
Would a test table be useful in a function like this? And, if so, would the test struct need to include the returned cmds string slice and the value of err?
Yes, it'd make sense to include both the expected strings to be returned as well as the expected error (if any).
How do you handle errors that are supposed to be returned for specific test cases? For example, os.Open() returns an error of type *PathError if an error is encountered. How do I initialize a *PathError that is equivalent to the one returned by os.Open()?
I don't think you'll be able to "initialize" an equivalent error for each case. Sometimes the libraries might use internal types for their errors making this impossible. Easiest would be to "initialize" a regular error with the same value returned in its Error() method, then just compare the returned error's Error() value with the expected one.

os.FileInfo mock not matching

in one of my golang Projects i went to mock the os.FileInfo for testcases.
I am not sure if I understand the interface handling of golang correctly.
As far as I know the following piece of code should work, but I get an Compilererror saying that the interface does not match.
I modified this go-doc example a bit in case you want to test it out yourself.
package main
import (
"fmt"
"io/ioutil"
"log"
)
type file interface{
Name() string
}
func readFiles() []file{
files, err := ioutil.ReadDir(".")
if err != nil {
log.Fatal(err)
}
return files
}
func main() {
files := readFiles()
for _, file := range files {
fmt.Println(file.Name())
}
}
Following at the golang doc, the ioutil.ReadDir(".") should Return a slice of os.FileInfo which should be a specialisation of my selfwritten file interface.
Can anyone help me out of this hell of misconceptions and entanglements, please?
Thank you very much guys!
os.FileInfo is interface and it can converted to file. But you try to convert []os.FileInfo to []file. Slice is not interface, and can not be asserted.
To elaborate on the other answer and comments, os.FileInfo does fulfill the file interface you've defined, but a slice of os.FileInfo does not implicitly convert to a slice of your file interface. A simple fix would be to have your readFiles do the conversion explicitly, e.g.
func readFiles() []file{
fileInfos, err := ioutil.ReadDir(".")
if err != nil {
log.Fatal(err)
}
files := make([]file, len(fileInfos))
for i, fi := range fileInfos {
files[i] = fi
}
return files
}

How to call Go function from pongo template

I need to create JSON data using few keys of map and need to incorporate into html generated. I am using pongo2 library and want to write custom filter to achieve the same.
<script> {{ CategoryMapping|MycustomFilter }} </script>
and coded custom filter like below.
func init() {
pongo2.RegisterFilter("superfilter", GetCategoryJsonData)
}
func GetCategoryJsonData(CatAttributeMapping *map[string]interface{}, param *int) (*string, *error) {
.....
}
But I am getting below error.
src/util/TemplateFilters.go:10: cannot use GetCategoryJsonData (type func(*int, *int) (*string, *error)) as type pongo2.FilterFunction in argument to pongo2.RegisterFilter
I am following below documentation - https://godoc.org/github.com/flosch/pongo2#FilterFunction
I am new to go and unable to understand what wrong I am doing here. Please guide me for the same.
The problem is that your filter function does not accept or return the right types to match what pongo2 is requiring. Let's walk through the docs and see what they want.
First, take a look at the godoc for RegisterFilterFunction. It says
func RegisterFilter(name string, fn FilterFunction)
This is in the pongo2 package so you should read this as RegisterFilter is a function that accepts two arguments and returns no values. The first argument name is of the builtin type string and the second argument fn is of the type pongo2.FilterFunction. But what is a pongo2.FilterFunction? Well clicking on it we see further down in the doc
type FilterFunction func(in *Value, param *Value) (out *Value, err *Error)
In Go you can make your own types based on any other types including functions. So what pongo2 has done is to create a named type called FilterFunction that is any func which accepts two arguments (both of type *pongo2.Value) and returns two values (one of type *pongo2.value and one of type *pongo2.Error).
To bring it all together we would do something like this:
package main
import (
"fmt"
"log"
"strings"
"github.com/flosch/pongo2"
)
func init() {
pongo2.RegisterFilter("scream", Scream)
}
// Scream is a silly example of a filter function that upper cases strings
func Scream(in *pongo2.Value, param *pongo2.Value) (out *pongo2.Value, err *pongo2.Error) {
if !in.IsString() {
return nil, &pongo2.Error{
ErrorMsg: "only strings should be sent to the scream filter",
}
}
s := in.String()
s = strings.ToUpper(s)
return pongo2.AsValue(s), nil
}
func main() {
tpl, err := pongo2.FromString("Hello {{ name|scream }}!")
if err != nil {
log.Fatal(err)
}
// Now you can render the template with the given
// pongo2.Context how often you want to.
out, err := tpl.Execute(pongo2.Context{"name": "stack overflow"})
if err != nil {
log.Fatal(err)
}
fmt.Println(out) // Output: Hello STACK OVERFLOW!
}