I am trying to find matching filesystem objects using Go and determine the type of path I received as input. Specifically, I need to perform actions on the object(s) if they match the path provided. Example input for the path could look like this:
/path/to/filename.ext
/path/to/dirname
/path/to/*.txt
I need to know if the path exists, is a file or a directory or a regex so I can process the input accordingly. Here's the solution I've devised so far:
func getPathType(path string) (bool, string, error) {
cpath := filepath.Clean(path)
l, err := filepath.Glob(cpath)
if err != nil {
return false, "", err
}
switch len(l) {
case 0:
return false, "", nil
case 1:
fsstat, fserr := os.Stat(cpath)
if fserr != nil {
return false, "", fserr
}
if fsstat.IsDir() {
return true, "dir", nil
}
return true, "file", nil
default:
return false, "regex", nil
}
}
I realize that the above code would allow a regex that returned a single value to be interpreted as a dir or file and not as a regex. For my purposes, I can let that slide but just curious if anyone has developed a better way of taking a path potentially containing regex as input and determining whether or not the last element is a regex or not.
Test for glob special characters to determine if the path is a glob pattern. Use filepath.Match to check for valid glob pattern syntax.
func getPathType(path string) (bool, string, error) {
cpath := filepath.Clean(path)
// Use Match to check glob syntax.
if _, err := filepath.Match(cpath, ""); err != nil {
return false, "", err
}
// If syntax is good and the path includes special
// glob characters, then it's a glob pattern.
special := `*?[`
if runtime.GOOS != "windows" {
special = `*?[\`
}
if strings.ContainsAny(cpath, special) {
return false, "regex", nil
}
fsstat, err := os.Stat(cpath)
if os.IsNotExist(err) {
return false, "", nil
} else if err != nil {
return false, "", err
}
if fsstat.IsDir() {
return true, "dir", nil
}
return true, "file", nil
}
Related
My program is as follows as a whole.
func main() {
flag.Parse()
if *token == "" {
log.Fatal(Red + "please provide a client token => -token={$token}")
}
tokenSource := oauth2.StaticTokenSource(&oauth2.Token{AccessToken: *token})
oauthClient := oauth2.NewClient(context.TODO(), tokenSource)
client := putio.NewClient(oauthClient)
//paths := make(chan string)
var wg = new(sync.WaitGroup)
for i := 0; i < 50; i++ {
wg.Add(1)
go worker(paths, wg, client)
}
WalkFilePath()
//if err := filepath.Walk(*rootpath, func(path string, info os.FileInfo, err error) error {
// if err != nil {
// return fmt.Errorf("Failed to walk directory: %T %w", err, err)
// }
// if !info.IsDir() {
// paths <- path
// }
// return nil
//}); err != nil {
// panic(fmt.Errorf("failed Walk: %w", err))
//}
close(paths)
wg.Wait()
}
// walks the file path and sends paths to channel
func WalkFilePath() {
if err := filepath.Walk(*rootpath, func(path string, info os.FileInfo, err error) error {
if err != nil {
return fmt.Errorf("Failed to walk directory: %T %w", err, err)
}
if !info.IsDir() {
paths <- path
}
return nil
}); err != nil {
panic(fmt.Errorf("failed Walk: %w", err))
}
}
func worker(paths <-chan string, wg *sync.WaitGroup, client *putio.Client) {
defer wg.Done()
for path := range paths {
f, err := os.Open(path)
if err != nil {
log.Printf(Red + "Failed to open file %v for reading" + Reset, f.Name())
}
upload, err := client.Files.Upload(context.TODO(), f, path, 0)
if err != nil {
log.Printf(Red + "Failed to upload file %v" + Reset, upload.File.Name)
}
log.Printf(Green+ "File %v has been uploaded succesfully" + Reset, upload.File.Name)
}
}
I did write the code. That's the cleanest I can do and I was told to write a unit test for the program. I'm confused. For example, considering the WalkFilePath function. What should I provide and what kind of result I should expect to test the function. Because it contains channel communication meaning goroutines. Is there any way to write unit tests for this program clearly? Or should I change the code structure which is not good in this case for me. Btw, the program runs properly.
Like most things, Go is very opinionated about how to test. Make sure to read https://go.dev/doc/tutorial/add-a-test
For example, considering the WalkFilePath function. What should I provide and what kind of result I should expect to test the function.
The input to WalkFilePath should be paths and a rootpath. Your WalkFilePath doesn't get paths or rootpath from anywhere, so this code wouldn't compile as is (testing will help catch that stuff of course).
A test for WalkFilePath might be done something like this:
Create a filesystem structure in your project under testdata/, a directory expressly set aside for data used for testing. Create subdirectories and files. For an example that might look like:
testdata/
walktest/
dir1/
file1.txt
dir2/
file2.txt
dir3/
file3.txt
Now you can define the expected data you'll be getting out of your channel.
expected_paths := []string{
"testdata/walktest/dir1/file1.txt",
"testdata/walktest/dir2/file2.txt",
"testdata/walktest/dir3/file3.txt"
}
Now you need to change WalkFilePath to take arguments for rootpath and paths.
func WalkFilePath(rootdir string, paths chan<- string) {
Now you're ready to write your test.
func TestWalkFilePath(t *testing.T(
paths := make(chan string)
go WalkFilePath("testdata/walktest")
results := make([]string,0)
for path := range paths {
results = append(results, path)
}
exp, res := strings.Join(expected_paths, ""), strings.Join(results, "")
if exp != res {
t.Errorf("Expected %s got %s", exp, res)
}
}
Because it contains channel communication meaning goroutines.
It's totally normal and valid to use channels and goroutines in unit tests.
I have the following IAM Policy:
{"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":"arn:aws:sts::<account>:assumed-role/custom_role/<role>"},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<account>"}}}]}
but the "AWS" portion can also be an array:
"AWS": [
"arn:aws:sts::<account>:assumed-role/custom_role/<role_1>",
"arn:aws:sts::<account>:assumed-role/custom_role/<role_2>"
]
What I need is a regex that can parse both structures and return the list of arn:aws:sts as a list of strings... how can I accomplish that using regex in Golang?
I tried to use json.Unmarshal but the object structure is different between []string and string
Edit:
I have the following snippet:
re := regexp.MustCompile(`arn:aws:sts::[a-z0-9]*:assumed-role/custom_role/[a-z0-9]-*`)
result := re.FindAll([]byte(arn), 10)
for _, res := range result {
fmt.Println(string(res))
}
>>> `arn:aws:sts::<account_id>:assumed-role/custom_role/`
Using JSON decoder
You can decode the AWS key directly into a custom type implementing the "json.Unmarshaler" interface and decode both inputs correctly.
Demo
type AWSRoles []string
func (r *AWSRoles) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err == nil {
*r = append(*r, s)
return nil
}
var ss []string
if err := json.Unmarshal(b, &ss); err == nil {
*r = ss
return nil
}
return errors.New("cannot unmarshal neither to a string nor a slice of strings")
}
type AWSPolicy struct {
Statement []struct {
Principal struct {
AWSRoles AWSRoles `json:"AWS"`
} `json:"Principal"`
} `json:"Statement"`
}
Here's a test for it
var testsAWSPolicyParsing = []struct {
name string
input []byte
wantRoles []string
}{
{
name: "unique role",
input: []byte(`{"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":"arn:aws:sts::<account>:assumed-role/custom_role/<role>"},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<account>"}}}]}`),
wantRoles: []string{"arn:aws:sts::<account>:assumed-role/custom_role/<role>"},
},
{
name: "multiple roles",
input: []byte(`{"Version":"2012-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":["arn:aws:sts::<account>:assumed-role/custom_role/<role_1>","arn:aws:sts::<account>:assumed-role/custom_role/<role_2>"]},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<account>"}}}]}`),
wantRoles: []string{
"arn:aws:sts::<account>:assumed-role/custom_role/<role_1>",
"arn:aws:sts::<account>:assumed-role/custom_role/<role_2>",
},
},
}
func TestParseAWSPolicy(t *testing.T) {
for _, tc := range testsAWSPolicyParsing {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var p AWSPolicy
err := json.Unmarshal(tc.input, &p)
if err != nil {
t.Fatal("unexpected error parsing AWSRoles policy", err)
}
if l := len(p.Statement); l != 1 {
t.Fatalf("unexpected Statement length. want 1, got %d", l)
}
if got := p.Statement[0].Principal.AWSRoles; !reflect.DeepEqual(got, tc.wantRoles) {
t.Fatalf("roles are not the same, got %v, want %v", got, tc.wantRoles)
}
})
}
}
Using a Regex
If you still want to use a regex, this one would parse it as long as:
AWS account has only numbers [0-9]
the custom role name has only alphanumeric characters and underscores
var awsRolesRegex = regexp.MustCompile("arn:aws:sts::[a-z0-9]+:assumed-role/custom_role/[a-zA-Z0-9_]+")
Demo
I have the following Golang code:
func getConfigFile() string {
var configFile string
flag.StringVar(&configFile, "config", "", "File containing configuration")
flag.Parse()
return configFile
}
This function is used elsewhere in my code, and I'd like to unit test what happens here when the user provides different values for the config argument (the config file name is used else where).
Is there a way to tell the flag package to return different values for the config argument while under test?
I have found that for testing custom flags is better to create a custom flag set, in that way I can fully test the flags, including the -h option without exiting the tests. hope the attached code could give you and idea of how you could implement test on your code:
package main
import (
"flag"
"fmt"
"os"
"reflect"
"testing"
)
// Test Helper
func expect(t *testing.T, a interface{}, b interface{}) {
if a != b {
t.Errorf("Expected: %v (type %v) Got: %v (type %v)", a, reflect.TypeOf(a), b, reflect.TypeOf(b))
}
}
type Flags struct {
ConfigFile string
}
func (self *Flags) Parse(fs *flag.FlagSet) (*Flags, error) {
fs.StringVar(&self.ConfigFile, "config", "", "File containing configuration")
err := fs.Parse(os.Args[1:])
if err != nil {
return nil, err
}
return self, nil
}
func main() {
fs := flag.NewFlagSet("test", flag.ContinueOnError)
parser := Flags{}
flags, err := parser.Parse(fs)
if err != nil {
panic(err)
}
fmt.Println(flags)
}
func TestFlags(t *testing.T) {
oldArgs := os.Args
defer func() { os.Args = oldArgs }()
var flagTest = []struct {
flag []string
name string
expected interface{}
}{
{[]string{"cmd", "-config", "config.yaml"}, "ConfigFile", "config.yaml"},
{[]string{"cmd", "-config", "config.json"}, "ConfigFile", "config.json"},
{[]string{"cmd", "-v"}, "Version", true},
}
for _, f := range flagTest {
os.Args = f.flag
p := &Flags{}
fs := flag.NewFlagSet("test", flag.ContinueOnError)
flags, err := p.Parse(fs)
if err != nil {
t.Error(err)
}
refValue := reflect.ValueOf(flags).Elem().FieldByName(f.name)
switch refValue.Kind() {
case reflect.Bool:
expect(t, f.expected, refValue.Bool())
case reflect.String:
expect(t, f.expected, refValue.String())
}
}
}
I put it also here: https://play.golang.org/p/h1nok1UMLA hope it can give you an idea.
If you change it like the code below, go test will fail but go test -config testconfig will pass. Not that we don't need to call flag.Parse() in the init() since it is be called by the testing package (as Rob Pike mentions in https://groups.google.com/d/msg/golang-nuts/uSFM8jG7yn4/PIQfEWOZx4EJ).
package main
import (
"flag"
"testing"
)
var configFile = flag.String("config", "", "File containing configuration")
func getConfigFile() string {
return *configFile
}
func TestConfig(t *testing.T) {
want := "testconfig"
if s := getConfigFile(); s != want {
t.Errorf("Got %s, want %s", s, want)
}
}
Test runs:
$ go test
--- FAIL: TestConfig (0.00s)
flag_test.go:17: Got , want testconfig
FAIL
exit status 1
FAIL github.com/dmitris/soflagtest 0.013s
$ go test -config testconfig
PASS
ok github.com/dmitris/soflagtest 0.012s
You can also use
var configFile string declaration and an init() function to assign the flag value to the variable:
func init() {
flag.StringVar(&configFile, "config", "", "File containing configuration")
}
(then no pointer dereferencing in getConfigFile since configFile is a string)
I am using Golang regex package, I want to use regex ReplaceAllStringFunc with argument, not only with the source string.
For example, I want to update this text
"<img src=\"/m/1.jpg\" /> <img src=\"/m/2.jpg\" /> <img src=\"/m/3.jpg\" />"
To (change "m" to "a" or anything else):
"<img src=\"/a/1.jpg\" /> <img src=\"/a/2.jpg\" /> <img src=\"/a/3.jpg\" />"
I would like to have something like:
func UpdateText(text string) string {
re, _ := regexp.Compile(`<img.*?src=\"(.*?)\"`)
text = re.ReplaceAllStringFunc(text, updateImgSrc)
return text
}
// update "/m/1.jpg" to "/a/1.jpg"
func updateImgSrc(imgSrcText, prefix string) string {
// replace "m" by prefix
return "<img src=\"" + newImgSrc + "\""
}
I checked the doc, ReplaceAllStringFunc doesn't support argument, but what would be the best way to achieve my goal?
More generally, I would like to find all occurrences of one pattern then update each with a new string which is composed by source string + a new parameter, could anyone give any idea?
I agree with the comments, you probably don't want to parse HTML with regular expressions (bad things will happen).
However, let's pretend it's not HTML, and you want to only replace submatches. You could do this
func UpdateText(input string) (string, error) {
re, err := regexp.Compile(`img.*?src=\"(.*?)\.(.*?)\"`)
if err != nil {
return "", err
}
indexes := re.FindAllStringSubmatchIndex(input, -1)
output := input
for _, match := range indexes {
imgStart := match[2]
imgEnd := match[3]
newImgName := strings.Replace(input[imgStart:imgEnd], "m", "a", -1)
output = output[:imgStart] + newImgName + input[imgEnd:]
}
return output, nil
}
see on playground
(note that I've slightly changed your regular expression to match the file extension separately)
thanks for kostix's advice, here is my solution using html parser.
func UpdateAllResourcePath(text, prefix string) (string, error) {
doc, err := goquery.NewDocumentFromReader(strings.NewReader(text))
if err != nil {
return "", err
}
sel := doc.Find("img")
length := len(sel.Nodes)
for index := 0; index < length; index++ {
imgSrc, ok := sel.Eq(index).Attr("src")
if !ok {
continue
}
newImgSrc, err := UpdateResourcePath(imgSrc, prefix) // change the imgsrc here
if err != nil {
return "", err
}
sel.Eq(index).SetAttr("src", newImgSrc)
}
newtext, err := doc.Find("body").Html()
if err != nil {
return "", err
}
return newtext, nil
}
I would like to update an item under certain conditions and then I would like to know whether the item was updated when UpdateItem returns.
The documentation seems contradictory to me.
On this page: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html in the "Conditional Update" example it says "All of the item's attributes, as they appear after the update, are returned in the response."
On this page: https://godoc.org/github.com/aws/aws-sdk-go/service/dynamodb#UpdateItemOutput it says that Attributes is "A map of attribute values as they appeared before the UpdateItem operation"
I don't really want either of these. What I want is a bool that says whether or not there was an update.
This is where my brain is at now:
out, err := db.DynamoDB.UpdateItem(&dynamodb.UpdateItemInput{
TableName: tableName,
Key: map[string]*dynamodb.AttributeValue{
"KeyName": {S: aws.String(keyname)},
},
ExpressionAttributeNames: map[string]*string{
"#lock": aws.String("Lock"),
},
ExpressionAttributeValues: map[string]*string{
":now": aws.String(compfmt(time.Now())),
":promise": aws.String(compfmt(time.Now().Add(30 * time.Second))),
},
ConditionExpression: aws.String("attribute_not_exist(#lock) OR :now > #lock"),
UpdateExpression: aws.String("SET #lock = :promise"),
})
One way to do this is to check the Code on the awserr
import "github.com/aws/aws-sdk-go/aws/awserr"
func Lock()(bool, error) {
//Create value v
_, err := db.DynamoDB.UpdateItem(v)
if err != nil {
if ae, ok := err.(awserr.RequestFailure); ok && ae.Code() == "ConditionalCheckFailedException" {
return false, nil
}
return false, err
}
return true, nil
}
There are now constants to compare the errors rather than using the hardcoded string as in other answers:
result, err := svc.UpdateItem(input)
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case dynamodb.ErrCodeConditionalCheckFailedException:
fmt.Println(dynamodb.ErrCodeConditionalCheckFailedException, aerr.Error())
default:
fmt.Println(aerr.Error())
}
}
}
Turns out what I wanted to do was check the error to see if it contained the string ConditionalCheckFailedException.
func Lock() (bool, error) {
...
_, err := db.DynamoDB.UpdateItem(v)
if err != nil {
if strings.Contains(err.Error(), "ConditionalCheckFailedException") {
return false, nil
}
return false, err
}
return true, nil
}