Matching serialized strings [duplicate] - regex

This question already has answers here:
Regexp won't match
(2 answers)
Closed 4 years ago.
I am receiving this string "\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80J\x13\x80SQ\x80L\xe0\x80#\x92\x80L?\x80H\xe0" from a function (which runs a GET command on a redis bitmap and gives me the serialized string)
But due to escape sequences I am having trouble matching this kind of pattern. Can some please tell me the regex sequence that will match this kind of string?

First, I'd try to find out how to get that same data from Redis in its direct, binary, form (as []byte); the rest would be way more simple then.
But to deal with this stuff in its present form,
I would first normalize the input string—replacing all those
backslash-escaped hex-encoded characters with their equivalent bytes.
This would allow easily searchig for the exact values of these
bytes—possibly using backslash-escaped hex-encoded characters
in the patterns:
package main
import (
"fmt"
"strconv"
"strings"
)
func main() {
s := "\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80J\x13\x80SQ\x80L\xe0\x80#\x92\x80L?\x80H\xe0"
s, err := strconv.Unquote(`"` + s + `"`)
if err != nil {
panic(err)
}
fmt.Println(strings.Index(s, "\x80SQ\x80L"))
}
Playground link.

An introduction to Redis data types and abstractions
Bitmaps
Bitmaps are not an actual data type, but a set of bit-oriented
operations defined on the String type.
Regular expressions are not the best solution. Write a simple Go function to do the conversion. For example,
package main
import (
"fmt"
"strconv"
)
func redisBits(s string) (string, error) {
s, err := strconv.Unquote(`"` + s + `"`)
if err != nil {
return "", err
}
return s, nil
}
func main() {
s := "\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80J\x13\x80SQ\x80L\xe0\x80#\x92\x80L?\x80H\xe0"
fmt.Printf("%x\n", s)
b, err := redisBits(s)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("%x\n", b)
}
Playground: https://play.golang.org/p/rbE9iG3tOTx
Output:
0100000000000000000000804a13805351804ce0804092804c3f8048e0
0100000000000000000000804a13805351804ce0804092804c3f8048e0

Related

Golang: How to test functions that use time.Now()? [duplicate]

This question already has answers here:
Is there an easy way to stub out time.Now() globally during test?
(10 answers)
Golang testing programs that involves time
(1 answer)
Closed 21 days ago.
I have a function that is calculating the number of days since a particular timestamp, where the timestamp is coming from an external API (parsed as string in json return from API)
I have been following this article on how to test functions that use time.Now():
https://medium.com/go-for-punks/how-to-test-functions-that-use-time-now-ea4f2453d430
My function looks like this:
type funcTimeType func() time.Time // per suggested in article
func ageOfReportDays(dateString string, funcTime funcTimeType) {
// date string will look like this:
//"2022-08-30 09:05:27.567995"
parseLayout := "2006-01-02 15:04:05.000000"
t, err := time.Parse(parseLayout, dateString)
if err != nil {
fmt.Printf("Error parsing datetime value %v: %w", timeStr, err)
}
days := int(time.Since(t).Abs().Hours() / 24)
//fmt.Println(days)
return days, nil
}
As you can see, I am not using the funcTime funcTimeType in my actual function, as indicated in the article, because I cannot figure out how my function would be implemented with that.
The unit test I would hope to run would be something like this:
func Test_ageOfReportDays(t *testing.T) {
t.Run("timestamp age in days test", func(t *testing.T) {
parseLayout := "2006-01-02 15:04:05.000000"
dateString := "2022-08-30 09:05:27.567995" // example of recent timestamp
mockNow := func() time.Time {
fakeTime, _ := time.Parse(parseLayout, "2023-01-20 09:00:00.000000")
return fakeTime
}
// now I want to use "fakeTime" to spoof "time.Now()" so I can test my function
got: ageOfReportDays(dateString, mockNow)
expected: 152
if got != expected {
t.Errorf("expected '%d' but got '%d'", expected, got)
}
}
Obviously the logic is not quite with my code vs article author's code.
Is there a good way for me to write a unit test for this funcition, based on how the article is suggesting to mock time.Now()?
You are pretty close. Changing time.Since(t) to funcTime().Sub(t) would probably get you passed the finish line.
From time package docs:
time.Since returns the time elapsed since t. It is shorthand for time.Now().Sub(t).
Example function:
import (
"fmt"
"time"
)
const parseLayout = "2006-01-02 15:04:05.000000"
type funcTimeType func() time.Time // per suggested in article
func ageOfReportDays(dateString string, funcTime funcTimeType) (int, error) {
t, err := time.Parse(parseLayout, dateString)
if err != nil {
return 0, fmt.Errorf("parsing datetime value %v: %w", dateString, err)
}
days := int(funcTime().Sub(t).Hours() / 24)
//fmt.Println(days)
return days, nil
}
And a test:
import (
"testing"
"time"
)
func Test_ageOfReportDays(t *testing.T) {
t.Run("timestamp age in days test", func(t *testing.T) {
dateString := "2022-08-30 09:05:27.567995" // example of recent timestamp
mockNow := func() time.Time {
fakeTime, _ := time.Parse(parseLayout, "2023-01-20 09:00:00.000000")
return fakeTime
}
// now I want to use "fakeTime" to spoof "time.Now()" so I can test my function
got, _ := ageOfReportDays(dateString, mockNow)
expected := 142
if got != expected {
t.Errorf("expected '%d' but got '%d'", expected, got)
}
})
}

How to extract substrings [duplicate]

This question already has answers here:
How to extract a floating number from a string [duplicate]
(7 answers)
Closed 2 years ago.
I wrote in Go the following code to extract two values ​​inside the string.
I used two regexp to seek the numbers (float64).
The first result is the correct, only de number. But the second is wrong.
This is the code:
package main
import (
"fmt"
"regexp"
)
func main() {
// RegExp utiliza la sintaxis RE2
pat1 := regexp.MustCompile(`[^m2!3d][\d\.-]+`)
s1 := pat1.FindString(`Torre+Eiffel!8m2!3d-48.8583701!4d-2.2944813!3m4!1s0x47e66e2964e34e2d:0x8ddca9ee380ef7e0!8m2!3d-48.8583701!4d-2.2944813`)
pat2 := regexp.MustCompile(`[^!4d][\d\.-]+`)
s2 := pat2.FindString(`Torre+Eiffel!8m2!3d-48.8583701!4d-2.2944813!3m4!1s0x47e66e2964e34e2d:0x8ddca9ee380ef7e0!8m2!3d-48.8583701!4d-2.2944813`)
fmt.Println(s1) // Print -> -48.8583701
fmt.Println(s2) // Print -> m2 (The correct answer is "-2.2944813")
}
Here I modify the syntax
pat2 := regexp.MustCompile(!4d[\d\.-]+)
and I get the following answer:
!4d-2.2944813
but it's not what I'm expecting.
It seems like you are only interessed in the latitude and longitute of an attraction and not really in the regex.
Maybe you just use something like this:
package main
import (
"fmt"
"strconv"
"strings"
)
var replacer = strings.NewReplacer("3d-", "", "4d-", "")
func main() {
var str = `Torre+Eiffel!8m2!3d-48.8583701!4d-2.2944813!3m4!1s0x47e66e2964e34e2d:0x8ddca9ee380ef7e0!8m2!3d-48.8583701!4d-2.2944813`
fmt.Println(getLatLong(str))
}
func getLatLong(str string) (float64, float64, error) {
parts := strings.Split(str, "!")
if latFloat, err := strconv.ParseFloat(replacer.Replace(parts[2]), 64); err != nil {
return 0, 0, err
} else if lngFloat, err := strconv.ParseFloat(replacer.Replace(parts[3]), 64); err != nil {
return 0, 0, err
} else {
return latFloat, lngFloat, nil
}
}
https://play.golang.org/p/UOIwGbl6nrb
You where almost there. Try (?m)(?:3d|4d)-([\d\.-]+)(?:!|$)
https://regex101.com/r/8KgirB/1
All you need is a matching group around the [\d\.-]+ part. With this group you are able to access it directly
package main
import (
"fmt"
"regexp"
)
func main() {
var re = regexp.MustCompile(`(?m)(?:3d|4d)-([\d\.-]+)!`)
var str = `Torre+Eiffel!8m2!3d-48.8583701!4d-2.2944813!3m4!1s0x47e66e2964e34e2d:0x8ddca9ee380ef7e0!8m2!3d-48.8583701!4d-2.2944813`
for _, match := range re.FindAllStringSubmatch(str, -1) {
fmt.Println(match[1])
}
}

Learning to write unit tests

I am trying to learn how to write tests for my code in order to write better code, but I just seem to have the hardest time figuring out how to actually test some code I have written. I have read so many tutorials, most of which seem to only cover functions that add two numbers or mock some database or server.
I have a simple function I wrote below that takes a text template and a CSV file as input and executes the template using the values of the CSV. I have "tested" the code by trial and error, passing files, and printing values, but I would like to learn how to write proper tests for it. I feel that learning to test my own code will help me understand and learn faster and better. Any help is appreciated.
// generateCmds generates configuration commands from a text template using
// the values from a CSV file. Multiple commands in the text template must
// be delimited by a semicolon. The first row of the CSV file is assumed to
// be the header row and the header values are used for key access in the
// text template.
func generateCmds(cmdTmpl string, filename string) ([]string, error) {
t, err := template.New("cmds").Parse(cmdTmpl)
if err != nil {
return nil, fmt.Errorf("parsing template: %v", err)
}
f, err := os.Open(filename)
if err != nil {
return nil, fmt.Errorf("reading file: %v", err)
}
defer f.Close()
records, err := csv.NewReader(f).ReadAll()
if err != nil {
return nil, fmt.Errorf("reading records: %v", err)
}
if len(records) == 0 {
return nil, errors.New("no records to process")
}
var (
b bytes.Buffer
cmds []string
keys = records[0]
vals = make(map[string]string, len(keys))
)
for _, rec := range records[1:] {
for k, v := range rec {
vals[keys[k]] = v
}
if err := t.Execute(&b, vals); err != nil {
return nil, fmt.Errorf("executing template: %v", err)
}
for _, s := range strings.Split(b.String(), ";") {
if cmd := strings.TrimSpace(s); cmd != "" {
cmds = append(cmds, cmd)
}
}
b.Reset()
}
return cmds, nil
}
Edit: Thanks for all the suggestions so far! My question was flagged as being too broad, so I have some specific questions regarding my example.
Would a test table be useful in a function like this? And, if so, would the test struct need to include the returned cmds string slice and the value of err? For example:
type tmplTest struct {
name string // test name
tmpl string // the text template
filename string // CSV file with template values
expected []string // expected configuration commands
err error // expected error
}
How do you handle errors that are supposed to be returned for specific test cases? For example, os.Open() returns an error of type *PathError if an error is encountered. How do I initialize a *PathError that is equivalent to the one returned by os.Open()? Same idea for template.Parse(), template.Execute(), etc.
Edit 2: Below is a test function I came up with. My two question from the first edit still stand.
package cmd
import (
"testing"
"strings"
"path/filepath"
)
type tmplTest struct {
name string // test name
tmpl string // text template to execute
filename string // CSV containing template text values
cmds []string // expected configuration commands
}
var tests = []tmplTest{
{"empty_error", ``, "", nil},
{"file_error", ``, "fake_file.csv", nil},
{"file_empty_error", ``, "empty.csv", nil},
{"file_fmt_error", ``, "fmt_err.csv", nil},
{"template_fmt_error", `{{ }{{`, "test_values.csv", nil},
{"template_key_error", `{{.InvalidKey}}`, "test_values.csv", nil},
}
func TestGenerateCmds(t *testing.T) {
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
cmds, err := generateCmds(tc.tmpl, filepath.Join("testdata", tc.filename))
if err != nil {
// Unexpected error. Fail the test.
if !strings.Contains(tc.name, "error") {
t.Fatal(err)
}
// TODO: Otherwise, check that the function failed at the expected point.
}
if tc.cmds == nil && cmds != nil {
t.Errorf("expected no commands; got %d", len(cmds))
}
if len(cmds) != len(tc.cmds) {
t.Errorf("expected %d commands; got %d", len(tc.cmds), len(cmds))
}
for i := range cmds {
if cmds[i] != tc.cmds[i] {
t.Errorf("expected %q; got %q", tc.cmds[i], cmds[i])
}
}
})
}
}
You basically need to have some sample files with the contents you want to test, then in your test code you can call the generateCmds function passing in the template string and the files to then verify that the results are what you expect.
It is not so much different as the examples you probably saw for simpler cases.
You can place the files under a testdata folder inside the same package (testdata is a special name that the Go tools will ignore during build).
Then you can do something like:
func TestCSVProcessing(t *testing.T) {
templateStr := `<your template here>`
testFile := "testdata/yourtestfile.csv"
result, err := generateCmds(templateStr, testFile)
if err != nil {
// fail the test here, unless you expected an error with this file
}
// compare the "result" contents with what you expected
// failing the test if it does not match
}
EDIT
About the specific questions you added later:
Would a test table be useful in a function like this? And, if so, would the test struct need to include the returned cmds string slice and the value of err?
Yes, it'd make sense to include both the expected strings to be returned as well as the expected error (if any).
How do you handle errors that are supposed to be returned for specific test cases? For example, os.Open() returns an error of type *PathError if an error is encountered. How do I initialize a *PathError that is equivalent to the one returned by os.Open()?
I don't think you'll be able to "initialize" an equivalent error for each case. Sometimes the libraries might use internal types for their errors making this impossible. Easiest would be to "initialize" a regular error with the same value returned in its Error() method, then just compare the returned error's Error() value with the expected one.

Golang regex replace does nothing

I want to replace any non-alphanumeric character sequences with a dash. A snippet of what I wrote is below. However it does not work and I'm completely clueless why. Could anyone explain me why the snippet behaves not like I expect it to and what would be the correct way to accomplish this?
package main
import (
"fmt"
"regexp"
"strings"
)
func main() {
reg, _ := regexp.Compile("/[^A-Za-z0-9]+/")
safe := reg.ReplaceAllString("a*-+fe5v9034,j*.AE6", "-")
safe = strings.ToLower(strings.Trim(safe, "-"))
fmt.Println(safe) // Output: a*-+fe5v9034,j*.ae6
}
The forward slashes are not matched by your string.
package main
import (
"fmt"
"log"
"regexp"
"strings"
)
func main() {
reg, err := regexp.Compile("[^A-Za-z0-9]+")
if err != nil {
log.Fatal(err)
}
safe := reg.ReplaceAllString("a*-+fe5v9034,j*.AE6", "-")
safe = strings.ToLower(strings.Trim(safe, "-"))
fmt.Println(safe) // Output: a*-+fe5v9034,j*.ae6
}
(Also here)
Output
a-fe5v9034-j-ae6

Golang Regular Expressions - what am I doing incorrectly?

As a personal project, trying to learn Go(lang) by applying it to something, I am writing an EMCAScript/JavaScript "compiler"; all it will (initially) do is allow you to include other .js files.
Functionality aside, I am pulling my hair out trying to figure out the regexp package. Here is the snippet that does not seem to be doing what I want it to:
// Note: "lines" is an array of strings.
var includeRegex, _ = regexp.Compile("^[ \t]*include[(]{1}\"([^\"]+)\"[)]{1};")
for _, line := range lines {
var isInclude = includeRegex.Match([]byte(line))
if isInclude {
var includeFile = includeRegex.FindString(line)
fmt.Println("INCLUDE", includeFile)
} else {
// ...
}
I have already stumbled across Go's subset of regular expressions, hence why the regex does not read as ^\s*include\("([^"]+)"\);. I have already tested both the preferred, and the Go-style regex, in RegexPal, and both definitely work. The match just never seems to occurr; what am I doing wrong?
For what it's worth, the include() statement I am trying to parse looks like so:
include("somefile.js");
EDIT: For what it's worth, I am keeping the code here.
This seems to work with the latest weekly
package main
import (
"fmt"
"log"
"regexp"
"strings"
)
func main() {
includeRegex, err := regexp.Compile(`^\s*include\("(\\\"|[^"])+"\);`)
if err != nil {
log.Fatal(err)
}
for _, line := range strings.Split(`
foo
include "abc.def"
include("file.js");
include "me\"to\""
include("please\"!\"");
nothing here
`, "\n") {
if includeRegex.Match([]byte(line)) {
includeFile := includeRegex.FindString(line)
fmt.Println("INCLUDE", includeFile)
} else {
fmt.Printf("no match for \"%s\"\n", line)
}
}
}
Output:
$ go build && ./tmp
no match for ""
no match for "foo"
no match for "include "abc.def""
INCLUDE include("file.js");
no match for " include "me\"to\"""
INCLUDE include("please\"!\"");
no match for " nothing here "
no match for ""
$
Try putting the following line at the start of your program:
println(runtime.Version())
It should print weekly.2012-03-13 or something close to that date.