Extract subdomain from URL using regexp in Golang - regex

In the code sample below, I use regex to extract the subdomain name from a given URL. This sample works, but I don't think I've done it correctly at the point where I compile the regex, mainly where I insert the 'virtualHost' variable. Any suggestions?
package main
import (
"fmt"
"regexp"
)
var (
virtualHost string
domainRegex *regexp.Regexp
)
func extractSubdomain(host string) string {
matches := domainRegex.FindStringSubmatch(host)
if matches != nil && len(matches) > 1 {
return matches[1]
}
return ""
}
func init() {
// virtualHost = os.GetEnv("VIRTUAL_HOST")
virtualHost = "login.localhost:3000"
domainRegex = regexp.MustCompile(`^(?:https?://)?([-a-z0-9]+)(?:\.` + virtualHost + `)*$`)
}
func main() {
// host := req.host
host := "http://acme.login.localhost:3000"
if result := extractSubdomain(host); result != "" {
fmt.Printf("Subdomain detected: %s\n", result)
return
}
fmt.Println("No subdomain detected")
}

The url package has a function parse that allows you to parse an URL. The parsed URL instance has a method Hostname which will return you the hostname.
package main
import (
"fmt"
"log"
"net/url"
)
func main() {
u, err := url.Parse("http://login.localhost:3000")
if err != nil {
log.Fatal(err)
}
fmt.Println(u.Hostname())
}
Output:
login.localhost
See https://play.golang.com/p/3R1TPyk8qck
Update:
My previous answer only dealt with parsing the host name. Since then I have been using the following library to parse the domain suffix from the host name. Once you have that, it is simple to strip the domain and leave only the subdomain prefix.
https://pkg.go.dev/golang.org/x/net/publicsuffix
I have found that it can be a bit tricky to exactly identify the difference between subdomain and host, without a little help first from this package that can identify common suffixes. For instance, internally we may have a domain coming from a kubernetes ingress:
foo.bar.host.kube.domain.com.au
The host is "host" and the subdomain is "foo.bar". Even with the help of the publicsuffix library it won't know that "kube" is part of the internal domain components. So you have to add some more of your own hinting to match.

This is what I've used
func getSubdomain(r *http.Request) string {
//The Host that the user queried.
host := r.URL.Host
host = strings.TrimSpace(host)
//Figure out if a subdomain exists in the host given.
hostParts := strings.Split(host, ".")
fmt.Println("host parts",hostParts)
lengthOfHostParts := len(hostParts)
// scenarios
// A. site.com -> length : 2
// B. www.site.com -> length : 3
// C. www.hello.site.com -> length : 4
if lengthOfHostParts == 4 {
return strings.Join([]string{hostParts[1]},"") // scenario C
}
if lengthOfHostParts == 3 { // scenario B with a check
subdomain := strings.Join([]string{hostParts[0]},"")
if subdomain == "www" {
return ""
} else {
return subdomain
}
}
return "" // scenario A
}

Related

How should I test functions that deal with setting a large number of environment configs/OS arguments?

I've written a Go application, and all of the packages have full test coverage. I'm in the process of writing my main package - which will handle all of the initial setup for the application in the main() function - this function currently reads in 14 environment variables and then sets the relevant variable in the application. A simple overview of the code is:
func main() {
myStruct1 := privatePackage.myStructType{}
myStruct2 := publicPackage.otherStructType{}
if config1 := os.Getenv("CONFIG_FOO"); config1 != "" {
myStruct1.attribute1 = config1
}
// ....
if config14 := os.Getenv("CONFIG_BAR"); config14 != "" {
myStruct2.attribute5 = config14
}
}
When I test unit env variables/OS args, I typically just set the env variable directly in the test function - so something like:
func TestMyArgument(t *testing.T) {
os.Setenv("CONFIG_BAZ", "apple")
//Invoke function that depends on CONFIG_BAZ
//Assert that expected outcome occurred
}
I pretty much always use table-driven tests, so the above snippet is a simplified example.
The issue is that my main() function takes in 14 (and growing) env variables, and whilst some env variables are essentially enums (so there's a small number of valid options - for example there's a small number of database drivers to choose from), other env variables have virtually unlimited potential values. So how can I effectively cover all of the (or enough of the) permutations of potential configs?
EDIT: When this application is deployed, it's going into a K8s cluster. Some of these variables are secrets that will be pulled in from secure store. Using a JSON file isn't viable because some of the values need to be encrypted/changed easily.
Also, using a JSON file would require me to store this file and share it between hundreds/thousands of running pods - this storage would then act as a point of failure.
To clarify, this question isn't about env vars VS config files; this question is about the best way to approach testing when there's a significant number of configurable variables - with each variables having a vast number of potential values - resulting in thousands of possible configuration permutations. How do I guarantee sufficient test coverage in such a scenario?
#Steven Penny is right: uses json
and use reflect can make the code more simple:
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
"strconv"
)
type MyStructType struct {
Attribute1 string `json:"CONFIG_FOO"`
Attribute2 string `json:"CONFIG_BAZ"`
Attribute3 int `json:"CONFIG_BAR"`
}
func NewMyStructTypeFormEnv() *MyStructType {
myStructType := MyStructType{}
ReflectMyStructType(&myStructType)
fmt.Println("myStructType is now", myStructType)
return &myStructType
}
func NewMyStructTypeFormJson() *MyStructType {
myStructType := MyStructType{}
f, e := os.Open("file.json")
if e != nil {
panic(e)
}
defer f.Close()
json.NewDecoder(f).Decode(&myStructType)
fmt.Println("myStructType is now", myStructType)
return &myStructType
}
func ReflectMyStructType(ptr interface{}){
v := reflect.ValueOf(ptr).Elem()
fmt.Printf("%v\n", v.Type())
for i := 0; i < v.NumField(); i++ {
env_str := v.Type().Field(i).Tag.Get("json")
if(env_str == ""){continue}
if config := os.Getenv(env_str); config != "" {
if v.Field(i).Kind() == reflect.String{
v.Field(i).SetString(config)
}else if v.Field(i).Kind() == reflect.Int{
iConfig,_ := strconv.Atoi(config)
v.Field(i).SetInt(int64(iConfig))
}
}
}
}
func main() {
NewMyStructTypeFormJson()
os.Setenv("CONFIG_FOO", "apple")
os.Setenv("CONFIG_BAZ", "apple")
os.Setenv("CONFIG_BAR", "1")
NewMyStructTypeFormEnv()
}
Beyond one or two, I don't think using environment variables is the right approach, unless it's required (calling something with os/exec). Instead, would be better to read from a config file. Here is an example with JSON:
{
"CONFIG_BAR": "east",
"CONFIG_BAZ": "south",
"CONFIG_FOO": "north"
}
package main
import (
"encoding/json"
"fmt"
"os"
)
func main() {
f, e := os.Open("file.json")
if e != nil {
panic(e)
}
defer f.Close()
var s struct { CONFIG_BAR, CONFIG_BAZ, CONFIG_FOO string }
json.NewDecoder(f).Decode(&s)
// {CONFIG_BAR:east CONFIG_BAZ:south CONFIG_FOO:north}
fmt.Printf("%+v\n", s)
}
TOML would be a good choice as well.
https://golang.org/pkg/encoding/json
https://pkg.go.dev/github.com/pelletier/go-toml

Prevent escaping forward slashes in templates

I'm working on converting a pet project of mine from Python to Go just to help me get a bit familiar with the language. An issue I am currently facing is that it's escaping my forward slashes. So it will receive a string like:
/location/to/something
and it then becomes
%2flocation%2fto%2fsomething
Now, it's only doing this when it's in a link (from what I've been reading this escaping is contextual) so this is what the line in the HTML template looks like:
<tr><td>{{.FileName}}</td></tr>
If possible, how can I prevent this in either the template or the code itself?
This is what my templating function looks like (yes, I know it's hackish)
func renderTemplate(w http.ResponseWriter, tmpl string) {
t, err := template.ParseFiles(templates_dir+"base.html", templates_dir+tmpl)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if tmpl == "view.html" {
err = t.Execute(w, FileList)
} else {
err = t.Execute(w, nil)
}
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
As the value of .FullFilePath, pass a value of type template.URL instead of string, which will tell the html/template package not to escape it.
For example:
func main() {
t := template.Must(template.New("").Parse(templ))
m := map[string]interface{}{
"FileName": "something.txt",
"FileFullPath": template.URL("/location/to/something"),
}
if err := t.Execute(os.Stdout, m); err != nil {
panic(err)
}
}
const templ = `<tr><td>{{.FileName}}</td></tr>`
Output (try it on the Go Playground):
<tr><td>something.txt</td></tr>
Note that even though forward slashes / are allowed in URLs, the reason why the template package still encodes them is because it analyses the URL and sees that the value you want to include is the value of a URL parameter (file=XXX), and so it also encodes the slashes (so that everything you pass in will be part of the value of the file URL parameter).
If you plan to acquire this file path at the server side from URL parameters, then what the template package does is the correct and proper way.
But know that by doing this, you'll lose the safety that prevents code injection into URLs. If you're the one providing the values and you know they are safe, there is no problem. But if the data comes from a user input for example, never do this.
Also note that if you pass the whole URL (and not just a part of it), it will work without using template.URL (try this variant on the Go Playground):
func main() {
t := template.Must(template.New("").Parse(templ))
m := map[string]interface{}{
"FileName": "something.txt",
"FileURL": "/file?file=/location/to/something",
}
if err := t.Execute(os.Stdout, m); err != nil {
panic(err)
}
}
const templ = `<tr><td>{{.FileName}}</td></tr>`
Also note that the recommended way in my opinion would be to include the file path as part of the URL path and not as the value of a parameter, so instead you should create urls like this:
/file/location/to/something
Map your handler (which serves the file content, see this answer as an example) to the /file/ pattern, and when it is matched and your handler is called, cut off the /file/ prefix from the path r.URL.Path, and the rest will be the full file path. If you choose this, you also won't need the template.URL conversion (because the value you include is not a value of a URL parameter anymore):
func main() {
t := template.Must(template.New("").Parse(templ))
m := map[string]interface{}{
"FileName": "something.txt",
"FileFullPath": "/location/to/something",
}
if err := t.Execute(os.Stdout, m); err != nil {
panic(err)
}
}
const templ = `<tr><td>{{.FileName}}</td></tr>`
Try this on the Go Playground.
Also very important: never parse templates in your handler functions! For details see:
It takes too much time when using "template" package to generate a dynamic web page to client in golang
OK, So the solution I've found (and please post if there's a better one) is based on an answer here.
I changed the struct I was using from:
type File struct {
FullFilePath string
FileName string
}
To this:
type File struct {
FullFilePath template.HTML
FileName string
}
And moved the html into the FullFilePath name, and then placed that in template.HTML so each FullFilePath name I was generating was done like so:
file := File{template.HTML("<a href=\"/file?file=" + path + "\"</a>"), f.Name()}
And my template file line was changed to this:
<tr><td>{{.FullFilePath}}{{.FileName}}</td></tr>

Cannot generate Amazon product API signature using Golang

Help. I can't get the right signature using the test parameters provided by Amazon and Go.
My signature hash function is as follows. I use SHA-256 and base64 encoding as per Amazon documentation.
func HashSignature(str string, secret string) string {
mac := hmac.New(sha256.New, []byte(secret))
_, err := mac.Write([]byte(str))
if err != nil { return "" }
hash := base64.StdEncoding.EncodeToString(mac.Sum(nil))
hash = url.QueryEscape(hash)
return hash
}
My signature test function is as follows. I use the canonical string below in Ruby code and it generates the correct expected signature. So the problem seems to be with the output of my HashSignature() function, but I don't see what I'm doing wrong there.
func TestAmazonSignature(t *testing.T) {
/* here is the canonical string from Amazon documentation which should yield the expected signature below
GET
webservices.amazon.com
/onca/xml
AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&AssociateTag=mytag-20&ItemId=0679722769&Operation=ItemLookup&ResponseGroup=Images%2CItemAttributes%2COffers%2CReviews&Service=AWSECommerceService&Timestamp=2014-08-18T12%3A00%3A00Z&Version=2013-08-01
*/
SECRET_KEY := "1234567890"
CANONICAL_STR := "GET\nwebservices.amazon.com\n/onca/xml\nAWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&AssociateTag=mytag-20&ItemId=0679722769&Operation=ItemLookup&ResponseGroup=Images%2CItemAttributes%2COffers%2CReviews&Service=AWSECommerceService&Timestamp=2014-08-18T12%3A00%3A00Z&Version=2013-08-01"
EXPECTED := "j7bZM0LXZ9eXeZruTqWm2DIvDYVUU3wxPPpp%2BiXxzQc%3D"
if RESULT := HashSignature(CANONICAL_STR, SECRET_KEY); RESULT != EXPECTED {
t.Errorf("\nEXPECTED:\n%v\nRESULT:\n%v", EXPECTED, RESULT)
} else { fmt.Println("TestAmazonSignature: Signature: OK") }
}
Here's a playground link with all this code.
Looks fine to me, try running:
https://play.golang.org/p/w0mQAYx2GQ
I added necessary imports and a main function

golang cast a string to net.IPNet type

I have a slice of strings that are in CIDR notation. They are both ipv4 and ipv6 and I need them cast into the type net.IPNet.
How would I do this in golang?
example strings:
192.168.1.1/24
fd04:3e42:4a4e:3381::/64
As cnicutar says use net.ParseCIDR.
This is a working example on how to actually use it.
http://play.golang.org/p/Wtqy56LS2Y
package main
import (
"fmt"
"net"
)
func main() {
ipList := []string{"192.168.1.1/24", "fd04:3e42:4a4e:3381::/64"}
for i := 0; i < len(ipList); i += 1 {
ip, ipnet, err := net.ParseCIDR(ipList[i])
if err != nil {
fmt.Println("Error", ipList[i], err)
continue
}
fmt.Println(ipList[i], "-> ip:", ip, " net:", ipnet)
}
}
I don't think you want casting; instead I think you want ParseCIDR
func ParseCIDR(s string) (IP, *IPNet, error)
This is how to do it using the IPAddress Go library. CIDR subnets, IP addresses and masks all use the same type in the IPAddress library, ipaddr.IPAddress, making code simpler. Disclaimer: I am the project manager.
package main
import (
"fmt"
"github.com/seancfoley/ipaddress-go/ipaddr"
"net"
)
func main() {
parseAddr("192.168.1.1/24")
parseAddr("fd04:3e42:4a4e:3381::/64")
}
func parseAddr(cidrStr string) {
cidr := ipaddr.NewIPAddressString(cidrStr)
subnet, addr := cidr.GetAddress().ToPrefixBlock(), cidr.GetHostAddress()
var ipNet = net.IPNet{
IP: subnet.GetNetIP(),
Mask: subnet.GetNetworkMask().Bytes(),
}
fmt.Printf("\nsubnet: %s\naddress: %s\nIPNet: %+v\n", subnet, addr, ipNet)
}
Output:
subnet: 192.168.1.0/24
address: 192.168.1.1
IPNet: {IP:192.168.1.0 Mask:ffffff00}
subnet: fd04:3e42:4a4e:3381::/64
address: fd04:3e42:4a4e:3381::
IPNet: {IP:fd04:3e42:4a4e:3381:: Mask:ffffffffffffffff0000000000000000}

How do I unit test command line flags in Go?

I would like a unit test that verifies a particular command line flag is within an enumeration.
Here is the code I would like to write tests against:
var formatType string
const (
text = "text"
json = "json"
hash = "hash"
)
func init() {
const (
defaultFormat = "text"
formatUsage = "desired output format"
)
flag.StringVar(&formatType, "format", defaultFormat, formatUsage)
flag.StringVar(&formatType, "f", defaultFormat, formatUsage+" (shorthand)")
}
func main() {
flag.Parse()
}
The desired test would pass only if -format equalled one of the const values given above. This value would be available in formatType. An example correct call would be: program -format text
What is the best way to test the desired behaviors?
Note: Perhaps I have phrased this poorly, but the displayed code it not the unit test itself, but the code I want to write unit tests against. This is a simple example from the tool I am writing and wanted to ask if there were a good way to test valid inputs to the tool.
Custom testing and processing of flags can be achieved with the flag.Var function in the flag package.
Flag.Var "defines a flag with the specified name and usage string. The type and value of the flag are represented by the first argument, of type Value, which typically holds a user-defined implementation of Value."
A flag.Value is any type that satisfies the Value interface, defined as:
type Value interface {
String() string
Set(string) error
}
There is a good example in the example_test.go file in the flag package source
For your use case you could use something like:
package main
import (
"errors"
"flag"
"fmt"
)
type formatType string
func (f *formatType) String() string {
return fmt.Sprint(*f)
}
func (f *formatType) Set(value string) error {
if len(*f) > 0 && *f != "text" {
return errors.New("format flag already set")
}
if value != "text" && value != "json" && value != "hash" {
return errors.New("Invalid Format Type")
}
*f = formatType(value)
return nil
}
var typeFlag formatType
func init() {
typeFlag = "text"
usage := `Format type. Must be "text", "json" or "hash". Defaults to "text".`
flag.Var(&typeFlag, "format", usage)
flag.Var(&typeFlag, "f", usage+" (shorthand)")
}
func main() {
flag.Parse()
fmt.Println("Format type is", typeFlag)
}
This is probably overkill for such a simple example, but may be very useful when defining more complex flag types (The linked example converts a comma separated list of intervals into a slice of a custom type based on time.Duration).
EDIT: In answer to how to run unit tests against flags, the most canonical example is flag_test.go in the flag package source. The section related to testing custom flag variables starts at Line 181.
You can do this
func main() {
var name string
var password string
flag.StringVar(&name, "name", "", "")
flag.StringVar(&password, "password", "", "")
flag.Parse()
for _, v := range os.Args {
fmt.Println(v)
}
if len(strings.TrimSpace(name)) == 0 || len(strings.TrimSpace(password)) == 0 {
log.Panicln("no name or no passward")
}
fmt.Printf("name:%s\n", name)
fmt.Printf("password:%s\n", password)
}
func TestMainApp(t *testing.T) {
os.Args = []string{"test", "-name", "Hello", "-password", "World"}
main()
}
You can test main() by:
Making a test that runs a command
Which then calls the app test binary, built from go test, directly
Passing the desired flags you want to test
Passing back the exit code, stdout, and stderr which you can assert on.
NOTE This only works when main exits, so that the test does not run infinitely, or gets caught in a recursive loop.
Given your main.go looks like:
package main
import (
"flag"
"fmt"
"os"
)
var formatType string
const (
text = "text"
json = "json"
hash = "hash"
)
func init() {
const (
defaultFormat = "text"
formatUsage = "desired output format"
)
flag.StringVar(&formatType, "format", defaultFormat, formatUsage)
flag.StringVar(&formatType, "f", defaultFormat, formatUsage+" (shorthand)")
}
func main() {
flag.Parse()
fmt.Printf("format type = %v\n", formatType)
os.Exit(0)
}
Your main_test.go may then look something like:
package main
import (
"fmt"
"os"
"os/exec"
"path"
"runtime"
"strings"
"testing"
)
// This will be used to pass args to app and keep the test framework from looping
const subCmdFlags = "FLAGS_FOR_MAIN"
func TestMain(m *testing.M) {
// Only runs when this environment variable is set.
if os.Getenv(subCmdFlags) != "" {
runAppMain()
}
// Run all tests
exitCode := m.Run()
// Clean up
os.Exit(exitCode)
}
func TestMainForCorrectness(tester *testing.T) {
var tests = []struct {
name string
wantCode int
args []string
}{
{"formatTypeJson", 0, []string{"-format", "json"}},
}
for _, test := range tests {
tester.Run(test.name, func(t *testing.T) {
cmd := getTestBinCmd(test.args)
cmdOut, cmdErr := cmd.CombinedOutput()
got := cmd.ProcessState.ExitCode()
// Debug
showCmdOutput(cmdOut, cmdErr)
if got != test.wantCode {
t.Errorf("unexpected error on exit. want %q, got %q", test.wantCode, got)
}
})
}
}
// private helper methods.
// Used for running the application's main function from other test.
func runAppMain() {
// the test framework has process its flags,
// so now we can remove them and replace them with the flags we want to pass to main.
// we are pulling them out of the environment var we set.
args := strings.Split(os.Getenv(subCmdFlags), " ")
os.Args = append([]string{os.Args[0]}, args...)
// Debug stmt, can be removed
fmt.Printf("\nos args = %v\n", os.Args)
main() // will run and exit, signaling the test framework to stop and return the exit code.
}
// getTestBinCmd return a command to run your app (test) binary directly; `TestMain`, will be run automatically.
func getTestBinCmd(args []string) *exec.Cmd {
// call the generated test binary directly
// Have it the function runAppMain.
cmd := exec.Command(os.Args[0], "-args", strings.Join(args, " "))
// Run in the context of the source directory.
_, filename, _, _ := runtime.Caller(0)
cmd.Dir = path.Dir(filename)
// Set an environment variable
// 1. Only exist for the life of the test that calls this function.
// 2. Passes arguments/flag to your app
// 3. Lets TestMain know when to run the main function.
subEnvVar := subCmdFlags + "=" + strings.Join(args, " ")
cmd.Env = append(os.Environ(), subEnvVar)
return cmd
}
func showCmdOutput(cmdOut []byte, cmdErr error) {
if cmdOut != nil {
fmt.Printf("\nBEGIN sub-command out:\n%v", string(cmdOut))
fmt.Print("END sub-command\n")
}
if cmdErr != nil {
fmt.Printf("\nBEGIN sub-command stderr:\n%v", cmdErr.Error())
fmt.Print("END sub-command\n")
}
}
I'm not sure whether we agree on the term 'unit test'. What you want to achieve seems to me
more like a pretty normal test in a program. You probably want to do something like this:
func main() {
flag.Parse()
if formatType != text || formatType != json || formatType != hash {
flag.Usage()
return
}
// ...
}
Sadly, it is not easily possible to extend the flag Parser with own value verifiers
so you have to stick with this for now.
See Intermernet for a solution which defines a custom format type and its validator.