I'm testing an http server in golang and everything seems pretty smooth except for one annoying thing.
At some point, when I'm configuring the server, before performing a http.ListenAndServe I register an handler with http.Handle("/", myrouter) , the problem is that on the following test, when the configuration method gets called again, I receive the following panic:
panic: http: multiple registrations for / [recovered]
I'd like to run each test in a clean environment, but haven't found a way to tear down http.DefaultServeMux, is there a convenient way for doing this by either "nulling" something or re-configuring the test environment so every test is executed on a fresh environment ?
Edit:
To give some context, as asked, I post here some code and the test I wanted to write, I have to be clear tough on the fact that I'm not even sure on the implementation choices made here (I think I would have configured the server in a slightly different way, but the code's not mine).
package httpserver
import (
"github.com/gorilla/mux"
"github.com/private/private/httpserver/rpchandler"
"net/http"
)
type HTTPServer struct {
router *mux.Router
port int
}
// Config is used to override the default port of the http server.
type Config struct {
Port *int
}
func NewHTTPServer(config *Config) (*HTTPServer, error) {
hs := &HTTPServer{
port: 80,
}
// Overwrite default port if passed.
if config != nil && config.Port != nil {
hs.port = *config.Port
}
err := hs.router = mux.NewRouter().StrictSlash(false)
if err != nil {
return nil, err
}
// other mux router configs here...
http.Handle("/", hs.router)
return hs, nil
}
Now, the tests I wanted to write were quite simple, like:
func TestThatServerIsInitializedWithDefaultPort(t *testing.T) {
sut, _ := NewHTTPServer(nil)
if sut.port != 80 {
t.Fatal("default port not configured")
}
}
func TestThatServerDefaultPortIsOverriddenWithConfig(t *testing.T) {
mockPort := 8080
c := Config{
Port: &mockPort,
}
sut, _ := NewHTTPServer(&c)
if sut.port != 8080 {
t.Fatal("the port has not been overridden with the one passed in configuration")
}
}
However, since I call the handle binding on http twice, I get the panic.
I think I've found a solution: basically after each test I reinitialize http.DefaultServeMux with http.DefaultServeMux = new(http.ServeMux)
I'm still not sure this is a clean way to workaround the problem tough.
I kindly ask you to give me any hints or point me some bad practices used here, since I'm quite new to the language and to backend development in general.
I think there might be some code smell in the configuration of the HTTP Server shown above, if you point it out I might make it clear to the rest of my team and work a better solution out.
I think you should look at using the httptest pkg. It has servers that you can start fresh every test.
Related
Is there a way to mock and/or spy on methods in golang without using dependency injection. For example lets say that I want to unit test the following method
import (
"github.com/spf13/viper"
"google.golang.org/grpc"
)
func startGrpcServer(ctx context.Context, wg *sync.WaitGroup) {
defer wg.Done()
listen, err := net.Listen("tcp", fmt.Sprintf(":%v", viper.GetString("GRPC_PORT")))
if err != nil {
log.Fatal(err)
}
var opts []grpc.ServerOption
grpcServer := grpc.NewServer(opts...)
reflection.Register(grpcServer)
s := health.NewHealthServer()
proto.RegisterHealthServer(grpcServer, s)
go func() {
<-ctx.Done()
log.Print("Stopping grpc server")
grpcServer.GracefulStop()
}()
log.Print("Starting grpc server")
grpcServer.Serve(listen)
}
I want to make sure of the following
The server is started (spying on the grpcServer.Serve method.
The environment value is ready from viper.GetString method.
As you can see the packages for these methods are global imports and I am not using DI. Also this is legacy code so I cannot go and change all the code to support DI. From my reading and initial researches done you cannot use mock/spy in golang without DI. Is this correct ? or is there some way to do this ?
I came across the following but this is fairly dated How do I mock a function from another package without using dependency injection? and from what I see in this, my options are fairly limited.
Is there a way to mock and/or spy on methods in golang without using dependency injection.
No, there isn't.
I'm pretty new to Go and still learning about how things work in Go, so with that said I've been looking in to Go testing approach and how mocking would work for the past few weeks and most of the information I found based on functions being concrete.
E.g. everything is a function is either passed as a receiver or a parameter, however, the problem I face is my function uses a switch case to determine what function it should be called, so it's not passed from outside.
func (n *Notification) Notify(m Message) error {
switch n.Service {
case "slack":
var s slack.Slack
s.User = m.User
s.Host = m.Host
s.Provider = m.Provider
s.SystemUser = m.SystemUser
return s.SlackSend(n.Url)
default:
return errors.New(codes.CODE5)
}
}
Above code is what the function I want to test looks like and I'm having a hard time figuring out how I could mock the SlackSend() function.
I've come across some article say I should write the function in the test file and when I'm happy with what function do, I should write the real code. This doesn't sound right to me because for me it feels like I have the same code in two places and test only uses the one in test and I could change the real code and break it without testes detecting that.
I mainly work on Python and I'm used to using things like Mock/MagicMock that can intercept the function call and replace on at runtime, so apologies in advance if I don't quite get the Go's TDD approach.
This is what test code would look like if anyone wonders:
type MockSlack struct {
*slack.Slack
}
func (s *MockSlack) SlackSend(url string) error {
if url != "" {
return nil
} else {
return errors.New("url empty")
}
}
func TestNotify(t *testing.T) {
m := Message{}
n := Notification{
Service: "slack",
Url: "https://dummy.io",
}
if err := n.Notify(m); err != nil {
t.Errorf("SlackSend, expected: %s, got: %s", "nil", err.Error())
}
}
Obviously the MockSlack structure is not taking effect because it's not really passed in anywhere.
If anyone has any advice on how I could mock this or what I should do instead would be much appreciated.
Thanks in advance.
UPDATE:
Background
This isn't a web server/applicaition of a sort. It's an SSH authentication plugin so it would be a server-side application. As for Notify method, it server's the purpose of a mapper. So it can call Slack, MS Teams, AWS SNS which give caller less conditions to process and how and where the notification it sent is decided by the Notify method.
If you are not able to change the Notify Method to make it testing friendly. One option would be to consider using monkeypatching. Here is an example provided for *net.Dialer:
func main() {
var d *net.Dialer // Has to be a pointer to because `Dial` has a pointer receiver
monkey.PatchInstanceMethod(reflect.TypeOf(d), "Dial", func(_ *net.Dialer, _, _ string) (net.Conn, error) {
return nil, fmt.Errorf("no dialing allowed")
})
_, err := http.Get("http://google.com")
fmt.Println(err) // Get http://google.com: no dialing allowed
}
WARNING: It's not safe to use it outside of a testing environment.
I'm learning go and am working on a simple service that ingests some data from a queue and sticks it in the database. It also runs a web server to allow scraping of data. Right now I have two go files (omitted some text for brevity):
func main() {
parseConfig()
s := &Service{ServiceConfig: config}
err := s.Run()
if err != nil {
panic(err)
}
}
And then the definition of the service (again left out some pieces for brevity):
func (s *Service) Run() error {
if err := s.validate(); err != nil {
return err
}
if err := s.initDB(); err != nil {
return err
}
defer s.db.Close()
// Same pattern with health check library (init, start, close)
// Same pattern starting queue consumer (init, start, close)
s.mux = http.NewServeMux()
s.registerHandlers(s.mux)
http.ListenAndServe(":8080", s.mux)
return nil
}
And the struct
type Service struct {
Config // Hold db connection info
db *sql.DB
hc *health
}
I'm able to test the individual pieces fine (like initDB or validate) but I'm not unclear how one would test the Run function because http.ListenAndServe blocks. I eventually time out. Previously, I would use httpTest and make a test server but that was when main would start the server (the application was more basic at first).
Some things I would test:
That I can hit the metrics endpoint once started.
That I can hit the health endpoint once started.
That I can push a message on the queue and it is received once started.
That Run actually starts w/o a panic.
Some notes: I am using docker to spin up a queue and database. The point of testing the Run function is to ensure that the bootstrapping works and the application can run successfully. Eventually I will want to push data through the queue and assert that its been processed correctly.
Question: How should I test this or refactor it so that it is more easily testable end to end?
You can build a test harness using a goroutine to execute Run in your unit test:
func TestRun(t *testing.T) {
service := Service{}
serviceRunning := make(chan struct{})
serviceDone := make(chan struct{})
go func() {
close(serviceRunning)
err := service.Run()
defer close(serviceDone)
}()
// wait until the goroutine started to run (1)
<-serviceRunning
//
// interact with your service to test whatever you want
//
// stop the service (2)
service.Shutdown()
// wait until the service is shutdown (3)
<-serviceDone
}
This is just a basic example to show how it could be done in principle. There are several points that should be improved for use in production:
(0) The most important detail: Do not use http.ListenAndServe in production! Create your own instance of http.Server instead. This saves you a lot of trouble.
s.httpServer := http.Server {
Addr: ":8080",
Handler: s.mux,
}
(1) The indication that the service is running should be moved into your Service type. The initialization part in Run might take a while and the indication channel should be closed right before ListenAndServe is called:
close(s.Running)
s.httpServer.ListenAndServe()
Of course, you need to add the indication channel to your Service type.
(2) Add a Shutdown method to Service that calls s.httpServer.Shutdown(). This will cause the call to s.httpServer.ListenAndServe to return with the error http.ErrServerClosed.
if err := s.httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
return err
}
return nil
It is important to shutdown your service at the end of the test. Otherwise you will not be able to have more than one unit test for your service. And it is good citizenship anyway to clean up the resources.
(3) You need to wait until service.Run returned to make sure that the service is actually shutdown.
The httptest package comes with a test server built for this purpose.
https://pkg.go.dev/net/http/httptest#example-Server
I am currently writing a workflow in Go that uses Google's API go client. I'm relatively new to Go and am having trouble unit testing the client's services. Here is an example method that enables an API in a Google Cloud Project.
func (gcloudService *GCloudService) EnableApi(projectId string, apiId string) error {
service, err := servicemanagement.New(gcloudService.Client)
if err != nil {
return err
}
requestBody := &servicemanagement.EnableServiceRequest{
ConsumerId: consumerId(projectId),
}
_, err = service.Services.Enable(apiId, requestBody).Do()
if err != nil {
return err
}
return nil
}
GCloudService is a simple struct that holds a Client.
type GCloudService struct {
Client *http.Client
}
This is my attempt at testing this method.
var (
mux *http.ServeMux
client *http.Client
server *httptest.Server
)
func setup() {
// test server
mux = http.NewServeMux()
server = httptest.NewServer(mux)
// client configured to use test server
client = server.Client()
}
func teardown() {
server.Close()
}
func TestGCloudService_EnableApi(t *testing.T) {
setup()
defer teardown()
projectName := "test"
apiId := "api"
testGcloudService := &GCloudService{
Client: client,
}
path := fmt.Sprintf("/v1/services/{%s}:enable", apiId)
mux.HandleFunc(path,
func(w http.ResponseWriter, r *http.Request) {
// test things...
})
err := testGcloudService.EnableApi(projectName, apiId)
if err != nil {
t.Errorf("EnableApi returned error: %v", err)
}
}
However, when I run this test it still hits the real Google endpoint instead of my localhost server because EnableApi uses the servicemanagement service which is configured with the API's base URL. How do I refactor this to call my server instead of the API? I am hoping to avoid mocking the entire servicemanagement service if possible.
What I'd recommend is creating your own interface that wraps the google api client and extract the methods that you're interested in.
type MyWrapperClient interface {
SomeMethodWithCorrectReturnType()
}
type myWrapperClient struct {
*GCloudService.Client // or whatever
}
In the directory I'd then run:
mockery -name=MyWrapperClient inside the directory (after installing mockery)
and then you can access your mocked version. Then on object creation substitute your mock in for your client - as the interface and the mock have the same methods they are interchangeable. Then you can test whether methods are called with specific params - leaving the google api client code alone.
More information on the mockery library is here: https://github.com/vektra/mockery
This article solves your same problem and it's absolutely fantastic in explaining how to mock and abstract your concerns away.
https://medium.com/agrea-technogies/mocking-dependencies-in-go-bb9739fef008
Make the base url in your servicemanagement service configurable or overwritable, and if that is hidden for you, then your code is not written for test convenience, change that, and if not possible, complain to who is responsible. If that does not help, take a deep breath, and write a mock service, which is mostly not needed to be very complicated
How I can test my NewClient constructor for my Client struct ?
package busybus
import (
"bufio"
"net"
)
type Client struct {
counter integer
conn net.Conn
bufin *bufio.Reader
bufout *bufio.Writer
messages chan string
state string
}
func NewClient(conn net.Conn, messages chan string) *Client {
return &Client{
counter: 0,
conn: conn,
bufin: bufio.NewReader(conn),
bufout: bufio.NewWriter(conn),
messages: messages,
state: "waiting",
}
}
I tried some testing like this:
package busybus
import (
"net"
"testing"
)
func TestNewClient(t *testing.T) {
ln, _ := net.Listen("tcp", ":65535")
conn, _ := ln.Accept()
messages := make(chan string)
client := NewClient(conn, messages)
if client.conn != conn {
t.Errorf("NewClient(%q, %q).conn == %q, want %q", conn, messages, client.conn, conn)
}
}
But this hangs during the test runs due to ln.Accept, it seems a totally wrong approach anyway... any suggestion how I can test this constructor ?
Some code is so short and simple that ensuring that the test is correct is more complicated than ensuring that the code itself is correct.
Your constructor is such code.
What I would do is either not test it at all, or just call it (using some dummy implementation of net.Conn) to ensure that it doesn't blow up in smoke when called (therefore called a smoke test).
Later you can test it as part of a bigger test (integration test), where you have a real server to connect to, and which uses your client struct to talk to it.
If you ever find a bug that is due to this constructor, then first add a test that demonstrates the problem, then fix it. Then you will have your test :-)
A test should check that the constructor "works as intended". Checking the value of client.conn is hardly checking any intended behavior, as the value of an unexported field is not behavior. It just checks HOW the struct and constructor is implemented, not WHAT it implements. Test the what, not the how.
By the way, you could maybe embed a *ReadWriter in your client.
client.buf = bufio.NewReadWriter(...)
In addition to what #bjarke-ebert said.
There are no constructors in Go, they are just normal functions. Test them as you would test any other function. If a function is too complex to test then probably there is a way to change the design to make it more testable.
Also from my experience tests that check internal details of implementation too much (like the test from the question) are very hard to maintain as they constantly need to be updated.