I am currently writing a workflow in Go that uses Google's API go client. I'm relatively new to Go and am having trouble unit testing the client's services. Here is an example method that enables an API in a Google Cloud Project.
func (gcloudService *GCloudService) EnableApi(projectId string, apiId string) error {
service, err := servicemanagement.New(gcloudService.Client)
if err != nil {
return err
}
requestBody := &servicemanagement.EnableServiceRequest{
ConsumerId: consumerId(projectId),
}
_, err = service.Services.Enable(apiId, requestBody).Do()
if err != nil {
return err
}
return nil
}
GCloudService is a simple struct that holds a Client.
type GCloudService struct {
Client *http.Client
}
This is my attempt at testing this method.
var (
mux *http.ServeMux
client *http.Client
server *httptest.Server
)
func setup() {
// test server
mux = http.NewServeMux()
server = httptest.NewServer(mux)
// client configured to use test server
client = server.Client()
}
func teardown() {
server.Close()
}
func TestGCloudService_EnableApi(t *testing.T) {
setup()
defer teardown()
projectName := "test"
apiId := "api"
testGcloudService := &GCloudService{
Client: client,
}
path := fmt.Sprintf("/v1/services/{%s}:enable", apiId)
mux.HandleFunc(path,
func(w http.ResponseWriter, r *http.Request) {
// test things...
})
err := testGcloudService.EnableApi(projectName, apiId)
if err != nil {
t.Errorf("EnableApi returned error: %v", err)
}
}
However, when I run this test it still hits the real Google endpoint instead of my localhost server because EnableApi uses the servicemanagement service which is configured with the API's base URL. How do I refactor this to call my server instead of the API? I am hoping to avoid mocking the entire servicemanagement service if possible.
What I'd recommend is creating your own interface that wraps the google api client and extract the methods that you're interested in.
type MyWrapperClient interface {
SomeMethodWithCorrectReturnType()
}
type myWrapperClient struct {
*GCloudService.Client // or whatever
}
In the directory I'd then run:
mockery -name=MyWrapperClient inside the directory (after installing mockery)
and then you can access your mocked version. Then on object creation substitute your mock in for your client - as the interface and the mock have the same methods they are interchangeable. Then you can test whether methods are called with specific params - leaving the google api client code alone.
More information on the mockery library is here: https://github.com/vektra/mockery
This article solves your same problem and it's absolutely fantastic in explaining how to mock and abstract your concerns away.
https://medium.com/agrea-technogies/mocking-dependencies-in-go-bb9739fef008
Make the base url in your servicemanagement service configurable or overwritable, and if that is hidden for you, then your code is not written for test convenience, change that, and if not possible, complain to who is responsible. If that does not help, take a deep breath, and write a mock service, which is mostly not needed to be very complicated
Related
I am learning to do unit testing using mocks in Go for the first time, using gomock's mockgen utility. My unit tests work fine except for one of them. The method-under-test has two dependencies: one on a database, and the other on an external service it makes rest api calls to. The mock for the database (mockRepo) works fine in that the method-under-test properly invokes the mock instead of the actual repo code. The mock for the rest client, however, continues to invoke the actual rest client and not the mock code. I can't figure out why. Can someone explain why and help fix?
Here is my unit test:
func TestService_CreateWorkspace(t *testing.T) {
ts := NewTestService(t)
defer ts.mockCtrl.Finish()
ts.mockClient.EXPECT().POST(gomock.Any(), gomock.Any(), gomock.Any()).Times(1).Return(&http.Response{StatusCode: 200}, nil)
testWs := TestWorkspaces()["max-ws"]
ts.mockRepo.EXPECT().Create(testWs).Times(1).Return(&testWs, nil)
ws, err := ts.service.CreateWorkspace(&testWs)
assert.Equal(t, testWs, ws)
assert.NoError(t, err)
}
Here is the code for NewTestService:
type TestService struct {
mockCtrl *gomock.Controller
mockClient *MockRestClient
mockRepo *MockRepository
service Service
}
func NewTestService(t *testing.T) *TestService {
mockCtrl := gomock.NewController(t)
mockRepo := NewMockRepository(mockCtrl)
mockClient := NewMockRestClient(mockCtrl)
return &TestService{
mockCtrl: mockCtrl,
mockClient: mockClient,
mockRepo: mockRepo,
service: NewService(mockRepo),
}
}
Is there an issue with assigning the same mock controller to two different mock objects? Not really sure what's going on here. Any help appreciated.
I resolved this as Adrian in the comments above suggested. I was missing a way to pass the mock client into the NewService and ended up adding a client parameter to NewService.
I'm learning go and am working on a simple service that ingests some data from a queue and sticks it in the database. It also runs a web server to allow scraping of data. Right now I have two go files (omitted some text for brevity):
func main() {
parseConfig()
s := &Service{ServiceConfig: config}
err := s.Run()
if err != nil {
panic(err)
}
}
And then the definition of the service (again left out some pieces for brevity):
func (s *Service) Run() error {
if err := s.validate(); err != nil {
return err
}
if err := s.initDB(); err != nil {
return err
}
defer s.db.Close()
// Same pattern with health check library (init, start, close)
// Same pattern starting queue consumer (init, start, close)
s.mux = http.NewServeMux()
s.registerHandlers(s.mux)
http.ListenAndServe(":8080", s.mux)
return nil
}
And the struct
type Service struct {
Config // Hold db connection info
db *sql.DB
hc *health
}
I'm able to test the individual pieces fine (like initDB or validate) but I'm not unclear how one would test the Run function because http.ListenAndServe blocks. I eventually time out. Previously, I would use httpTest and make a test server but that was when main would start the server (the application was more basic at first).
Some things I would test:
That I can hit the metrics endpoint once started.
That I can hit the health endpoint once started.
That I can push a message on the queue and it is received once started.
That Run actually starts w/o a panic.
Some notes: I am using docker to spin up a queue and database. The point of testing the Run function is to ensure that the bootstrapping works and the application can run successfully. Eventually I will want to push data through the queue and assert that its been processed correctly.
Question: How should I test this or refactor it so that it is more easily testable end to end?
You can build a test harness using a goroutine to execute Run in your unit test:
func TestRun(t *testing.T) {
service := Service{}
serviceRunning := make(chan struct{})
serviceDone := make(chan struct{})
go func() {
close(serviceRunning)
err := service.Run()
defer close(serviceDone)
}()
// wait until the goroutine started to run (1)
<-serviceRunning
//
// interact with your service to test whatever you want
//
// stop the service (2)
service.Shutdown()
// wait until the service is shutdown (3)
<-serviceDone
}
This is just a basic example to show how it could be done in principle. There are several points that should be improved for use in production:
(0) The most important detail: Do not use http.ListenAndServe in production! Create your own instance of http.Server instead. This saves you a lot of trouble.
s.httpServer := http.Server {
Addr: ":8080",
Handler: s.mux,
}
(1) The indication that the service is running should be moved into your Service type. The initialization part in Run might take a while and the indication channel should be closed right before ListenAndServe is called:
close(s.Running)
s.httpServer.ListenAndServe()
Of course, you need to add the indication channel to your Service type.
(2) Add a Shutdown method to Service that calls s.httpServer.Shutdown(). This will cause the call to s.httpServer.ListenAndServe to return with the error http.ErrServerClosed.
if err := s.httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
return err
}
return nil
It is important to shutdown your service at the end of the test. Otherwise you will not be able to have more than one unit test for your service. And it is good citizenship anyway to clean up the resources.
(3) You need to wait until service.Run returned to make sure that the service is actually shutdown.
The httptest package comes with a test server built for this purpose.
https://pkg.go.dev/net/http/httptest#example-Server
I'm trying to implement unit tests in Go for an existing service which uses a connection pool struct and a connection struct from an existing library (call these LibraryPool and LibraryConnection) to connect to an external service.
To use these, the service functions in the main code uses a unique, global instance of the pool, which has a GetConnection() method, like this:
// Current Main Code
var pool LibraryPool // global, instantiated in main()
func someServiceFunction(w http.ResponseWriter, r *http.Request) {
// read request
// ...
conn := pool.GetConnection()
conn.Do("some command")
// write response
// ...
}
func main() {
pool := makePool() // builds and returns a LibraryPool
// sets up endpoints that use the service functions as handlers
// ...
}
I'd like to unit-test these service functions without connecting to the external service, and so I'd like to mock the LibraryPool and LibraryConnection. To allow for this, I was thinking of changing the main code to something like this:
// Tentative New Main Code
type poolInterface interface {
GetConnection() connInterface
}
type connInterface interface {
Do(command string)
}
var pool poolInterface
func someServiceFunction(w http.ResponseWriter, r *http.Request) {
// read request
// ...
conn := pool.GetConnection()
conn.Do("some command")
// write response
// ...
}
func main() {
pool := makePool() // still builds a LibraryPool
}
In the tests, I would use mock implementations MockPool and MockConnection of these interfaces, and the global pool variable would be instantiated using MockPool. I would instantiate this global pool in a setup() function, inside of a TestMain() function.
The problem is that in the new main code, LibraryPool does not properly implement poolInterface, because GetConnection() returns a connInterface instead of a LibraryConnection (even though LibraryConnection is a valid implementation of connInterface).
What would be a good way to approach this kind of testing? The main code is flexible too, by the way.
Well, I'll try to answer by completely explain how I see this design. Sorry in advance if this is too much and not to the point..
Entity / Domain
The core of the app, will include the entity struct, won't import ANY outer layer package, but can be imported by every package (almost)
Application / Use case
The "service". Will be responsible mainly for the app logic, won't know about the transport(http), will "talk" with the DB through interface. Here you can have the domain validation, for example if resource is not found, or text is too short. Anything related to business logic.
transport
Will handle the http request, decode the request, get the service to do his stuff, and encode the response. Here you can return 401 if there is a missing required param in the request, or the user is not authorized, or something...
infrastructure
DB connection
Maybe some http engine and router and stuff.
Totally app-agnostic, don't import any inner package, not even Pseron
For example, let's say we want to do something as simple as insert person to the db.
package person will only include the person struct
package person
type Person struct{
name string
}
func New(name string) Person {
return Person{
name: name,
{
}
About the db, let's say you use sql, I recommend to make a package named sql to handle the repo. (if you use postgress, use 'postgress package...).
The personRepo will get the dbConnection which will be initialized in main and implement DBAndler. only the connection will "talk" with the db directly, the repository main goal is to be gateway to the db, and speak in application-terms. (the connection is app-agnostic)
package sql
type DBAndler interface{
exec(string, ...interface{}) (int64, error)
}
type personRepo struct{
dbHandler DBHandler
}
func NewPersonRepo(dbHandler DBHandler) &personRepo {
return &personRepo{
dbHandler: dbHandler,
}
}
func (p *personRepo) InsertPerson(p person.Person) (int64, error) {
return p.dbHandler.Exec("command to insert person", p)
}
The service will get this repository as a dependancy (as interface) in the initailzer, and will interact with it to accomplish the business logic
package service
type PersonRepo interface{
InsertPerson(person.Person) error
}
type service struct {
repo PersonRepo
}
func New(repo PersonRepo) *service {
return &service{
repo: repo
}
}
func (s *service) AddPerson(name string) (int64, error) {
person := person.New(name)
return s.repo.InsertPerson(person)
}
Your transport handler will be initialized with the service as a dependancy, and he will handle the http request.
package http
type Service interface{
AddPerson(name string) (int64, error)
}
type handler struct{
service Service
}
func NewHandler(s Service) *handler {
return &handler{
service: s,
}
}
func (h *handler) HandleHTTP(w http.ResponseWriter, r *http.Request) {
// read request
// decode name
id, err := h.service.AddPerson(name)
// write response
// ...
}
And in main.go you will tie everything together:
Initialize db connection
Initialize personRepo with this connection
Initialize service with the repo
Initialize the transport with the service
package main
func main() {
pool := makePool()
conn := pool.GetConnection()
// repo
personRepo := sql.NewPersonRepo(conn)
// service
personService := service.New(personRepo)
// handler
personHandler := http.NewPersonHandler(personService)
// Do the rest of the stuff, init the http engine/router by passing this handler.
}
Note that every package struct was initialized with an interface but returned a struct, and also the interfaces were declared in the package which used them, not in the package which implemented them.
This makes it easy to unit test these package. for example, if you want to test the service, you don't need to worry about the http request, just use some 'mock' struct that implements the interface that the service depend on (PersonRepo), and you good to go..
Well, I hope it helped you even a little bit, it may seem confusing at first, but in time you will see how this seems like a large piece of code, but it helps when you need to add functionality or switching the db driver and such.. I recommend you to read about domain driven design in go, and also hexagonal arch.
edit:
In addition, this way you pass the connection to the service, the service doesn't import and use the global DB pool. Honestly, I don't know why it is so common, I guess it has its advantages and it is better to some application, but generally I think that letting your service depend on some interface, without actually know what is going on, is much a better practice.
I'm testing an http server in golang and everything seems pretty smooth except for one annoying thing.
At some point, when I'm configuring the server, before performing a http.ListenAndServe I register an handler with http.Handle("/", myrouter) , the problem is that on the following test, when the configuration method gets called again, I receive the following panic:
panic: http: multiple registrations for / [recovered]
I'd like to run each test in a clean environment, but haven't found a way to tear down http.DefaultServeMux, is there a convenient way for doing this by either "nulling" something or re-configuring the test environment so every test is executed on a fresh environment ?
Edit:
To give some context, as asked, I post here some code and the test I wanted to write, I have to be clear tough on the fact that I'm not even sure on the implementation choices made here (I think I would have configured the server in a slightly different way, but the code's not mine).
package httpserver
import (
"github.com/gorilla/mux"
"github.com/private/private/httpserver/rpchandler"
"net/http"
)
type HTTPServer struct {
router *mux.Router
port int
}
// Config is used to override the default port of the http server.
type Config struct {
Port *int
}
func NewHTTPServer(config *Config) (*HTTPServer, error) {
hs := &HTTPServer{
port: 80,
}
// Overwrite default port if passed.
if config != nil && config.Port != nil {
hs.port = *config.Port
}
err := hs.router = mux.NewRouter().StrictSlash(false)
if err != nil {
return nil, err
}
// other mux router configs here...
http.Handle("/", hs.router)
return hs, nil
}
Now, the tests I wanted to write were quite simple, like:
func TestThatServerIsInitializedWithDefaultPort(t *testing.T) {
sut, _ := NewHTTPServer(nil)
if sut.port != 80 {
t.Fatal("default port not configured")
}
}
func TestThatServerDefaultPortIsOverriddenWithConfig(t *testing.T) {
mockPort := 8080
c := Config{
Port: &mockPort,
}
sut, _ := NewHTTPServer(&c)
if sut.port != 8080 {
t.Fatal("the port has not been overridden with the one passed in configuration")
}
}
However, since I call the handle binding on http twice, I get the panic.
I think I've found a solution: basically after each test I reinitialize http.DefaultServeMux with http.DefaultServeMux = new(http.ServeMux)
I'm still not sure this is a clean way to workaround the problem tough.
I kindly ask you to give me any hints or point me some bad practices used here, since I'm quite new to the language and to backend development in general.
I think there might be some code smell in the configuration of the HTTP Server shown above, if you point it out I might make it clear to the rest of my team and work a better solution out.
I think you should look at using the httptest pkg. It has servers that you can start fresh every test.
I'm trying to create a simple mock for unit testing some code using the VMware vSphere API client - govmomi - but I'm having trouble finding a usable pattern.
A simple use case for the client library would be to retrieve the installed licenses for a vSphere cluster:
vclient, err := govmomi.NewClient(*vcurl, true)
if err != nil {
return err
}
lic, err := vclient.LicenseManager().ListLicenses()
NewClient() returns a pointer to a Client structure, Client.LicenseManager() returns an instance of a LicenseManager structure, and LicenseManager.ListLicenses() returns a slice of structures containing the license info. Coming from a Python background, I'd usually monkey patch the ListLicenses() method on LicenseManger for a mock, but I can't seem to come up with a comparable pattern or methodology in Go.
To this point, I've tried creating a wrapper structure VCenterClient with the govmomi Client structure as an anonymous member and a "constructor" function NewVCenter() to create new instances of the wrapper structure with logic for mocks:
import (
"net/url"
"github.com/vmware/govmomi"
"github.com/vmware/govmomi/vim25/types"
)
type VCenterClient struct {
VCenterClientInterface
}
type VCenterClientInterface interface {
LicenseManager() LicenseManager
}
type LicenseManager interface {
ListLicenses() ([]types.LicenseManagerLicenseInfo, error)
}
type VCenterClientMock struct{}
type LicenseManagerMock struct{}
func (v *VCenterClientMock) LicenseManager() LicenseManager {
return LicenseManagerMock{}
}
func (l LicenseManagerMock) ListLicenses() ([]types.LicenseManagerLicenseInfo, error) {
return make([]types.LicenseManagerLicenseInfo, 0), nil
}
func NewVCenterClient(uri string, mock bool) *VCenterClient {
if mock {
return &VCenterClient{&VCenterClientMock{}}
}
vcurl, _ := url.Parse(uri)
vclient, _ := govmomi.NewClient(*vcurl, true)
return &VCenterClient{vclient}
}
...but I having trouble using interfaces to properly abstract the nested structures in the govmomi library. I know the above will not work as govmomi.LicenseManager() returns a structure of type govmomi.LicenseManager and my VCenterClientInterface.LicenseManager() method returns an interface of type LicenseManager. However, I'm struggling to find an alternative.
Any help on a better design pattern or proper use of interfaces in this case would be much appreciated.
This library is a SOAP client (http://godoc.org/github.com/vmware/govmomi/vim25/soap#Client). Abstract at the HTTP layer with net/http/httptest (http://golang.org/pkg/net/http/httptest/) or by using your own HTTPRoundtripper to mock the response.