Gomock not working for sending data to cockroach DB - unit-testing

I am writing a unit test for a function which calls a goroutine to send data to cockroach db. I am trying to mock the call to cockroach db which doesn't seem to work.The test case results in time out in the exact same manner as it would have if there was no mocking done for sendBatch to cockroach db.
here are the functions:
func DrainQueue() {
for range time.Tick(time.Second * time.Duration(1)) {
if q.Length() > 0 {
var queuedItems []interface{}
// Read the queue up to the size specified.
for qlen := 0; qlen < QueueSize && q.Length() > 0; qlen++ {
itemFromQueue := q.Pop()
queuedItems = append(queuedItems, itemFromQueue)
}
logging.Debug("Sending to cockroachDB...")
go SendToCockRoachDB(queuedItems...)
queuedItems = nil
}
}
}
func SendToCockRoachDB(toSend ...interface{}) ([][][]byte, int64, error) {
var nor int64
var qErr error
var rows pgx.Rows
var results = make([][][]byte, 0)
batch := &pgx.Batch{}
for _, item := range toSend {
qs := item.(*QueryStruct)
batch.Queue(qs.Query, qs.Items...)
}
// Send to cockroachDB
br := dbConn.dbpool.SendBatch(dbConn.ctx, batch)
defer br.Close()
rows, qErr = br.Query()
if qErr != nil {
return nil, 0, qErr
}
for rows.Next() {
results = append(results, rows.RawValues())
}
nor += rows.CommandTag().RowsAffected()
if nor > 0 {
logging.Log.Printf("Inserted: %d rows", nor)
}
return results, nor, qErr
}
The test function I have created works like this:
func TestDrainQueue(t *testing.T) {
var results = make([][][]byte, 0)
var nor int64 = 0
controller := gomock.NewController(t)
defer controller.Finish()
mockSendBatch := Mocks.NewMockValues(controller)
mockSendBatch.EXPECT().SendToCockRoachDB(mockSendBatch).Return(results, nor, nil)
DrainQueue()
}
I have created a mock function using gomock for SendToCockRoachDB and have called it in the unit test

Related

assert: mock: I don't know what to return (even if I've declared the mock function & the return)

I use Testify to create a unit test for my golang app. I need to create a unit test for this function where it calls a variadic function (function with trailing arguments). I encountered an error when I test it. I'm actually not sure if the error is because of the trailing argument itself or not, but I feel like there's something wrong with the mock.
// svc/callThisFunction.go
// data type of args is []sqkit.SelectOption
func CallThisFunction(ctx context.Context, args ...sqkit.SelectFunctiom) (result string, err error) {
return result, nil
}
// svc/functionToTest.go
// This is the function that I wanna test
func FunctionToTest(ctx context.Context, id int64) (result string, err error) {
args := []sqkit.SelectOption{
sqkit.Where{
fmt.Sprintf("id = %d", id),
},
}
newResult, err := callThisFunctionService.CallThisFunction(ctx, args)
if err != nil {
return newResult, err
}
return newResult, nil
}
// svc/functionToTest_test.go
func Test_FunctionToTest(t *testing.T) {
testCase := []struct {
name string
id int64
onCallThisFunctionMock func(callThisFunctionSvc *mocks.CallThisFunctionSvc)
expectedResult string
wantError bool
expectedError error
}{
{
name: "Success",
id: 1,
onCallThisFunctionMock: func(callThisFunctionSvc *mocks.CallThisFunctionSvc) {
// NOTE: I've created 2 different versions (used separately, not at the same), using mock.Anything() and using actual arguments
// Both of these give the same errors
// Using actual arguments
args := []sqkit.SelectOption{
sqkit.Where{
fmt.Sprintf("id = %d", 1},
},
}
callThisFunctionSvc.On("CallThisFunction", context.Background(), args).Return("Success", nil)
// Using mock.Anything
callThisFunctionSvc.On("CallThisFunction", context.Background(), mock.Anything).Return("Success", nil)
}
}
}
for _, tc := range testCases {
var callThisFunctionSvc = new(mocks.CallThisFunctionSvc)
tc.onCallThisFunctionMock(callThisFunctionSvc)
svc := &svc.FunctionToTest{
CallThisFunction: callThisFunctionSvc,
}
actualResult, actualError := svc.FunctionToTest(context.Background(), tc.id)
if tc.wantEror {
require.Error(t, actualError, tc.expectedError)
} else {
require.NoError(t, actualError)
}
require.Equal(t, tc.expectedResult, actualResult)
}
}
This is the error it gives
=== RUN Test_GenerateDocument
--- FAIL: Test_GenerateDocument (0.00s)
panic:
assert: mock: I don't know what to return because the method call was unexpected.
Either do Mock.On("CallThisFunction").Return(...) first, or remove the GetTemplates() call.
This method was unexpected:
CallThisFunction(*context.emptyCtx,sqkit.Where)
0: (*context.emptyCtx)(0xc0000a4010)
1: sqkit.Where{"id = 1"}
Usually, when I encountered an error like this, it's because I haven't defined the return values of the function calls inside the function I wanna test. But this time I've created it, but it somehow can't read the return. Any idea why?
The error indicates you called CallThisFuncion with params (context.Context, sqkit.Where), but your example is using and setting the expectation for (context.Context, []sqkit.Option). The example with mock.Anything should work, but I believe it's failing because of the context. You'll need to set the expectation with the same context you're passing down. If FunctionToTest is going to be altering the context, I believe you'll need to use mock.Anything instead.
func Test_FunctionToTest(t *testing.T) {
testCase := []struct {
name string
id int64
onCallThisFunctionMock func(context.Context, *mocks.CallThisFunctionSvc)
expectedResult string
wantError bool
expectedError error
}{
{
name: "Success",
id: 1,
onCallThisFunctionMock: func(ctx context.Context, callThisFunctionSvc *mocks.CallThisFunctionSvc) {
args := []sqkit.SelectOption{
sqkit.Where{
fmt.Sprintf("id = %d", 1},
},
}
callThisFunctionSvc.On("CallThisFunction", ctx, args).Return("Success", nil)
}
}
}
for _, tc := range testCases {
var callThisFunctionSvc = new(mocks.CallThisFunctionSvc)
var ctx = context.Background()
tc.onCallThisFunctionMock(ctx, callThisFunctionSvc)
svc := &svc.FunctionToTest{
CallThisFunction: callThisFunctionSvc,
}
actualResult, actualError := svc.FunctionToTest(ctx, tc.id)
if tc.wantEror {
require.Error(t, actualError, tc.expectedError)
} else {
require.NoError(t, actualError)
}
require.Equal(t, tc.expectedResult, actualResult)
}
}
If you want to ensure a context.Context was passed as the first parameter but don't care what context, you could use AnythingOfType.
callThisFunctionSvc.On("CallThisFunction", mock.AnythingOfType("context.Context"), args).Return("Success", nil)

How to solve this syncing issue in Unit testing golang?

Here when I'm printing the activity it is printing them in the order they are getting created but at the time of assertion it is picking up activities in random order and also it is picking expected values in random order. The api that I'm calling mastercontroller have some goroutines and could take time maybe that is the reason but not sure.
for i, param := range params {
gin.SetMode(gin.TestMode)
w := httptest.NewRecorder()
ctx, _ := gin.CreateTestContext(w)
ctx.Request = &http.Request{
URL: &url.URL{},
Header: make(http.Header),
}
MockJsonPost(ctx, param)
MasterController(ctx)
time.Sleep(3 * time.Second)
fmt.Println("response body", string(w.Body.Bytes()))
fmt.Println("status", w.Code)
// var activity *activity.Activity
activity, err := activityController.GetLastActivity(nil)
//tx.Raw("select * from activity order by id desc limit 1").Find(&activity)
if err != nil {
fmt.Println("No activity found")
}
activityJson, err := activity.ToJsonTest()
if err != nil {
fmt.Println("error converting in json")
}
fmt.Printf("reponse activity %+v", string(activityJson))
assert.EqualValues(t, string(expected[i]), string(activityJson))
}
func MockJsonPost(c *gin.Context, content interface{}) {
c.Request.Method = "POST" // or PUT
c.Request.Header.Set("Content-Type", "application/json")
jsonbytes, err := json.Marshal(content)
if err != nil {
panic(err)
}
// the request body must be an io.ReadCloser
// the bytes buffer though doesn't implement io.Closer,
// so you wrap it in a no-op closer
c.Request.Body = io.NopCloser(bytes.NewBuffer(jsonbytes))
}

How do I mock a function that write result to it's argument in Go

I am writing unit test in golang by https://github.com/stretchr/testify
Suppose I have a method below,
func DoSomething(result interface{}) error {
// write some data to result
return nil
}
so the caller can call DoSomething as following
result := &SomeStruct{}
err := DoSomething(result)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("The result is", result)
}
Now I know how to use testify or some other mocking tools to mock the returns value (it's err here) by something like
mockObj.On("DoSomething", mock.Anything).Return(errors.New("mock error"))
My question is "how do i mock the result argument" in this kind of scenario?
Since result is not a return value but a argument, the caller calls it by passing a pointer of a struct, and the function modify it.
You can use the (*Call).Run method:
Run sets a handler to be called before returning. It can be used when
mocking a method (such as an unmarshaler) that takes a pointer to a
struct and sets properties in such struct
Example:
mockObj.On("Unmarshal", mock.AnythingOfType("*map[string]interface{}")).Return().Run(func(args Arguments) {
arg := args.Get(0).(*map[string]interface{})
arg["foo"] = "bar"
})
As #bikbah said, here is an example:
services/message.go:
type messageService struct {
HttpClient http.Client
BaseURL string
}
func (m *messageService) MarkAllMessages(accesstoken string) []*model.MarkedMessage {
endpoint := m.BaseURL + "/message/mark_all"
var res model.MarkAllMessagesResponse
if err := m.HttpClient.Post(endpoint, &MarkAllMessagesRequestPayload{Accesstoken: accesstoken}, &res); err != nil {
fmt.Println(err)
return res.MarkedMsgs
}
return res.MarkedMsgs
}
We passes res to the m.HttpClient.Post method. In this method, the res will be populated with json.unmarshal method.
mocks/http.go:
package mocks
import (
"io"
"github.com/stretchr/testify/mock"
)
type MockedHttp struct {
mock.Mock
}
func (m *MockedHttp) Get(url string, data interface{}) error {
args := m.Called(url, data)
return args.Error(0)
}
func (m *MockedHttp) Post(url string, body interface{}, data interface{}) error {
args := m.Called(url, body, data)
return args.Error(0)
}
services/message_test.go:
package services_test
import (
"errors"
"reflect"
"strconv"
"testing"
"github.com/stretchr/testify/mock"
"github.com/mrdulin/gqlgen-cnode/graph/model"
"github.com/mrdulin/gqlgen-cnode/services"
"github.com/mrdulin/gqlgen-cnode/mocks"
)
const (
baseURL string = "http://localhost/api/v1"
accesstoken string = "123"
)
func TestMessageService_MarkAllMessages(t *testing.T) {
t.Run("should mark all messaages", func(t *testing.T) {
testHttp := new(mocks.MockedHttp)
var res model.MarkAllMessagesResponse
var markedMsgs []*model.MarkedMessage
for i := 1; i <= 3; i++ {
markedMsgs = append(markedMsgs, &model.MarkedMessage{ID: strconv.Itoa(i)})
}
postBody := services.MarkAllMessagesRequestPayload{Accesstoken: accesstoken}
testHttp.On("Post", baseURL+"/message/mark_all", &postBody, &res).Return(nil).Run(func(args mock.Arguments) {
arg := args.Get(2).(*model.MarkAllMessagesResponse)
arg.MarkedMsgs = markedMsgs
})
service := services.NewMessageService(testHttp, baseURL)
got := service.MarkAllMessages(accesstoken)
want := markedMsgs
testHttp.AssertExpectations(t)
if !reflect.DeepEqual(got, want) {
t.Errorf("got wrong return value. got: %#v, want: %#v", got, want)
}
})
t.Run("should print error and return empty slice", func(t *testing.T) {
var res model.MarkAllMessagesResponse
testHttp := new(mocks.MockedHttp)
postBody := services.MarkAllMessagesRequestPayload{Accesstoken: accesstoken}
testHttp.On("Post", baseURL+"/message/mark_all", &postBody, &res).Return(errors.New("network"))
service := services.NewMessageService(testHttp, baseURL)
got := service.MarkAllMessages(accesstoken)
var want []*model.MarkedMessage
testHttp.AssertExpectations(t)
if !reflect.DeepEqual(got, want) {
t.Errorf("got wrong return value. got: %#v, want: %#v", got, want)
}
})
}
In the unit test case, we populated the res in #Call.Run method and assigned the return value(res.MarkedMsgs) of service.MarkAllMessages(accesstoken) to got variable.
unit test result and coverage:
=== RUN TestMessageService_MarkAllMessages
--- PASS: TestMessageService_MarkAllMessages (0.00s)
=== RUN TestMessageService_MarkAllMessages/should_mark_all_messaages
TestMessageService_MarkAllMessages/should_mark_all_messaages: message_test.go:39: PASS: Post(string,*services.MarkAllMessagesRequestPayload,*model.MarkAllMessagesResponse)
--- PASS: TestMessageService_MarkAllMessages/should_mark_all_messaages (0.00s)
=== RUN TestMessageService_MarkAllMessages/should_print_error_and_return_empty_slice
network
TestMessageService_MarkAllMessages/should_print_error_and_return_empty_slice: message_test.go:53: PASS: Post(string,*services.MarkAllMessagesRequestPayload,*model.MarkAllMessagesResponse)
--- PASS: TestMessageService_MarkAllMessages/should_print_error_and_return_empty_slice (0.00s)
PASS
coverage: 5.6% of statements in ../../gqlgen-cnode/...
Process finished with exit code 0
I highly recommend to get familiar with the gomock framework and develop towards interfaces. What you need would look something like this.
// SetArg does the job
myObj.EXPECT().DoSomething(gomock.Any()).SetArg(0, <value you want to r eturn>).Return(nil)
https://github.com/golang/mock#building-mocks

Go webcrawler hangs after checking about 2000 urls

I have a program to check whether keywords are on a web page. But after checking 1000-3000 urls, it hangs. There is no output, it does not exit, and the number of tcp connections is zero. I don't know why there are no new connections.
Would you give me some advice how to debug it?
type requestReturn struct {
url string
status bool
}
var timeout = time.Duration(800 * time.Millisecond)
func checkUrls(urls []string, kws string, threadLimit int) []string {
limitChan := make(chan int, threadLimit)
ok := make(chan requestReturn, 1)
var result []string
i := 0
for ; i < threadLimit; i++ {
go func(u string) {
request(u, limitChan, ok, kws)
}(urls[i])
}
for o := range ok {
if o.status {
result = append(result, o.url)
log.Printf("success %s,remain %d", o.url, len(urls)-i)
} else {
log.Printf("fail %s,remain %d", o.url, len(urls)-i)
}
if i < len(urls) {
go func(u string) {
request(u, limitChan, ok, kws)
}(urls[i])
i++
}
}
close(limitChan)
return result
}
func dialTimeout(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
func request(url string, threadLimit chan int, ok chan requestReturn, kws string) {
threadLimit <- 1
log.Printf("%s, start...", url)
//startTime := time.Now().UnixNano()
rr := requestReturn{url: url}
transport := http.Transport{
Dial: dialTimeout,
DisableKeepAlives: true,
}
client := http.Client{
Transport: &transport,
Timeout: time.Duration(15 * time.Second),
}
resp, e := client.Get(url)
if e != nil {
log.Printf("%q", e)
rr.status = false
return
}
if resp.StatusCode == 200 {
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Printf("%q", err)
rr.status = false
return
}
content := bytes.NewBuffer(body).String()
matched, err1 := regexp.MatchString(kws, content)
if err1 != nil {
log.Printf("%q", err1)
rr.status = false
} else if matched {
rr.status = true
log.Println(rr.url)
} else {
rr.status = false
}
} else {
rr.status = false
}
defer (func() {
resp.Body.Close()
ok <- rr
//processed := float32(time.Now().UnixNano()-startTime) / 1e9
//log.Printf("%s, status:%t,time:%.3fs", rr.url, rr.status, processed)
<-threadLimit
})()
}
You seem to be using two forms of concurrency control in this code, and both have problems.
You've got limitChan, which looks like it is being used as a semaphore (request sends a value at its start, and receives a value in a defer in that function). But checkUrls is also trying to make sure it only has threadLimit goroutines running at once (by spawning that number first up, and only spawning more when one reports its results on the ok channel). Only one of these should be necessary to limit the concurrency.
Both methods fail due to the way the defer is set up in request. There are a number of return statements that occur before defer, so it is possible for the function to complete without sending the result to the ok channel, and without freeing up its slot in limitChan. After a sufficient number of errors, checkUrls will stop spawning new goroutines and you'll see your hang.
The fix is to place the defer statement before any of the return statements so you know it will always be run. Something like this:
func request(url string, threadLimit chan int, ok chan requestReturn, kws string) {
threadLimit <- 1
rr := requestReturn{url: url}
var resp *http.Response
defer func() {
if resp != nil {
resp.Body.Close()
}
ok <- rr
<-threadLimit
}()
...
}

Why does the function return early?

I've just started learning go, and have been working through the tour. The last exercise is to edit a web crawler to crawl in parallel and without repeats.
Here is the link to the exercise: http://tour.golang.org/#70
Here is the code. I only changed the crawl and the main function. So I'll just post those to keep it neat.
// Crawl uses fetcher to recursively crawl
// pages starting with url, to a maximum of depth.
var used = make(map[string]bool)
var urlchan = make(chan string)
func Crawl(url string, depth int, fetcher Fetcher) {
// TODO: Fetch URLs in parallel.
// Done: Don't fetch the same URL twice.
// This implementation doesn't do either:
done := make(chan bool)
if depth <= 0 {
return
}
body, urls, err := fetcher.Fetch(url)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("\nfound: %s %q\n\n", url, body)
go func() {
for _, i := range urls {
urlchan <- i
}
done <- true
}()
for u := range urlchan {
if used[u] == false {
used[u] = true
go Crawl(u, depth-1, fetcher)
}
if <-done == true {
break
}
}
return
}
func main() {
used["http://golang.org/"] = true
Crawl("http://golang.org/", 4, fetcher)
}
The problem is that when I run the program the crawler stops after printing
not found: http://golang.org/cmd/
This only happens when I try to make the program run in parallel. If I have it run linearly then all the urls are found correctly.
Note: If I am not doing this right (parallelism I mean) then I apologise.
Be careful with goroutine.
Because when the main routine, or main() func, returns, all others go routine would be killed immediately.
Your Crawl() seems like recursive, however it is not, which means it would return immediately, not awaiting for other Crawl() routines. And you know that if the first Crawl(), called by main(), returns, the main() func regards its mission fulfilled.
What you could do is to let main() func wait until the last Crawl() returns. The sync package, or a chan would help.
You could probably take a look at the last solution of this, which I did months ago:
var store map[string]bool
func Krawl(url string, fetcher Fetcher, Urls chan []string) {
body, urls, err := fetcher.Fetch(url)
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("found: %s %q\n", url, body)
}
Urls <- urls
}
func Crawl(url string, depth int, fetcher Fetcher) {
Urls := make(chan []string)
go Krawl(url, fetcher, Urls)
band := 1
store[url] = true // init for level 0 done
for i := 0; i < depth; i++ {
for band > 0 {
band--
next := <- Urls
for _, url := range next {
if _, done := store[url] ; !done {
store[url] = true
band++
go Krawl(url, fetcher, Urls)
}
}
}
}
return
}
func main() {
store = make(map[string]bool)
Crawl("http://golang.org/", 4, fetcher)
}