Related
I use Testify to create a unit test for my golang app. I need to create a unit test for this function where it calls a variadic function (function with trailing arguments). I encountered an error when I test it. I'm actually not sure if the error is because of the trailing argument itself or not, but I feel like there's something wrong with the mock.
// svc/callThisFunction.go
// data type of args is []sqkit.SelectOption
func CallThisFunction(ctx context.Context, args ...sqkit.SelectFunctiom) (result string, err error) {
return result, nil
}
// svc/functionToTest.go
// This is the function that I wanna test
func FunctionToTest(ctx context.Context, id int64) (result string, err error) {
args := []sqkit.SelectOption{
sqkit.Where{
fmt.Sprintf("id = %d", id),
},
}
newResult, err := callThisFunctionService.CallThisFunction(ctx, args)
if err != nil {
return newResult, err
}
return newResult, nil
}
// svc/functionToTest_test.go
func Test_FunctionToTest(t *testing.T) {
testCase := []struct {
name string
id int64
onCallThisFunctionMock func(callThisFunctionSvc *mocks.CallThisFunctionSvc)
expectedResult string
wantError bool
expectedError error
}{
{
name: "Success",
id: 1,
onCallThisFunctionMock: func(callThisFunctionSvc *mocks.CallThisFunctionSvc) {
// NOTE: I've created 2 different versions (used separately, not at the same), using mock.Anything() and using actual arguments
// Both of these give the same errors
// Using actual arguments
args := []sqkit.SelectOption{
sqkit.Where{
fmt.Sprintf("id = %d", 1},
},
}
callThisFunctionSvc.On("CallThisFunction", context.Background(), args).Return("Success", nil)
// Using mock.Anything
callThisFunctionSvc.On("CallThisFunction", context.Background(), mock.Anything).Return("Success", nil)
}
}
}
for _, tc := range testCases {
var callThisFunctionSvc = new(mocks.CallThisFunctionSvc)
tc.onCallThisFunctionMock(callThisFunctionSvc)
svc := &svc.FunctionToTest{
CallThisFunction: callThisFunctionSvc,
}
actualResult, actualError := svc.FunctionToTest(context.Background(), tc.id)
if tc.wantEror {
require.Error(t, actualError, tc.expectedError)
} else {
require.NoError(t, actualError)
}
require.Equal(t, tc.expectedResult, actualResult)
}
}
This is the error it gives
=== RUN Test_GenerateDocument
--- FAIL: Test_GenerateDocument (0.00s)
panic:
assert: mock: I don't know what to return because the method call was unexpected.
Either do Mock.On("CallThisFunction").Return(...) first, or remove the GetTemplates() call.
This method was unexpected:
CallThisFunction(*context.emptyCtx,sqkit.Where)
0: (*context.emptyCtx)(0xc0000a4010)
1: sqkit.Where{"id = 1"}
Usually, when I encountered an error like this, it's because I haven't defined the return values of the function calls inside the function I wanna test. But this time I've created it, but it somehow can't read the return. Any idea why?
The error indicates you called CallThisFuncion with params (context.Context, sqkit.Where), but your example is using and setting the expectation for (context.Context, []sqkit.Option). The example with mock.Anything should work, but I believe it's failing because of the context. You'll need to set the expectation with the same context you're passing down. If FunctionToTest is going to be altering the context, I believe you'll need to use mock.Anything instead.
func Test_FunctionToTest(t *testing.T) {
testCase := []struct {
name string
id int64
onCallThisFunctionMock func(context.Context, *mocks.CallThisFunctionSvc)
expectedResult string
wantError bool
expectedError error
}{
{
name: "Success",
id: 1,
onCallThisFunctionMock: func(ctx context.Context, callThisFunctionSvc *mocks.CallThisFunctionSvc) {
args := []sqkit.SelectOption{
sqkit.Where{
fmt.Sprintf("id = %d", 1},
},
}
callThisFunctionSvc.On("CallThisFunction", ctx, args).Return("Success", nil)
}
}
}
for _, tc := range testCases {
var callThisFunctionSvc = new(mocks.CallThisFunctionSvc)
var ctx = context.Background()
tc.onCallThisFunctionMock(ctx, callThisFunctionSvc)
svc := &svc.FunctionToTest{
CallThisFunction: callThisFunctionSvc,
}
actualResult, actualError := svc.FunctionToTest(ctx, tc.id)
if tc.wantEror {
require.Error(t, actualError, tc.expectedError)
} else {
require.NoError(t, actualError)
}
require.Equal(t, tc.expectedResult, actualResult)
}
}
If you want to ensure a context.Context was passed as the first parameter but don't care what context, you could use AnythingOfType.
callThisFunctionSvc.On("CallThisFunction", mock.AnythingOfType("context.Context"), args).Return("Success", nil)
I am writing unit test in golang by https://github.com/stretchr/testify
Suppose I have a method below,
func DoSomething(result interface{}) error {
// write some data to result
return nil
}
so the caller can call DoSomething as following
result := &SomeStruct{}
err := DoSomething(result)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("The result is", result)
}
Now I know how to use testify or some other mocking tools to mock the returns value (it's err here) by something like
mockObj.On("DoSomething", mock.Anything).Return(errors.New("mock error"))
My question is "how do i mock the result argument" in this kind of scenario?
Since result is not a return value but a argument, the caller calls it by passing a pointer of a struct, and the function modify it.
You can use the (*Call).Run method:
Run sets a handler to be called before returning. It can be used when
mocking a method (such as an unmarshaler) that takes a pointer to a
struct and sets properties in such struct
Example:
mockObj.On("Unmarshal", mock.AnythingOfType("*map[string]interface{}")).Return().Run(func(args Arguments) {
arg := args.Get(0).(*map[string]interface{})
arg["foo"] = "bar"
})
As #bikbah said, here is an example:
services/message.go:
type messageService struct {
HttpClient http.Client
BaseURL string
}
func (m *messageService) MarkAllMessages(accesstoken string) []*model.MarkedMessage {
endpoint := m.BaseURL + "/message/mark_all"
var res model.MarkAllMessagesResponse
if err := m.HttpClient.Post(endpoint, &MarkAllMessagesRequestPayload{Accesstoken: accesstoken}, &res); err != nil {
fmt.Println(err)
return res.MarkedMsgs
}
return res.MarkedMsgs
}
We passes res to the m.HttpClient.Post method. In this method, the res will be populated with json.unmarshal method.
mocks/http.go:
package mocks
import (
"io"
"github.com/stretchr/testify/mock"
)
type MockedHttp struct {
mock.Mock
}
func (m *MockedHttp) Get(url string, data interface{}) error {
args := m.Called(url, data)
return args.Error(0)
}
func (m *MockedHttp) Post(url string, body interface{}, data interface{}) error {
args := m.Called(url, body, data)
return args.Error(0)
}
services/message_test.go:
package services_test
import (
"errors"
"reflect"
"strconv"
"testing"
"github.com/stretchr/testify/mock"
"github.com/mrdulin/gqlgen-cnode/graph/model"
"github.com/mrdulin/gqlgen-cnode/services"
"github.com/mrdulin/gqlgen-cnode/mocks"
)
const (
baseURL string = "http://localhost/api/v1"
accesstoken string = "123"
)
func TestMessageService_MarkAllMessages(t *testing.T) {
t.Run("should mark all messaages", func(t *testing.T) {
testHttp := new(mocks.MockedHttp)
var res model.MarkAllMessagesResponse
var markedMsgs []*model.MarkedMessage
for i := 1; i <= 3; i++ {
markedMsgs = append(markedMsgs, &model.MarkedMessage{ID: strconv.Itoa(i)})
}
postBody := services.MarkAllMessagesRequestPayload{Accesstoken: accesstoken}
testHttp.On("Post", baseURL+"/message/mark_all", &postBody, &res).Return(nil).Run(func(args mock.Arguments) {
arg := args.Get(2).(*model.MarkAllMessagesResponse)
arg.MarkedMsgs = markedMsgs
})
service := services.NewMessageService(testHttp, baseURL)
got := service.MarkAllMessages(accesstoken)
want := markedMsgs
testHttp.AssertExpectations(t)
if !reflect.DeepEqual(got, want) {
t.Errorf("got wrong return value. got: %#v, want: %#v", got, want)
}
})
t.Run("should print error and return empty slice", func(t *testing.T) {
var res model.MarkAllMessagesResponse
testHttp := new(mocks.MockedHttp)
postBody := services.MarkAllMessagesRequestPayload{Accesstoken: accesstoken}
testHttp.On("Post", baseURL+"/message/mark_all", &postBody, &res).Return(errors.New("network"))
service := services.NewMessageService(testHttp, baseURL)
got := service.MarkAllMessages(accesstoken)
var want []*model.MarkedMessage
testHttp.AssertExpectations(t)
if !reflect.DeepEqual(got, want) {
t.Errorf("got wrong return value. got: %#v, want: %#v", got, want)
}
})
}
In the unit test case, we populated the res in #Call.Run method and assigned the return value(res.MarkedMsgs) of service.MarkAllMessages(accesstoken) to got variable.
unit test result and coverage:
=== RUN TestMessageService_MarkAllMessages
--- PASS: TestMessageService_MarkAllMessages (0.00s)
=== RUN TestMessageService_MarkAllMessages/should_mark_all_messaages
TestMessageService_MarkAllMessages/should_mark_all_messaages: message_test.go:39: PASS: Post(string,*services.MarkAllMessagesRequestPayload,*model.MarkAllMessagesResponse)
--- PASS: TestMessageService_MarkAllMessages/should_mark_all_messaages (0.00s)
=== RUN TestMessageService_MarkAllMessages/should_print_error_and_return_empty_slice
network
TestMessageService_MarkAllMessages/should_print_error_and_return_empty_slice: message_test.go:53: PASS: Post(string,*services.MarkAllMessagesRequestPayload,*model.MarkAllMessagesResponse)
--- PASS: TestMessageService_MarkAllMessages/should_print_error_and_return_empty_slice (0.00s)
PASS
coverage: 5.6% of statements in ../../gqlgen-cnode/...
Process finished with exit code 0
I highly recommend to get familiar with the gomock framework and develop towards interfaces. What you need would look something like this.
// SetArg does the job
myObj.EXPECT().DoSomething(gomock.Any()).SetArg(0, <value you want to r eturn>).Return(nil)
https://github.com/golang/mock#building-mocks
How can I do overall test setup processing which sets the stage for all the tests when using the testing package?
As an example in Nunit there is a [SetUp] attribute.
[TestFixture]
public class SuccessTests
{
[SetUp] public void Init()
{ /* Load test data */ }
}
Starting with Go 1.4 you can implement setup/teardown (no need to copy your functions before/after each test). The documentation is outlined here in the Main section:
TestMain runs in the main goroutine and can do whatever setup and
teardown is necessary around a call to m.Run. It should then call
os.Exit with the result of m.Run
It took me some time to figure out that this means that if a test contains a function func TestMain(m *testing.M) then this function will be called instead of running the test. And in this function I can define how the tests will run. For example I can implement global setup and teardown:
func TestMain(m *testing.M) {
setup()
code := m.Run()
shutdown()
os.Exit(code)
}
A couple of other examples can be found here.
The TestMain feature added to Go’s testing framework in the latest
release is a simple solution for several testing use cases. TestMain
provides a global hook to perform setup and shutdown, control the
testing environment, run different code in a child process, or check
for resources leaked by test code. Most packages will not need a
TestMain, but it is a welcome addition for those times when it is
needed.
This can be achieved by putting a init() function in the myfile_test.go file. This will be run before the init() function.
// myfile_test.go
package main
func init() {
/* load test data */
}
The myfile_test.init() will be called before the package init() function.
Given a simple function to unit test:
package math
func Sum(a, b int) int {
return a + b
}
You can test it with a setup function that returns teardown function. And after calling setup() you can make a deferred call to teardown().
package math
import "testing"
func setupTestCase(t *testing.T) func(t *testing.T) {
t.Log("setup test case")
return func(t *testing.T) {
t.Log("teardown test case")
}
}
func setupSubTest(t *testing.T) func(t *testing.T) {
t.Log("setup sub test")
return func(t *testing.T) {
t.Log("teardown sub test")
}
}
func TestAddition(t *testing.T) {
cases := []struct {
name string
a int
b int
expected int
}{
{"add", 2, 2, 4},
{"minus", 0, -2, -2},
{"zero", 0, 0, 0},
}
teardownTestCase := setupTestCase(t)
defer teardownTestCase(t)
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
teardownSubTest := setupSubTest(t)
defer teardownSubTest(t)
result := Sum(tc.a, tc.b)
if result != tc.expected {
t.Fatalf("expected sum %v, but got %v", tc.expected, result)
}
})
}
}
Go testing tool will report the logging statements in the shell console:
% go test -v
=== RUN TestAddition
=== RUN TestAddition/add
=== RUN TestAddition/minus
=== RUN TestAddition/zero
--- PASS: TestAddition (0.00s)
math_test.go:6: setup test case
--- PASS: TestAddition/add (0.00s)
math_test.go:13: setup sub test
math_test.go:15: teardown sub test
--- PASS: TestAddition/minus (0.00s)
math_test.go:13: setup sub test
math_test.go:15: teardown sub test
--- PASS: TestAddition/zero (0.00s)
math_test.go:13: setup sub test
math_test.go:15: teardown sub test
math_test.go:8: teardown test case
PASS
ok github.com/kare/go-unit-test-setup-teardown 0.010s
%
You can pass some additional parameters to setup/teardown with this approach.
The Go testing framework doesn't have anything equivalent to NUnit's SetUp attribute (marking a function to be called before each test in the suite). There are a few options though:
Simply call your SetUp function from each test where it is needed.
Use an extension to Go's testing framework that implements xUnit paradigms and concepts. Three strong options come to mind:
gocheck
testify
gunit
Each of these libraries encourage you to organize your tests into suites/fixtures similar to other xUnit frameworks, and will call the setup methods on the suite/fixture type before each of the Test* methods.
With the following template, you can make a one line call in each TestMethod that does both setup and tear-down.
func setupTest() func() {
// Setup code here
// tear down later
return func() {
// tear-down code here
}
}
func TestMethod(t *testing.T) {
defer setupTest()()
// asserts, ensures, requires... here
}
Typically, tests in go aren't written in the same style as other languages. Often, there's relatively fewer test functions, but each contains a table-driven set of test cases. See this article written by one of the Go team.
With a table-driven test, you simply put any setup code before the loop that executes the individual test-cases specified in the table, and put any cleanup code afterwards.
If you still have shared setup code between test functions, you can extract the shared setup code into a function, and use a sync.Once if it's important that it's executed exactly once (or as another answer suggests, use init(), but this has the disadvantage that the setup will be done even if the test cases aren't run (perhaps because you've limited the test cases by using go test -run <regexp>.)
I'd say if you think you need shared setup between different tests that gets executed exactly once you should have a think if you really need it, and if a table-driven test wouldn't be better.
In case someone is looking an alternative of #BeforeEach (which runs before each test in a test file) and #AfterEach (which runs after test in a test file), here's a helper snippet.
func CreateForEach(setUp func(), tearDown func()) func(func()) {
return func(testFunc func()) {
setUp()
testFunc()
tearDown()
}
}
You can use it like below with help of TestMain
var RunTest = CreateForEach(setUp, tearDown)
func setUp() {
// SETUP METHOD WHICH IS REQUIRED TO RUN FOR EACH TEST METHOD
// your code here
}
func tearDown() {
// TEAR DOWN METHOD WHICH IS REQUIRED TO RUN FOR EACH TEST METHOD
// your code here
}
fun TestSample(t *testing.T) {
RunTest(func() {
// YOUR CODE HERE
})
}
also you can check: go-beforeeach
You can use the testing package for test setup - which will set the stage for all tests and teardown - which will cleanup the stage after tests have run.
The below calculates the area of a rectangle:
package main
import (
"errors"
"fmt"
)
func area(height float64, width float64) (float64, error) {
if height == width {
fmt.Printf("Rectangle with given dimension is a square. Area is: %f\n", height * width)
return height * width, nil
} else if height <= 0 || width <= 0 {
return 0, errors.New("Both dimensions need to be positive")
} else {
fmt.Printf("Area is: %f\n", height * width)
return height * width, nil
}
}
func main() {
var length float64 = 4.0
var breadth float64 = 5.0
area(length, breadth)
}
This is the implementation for test setup and teardown using TestMain as Salvador Dali's explains. (Note that since v1.15 the TestMain function is no longer required to call os.Exit [ref])
package main
import (
"log"
"testing"
)
var length float64
var breadth float64
func TestMain(m *testing.M) {
setup()
m.Run()
teardown()
}
func setup() {
length = 2.0
breadth = 3.0
log.Println("\n-----Setup complete-----")
}
func teardown() {
length = 0
breadth = 0
log.Println("\n----Teardown complete----")
}
func TestAreaOfRectangle(t *testing.T) {
val, err := area(length, breadth)
want := 6.0
if val != want && err != nil {
t.Errorf("Got %f and %v; Want %f and %v", val, err, want, nil)
}
}
And this is the implementation for test setup and teardown using sub-tests:
package main
import "testing"
func TestInvalidRectangle(t *testing.T) {
// setup
var length float64 = -2.0
var breadth float64 = 3.0
t.Log("\n-----Setup complete for invalid rectangle-----")
// sub-tests
t.Run("invalid dimensions return value", func(t *testing.T) {
val, _ := area(length, breadth)
area := 0.0
if val != area {
t.Errorf("Got %f; Want %f", val, area)
}
})
t.Run("invalid dimensions message", func(t *testing.T) {
_, err := area(length, breadth)
want := "Both dimensions need to be positive"
if err.Error() != want {
t.Errorf("Got error: %v; Want error: %v", err.Error(), want)
}
})
// teardown
t.Cleanup(func(){
length = 0
breadth = 0
t.Log("\n----Teardown complete for invalid rectangle----")
})
}
func TestRectangleIsSquare(t *testing.T) {
var length float64 = 3.0
var breadth float64 = 3.0
t.Log("\n-----Rectangle is square setup complete-----")
t.Run("valid dimensions value and message", func(t *testing.T) {
val, msg := area(length, breadth)
area := 9.0
if val != area && msg != nil {
t.Errorf("Got %f and %v; Want %f and %v", val, msg, area, nil)
}
})
t.Cleanup(func(){
length = 0
breadth = 0
t.Log("\n----Rectangle is square teardown Complete----")
})
}
Shameless plug, I created https://github.com/houqp/gtest to help solve exactly this problem.
Here is a quick example:
import (
"strings"
"testing"
"github.com/houqp/gtest"
)
type SampleTests struct{}
// Setup and Teardown are invoked per test group run
func (s *SampleTests) Setup(t *testing.T) {}
func (s *SampleTests) Teardown(t *testing.T) {}
// BeforeEach and AfterEach are invoked per test run
func (s *SampleTests) BeforeEach(t *testing.T) {}
func (s *SampleTests) AfterEach(t *testing.T) {}
func (s *SampleTests) SubTestCompare(t *testing.T) {
if 1 != 1 {
t.FailNow()
}
}
func (s *SampleTests) SubTestCheckPrefix(t *testing.T) {
if !strings.HasPrefix("abc", "ab") {
t.FailNow()
}
}
func TestSampleTests(t *testing.T) {
gtest.RunSubTests(t, &SampleTests{})
}
You can create as any test group you want within a package with each of them using a different set of setup/teardown routines.
Here is the minimal test suit framework to run subtests with
BeforeAll, ran before all subtests in a test
BeforeEach, ran before each subtest
AfterEach, ran after each subtest
AfterAll, ran after all subtests in a test
package suit
import "testing"
func Of(subTests *SubTests) *SubTests {
if subTests.AfterAll != nil {
subTests.T.Cleanup(subTests.AfterAll)
}
return subTests
}
type SubTests struct {
T *testing.T
BeforeEach func()
AfterEach func()
AfterAll func()
}
func (s *SubTests) TestIt(name string, f func(t *testing.T)) {
if s.AfterEach != nil {
defer s.AfterEach()
}
if s.BeforeEach != nil {
s.BeforeEach()
}
s.T.Run(name, f)
}
Usage
func TestFoo(t *testing.T) {
// BeforeAll setup goes here
s := suit.Of(&suit.SubTests{
T: t,
BeforeEach: func() { ... },
AfterEach: func() { ... },
AfterAll: func() { ... },
})
s.TestIt("returns true", func(t *testing.T) {
assert.Equal(t, 1, 1)
})
}
I have a program to check whether keywords are on a web page. But after checking 1000-3000 urls, it hangs. There is no output, it does not exit, and the number of tcp connections is zero. I don't know why there are no new connections.
Would you give me some advice how to debug it?
type requestReturn struct {
url string
status bool
}
var timeout = time.Duration(800 * time.Millisecond)
func checkUrls(urls []string, kws string, threadLimit int) []string {
limitChan := make(chan int, threadLimit)
ok := make(chan requestReturn, 1)
var result []string
i := 0
for ; i < threadLimit; i++ {
go func(u string) {
request(u, limitChan, ok, kws)
}(urls[i])
}
for o := range ok {
if o.status {
result = append(result, o.url)
log.Printf("success %s,remain %d", o.url, len(urls)-i)
} else {
log.Printf("fail %s,remain %d", o.url, len(urls)-i)
}
if i < len(urls) {
go func(u string) {
request(u, limitChan, ok, kws)
}(urls[i])
i++
}
}
close(limitChan)
return result
}
func dialTimeout(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
func request(url string, threadLimit chan int, ok chan requestReturn, kws string) {
threadLimit <- 1
log.Printf("%s, start...", url)
//startTime := time.Now().UnixNano()
rr := requestReturn{url: url}
transport := http.Transport{
Dial: dialTimeout,
DisableKeepAlives: true,
}
client := http.Client{
Transport: &transport,
Timeout: time.Duration(15 * time.Second),
}
resp, e := client.Get(url)
if e != nil {
log.Printf("%q", e)
rr.status = false
return
}
if resp.StatusCode == 200 {
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Printf("%q", err)
rr.status = false
return
}
content := bytes.NewBuffer(body).String()
matched, err1 := regexp.MatchString(kws, content)
if err1 != nil {
log.Printf("%q", err1)
rr.status = false
} else if matched {
rr.status = true
log.Println(rr.url)
} else {
rr.status = false
}
} else {
rr.status = false
}
defer (func() {
resp.Body.Close()
ok <- rr
//processed := float32(time.Now().UnixNano()-startTime) / 1e9
//log.Printf("%s, status:%t,time:%.3fs", rr.url, rr.status, processed)
<-threadLimit
})()
}
You seem to be using two forms of concurrency control in this code, and both have problems.
You've got limitChan, which looks like it is being used as a semaphore (request sends a value at its start, and receives a value in a defer in that function). But checkUrls is also trying to make sure it only has threadLimit goroutines running at once (by spawning that number first up, and only spawning more when one reports its results on the ok channel). Only one of these should be necessary to limit the concurrency.
Both methods fail due to the way the defer is set up in request. There are a number of return statements that occur before defer, so it is possible for the function to complete without sending the result to the ok channel, and without freeing up its slot in limitChan. After a sufficient number of errors, checkUrls will stop spawning new goroutines and you'll see your hang.
The fix is to place the defer statement before any of the return statements so you know it will always be run. Something like this:
func request(url string, threadLimit chan int, ok chan requestReturn, kws string) {
threadLimit <- 1
rr := requestReturn{url: url}
var resp *http.Response
defer func() {
if resp != nil {
resp.Body.Close()
}
ok <- rr
<-threadLimit
}()
...
}
I've just started learning go, and have been working through the tour. The last exercise is to edit a web crawler to crawl in parallel and without repeats.
Here is the link to the exercise: http://tour.golang.org/#70
Here is the code. I only changed the crawl and the main function. So I'll just post those to keep it neat.
// Crawl uses fetcher to recursively crawl
// pages starting with url, to a maximum of depth.
var used = make(map[string]bool)
var urlchan = make(chan string)
func Crawl(url string, depth int, fetcher Fetcher) {
// TODO: Fetch URLs in parallel.
// Done: Don't fetch the same URL twice.
// This implementation doesn't do either:
done := make(chan bool)
if depth <= 0 {
return
}
body, urls, err := fetcher.Fetch(url)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("\nfound: %s %q\n\n", url, body)
go func() {
for _, i := range urls {
urlchan <- i
}
done <- true
}()
for u := range urlchan {
if used[u] == false {
used[u] = true
go Crawl(u, depth-1, fetcher)
}
if <-done == true {
break
}
}
return
}
func main() {
used["http://golang.org/"] = true
Crawl("http://golang.org/", 4, fetcher)
}
The problem is that when I run the program the crawler stops after printing
not found: http://golang.org/cmd/
This only happens when I try to make the program run in parallel. If I have it run linearly then all the urls are found correctly.
Note: If I am not doing this right (parallelism I mean) then I apologise.
Be careful with goroutine.
Because when the main routine, or main() func, returns, all others go routine would be killed immediately.
Your Crawl() seems like recursive, however it is not, which means it would return immediately, not awaiting for other Crawl() routines. And you know that if the first Crawl(), called by main(), returns, the main() func regards its mission fulfilled.
What you could do is to let main() func wait until the last Crawl() returns. The sync package, or a chan would help.
You could probably take a look at the last solution of this, which I did months ago:
var store map[string]bool
func Krawl(url string, fetcher Fetcher, Urls chan []string) {
body, urls, err := fetcher.Fetch(url)
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("found: %s %q\n", url, body)
}
Urls <- urls
}
func Crawl(url string, depth int, fetcher Fetcher) {
Urls := make(chan []string)
go Krawl(url, fetcher, Urls)
band := 1
store[url] = true // init for level 0 done
for i := 0; i < depth; i++ {
for band > 0 {
band--
next := <- Urls
for _, url := range next {
if _, done := store[url] ; !done {
store[url] = true
band++
go Krawl(url, fetcher, Urls)
}
}
}
}
return
}
func main() {
store = make(map[string]bool)
Crawl("http://golang.org/", 4, fetcher)
}