Go times out on sleep but not on busy-wait - concurrency

In Go, I can use time.After to time out a sleeping function, but I can't do the same to a function that is busy-waiting (or working). The following code returns timed out after one second, and then hangs.
package main
import (
"fmt"
"time"
)
func main() {
sleepChan := make(chan int)
go sleep(sleepChan)
select {
case sleepResult := <-sleepChan:
fmt.Println(sleepResult)
case <-time.After(time.Second):
fmt.Println("timed out")
}
busyChan := make(chan int)
go busyWait(busyChan)
select {
case busyResult := <-busyChan:
fmt.Println(busyResult)
case <-time.After(time.Second):
fmt.Println("timed out")
}
}
func sleep(c chan<- int) {
time.Sleep(10 * time.Second)
c <- 0
}
func busyWait(c chan<- int) {
for {
}
c <- 0
}
Why doesn't the timeout fire in the second case, and what alternative do I need to use to interrupt working goroutines?

The for {} statement is an infinite loop which monopolizes a single processor. Set runtime.GOMAXPROCS to 2 or more to allow the timer to run.
For example,
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.GOMAXPROCS(0))
runtime.GOMAXPROCS(runtime.NumCPU())
fmt.Println(runtime.GOMAXPROCS(0))
sleepChan := make(chan int)
go sleep(sleepChan)
select {
case sleepResult := <-sleepChan:
fmt.Println(sleepResult)
case <-time.After(time.Second):
fmt.Println("timed out")
}
busyChan := make(chan int)
go busyWait(busyChan)
select {
case busyResult := <-busyChan:
fmt.Println(busyResult)
case <-time.After(time.Second):
fmt.Println("timed out")
}
}
func sleep(c chan<- int) {
time.Sleep(10 * time.Second)
c <- 0
}
func busyWait(c chan<- int) {
for {
}
c <- 0
}
Output (4 CPU processor):
1
4
timed out
timed out

Related

Test to check if a function didn't run?

So I'm new to testing in general and I'm stuck trying to write a test for a function that triggers another function. This is what I have so far but it's kind of backwards and blocks forever if the function doesn't run:
var cha = make(chan bool, 1)
func TestFd(t *testing.T) {
c := &fd.Fdcount{Interval: 1, MaxFiles: 1}
c.Start(trigger)
if <- cha {
}
}
func trigger(i int) {
cha <- true
}
c.Start will trigger the trigger() function when certain criteria is met. It tests whether or not the criteria is met every 1 second.
The error case is when the function fails to run. Is there a way to test for this or is there a way to use the testing package to test for success (e.g. t.Pass())?
If c.Start is synchronous, you can simply pass a function that sets a value in the test case's scope, and then test against that value. Condider the functionCalled variable in the example below that is set by the trigger function (playground):
func TestFd(t *testing.T) {
functionCalled := false
trigger := func(i int) {
functionCalled = true;
}
c := &fd.Fdcount{Interval: 1, MaxFiles: 1}
c.Start(trigger)
if !functionCalled {
t.FatalF("function was not called")
}
}
If c.Start is asynchronous, you can use a select statement to implement a timeout that will fail the test when the passed function was not called within a given time frame (playground):
func TestFd(t *testing.T) {
functionCalled := make(chan bool)
timeoutSeconds := 1 * time.Second
trigger := func(i int) {
functionCalled <- true
}
timeout := time.After(timeoutSeconds)
c := &SomeStruct{}
c.Start(trigger)
select {
case <- functionCalled:
t.Logf("function was called")
case <- timeout:
t.Fatalf("function was not called within timeout")
}
}

Golang unit testing user input

I am trying to learn go with a TDD mindset. I am stuck getting my head wrapped around testing.
In the example below, I am prompting a user for input, doing a little validation and printing the results. I wrote a test for it (which is passing) however I don't feel like it is hitting the validation portion, so I am doing something wrong. Any advice would be appreciated.
https://play.golang.org/p/FDpbof9Y20
package main
import (
"bufio"
"fmt"
"io"
"os"
"regexp"
"strings"
)
func main() {
response := askQuestion("What is your name?")
fmt.Printf("Hello %s\n",response)
}
func askQuestion(question string) string {
reader := bufio.NewReader(os.Stdin)
answer := ""
for {
fmt.Printf("%s\n", question)
input, err := reader.ReadString('\n')
if err != nil {
if err != io.EOF {
panic(err)
}
break
}
if regexp.MustCompile(`[A-Z]{5}`).MatchString(strings.TrimSpace(input)) == true {
answer = strings.TrimSpace(input)
fmt.Printf("You entered %s\n", answer)
break
} else {
fmt.Printf("\033[31mYou must enter only 5 upper case letters.\n\033[0m")
continue
}
}
return answer
}
https://play.golang.org/p/WcI4CRfle5
package main
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"os"
"reflect"
"strings"
"testing"
)
func TestAskQuestion(t *testing.T) {
expected := "foo"
entered := "foo"
askQuestion("What is your last name?")
oldStdout := os.Stdout
r, w, _ := os.Pipe()
os.Stdout = w
fmt.Println(entered)
outC := make(chan string)
go func() {
var buf bytes.Buffer
io.Copy(&buf, r)
outC <- buf.String()
}()
w.Close()
os.Stdout = oldStdout
out := strings.TrimSpace(<-outC)
b, _ := ioutil.ReadAll(os.Stdin)
t.Log(string(b))
if !reflect.DeepEqual(expected, out) {
t.Fatalf("Test Status Failure Issue. Got: '%v' expected %s", out, expected)
}
}
Go's tests need to live in files which are named xyz_test.go, so the playground is not the right place to familiarize yourself with the unit testing feature.
If you have go installed locally, run the command go help test, to get a very brief introduction.

Go webcrawler hangs after checking about 2000 urls

I have a program to check whether keywords are on a web page. But after checking 1000-3000 urls, it hangs. There is no output, it does not exit, and the number of tcp connections is zero. I don't know why there are no new connections.
Would you give me some advice how to debug it?
type requestReturn struct {
url string
status bool
}
var timeout = time.Duration(800 * time.Millisecond)
func checkUrls(urls []string, kws string, threadLimit int) []string {
limitChan := make(chan int, threadLimit)
ok := make(chan requestReturn, 1)
var result []string
i := 0
for ; i < threadLimit; i++ {
go func(u string) {
request(u, limitChan, ok, kws)
}(urls[i])
}
for o := range ok {
if o.status {
result = append(result, o.url)
log.Printf("success %s,remain %d", o.url, len(urls)-i)
} else {
log.Printf("fail %s,remain %d", o.url, len(urls)-i)
}
if i < len(urls) {
go func(u string) {
request(u, limitChan, ok, kws)
}(urls[i])
i++
}
}
close(limitChan)
return result
}
func dialTimeout(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
func request(url string, threadLimit chan int, ok chan requestReturn, kws string) {
threadLimit <- 1
log.Printf("%s, start...", url)
//startTime := time.Now().UnixNano()
rr := requestReturn{url: url}
transport := http.Transport{
Dial: dialTimeout,
DisableKeepAlives: true,
}
client := http.Client{
Transport: &transport,
Timeout: time.Duration(15 * time.Second),
}
resp, e := client.Get(url)
if e != nil {
log.Printf("%q", e)
rr.status = false
return
}
if resp.StatusCode == 200 {
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Printf("%q", err)
rr.status = false
return
}
content := bytes.NewBuffer(body).String()
matched, err1 := regexp.MatchString(kws, content)
if err1 != nil {
log.Printf("%q", err1)
rr.status = false
} else if matched {
rr.status = true
log.Println(rr.url)
} else {
rr.status = false
}
} else {
rr.status = false
}
defer (func() {
resp.Body.Close()
ok <- rr
//processed := float32(time.Now().UnixNano()-startTime) / 1e9
//log.Printf("%s, status:%t,time:%.3fs", rr.url, rr.status, processed)
<-threadLimit
})()
}
You seem to be using two forms of concurrency control in this code, and both have problems.
You've got limitChan, which looks like it is being used as a semaphore (request sends a value at its start, and receives a value in a defer in that function). But checkUrls is also trying to make sure it only has threadLimit goroutines running at once (by spawning that number first up, and only spawning more when one reports its results on the ok channel). Only one of these should be necessary to limit the concurrency.
Both methods fail due to the way the defer is set up in request. There are a number of return statements that occur before defer, so it is possible for the function to complete without sending the result to the ok channel, and without freeing up its slot in limitChan. After a sufficient number of errors, checkUrls will stop spawning new goroutines and you'll see your hang.
The fix is to place the defer statement before any of the return statements so you know it will always be run. Something like this:
func request(url string, threadLimit chan int, ok chan requestReturn, kws string) {
threadLimit <- 1
rr := requestReturn{url: url}
var resp *http.Response
defer func() {
if resp != nil {
resp.Body.Close()
}
ok <- rr
<-threadLimit
}()
...
}

Regex end modifier not returning results in a Go program

I have a simple program in Go to aid in learning regular expressions. It runs in an infinite loop and has 2 channels, one which is used to provide input (input contains regex pattern and subject), and the second one, which provides the output.
usage: main.exe (cat)+ catcatdog
However there is propably something wrong in the code, as i can't seem to get any results with the $ modifier.
For example, i expect "cat" output from
main.exe cat$ cat\ndog
yet receive zero results.
Code:
package main
import (
"fmt"
"regexp"
"bufio"
"os"
"strings"
)
type RegexRequest struct {
regex string
subject string
}
func main() {
regexRequests := make(chan *RegexRequest)
defer close(regexRequests)
regexAnswers, err := createResolver(regexRequests)
defer close(regexAnswers)
if(err != nil) { // TODO: Panics when exited via ctrl+c
panic(err)
}
interact(regexRequests, regexAnswers)
}
func interact(regexRequests chan *RegexRequest, regexAnswers chan []string) {
for {
fmt.Println("Enter regex and subject: ")
reader := bufio.NewReader(os.Stdin)
line, err := reader.ReadString('\n')
if(err != nil) {
panic(err)
}
regAndString := strings.SplitN(line, " ", 2);
if len(regAndString) != 2 {
fmt.Println("Invalid input, expected [regex][space][subject]")
continue
}
regexRequests <- &RegexRequest{ regAndString[0], regAndString[1] }
result := <- regexAnswers
var filteredResult []string
for _, element := range result {
if(element != "") {
filteredResult = append(filteredResult, element)
} else {
filteredResult = append(filteredResult, "EMPTY");
}
}
fmt.Println(strings.Join(filteredResult, " "))
}
}
func createResolver(inputChan chan *RegexRequest)(outputChan chan []string, err error) {
if(cap(inputChan) > 0) {
return nil, fmt.Errorf("Expected an unbuffered channel")
}
outputChan = make(chan []string)
err = nil
go func() {
for {
var regReq *RegexRequest= (<- inputChan);
var regex *regexp.Regexp = regexp.MustCompile(regReq.regex)
outputChan <- regex.FindAllString(regReq.subject, -1)
}
}()
return
}
Check your regex pattern. For example,
Enter regex and subject:
cat$ cat\ndog
Enter regex and subject:
^cat cat\ndog
cat

Why does the function return early?

I've just started learning go, and have been working through the tour. The last exercise is to edit a web crawler to crawl in parallel and without repeats.
Here is the link to the exercise: http://tour.golang.org/#70
Here is the code. I only changed the crawl and the main function. So I'll just post those to keep it neat.
// Crawl uses fetcher to recursively crawl
// pages starting with url, to a maximum of depth.
var used = make(map[string]bool)
var urlchan = make(chan string)
func Crawl(url string, depth int, fetcher Fetcher) {
// TODO: Fetch URLs in parallel.
// Done: Don't fetch the same URL twice.
// This implementation doesn't do either:
done := make(chan bool)
if depth <= 0 {
return
}
body, urls, err := fetcher.Fetch(url)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("\nfound: %s %q\n\n", url, body)
go func() {
for _, i := range urls {
urlchan <- i
}
done <- true
}()
for u := range urlchan {
if used[u] == false {
used[u] = true
go Crawl(u, depth-1, fetcher)
}
if <-done == true {
break
}
}
return
}
func main() {
used["http://golang.org/"] = true
Crawl("http://golang.org/", 4, fetcher)
}
The problem is that when I run the program the crawler stops after printing
not found: http://golang.org/cmd/
This only happens when I try to make the program run in parallel. If I have it run linearly then all the urls are found correctly.
Note: If I am not doing this right (parallelism I mean) then I apologise.
Be careful with goroutine.
Because when the main routine, or main() func, returns, all others go routine would be killed immediately.
Your Crawl() seems like recursive, however it is not, which means it would return immediately, not awaiting for other Crawl() routines. And you know that if the first Crawl(), called by main(), returns, the main() func regards its mission fulfilled.
What you could do is to let main() func wait until the last Crawl() returns. The sync package, or a chan would help.
You could probably take a look at the last solution of this, which I did months ago:
var store map[string]bool
func Krawl(url string, fetcher Fetcher, Urls chan []string) {
body, urls, err := fetcher.Fetch(url)
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("found: %s %q\n", url, body)
}
Urls <- urls
}
func Crawl(url string, depth int, fetcher Fetcher) {
Urls := make(chan []string)
go Krawl(url, fetcher, Urls)
band := 1
store[url] = true // init for level 0 done
for i := 0; i < depth; i++ {
for band > 0 {
band--
next := <- Urls
for _, url := range next {
if _, done := store[url] ; !done {
store[url] = true
band++
go Krawl(url, fetcher, Urls)
}
}
}
}
return
}
func main() {
store = make(map[string]bool)
Crawl("http://golang.org/", 4, fetcher)
}