Why does this concurrent HTTP client randomly crash? - concurrency

Question

I'm learning Go by writing an HTTP testing client like Apache's ab. The code below seems pretty straightforward: I create a configurable number of goroutines, each of which sends a portion of the overall HTTP requests and records the result. I iterate over the resultChan channel and inspect/record each result. This works find when the number of messages is, say, 100. When I increase the number of messages, however, it hangs and htop shows VIRT of 138G for the process.
Here's the code in question:
package main
import "net/http"
import "fmt"
import "time"
const (
SUCCESS = iota
TOTAL = iota
TIMEOUT = iota
ERROR = iota
)
type Result struct {
successful int
total int
timeouts int
errors int
duration time.Duration
}
func makeRequests(url string, messages int, resultChan chan<- *http.Response) {
for i := 0; i < messages; i++ {
resp, _ := http.Get(url)
if resp != nil {
resultChan <- resp
}
}
}
func deployRequests(url string, threads int, messages int) *Result {
results := new (Result)
resultChan := make(chan *http.Response)
start := time.Now()
defer func() {
fmt.Printf("%s\n", time.Since(start))
}()
for i := 0; i < threads; i++ {
go makeRequests(url, (messages/threads) + 1, resultChan)
}
for response := range resultChan {
if response.StatusCode != 200 {
results.errors += 1
} else {
results.successful += 1
}
results.total += 1
if results.total == messages {
return results
}
}
return results
}
func main () {
results := deployRequests("http://www.google.com", 10, 1000)
fmt.Printf("Total: %d\n", results.total)
fmt.Printf("Successful: %d\n", results.successful)
fmt.Printf("Error: %d\n", results.errors)
fmt.Printf("Timeouts: %d\n", results.timeouts)
fmt.Printf("%s", results.duration)
}
There are obviously some things missing or stupidly done (no timeout checking, channel is synchronous, etc) but I wanted to get the basic case working before fixing those. What is it about the program as written that causes so much memory allocation?
As far as I can tell, there are just 10 goroutines. If one is created per HTTP request, which would make sense, how does one perform operations that would create many goroutines in a loop? Or is the issue totally unrelated.

Answer 1

I think the sequence leading to the hang is:
http.Get in makeRequests fails (connection denied, request timeout, etc.), returning a nil response and an error value
The error is ignored and makeRequests moves on to the next request
If any errors occur, makeRequests posts less than the expected number of results to resultChan
The for .. range .. chan loop in deployRequests never breaks because results.total is always less than messages
One workaround would be:
If http.Get returns an error value, post a nil response to resultChan:
resp, err := http.Get(url)
if err != nil {
resultChan <- nil
} else if resp != nil {
resultChan <- resp
}
In deployRequests, if the for loop reads a nil value from resultChan, count that as an error:
for response := range resultChan {
if response == nil {
results.errors += 1
} else if response.StatusCode != 200 {
// ...