Problem with testing cli prompts interactively, scanner is not waiting for user input - unit-testing

I try to write a test to verify cli prompts, emulate user inputs in response to some program outputs.
How to make scanner.Scan wait for the rest of the writes?
What I have so far:
b := &bytes.Buffer{}
fmt.Fprint(b, "0")
go func() {
time.Sleep(1 * time.Second)
for i := 1; i < 4; i++ {
fmt.Fprint(b, i)
time.Sleep(1 * time.Second)
}
}()
scanner := bufio.NewScanner(b)
for scanner.Scan() {
log.Print(scanner.Text())
}
if err := scanner.Err(); err != nil {
log.Println("problem while scanning:", err)
}
Expected result is: 0123
Actual result is: 0
I tried a version with io.Pipe
r, w := io.Pipe()
fmt.Fprint(w, "0")
go func() {
time.Sleep(1 * time.Second)
for i := 1; i < 4; i++ {
fmt.Fprint(w, i)
time.Sleep(1 * time.Second)
}
}()
scanner := bufio.NewScanner(r)
for scanner.Scan() {
log.Print(scanner.Text())
}
if err := scanner.Err(); err != nil {
log.Println("problem while scanning:", err)
}
result: fatal error: all goroutines are asleep - deadlock!

When using a pipe, writes and reads are synchronous. A Write cannot complete without a matching Read. Move the first write into the goroutine. And Close the writing end of the pipe to allow the scanner to stop scanning.
r, w := io.Pipe()
go func() {
defer w.Close()
fmt.Fprint(w, "0")
time.Sleep(1 * time.Second)
for i := 1; i < 4; i++ {
fmt.Fprint(w, i)
time.Sleep(1 * time.Second)
}
}()
scanner := bufio.NewScanner(r)
for scanner.Scan() {
log.Print(scanner.Text())
}
if err := scanner.Err(); err != nil {
log.Println("problem while scanning:", err)
}

Related

Golang FormFile unit test

I have the following function:
func CreateByID(w http.ResponseWriter, r *http.Request) {
formFile, _, err := r.FormFile("audio")
if err != nil {
return
}
defer formFile.Close()
defer r.MultipartForm.RemoveAll()
tmpFile, err := os.CreateTemp("/tmp")
if err != nil {
return
}
defer os.Remove(tmpFile.Name())
io.Copy(tmpFile, formFile)
err = validateFile(tmpFile.Name())
if err != nil {
return
}
}
func validateFile(path string) error {
fs, err := os.Stat(path)
if err != nil {
return
}
if fs.Size() < allowedSize {
return fmt.Errorf("file is too small")
}
}
which is basically creating a temp file from the HTTP request and validating it before further processing. This works fine except for the unit test. When I'm testing this function - I'm getting always filesize = 0.
Below you can see the Test case:
func TestCreateByID(t *testing.T) {
pr, pw := io.Pipe()
writer := multipart.NewWriter(pw)
go func() {
defer writer.Close()
_, err := writer.CreateFormFile("audio", "/tmp/audioFile")
if err != nil {
t.Error(err)
}
generateAudioSample("/tmp/audioFile")
}()
request := httptest.NewRequest(http.MethodGet, "/{id}")
request.Header.Add("Content-Type", writer.FormDataContentType())
response := httptest.NewRecorder()
# Calling The Actual Function
CreateByID(response, request)
handler := http.NewServeMux()
handler.ServeHTTP(response, request)
if response.Code != 200 {
t.Errorf("Expected %d, received %d", 200, response.Code)
return
}
}
generateAudioSample function is working fine because I can see the file on the filesystem and its size it bigger than 0 but for some reason this function io.Copy(tmpFile, formFile) doesn't seem to handle FormFile correctly, and is creating just an empty file on the filesystem and obviously fail the test. What else should I add to the test function in order to pass the test?

Channel race condition

I'm trying to write tests for a simple program that reads events from the inotify bus, filters them, then puts them on a channel. I have seen no issues when actually running the code, but in the last line of the following test, it will occasionally deadlock (about 30% of the time) on reading from that channel, even though as best I can tell, the event is always being put on the channel.
Here is the function:
func eventFilter(watcher *rfsnotify.RWatcher, excludes []string, out chan<- fsnotify.Event) {
for {
select {
case event := <-watcher.Events:
log.Debug(fmt.Sprintf("Got event %v", event))
if isExcluded(event.Name, excludes) {
log.Info("Ignoring excluded file: %v", event)
} else if isRelevantOp(event) {
log.Info(fmt.Sprintf("Handling event %v", event))
out <- event
} else {
log.Info(fmt.Sprintf("Ignoring event %v", event))
}
case err := <-watcher.Errors:
log.Error(fmt.Sprintf("Error: %v", err))
}
}
}
Here is the testing code:
var hook *logrusTest.Hook
var testDir string
var rWatcher *rfsnotify.RWatcher
func TestMain(m *testing.M) {
// initialize bad globals, amongst other things
log.SetLevel(log.DebugLevel)
hook = logrusTest.NewGlobal()
// Create directory to watch
testDir = tempMkdir()
defer os.RemoveAll(testDir)
rWatcher, _ = rfsnotify.NewWatcher()
defer rWatcher.Close()
err := rWatcher.AddRecursive(testDir)
if err != nil {
log.Fatal("Failed to add watcher: ", err)
}
os.Exit(m.Run())
}
func TestEventFilterMove(t *testing.T) {
ch := make(chan fsnotify.Event, 10)
go eventFilter(rWatcher, []string{"test"}, ch)
testFileSrc := filepath.Join(testDir, "TestFsnotifyEvents.testfileSrc")
testFileDest := filepath.Join(testDir, "TestFsnotifyEvents.testfileDest")
f, err := os.OpenFile(testFileSrc, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
t.Fatalf("creating test file failed: %s", err)
}
f.WriteString("data")
f.Sync()
hook.Reset()
err = os.Rename(testFileSrc, testFileDest)
if err != nil {
t.Fatalf("renaming test file failed: %s", err)
}
time.Sleep(100 * time.Millisecond)
expectedLogs := []string{
fmt.Sprintf("Got event \"%v\": CREATE|UPDATE", testFileDest),
fmt.Sprintf("Handling event \"%v\": CREATE|UPDATE", testFileDest),
fmt.Sprintf("Got event \"%v\": RENAME", testFileSrc),
fmt.Sprintf("Ignoring event \"%v\": RENAME", testFileSrc),
}
var actualLogs []string
for _, logEntry := range hook.AllEntries() {
actualLogs = append(actualLogs, logEntry.Message)
}
assert.ElementsMatch(t, actualLogs, expectedLogs)
hook.Reset()
_ = <-ch
assert.Equal(t, 0, len(ch))
}

Read two files simultsneously value by value using regexp

Doing small helping tool for combining two text files into one.
These files stores a big 2D arrays of float values. Here is some of them:
File 1
-0,1296169 -0,1286087 -0,1276232 ...
-0,1288124 -0,1278683 -0,1269373 ...
-0,1280221 -0,1271375 -0,12626 ...
...
File 2
-0,1181779 -0,1200798 -0,1219472 ...
-0,1198357 -0,1216468 -0,1234369 ...
-0,1214746 -0,1232006 -0,1249159 ...
...
both may have hunderds of rows and columns ...
Values also can be in scientific form (etc. 1.234e-003).
My goal is to read two files simultaneously value by value and write output, while fixing delimeter from comma to point and conver from scientific form to standard in the process.
This version of program combines only prepeared files (delimeter changed to point, values represented in standard form and values moved "one value per line"), but making these preparation is unreal if file have more than million of values.
Here is what i have for now:
import (
"bufio"
"fmt"
"io"
"os"
"regexp"
)
func main() {
file_dB, err := os.Open("d:/dB.txt")
if err != nil {
fmt.Printf("error opening file: %v\n", err)
os.Exit(1)
}
file_dL, err := os.Open("d:/dL.txt")
if err != nil {
fmt.Printf("error opening file: %v\n", err)
os.Exit(1)
}
file_out, err := os.Create("d:/out.txt") // also rewrite existing !
if err != nil {
fmt.Printf("error opening file: %v\n", err)
os.Exit(1)
}
dB := bufio.NewReader(file_dB)
dL := bufio.NewReader(file_dL)
err = nil
i := 1
for {
line1, _, err := dB.ReadLine()
if len(line1) > 0 && line1[len(line1)-1] == '\n' {
line1 = line1[:len(line1)-1]
}
line2, _, err := dL.ReadLine()
if len(line2) > 0 && line2[len(line2)-1] == '\n' {
line2 = line2[:len(line2)-1]
}
if len(line1) == 0 || len(line2) == 0 || err == io.EOF {
fmt.Println("Total lines done: ", i)
break
} else if err != nil {
fmt.Printf("Error while reading files: %v\n", err)
os.Exit(1)
}
i++
str := string(line1) + ";" + string(line2) + "\n"
if _, err := file_out.WriteString(str); err != nil {
panic(err)
}
}
}
How can i use regexp to make this program read unprepeared files (first listing) value by value and form it like:
-0.129617;-0.118178
-0.128609;-0.120080
-0.127623;-0.121947
...
Input files always formed in same way:
-decimal separator is comma
-one space after value (even if it last in a row)
-newline in the end of line
Previously used expression like ([-?])([0-9]{1})([,]{1})([0-9]{1,12})( {1}) and Notepad++ replace function to split line-of-values into one-value-per-line (combined to new vaules used expression like $1$2.$4\r\n\), but its mess if 'scientific form' value happens.
So is there any way to read files value by value without messing with splitting line into slices/substrings and working over them?
Thanks for help, with points of view of another peoples i've found my own solution.
What this tool does? Generally it combines two text files to one.
Where i've used it? Creating "Generic ASCII" text file for "Country specific coordinate system tool". Input text files are ASCII export of GRID files from GIS applications (values in arc degrees expected). Later this file may be used to fix local coordinate shifts when working with precise GPS/GNSS receivers.
Here what i've "developed":
package main
import (
"bufio"
"fmt"
"os"
"regexp"
"strconv"
"strings"
)
func main() {
file_dB, err := os.Open("d:/dB.txt")
if err != nil {
fmt.Printf("error opening file: %v\n", err)
os.Exit(1)
}
defer file_dB.Close()
file_dL, err := os.Open("d:/dL.txt")
if err != nil {
fmt.Printf("error opening file: %v\n", err)
os.Exit(1)
}
defer file_dL.Close()
file_out, err := os.Create("d:/out.txt") // also rewrite existing !
if err != nil {
fmt.Printf("error opening file: %v\n", err)
os.Exit(1)
}
defer file_out.Close()
dB := bufio.NewReader(file_dB)
dL := bufio.NewReader(file_dL)
err = nil
xcorn_float := 0.0
ycorn_float := 0.0
cellsize_float := 0.0
ncols := regexp.MustCompile("[0-9]+")
nrows := regexp.MustCompile("[0-9]+")
xcorn := regexp.MustCompile("[0-9]*,[0-9]*")
ycorn := regexp.MustCompile("[0-9]*,[0-9]*")
cellsize := regexp.MustCompile("[0-9]*,[0-9]*")
nodataval := regexp.MustCompile("-?d+")
tmp := 0.0
// n cols --------------------
ncols_dB, err := dB.ReadString('\n')
if err != nil {
panic(err)
}
ncols_dL, err := dL.ReadString('\n')
if err != nil {
panic(err)
}
if ncols.FindString(ncols_dB) != ncols.FindString(ncols_dL) {
panic(err)
}
ncols_dB = ncols.FindString(ncols_dB)
// n rows --------------------
nrows_dB, err := dB.ReadString('\n')
if err != nil {
panic(err)
}
nrows_dL, err := dL.ReadString('\n')
if err != nil {
panic(err)
}
if nrows.FindString(nrows_dB) != nrows.FindString(nrows_dL) {
panic(err)
}
nrows_dB = nrows.FindString(nrows_dB)
// X --------------------
xcorn_dB, err := dB.ReadString('\n')
if err != nil {
panic(err)
}
xcorn_dL, err := dL.ReadString('\n')
if err != nil {
panic(err)
}
if xcorn.FindString(xcorn_dB) != xcorn.FindString(xcorn_dL) {
panic(err)
}
xcorn_float, err = strconv.ParseFloat(strings.Replace(cellsize.FindString(xcorn_dB), ",", ".", 1), 8)
xcorn_float *= 3600.0
// Y --------------------
ycorn_dB, err := dB.ReadString('\n')
if err != nil {
panic(err)
}
ycorn_dL, err := dL.ReadString('\n')
if err != nil {
panic(err)
}
if ycorn.FindString(ycorn_dB) != ycorn.FindString(ycorn_dL) {
panic(err)
}
ycorn_float, err = strconv.ParseFloat(strings.Replace(cellsize.FindString(ycorn_dB), ",", ".", 1), 8)
ycorn_float *= 3600.0
// cell size --------------------
cellsize_dB, err := dB.ReadString('\n')
if err != nil {
panic(err)
}
cellsize_dL, err := dL.ReadString('\n')
if err != nil {
panic(err)
}
if cellsize.FindString(cellsize_dB) != cellsize.FindString(cellsize_dL) {
panic(err)
}
cellsize_float, err = strconv.ParseFloat(strings.Replace(cellsize.FindString(cellsize_dB), ",", ".", 1), 8)
cellsize_float *= 3600.0
// nodata value --------------------
nodataval_dB, err := dB.ReadString('\n')
if err != nil {
panic(err)
}
nodataval_dL, err := dL.ReadString('\n')
if err != nil {
panic(err)
}
if nodataval.FindString(nodataval_dB) != nodataval.FindString(nodataval_dL) {
panic(err)
}
nodataval_dB = nodataval.FindString(nodataval_dB)
fmt.Print(nodataval_dB)
//making header
if _, err := file_out.WriteString("name\n3;0;2\n1;2;" + nrows_dB + ";" + ncols_dB + "\n" + strconv.FormatFloat(xcorn_float, 'f', -1, 32) + ";" + strconv.FormatFloat(ycorn_float, 'f', -1, 32) + ";" + strconv.FormatFloat(cellsize_float, 'f', -1, 32) + ";" + strconv.FormatFloat(cellsize_float, 'f', -1, 32) + "\n1\n"); err != nil {
panic(err)
}
// valuses --------------------
for {
line1, err := dB.ReadString(' ')
if err != nil {
break
}
if tmp, err = strconv.ParseFloat(strings.TrimSpace(strings.Replace(line1, ",", ".", 1)), 64); err == nil {
line1 = strconv.FormatFloat(tmp, 'f', 8, 64)
}
line2, err := dL.ReadString(' ')
if err != nil {
break
}
if tmp, err = strconv.ParseFloat(strings.TrimSpace(strings.Replace(line2, ",", ".", 1)), 64); err == nil {
line2 = strconv.FormatFloat(tmp, 'f', 8, 64)
}
if err != nil {
panic(err)
}
str := string(line1) + ";" + string(line2) + "\n"
if _, err := file_out.WriteString(str); err != nil {
panic(err)
}
}
}
If you have any recomendations - feel free to leave a comment!
For example,
package main
import (
"bufio"
"bytes"
"fmt"
"io"
"os"
"strconv"
"strings"
)
var comma, period = []byte{','}, []byte{'.'}
func readNext(r io.Reader) func() (float64, error) {
s := bufio.NewScanner(r)
var fields []string
return func() (float64, error) {
if len(fields) == 0 {
err := io.EOF
for s.Scan() {
line := bytes.Replace(s.Bytes(), comma, period, -1)
fields = strings.Fields(string(line))
if len(fields) > 0 {
err = nil
break
}
}
if err := s.Err(); err != nil {
return 0, err
}
if err == io.EOF {
return 0, err
}
}
n, err := strconv.ParseFloat(fields[0], 64)
fields = fields[1:]
if err != nil {
return 0, err
}
return n, nil
}
}
func main() {
in1Name := `in1.data`
in2Name := `in2.data`
outName := `out.data`
in1, err := os.Open(in1Name)
if err != nil {
fmt.Fprint(os.Stderr, err)
return
}
defer in1.Close()
in2, err := os.Open(in2Name)
if err != nil {
fmt.Fprint(os.Stderr, err)
return
}
defer in2.Close()
out, err := os.Create(outName)
if err != nil {
fmt.Fprint(os.Stderr, err)
return
}
defer out.Close()
outw := bufio.NewWriter(out)
defer outw.Flush()
next1 := readNext(in1)
next2 := readNext(in2)
for {
n1, err1 := next1()
n2, err2 := next2()
if err1 == io.EOF && err2 == io.EOF {
break
}
if err1 != nil || err2 != nil {
fmt.Fprint(os.Stderr, err1, err2)
return
}
_, err := fmt.Fprintf(outw, "%g;%g\n", n1, n2)
if err != nil {
fmt.Fprint(os.Stderr, err)
return
}
}
}
Playground: https://play.golang.org/p/I_sT_EPFI_W
Output:
$ go run data.go
$ cat in1.data
-0,1296169 -0,1286087 -0,1276232
-0,1288124 -0,1278683 -0,1269373
-0,1280221 -0,1271375 -0,12626
$ cat in2.data
-0,1296169 -0,1286087 -0,1276232
-0,1288124 -0,1278683 -0,1269373
-0,1280221 -0,1271375 -0,12626
$ cat out.data
-0.1296169;-0.1296169
-0.1286087;-0.1286087
-0.1276232;-0.1276232
-0.1288124;-0.1288124
-0.1278683;-0.1278683
-0.1269373;-0.1269373
-0.1280221;-0.1280221
-0.1271375;-0.1271375
-0.12626;-0.12626
$
Something like this. Note the limitation that assumes same number of values per line. Be careful it would blowup with the error if this assumption is wrong :)
package main
import (
"bufio"
"fmt"
"os"
"strconv"
"strings"
)
func main() {
file_dB, err := os.Open("dB.txt")
if err != nil {
fmt.Printf("error opening file: %v\n", err)
return
}
defer file_dB.Close()
file_dL, err := os.Open("dL.txt")
if err != nil {
fmt.Printf("error opening file: %v\n", err)
return
}
defer file_dL.Close()
file_out, err := os.Create("out.txt") // also rewrite existing !
if err != nil {
fmt.Printf("error opening file: %v\n", err)
return
}
defer file_out.Close()
dB := bufio.NewReader(file_dB)
dL := bufio.NewReader(file_dL)
lc := 0
for {
lc++
line1, _, err := dB.ReadLine()
vals1 := strings.Split(string(line1), " ")
if err != nil {
fmt.Println(lc, err)
return
}
line2, _, err := dL.ReadLine()
vals2 := strings.Split(string(line2), " ")
if err != nil {
fmt.Println(lc, err)
return
}
// Limitation: assumes line1 and line2 have same number of values per line
for i := range vals1 {
dot1 := strings.Replace(vals1[i], ",", ".", 1)
v1, err := strconv.ParseFloat(dot1, 64)
if err != nil {
fmt.Println(lc, err)
continue
}
dot2 := strings.Replace(vals2[i], ",", ".", 1)
v2, err := strconv.ParseFloat(dot2, 64)
if err != nil {
fmt.Println(lc, err)
continue
}
_, err = fmt.Fprintf(file_out, "%v; %v\n", v1, v2)
if err != nil {
fmt.Println(lc, err)
return
}
}
}
}

how can I reduce cpu usage in a golang tcp server?

I try to implement a golang tcp server, and I found the concurrency is satisfied for me, but the CPU usage is too high(concurrency is 15W+/s, but the CPU usage is about 800% in a 24 cores linux machine). At the same time, a C++ tcp server is only about 200% usage with a similar concurrency(with libevent).
The following code is the demo of golang:
func main() {
listen, err := net.Listen("tcp", "0.0.0.0:17379")
if err != nil {
fmt.Errorf(err.Error())
}
go acceptClient(listen)
var channel2 = make(chan bool)
<-channel2
}
func acceptClient(listen net.Listener) {
for {
sock, err := listen.Accept()
if err != nil {
fmt.Errorf(err.Error())
}
tcp := sock.(*net.TCPConn)
tcp.SetNoDelay(true)
var channel = make(chan bool, 10)
go read(channel, sock.(*net.TCPConn))
go write(channel, sock.(*net.TCPConn))
}
}
func read(channel chan bool, sock *net.TCPConn) {
count := 0
for {
var buf = make([]byte, 1024)
n, err := sock.Read(buf)
if err != nil {
close(channel)
sock.CloseRead()
return
}
count += n
x := count / 58
count = count % 58
for i := 0; i < x; i++ {
channel <- true
}
}
}
func write(channel chan bool, sock *net.TCPConn) {
buf := []byte("+OK\r\n")
defer func() {
sock.CloseWrite()
recover()
}()
for {
_, ok := <-channel
if !ok {
return
}
_, writeError := sock.Write(buf)
if writeError != nil {
return
}
}
}
And I test this tcp server by the redis-benchmark with multi-clients:
redis-benchmark -h 10.100.45.2 -p 17379 -n 1000 -q script load "redis.call('set','aaa','aaa')"
I also analyzed my golang code by the pprof, it is said CPU cost a lot of time on syscall:
enter image description here
I don't think parallelise the read and write with channel will provide you better performance in this case. You should try to do less memory allocation and less syscall (The write function may do a lot of syscalls)
Can you try this version?
package main
import (
"bytes"
"fmt"
"net"
)
func main() {
listen, err := net.Listen("tcp", "0.0.0.0:17379")
if err != nil {
fmt.Errorf(err.Error())
}
acceptClient(listen)
}
func acceptClient(listen net.Listener) {
for {
sock, err := listen.Accept()
if err != nil {
fmt.Errorf(err.Error())
}
tcp := sock.(*net.TCPConn)
tcp.SetNoDelay(true)
go handleConn(tcp) // less go routine creation but no concurrent read/write on the same conn
}
}
var respPattern = []byte("+OK\r\n")
// just one goroutine per conn
func handleConn(sock *net.TCPConn) {
count := 0
buf := make([]byte, 4098) // Do not create a new buffer each time & increase the buff size
defer sock.Close()
for {
n, err := sock.Read(buf)
if err != nil {
return
}
count += n
x := count / 58
count = count % 58
resp := bytes.Repeat(respPattern, x) // can be optimize
_, writeError := sock.Write(resp) // do less syscall
if writeError != nil {
return
}
}
}

Go webcrawler hangs after checking about 2000 urls

I have a program to check whether keywords are on a web page. But after checking 1000-3000 urls, it hangs. There is no output, it does not exit, and the number of tcp connections is zero. I don't know why there are no new connections.
Would you give me some advice how to debug it?
type requestReturn struct {
url string
status bool
}
var timeout = time.Duration(800 * time.Millisecond)
func checkUrls(urls []string, kws string, threadLimit int) []string {
limitChan := make(chan int, threadLimit)
ok := make(chan requestReturn, 1)
var result []string
i := 0
for ; i < threadLimit; i++ {
go func(u string) {
request(u, limitChan, ok, kws)
}(urls[i])
}
for o := range ok {
if o.status {
result = append(result, o.url)
log.Printf("success %s,remain %d", o.url, len(urls)-i)
} else {
log.Printf("fail %s,remain %d", o.url, len(urls)-i)
}
if i < len(urls) {
go func(u string) {
request(u, limitChan, ok, kws)
}(urls[i])
i++
}
}
close(limitChan)
return result
}
func dialTimeout(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
func request(url string, threadLimit chan int, ok chan requestReturn, kws string) {
threadLimit <- 1
log.Printf("%s, start...", url)
//startTime := time.Now().UnixNano()
rr := requestReturn{url: url}
transport := http.Transport{
Dial: dialTimeout,
DisableKeepAlives: true,
}
client := http.Client{
Transport: &transport,
Timeout: time.Duration(15 * time.Second),
}
resp, e := client.Get(url)
if e != nil {
log.Printf("%q", e)
rr.status = false
return
}
if resp.StatusCode == 200 {
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Printf("%q", err)
rr.status = false
return
}
content := bytes.NewBuffer(body).String()
matched, err1 := regexp.MatchString(kws, content)
if err1 != nil {
log.Printf("%q", err1)
rr.status = false
} else if matched {
rr.status = true
log.Println(rr.url)
} else {
rr.status = false
}
} else {
rr.status = false
}
defer (func() {
resp.Body.Close()
ok <- rr
//processed := float32(time.Now().UnixNano()-startTime) / 1e9
//log.Printf("%s, status:%t,time:%.3fs", rr.url, rr.status, processed)
<-threadLimit
})()
}
You seem to be using two forms of concurrency control in this code, and both have problems.
You've got limitChan, which looks like it is being used as a semaphore (request sends a value at its start, and receives a value in a defer in that function). But checkUrls is also trying to make sure it only has threadLimit goroutines running at once (by spawning that number first up, and only spawning more when one reports its results on the ok channel). Only one of these should be necessary to limit the concurrency.
Both methods fail due to the way the defer is set up in request. There are a number of return statements that occur before defer, so it is possible for the function to complete without sending the result to the ok channel, and without freeing up its slot in limitChan. After a sufficient number of errors, checkUrls will stop spawning new goroutines and you'll see your hang.
The fix is to place the defer statement before any of the return statements so you know it will always be run. Something like this:
func request(url string, threadLimit chan int, ok chan requestReturn, kws string) {
threadLimit <- 1
rr := requestReturn{url: url}
var resp *http.Response
defer func() {
if resp != nil {
resp.Body.Close()
}
ok <- rr
<-threadLimit
}()
...
}