sync.WaitGroup doesnt waits - concurrency

May be I can not see obvious thing, what am I doing wrong:
func printSize (listOfUrls []string){
var wg sync.WaitGroup
wg.Add(len(listOfUrl))
for _, myurl := range(listOfUrls){
go func(){
body := getUrlBody(myurl)
fmt.Println(len(body))
wg.Done()
}()
}
wg.Wait()
}
If I remove wg and go, I receive the size of each url body correctly. If I do it as in above, it prints zeroes almost instantly. The getUrlBody() takes time to execute sometimes minutes.
Answering comment: I also tried it this way, to be sure, and it demonstrate same behaviour.
I found the error was in getUrlBody and main() function...
func printSize(listOfUrls []string) {
var wg sync.WaitGroup
wg.Add(len(listOfUrls))
for _, myurl := range listOfUrls {
go f(myurl, &wg)
}
wg.Wait()
}
func f(myurl string, wg *sync.WaitGroup) {
body := getUrlBody(myurl)
fmt.Println(len(body))
wg.Done()
}

All of the goroutines are sharing the single myurl variable. See https://golang.org/doc/faq#closures_and_goroutines for more information.
Change the code to:
func f(listOfUrls []string){
var wg sync.WaitGroup
wg.Add(len(listOfUrl))
for _, myurl := range(listOfUrls){
go func(myurl string){
body := getUrlBody(myurl)
fmt.Println(len(body))
wg.Done()
}(myurl)
}
wg.Wait()
}
or
func f(listOfUrls []string){
var wg sync.WaitGroup
wg.Add(len(listOfUrl))
for _, myurl := range(listOfUrls){
myurl := myurl
go func(){
body := getUrlBody(myurl)
fmt.Println(len(body))
wg.Done()
}()
}
wg.Wait()
}

Related

How to use go-routines while parsing text file for URLs with regex

I've been given a task to search for URLs in text file useng regex and goroutines with waitgroup in the way the given way: text should be devided between N workers (goroutines), each goroutine search for //https://, goroutines in waitgroup, final result should be a slice of strings (URLs) from all goroutines together.
Iam wotking with a txt.file with dozens of stuff in a single string but including URLs
right for now i know how to extract a slice of URLs from the text but without deviding a text and goroutines...
import (
"fmt"
"os"
"regexp"
"sync"
"time"
)
func Parser1(wg *sync.WaitGroup) {
time.Sleep((1 * time.Second))
b, err := os.ReadFile("repitations")
if err != nil {
fmt.Print(err)
}
str := string(b)
re := regexp.MustCompile(`(?:https?://)?(?:[^/.]+\.)*google\.com(?:/[^/\s]+)*/?`)
fmt.Printf("%q\n", re.FindAllString(str, -1))
wg.Done()
}
func Parser2(wg *sync.WaitGroup) {
time.Sleep((1 * time.Second))
b, err := os.ReadFile("repitations")
if err != nil {
fmt.Print(err)
}
str := string(b)
re := regexp.MustCompile(`(?:https?://)?(?:[^/.]+\.)*google\.com(?:/[^/\s]+)*/?`)
fmt.Printf("%q\n", re.FindAllString(str, -1))
wg.Done()
}
func main() {
var wg sync.WaitGroup
wg.Add(2)
go Parser1(&wg)
go Parser2(&wg)
wg.Wait()
fmt.Println("Well done!")
}````
Split your read process.
Open file with os.Open() and read sequentially with file.ReadAt().
Pass length to read and offset from start to Parser()
func Parser(wg *sync.WaitGroup, f *os.File, length int64, offset int64) {
defer wg.Done()
content := make([]byte, length)
_, err := f.ReadAt(content, offset)
if err != nil {
log.Fatal(err)
}
log.Printf("%s", content)
....
}

How to use http.ResponseWriter and http.request as argument(Unit Testing) with Golang language

I just getting started learning Golang and PostgreSQL. For now, I tried to make Unit testing for CreateTodo function.
My CreateTodo function is
func CreateTodo(w http.ResponseWriter, r *http.Request) {
CreateTodo := &models.Todo{}
utils.ParseBody(r, CreateTodo)
CreateTodoList := CreateTodo.CreateTodo()
res, _ := json.Marshal(CreateTodoList)
w.WriteHeader(http.StatusOK)
w.Write(res)
}
I tried to make Unit Test for this function... So far I wrote some codes like
func TestCreateTodo(t *testing.T) {
dbData := &models.Todo{
Title: "test-title-console-check",
Description: "test-description-console-check",
Condition: true,
}
utils.ParseBody(r, dbData) // r should be r *http.Request
submittedTodo := dbData.CreateTodo()
res, _ := json.Marshal(submittedTodo)
r.WriteHeader(http.StatusOK) // r should be r *http.Request
r.Write(res)
fmt.Println("res: ", res)
}
This is ParseBodu function in utils folder
func ParseBody(r *http.Request, x interface{}) {
if body, err := ioutil.ReadAll(r.Body); err == nil {
if err := json.Unmarshal([]byte(body), x); err != nil {
return
}
}
}
Here, I have a problem with passing net/http(r *http.Request). I am not sure how to pass this function like argument... I tried to receive it in TestCreateTodo(t *testing.T, r *http.Request) but not working what I expected.
Is there any way to unit test for CreateTodo function??
I really appreciate your help!
Edit 1]
I tried to make a global variable
var readData *http.Request
var writeData http.ResponseWriter
and using it in the function. The reason why I make it global variables is that I usually use it in the funcs like <w http.ResponseWriter, r *http.Request>, so I thought I can use as global vars too.
so I edit my code as
var readData *http.Request
var writeData http.ResponseWriter
func TestCreateTodo(t *testing.T) {
// w := httptest.NewRecorder()
dbData := &models.Todo{
Title: "test-title-console-check",
Description: "test-description-console-check",
Condition: true,
}
utils.ParseBody(readData, dbData)
submittedTodo := dbData.CreateTodo()
res, _ := json.Marshal(submittedTodo)
writeData.WriteHeader(http.StatusOK)
writeData.Write(res)
fmt.Println("res: ", res)
}
But I got this error
As mentioned by Volker, you need to create an http request. So you are missing this line:
req, err := http.NewRequest("GET", <your endpoint>, <your body>)
As shown by the Go http package documentation, the body must be passed as a stream of bytes. You can use bytes.Buffer for this:
var body bytes.Buffer
err := json.NewEncoder(&body).Encode(dbData)
After making your request, you need to initiate a response recorder and define the handler:
res := httptest.NewRecorder()
handler := http.HandlerFunc(<your handler>)
handler.ServeHTTP(res, req)
Then you can check if your response was as expected with the assert package.
~ Zoe ~

How to use go-sqlmock when I have concurrent query in my program?

sqlmock needs to match the SQL in order. But if I have concurrent query in my code just like this:
condition1 := make(map[string]string)
condition2 := make(map[string]string)
var count int64
var user User
var task Task
var wg sync.WaitGroup
wg.Add(3)
wgDone := make(chan interface{})
errCh := make(chan error)
go func(wg *sync.WaitGroup) {
defer wg.Done()
err := conn.Where(condition1).Find(&user).Error
if err != nil {
errCh <- err
}
}(&wg)
go func(wg *sync.WaitGroup) {
defer wg.Done()
err := conn.Where(condition2).Find(&task).Error
if err != nil {
errCh <- err
}
}(&wg)
go func(wg *sync.WaitGroup) {
defer wg.Done()
err := conn.Count(&count).Error
if err != nil {
errCh <- err
}
}(&wg)
go func() {
wg.Wait()
close(wgDone)
}()
select {
case err := <-errCh:
return err
case <-wgDone:
break
}
...
It is said that we can't know the execution order of the SQL. So I dont't know how to use sqlmock to match the sql correctly.
The MatchExpectationsInOrder method disables in-order checking for exactly this scenario.

Golang concurrent download deadlock

I want to download files in parallel in go, but my code never exits:
package main
import (
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"sync"
)
func download_file(file_path string, wg sync.WaitGroup) {
defer wg.Done()
resp, _ := http.Get(file_path)
defer resp.Body.Close()
filename := filepath.Base(file_path)
file, _ := os.Create(filename)
defer file.Close()
size, _ := io.Copy(file, resp.Body)
fmt.Println(filename, size, resp.Status)
}
func main() {
var wg sync.WaitGroup
file_list := []string{
"http://i.imgur.com/dxGb2uZ.jpg",
"http://i.imgur.com/RSU6NxX.jpg",
"http://i.imgur.com/hUWgS2S.jpg",
"http://i.imgur.com/U8kaix0.jpg",
"http://i.imgur.com/w3cEYpY.jpg",
"http://i.imgur.com/ooSCD9T.jpg"}
fmt.Println(len(file_list))
for _, url := range file_list {
wg.Add(1)
fmt.Println(wg)
go download_file(url, wg)
}
wg.Wait()
}
What's the reason? I've looked here: Golang download multiple files in parallel using goroutines but I found no solution.
What is the best way to debug such code?
As Tim Cooper said you need to pass the WaitGroup as a pointer. If you run the go vet tool on your code it will give you this warning:
$ go vet ex.go
ex.go:12: download_file passes Lock by value: sync.WaitGroup contains sync.Mutex
exit status 1
I recommend using an editor that can do this for you when you save a file. For example go-plus for Atom.
As for the code I think you should restructure it like this:
package main
import (
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"sync"
)
func downloadFile(filePath string) error {
resp, err := http.Get(filePath)
if err != nil {
return err
}
defer resp.Body.Close()
name := filepath.Base(filePath)
file, err := os.Create(name)
if err != nil {
return err
}
defer file.Close()
size, err := io.Copy(file, resp.Body)
if err != nil {
return err
}
fmt.Println(name, size, resp.Status)
return nil
}
func main() {
var wg sync.WaitGroup
fileList := []string{
"http://i.imgur.com/dxGb2uZ.jpg",
"http://i.imgur.com/RSU6NxX.jpg",
"http://i.imgur.com/hUWgS2S.jpg",
"http://i.imgur.com/U8kaix0.jpg",
"http://i.imgur.com/w3cEYpY.jpg",
"http://i.imgur.com/ooSCD9T.jpg"}
fmt.Println("downloading", len(fileList), "files")
for _, url := range fileList {
wg.Add(1)
go func(url string) {
err := downloadFile(url)
if err != nil {
fmt.Println("[error]", url, err)
}
wg.Done()
}(url)
}
wg.Wait()
}
I don't like passing WaitGroups around and prefer to keep functions simple, blocking and sequential and then stitch together the concurrency at a higher level. This gives you the option of doing it all sequentially without having to change downloadFile.
I also added error handling and fixed names so they are camelCase.
Adding to Calab's response, there's absolutely nothing wrong with your approach, all you had to do is to pass a pointer to the sync.WaitGroup.
func download_file(file_path string, wg *sync.WaitGroup) {
defer wg.Done()
......
}
.....
go download_file(url, &wg)
.....
playground

GoLang FanIn function not working

So I'm trying to write a webcrawler using Rob Pike's fanin function.
This is my code -
package main
import (
"net/http"
"encoding/json"
"fmt"
"io/ioutil"
)
func main() {
fanIn(getDuckDuckGo("food"), getGitHub("defunkt"))
}
type DuckDuckGoResponse struct {
RelatedTopics []struct {
Result string `json:"Result"`
FirstUrl string `json:"FirstURL"`
Text string `json:"Text"`
} `json:"RelatedTopics"`
}
type GitHubResponse struct {
Login string `json:"login"`
Email string `json:"email"`
Name string `json:"name"`
}
func fanIn(input1 <-chan DuckDuckGoResponse, input2 <-chan GitHubResponse) <-chan string {
c := make(chan string)
go func() {
for {
select {
case s := <-input1:
fmt.Println(s)
case s := <-input2:
fmt.Println(s)
}
}
}()
return c
}
func getDuckDuckGo(k string) <-chan DuckDuckGoResponse {
resp, err := http.Get("http://api.duckduckgo.com/?q=" + k + "&format=json&pretty=1")
if err != nil {
return nil
}
c := make(chan DuckDuckGoResponse)
var duckDuckParsed DuckDuckGoResponse
jsonDataFromHttp, jsonErr := ioutil.ReadAll(resp.Body)
if jsonErr != nil {
fmt.Println("Json error!")
}
defer resp.Body.Close()
if err:= json.Unmarshal(jsonDataFromHttp, &duckDuckParsed); err != nil {
panic(err)
}
return c
}
func getGitHub(k string) <-chan GitHubResponse {
resp, err := http.Get("https://api.github.com/users/?q=" + k)
if err != nil {
return nil
}
c := make(chan GitHubResponse)
var githubParsed GitHubResponse
jsonDataFromHttp, jsonErr := ioutil.ReadAll(resp.Body)
if jsonErr != nil {
fmt.Println("Json error!")
}
defer resp.Body.Close()
if err:= json.Unmarshal(jsonDataFromHttp, &githubParsed); err != nil {
panic(err)
}
return c
}
I run this program, and nothing prints.
Why?
Thanks
At first glance, the fanIn function returns a channel that is not being read from in your main loop. So yes, you are invoking the fanIn function which returns a channel, but there is nothing reading off of that channel. For a channel to be useful there needs to be a consumer consuming from the channel while on the other end there needs to be a producer producing on that channel. In other words, sending on a channel can't make progress unless someone on the other end is ready to receive on it.
Next, your getGitHub and getDuckDuckGo return channels, but they don't actually send anything on those channels that they return. Also, what you really need is a way to invoke those functions, have them return a channel and still execute your work. You need to use additional goroutines in order be able to have the http.Get calls do their work.
Lastly, your fanIn function also creates a channel and returns it, however it doesn't actually "fan-in" the results from input1 or input2. Since the fanIn returns a channel of type string, you'll need to write a string into them which could be a field off of DuckDuckGoResponse and GitHubResponse.
I urge you too look at this streamlined example of what you are trying to accomplish: https://talks.golang.org/2012/go-docs/faninboring.go
Last observation you are checking that jsonErr != nil and printing it out but you probably want to return nil as well to prevent the code from continuing on.
I hope this gives you just enough insight to get your code working. Good luck!