Imagine a situation where I'd like to call a function that does some amount of processing, but is time-bound. I could write a function in golang using context.Context and select. I'd imagine something as follows:
package main
import (
"context"
"fmt"
"time"
)
func longRunning(ctx context.Context, msg string) {
stop := make(chan bool)
done := make(chan bool)
go func() {
for {
fmt.Printf("long running calculation %v...", msg)
select {
case <-stop:
fmt.Println("time to stop early!")
return
default:
}
}
done <- true
}()
select {
case <-done:
return
case <-ctx.Done():
stop <- true
return
}
}
func main() {
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
longRunning(ctx, "wheeee")
}
Is there a pattern I can use to achieve similar results in C++? in the sample above select is able to listen on a channel in a non-blocking way. Is having a eventfd file descriptor of some kind and listening to events the way to do it?
Any suggestions or tips would be much appreciated.
Something along these lines, perhaps:
void longRunning(std::atomic<bool>& stop) {
for (;;) {
if (stop) return;
// Do a bit of work
}
}
int main() {
std::atomic<bool> stop = false;
auto future = std::async(std::launch::async, longRunning, std::ref(stop));
future.wait_for(std::chrono::seconds(num_seconds));
stop = true;
future.get(); // wait for the task to finish or exit early.
}
Demo
Related
This is a follow up from my previous question.
I am trying to build a prototype for a webcrawler and I want to use a chan to block the execution until all the jobs are done, just as in
func main() {
go func() {
do_stuff()
stop <- true
}
fmt.Println(<-stop)
}
There is a queue function that dispatch the jobs to the workers. When all jobs are finished, the function will also the channel and send a signal.
type Job int
//simulating a worker that processes a html page and returns some more links
func worker(in chan Job, out chan Job, num int) {
for element := range in {
if element%2 == 0 {
out <- 100*element + 5
out <- 100*element + 3
out <- 100*element + 1
}
}
}
func queue(toWorkers chan<- Job, fromWorkers <-chan Job, init Job, stop chan bool) {
var list []Job
var currentJobs int
currentJobs = 0
list = append(list, init)
done := make(map[Job]bool)
for {
var send chan<- Job
var item Job
if len(list) > 0 {
send = toWorkers
item = list[0]
} else if currentJobs == 0 {
close(toWorkers)
// this messes up everything!
stop <- true
return
}
select {
case send <- item:
currentJobs += 1
// We sent an item, remove it
list = list[1:]
case thing := <-fromWorkers:
currentJobs -= 1
// Got a new thing
if !done[thing] {
list = append(list, thing)
done[thing] = true
}
}
}
}
func main() {
in := make(chan Job, 1)
out := make(chan Job, 1)
stop := make(chan bool)
// dispatches jobs to workers
go queue(in, out, 0, stop)
for i := 0; i < max_workers; i++ {
go worker(in, out, i)
}
duration := time.Second
time.Sleep(duration)
// this cause deadlock
fmt.Println(<-stop)
}
Link to playground
If I understand correctly, the problem is with the stop channel: when the workers still have jobs, go thinks that no one will send to that channel and declares deadlock. The function queue will both close the toWorkers channel and send a signal to stop, but not while there are outstanding jobs.
What am I missing?
Use sync.WaitGroup to wait for all the go routines to end.
http://golang.org/pkg/sync/#WaitGroup
http://blog.golang.org/pipelines
I made a small example here: http://play.golang.org/p/P30LdV0Gfe
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
routinesNo := 10
wg.Add(routinesNo)
for i := 0; i < routinesNo; i++ {
go func(n int) {
fmt.Printf("%d ", n)
wg.Done()
}(i)
}
wg.Wait()
fmt.Println("\nThe end!")
}
I'm wondering how can I drain / close the buffered channels so that I don't get into the deadlock? I'm using range to loop through the channels but it seems that although they are "read" they don't get closed like the non-buffered channels do.
package main
func main() {
cp := 2
ch := make(chan string, cp)
for i := 0; i < cp; i++ {
go send(ch)
}
go send(ch)
for lc := range ch {
print(lc)
}
}
func send(ch chan string) {
ch <- "hello\n"
}
Play
You can close channels using the close() builtin. This has to be called after all of your concurrent processing is done. How you're doing that depends on what you want to do.
In your current architecture it seems that you have to establish a global state, something that tracks all your processes and determines that the last one finished. Such a state can be achieved by using a sync.WaitGroup for example.
func send(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
// ...
}
wg := &sync.WaitGroup{}
for i := 0; i < cp; i++ {
wg.Add(1)
go send(ch, wg)
}
wg.Add(1)
go send(ch, wg)
wg.Wait()
close(ch)
for e := range(ch) {
// ...
}
Note that closing the channel and then iterating over it will give you only the elements that are queued in the channel. This means that any goroutine that wanted to put a value in the channel can't do this anymore as the channel is closed.
I have concurrent goroutines which want to append a (pointer to a) struct to the same slice.
How do you write that in Go to make it concurrency-safe?
This would be my concurrency-unsafe code, using a wait group:
var wg sync.WaitGroup
MySlice = make([]*MyStruct)
for _, param := range params {
wg.Add(1)
go func(param string) {
defer wg.Done()
OneOfMyStructs := getMyStruct(param)
MySlice = append(MySlice, &OneOfMyStructs)
}(param)
}
wg.Wait()
I guess you would need to use go channels for concurrency-safety. Can anyone contribute with an example?
There is nothing wrong with guarding the MySlice = append(MySlice, &OneOfMyStructs) with a sync.Mutex. But of course you can have a result channel with buffer size len(params) all goroutines send their answers and once your work is finished you collect from this result channel.
If your params has a fixed size:
MySlice = make([]*MyStruct, len(params))
for i, param := range params {
wg.Add(1)
go func(i int, param string) {
defer wg.Done()
OneOfMyStructs := getMyStruct(param)
MySlice[i] = &OneOfMyStructs
}(i, param)
}
As all goroutines write to different memory this isn't racy.
The answer posted by #jimt is not quite right, in that it misses the last value sent in the channel and the last defer wg.Done() is never called. The snippet below has the corrections.
https://play.golang.org/p/7N4sxD-Bai
package main
import "fmt"
import "sync"
type T int
func main() {
var slice []T
var wg sync.WaitGroup
queue := make(chan T, 1)
// Create our data and send it into the queue.
wg.Add(100)
for i := 0; i < 100; i++ {
go func(i int) {
// defer wg.Done() <- will result in the last int to be missed in the receiving channel
queue <- T(i)
}(i)
}
go func() {
// defer wg.Done() <- Never gets called since the 100 `Done()` calls are made above, resulting in the `Wait()` to continue on before this is executed
for t := range queue {
slice = append(slice, t)
wg.Done() // ** move the `Done()` call here
}
}()
wg.Wait()
// now prints off all 100 int values
fmt.Println(slice)
}
I wanted to add that since you know how many values you are expecting from the channel, you may not need to make use of any synchronization primitives. Just read from the channel as much data as you are expecting and leave it alone:
borrowing #chris' answer
package main
import "fmt"
type T int
func main() {
var slice []T
queue := make(chan T)
// Create our data and send it into the queue.
for i := 0; i < 100; i++ {
go func(i int) {
queue <- T(i)
}(i)
}
for i := 0; i < 100; i++ {
select {
case t := <-queue:
slice = append(slice, t)
}
}
// now prints off all 100 int values
fmt.Println(slice)
}
The select will block until the channels receives some data, so we can rely on this behaviour to just read from the channel 100 times before exiting.
In your case, you can just do:
package main
func main() {
MySlice = []*MyStruct{}
queue := make(chan *MyStruct)
for _, param := range params {
go func(param string) {
OneOfMyStructs := getMyStruct(param)
queue <- &OneOfMyStructs
}(param)
}
for _ := range params {
select {
case OneOfMyStructs := <-queue:
MySlice = append(MySlice, OneOfMyStructs)
}
}
}
I'm doing some stream processing in Go and got stuck trying to figure out how to do this the "Go way" without locks.
This contrived example shows the problem I'm facing.
We get one thing at a time.
There is a goroutine which buffers them into a slice called things.
When things becomes full len(things) == 100 then it is processed somehow and reset
There are n number of concurrent goroutines that need to access things before it's full
Access to the "incomplete" things from other goroutines is not predictable.
Neither doSomethingWithPartial nor doSomethingWithComplete needs to mutate things
Code:
var m sync.Mutex
var count int64
things := make([]int64, 0, 100)
// slices of data are constantly being generated and used
go func() {
for {
m.Lock()
if len(things) == 100 {
// doSomethingWithComplete does not modify things
doSomethingWithComplete(things)
things = make([]int64, 0, 100)
}
things = append(things, count)
m.Unlock()
count++
}
}()
// doSomethingWithPartial needs to access the things before they're ready
for {
m.Lock()
// doSomethingWithPartial does not modify things
doSomethingWithPartial(things)
m.Unlock()
}
I know that slices are immutable so does that mean I can remove the mutex and expect it to still work (I assume no).
How can I refactor this to use channels instead of a mutex.
Edit: Here's the solution I came up with that does not use a mutex
package main
import (
"fmt"
"sync"
"time"
)
func Incrementor() chan int {
ch := make(chan int)
go func() {
count := 0
for {
ch <- count
count++
}
}()
return ch
}
type Foo struct {
things []int
requests chan chan []int
stream chan int
C chan []int
}
func NewFoo() *Foo {
foo := &Foo{
things: make([]int, 0, 100),
requests: make(chan chan []int),
stream: Incrementor(),
C: make(chan []int),
}
go foo.Launch()
return foo
}
func (f *Foo) Launch() {
for {
select {
case ch := <-f.requests:
ch <- f.things
case thing := <-f.stream:
if len(f.things) == 100 {
f.C <- f.things
f.things = make([]int, 0, 100)
}
f.things = append(f.things, thing)
}
}
}
func (f *Foo) Things() []int {
ch := make(chan []int)
f.requests <- ch
return <-ch
}
func main() {
foo := NewFoo()
var wg sync.WaitGroup
wg.Add(10)
for i := 0; i < 10; i++ {
go func(i int) {
time.Sleep(time.Millisecond * time.Duration(i) * 100)
things := foo.Things()
fmt.Println("got things:", len(things))
wg.Done()
}(i)
}
go func() {
for _ = range foo.C {
// do something with things
}
}()
wg.Wait()
}
It should be noted that the "Go way" is probably just to use a mutex for this. It's fun to work out how to do it with a channel but a mutex is probably simpler and easier to reason about for this particular problem.
What I would like to do is have a set of producer goroutines (of which some may or may not complete) and a consumer routine. The issue is with that caveat in parentheses - we don't know the total number that will return an answer.
So what I want to do is this:
package main
import (
"fmt"
"math/rand"
)
func producer(c chan int) {
// May or may not produce.
success := rand.Float32() > 0.5
if success {
c <- rand.Int()
}
}
func main() {
c := make(chan int, 10)
for i := 0; i < 10; i++ {
go producer(c, signal)
}
// If we include a close, then that's WRONG. Chan will be closed
// but a producer will try to write to it. Runtime error.
close(c)
// If we don't close, then that's WRONG. All goroutines will
// deadlock, since the range keyword will look for a close.
for num := range c {
fmt.Printf("Producer produced: %d\n", num)
}
fmt.Println("All done.")
}
So the issue is, if I close it's wrong, if I don't close - it's still wrong (see comments in code).
Now, the solution would be an out-of-band signal channel, that ALL producers write to:
package main
import (
"fmt"
"math/rand"
)
func producer(c chan int, signal chan bool) {
success := rand.Float32() > 0.5
if success {
c <- rand.Int()
}
signal <- true
}
func main() {
c := make(chan int, 10)
signal := make(chan bool, 10)
for i := 0; i < 10; i++ {
go producer(c, signal)
}
// This is basically a 'join'.
num_done := 0
for num_done < 10 {
<- signal
num_done++
}
close(c)
for num := range c {
fmt.Printf("Producer produced: %d\n", num)
}
fmt.Println("All done.")
}
And that totally does what I want! But to me it seems like a mouthful. My question is: Is there any idiom/trick that lets me do something similar in an easier way?
I had a look here: http://golang.org/doc/codewalk/sharemem/
And it seems like the complete chan (initialised at the start of main) is used in a range but never closed. I do not understand how.
If anyone has any insights, I would greatly appreciate it. Cheers!
Edit: fls0815 has the answer, and has also answered the question of how the close-less channel range works.
My code above modifed to work (done before fls0815 kindly supplied code):
package main
import (
"fmt"
"math/rand"
"sync"
)
var wg_prod sync.WaitGroup
var wg_cons sync.WaitGroup
func producer(c chan int) {
success := rand.Float32() > 0.5
if success {
c <- rand.Int()
}
wg_prod.Done()
}
func main() {
c := make(chan int, 10)
wg_prod.Add(10)
for i := 0; i < 10; i++ {
go producer(c)
}
wg_cons.Add(1)
go func() {
for num := range c {
fmt.Printf("Producer produced: %d\n", num)
}
wg_cons.Done()
} ()
wg_prod.Wait()
close(c)
wg_cons.Wait()
fmt.Println("All done.")
}
Only producers should close channels. You could achieve your goal by invoking consumer(s) which iterates (range) over the resulting channel once your producers were started. In your main thread you wait (see sync.WaitGroup) until your consumers/producers finished their work. After producers finished you close the resulting channel which will force your consumers to exit (range will exit when channels are closed and no buffered item is left).
Example code:
package main
import (
"log"
"sync"
"time"
"math/rand"
"runtime"
)
func consumer() {
defer consumer_wg.Done()
for item := range resultingChannel {
log.Println("Consumed:", item)
}
}
func producer() {
defer producer_wg.Done()
success := rand.Float32() > 0.5
if success {
resultingChannel <- rand.Int()
}
}
var resultingChannel = make(chan int)
var producer_wg sync.WaitGroup
var consumer_wg sync.WaitGroup
func main() {
rand.Seed(time.Now().Unix())
for c := 0; c < runtime.NumCPU(); c++ {
producer_wg.Add(1)
go producer()
}
for c := 0; c < runtime.NumCPU(); c++ {
consumer_wg.Add(1)
go consumer()
}
producer_wg.Wait()
close(resultingChannel)
consumer_wg.Wait()
}
The reason I put the close-statement into the main function is because we have more than one producer. Closing the channel in one producer in the example above would lead to the problem you already ran into (writing on closed channels; the reason is that there could one producer left who still produces data). Channels should only be closed when there is no producer left (therefore my suggestion on closing the channel only by the producer). This is how channels are constructed in Go. Here you'll find some more information on closing channels.
Related to the sharemem example: AFAICS this example runs endless by re-queuing the Resources again and again (from pending -> complete -> pending -> complete... and so on). This is what the iteration at the end of the main-func does. It receives the completed Resources and re-queues them using Resource.Sleep() to pending. When there is no completed Resource it waits and blocks for new Resources being completed. Therefore there is no need to close the channels because they are in use all the time.
There are always lots of ways to solve these problems. Here's a solution using the simple synchronous channels that are fundamental in Go. No buffered channels, no closing channels, no WaitGroups.
It's really not that far from your "mouthful" solution, and--sorry to disappoint--not that much smaller. It does put the consumer in it's own goroutine, so that the consumer can consume numbers as the producer produces them. It also makes the distinction that a production "try" can end in either success or failure. If production fails, the try is done immediately. If it succeeds, the try is not done until the number is consumed.
package main
import (
"fmt"
"math/rand"
)
func producer(c chan int, fail chan bool) {
if success := rand.Float32() > 0.5; success {
c <- rand.Int()
} else {
fail <- true
}
}
func consumer(c chan int, success chan bool) {
for {
num := <-c
fmt.Printf("Producer produced: %d\n", num)
success <- true
}
}
func main() {
const nTries = 10
c := make(chan int)
done := make(chan bool)
for i := 0; i < nTries; i++ {
go producer(c, done)
}
go consumer(c, done)
for i := 0; i < nTries; i++ {
<-done
}
fmt.Println("All done.")
}
I'm adding this because the extant answers don't make a couple things clear. First, the range loop in the codewalk example is just an infinite event loop, there to keep re-checking and updating the same url list forever.
Next, a channel, all by itself, already is the idiomatic consumer-producer queue in Go. The size of the async buffer backing the channel determines how much producers can produce before getting backpressure. Set N = 0 below to see lock-step producer consumer without anyone racing ahead or getting behind. As it is, N = 10 will let the producer produce up to 10 products before blocking.
Last, there are some nice idioms for writing communicating sequential processees in Go (e.g. functions that start go routines for you, and using the for/select pattern to communicate and accept control commands). I think of WaitGroups as clumsy, and would like to see idiomatic examples instead.
package main
import (
"fmt"
"time"
)
type control int
const (
sleep control = iota
die // receiver will close the control chan in response to die, to ack.
)
func (cmd control) String() string {
switch cmd {
case sleep: return "sleep"
case die: return "die"
}
return fmt.Sprintf("%d",cmd)
}
func ProduceTo(writechan chan<- int, ctrl chan control, done chan bool) {
var product int
go func() {
for {
select {
case writechan <- product:
fmt.Printf("Producer produced %v\n", product)
product++
case cmd:= <- ctrl:
fmt.Printf("Producer got control cmd: %v\n", cmd)
switch cmd {
case sleep:
fmt.Printf("Producer sleeping 2 sec.\n")
time.Sleep(2000 * time.Millisecond)
case die:
fmt.Printf("Producer dies.\n")
close(done)
return
}
}
}
}()
}
func ConsumeFrom(readchan <-chan int, ctrl chan control, done chan bool) {
go func() {
var product int
for {
select {
case product = <-readchan:
fmt.Printf("Consumer consumed %v\n", product)
case cmd:= <- ctrl:
fmt.Printf("Consumer got control cmd: %v\n", cmd)
switch cmd {
case sleep:
fmt.Printf("Consumer sleeping 2 sec.\n")
time.Sleep(2000 * time.Millisecond)
case die:
fmt.Printf("Consumer dies.\n")
close(done)
return
}
}
}
}()
}
func main() {
N := 10
q := make(chan int, N)
prodCtrl := make(chan control)
consCtrl := make(chan control)
prodDone := make(chan bool)
consDone := make(chan bool)
ProduceTo(q, prodCtrl, prodDone)
ConsumeFrom(q, consCtrl, consDone)
// wait for a moment, to let them produce and consume
timer := time.NewTimer(10 * time.Millisecond)
<-timer.C
// tell producer to pause
fmt.Printf("telling producer to pause\n")
prodCtrl <- sleep
// wait for a second
timer = time.NewTimer(1 * time.Second)
<-timer.C
// tell consumer to pause
fmt.Printf("telling consumer to pause\n")
consCtrl <- sleep
// tell them both to finish
prodCtrl <- die
consCtrl <- die
// wait for that to actually happen
<-prodDone
<-consDone
}
You can use simple unbuffered channels without wait groups if you use the generator pattern with a fanIn function.
In the generator pattern, each producer returns a channel and is responsible for closing it. A fanIn function then iterates over these channels and forwards the values returned on them down a single channel that it returns.
The problem of course, is that the fanIn function forwards the zero value of the channel type (int) when each channel is closed.
You can work around it by using the zero value of your channel type as a sentinel value and only using the results from the fanIn channel if they are not the zero value.
Here's an example:
package main
import (
"fmt"
"math/rand"
)
const offset = 1
func producer() chan int {
cout := make(chan int)
go func() {
defer close(cout)
// May or may not produce.
success := rand.Float32() > 0.5
if success {
cout <- rand.Int() + offset
}
}()
return cout
}
func fanIn(cin []chan int) chan int {
cout := make(chan int)
go func() {
defer close(cout)
for _, c := range cin {
cout <- <-c
}
}()
return cout
}
func main() {
chans := make([]chan int, 0)
for i := 0; i < 10; i++ {
chans = append(chans, producer())
}
for num := range fanIn(chans) {
if num > offset {
fmt.Printf("Producer produced: %d\n", num)
}
}
fmt.Println("All done.")
}
producer-consumer is such a common pattern that I write a library prosumer for convenience with dealing with chan communication carefully. Eg:
func main() {
maxLoop := 10
var wg sync.WaitGroup
wg.Add(maxLoop)
defer wg.Wait()
consumer := func(ls []interface{}) error {
fmt.Printf("get %+v \n", ls)
wg.Add(-len(ls))
return nil
}
conf := prosumer.DefaultConfig(prosumer.Consumer(consumer))
c := prosumer.NewCoordinator(conf)
c.Start()
defer c.Close(true)
for i := 0; i < maxLoop; i++ {
fmt.Printf("try put %v\n", i)
discarded, err := c.Put(i)
if err != nil {
fmt.Errorf("discarded elements %+v for err %v", discarded, err)
wg.Add(-len(discarded))
}
time.Sleep(time.Second)
}
}
close has a param called graceful, which means whether drain the underlying chan.