I have two go routines:
func f1 {
... some code
// critical section 1 (CS1)
... critical section code
// end criticla section 1
... more code
}
func f2 {
... some code
// critical section 2 (CS2)
... critical section code
// end criticla section 2
... more code
}
func main() {
go f1()
go f2()
}
What is proper way to ensure that the critical sections in these routines always execute alternately?
In other words, CS1 should be executed only after CS2 and vice versa: CS1, CS2, CS1, CS2, CS1, etc.
If you are running the functions in different goroutines, I would suggest dual channels. It is like passing on a little bool-ball. Each function has a channel they listen on, and another channel where they pass on the ball once the critical section is done. Then you can be sure that, no matter when they are called, they will always run alternately.
This pattern also allows you to extend the cycle with f3, f4 ... as well.
package main
func f1(do chan bool, next chan bool) {
//... some code
<-do // Waits for the ball
// critical section 1 (CS1)
//... critical section code
// end criticla section 1
next <- true // Pass on the ball to the next function
//... more code
}
func f2(do chan bool, next chan bool) {
//... some code
<-do
// critical section 2 (CS2)
//... critical section code
// end criticla section 2
next <- true
//... more code
}
func main() {
cf1 := make(chan bool, 1)
cf2 := make(chan bool, 1)
cf1 <- true // Let cf1 start with the ball
go f1(cf1, cf2)
go f2(cf2, cf1)
// Wait here, otherwise it will just exit
}
Related
Please consider the following function:
pub fn shiny_function(&mut self, cb: Arc<Mutex<dyn FnMut(usize) + Send>>) {
// Do stuff here...
}
Now, the question is, how do I write a Unit Test that checks that the callback (closure) parameter is equal to some value?
The obvious solution looks something like this:
#[test]
fn progress_cb() {
let cut = ... // cut stands for Class Under Test
cut.shiny_function(Arc::new(Mutex::new(move |percent| {
// Assert here maybe? I don't know.
})));
cut.shiny_function();
// Or maybe assert it somehow here? I don't know.
}
But the thing is the test finishes before the callback is even called. How can I tell the test harness to wait until the callback is called?
You can use the regular concurrency structs provided in the standard library to fix this issue. In this example, I use a barrier to ensure that the end of the closure is reached before the test function exits. I create the barrier with a value of 2 since wait must be called twice before the barrier is released on both threads. This behavior might not be desirable when calling shiny_function multiple times so you could also substitute another concurrency structure that only blocks in a single location.
use std::sync::{Arc, Barrier};
#[test]
fn progress_cb() {
let cut = ... // cut stands for Class Under Test
// Create a barrier for this thread and clone it to move into the closure
let barrier = Arc::new(Barrier::new(2));
let barrier_clone = barrier.clone();
cut.shiny_function(Arc::new(Mutex::new(move |percent| {
// Perform tests
assert_eq!(percent, foo);
// Once we finish we can trigger the barrier so the outer thread can continue
barrier_clone.wait();
})));
// Don't exit the function until the barrier has been resolved in the callback
barrier.wait();
}
Edit: Here is a struct you could use to if the barrier starts becoming an issue due to the closure blocking on every call and holding up later calls to shiny_function in a single test function.
use std::sync::{Arc, Mutex, Condvar};
pub struct SingleBlockingBarrier {
target: u32,
counter: Mutex<u32>,
lock: Condvar,
}
impl SingleBlockingBarrier {
pub fn new(target: u32) -> Arc<Self> {
Arc::new(SingleBlockingBarrier {
target,
counter: Mutex::new(0),
lock: Condvar::new(),
})
}
pub fn signal(&self) {
let mut guard = self.counter.lock().unwrap();
*guard += 1;
if *guard >= self.target {
self.lock.notify_all();
}
}
// Block until signal has been called the target number of times
pub fn block_until_finished(&self) {
let mut guard = self.counter.lock().unwrap();
loop {
if *guard >= self.target {
return;
}
guard = self.lock.wait(guard).unwrap();
}
}
}
#[test]
fn progress_cb() {
let cut = ... // cut stands for Class Under Test
// Create a barrier for this thread and clone it to move into the closure
let barrier = SingleBlockingBarrier::new(10);
for _ in 0..10 {
let barrier_clone = barrier.clone();
cut.shiny_function(Arc::new(Mutex::new(move |percent| {
// Perform tests
assert_eq!(percent, foo);
// Notify barrier that a worker has finished without blocking
barrier_clone.signal();
})));
}
// Block until all non-blocking barriers have been reached
barrier.block_until_finished();
}
I'm writing an application that the user can start with a number of "jobs" (URLs actually). At the beginning (main routine), I add these URLs to a queue, then start x goroutines that work on these URLs.
In special cases, the resource a URL points to may contain even more URLs which have to be added to the queue. The 3 workers are waiting for new jobs to come in and process them. The problem is: once EVERY worker is waiting for a job (and none is producing any), the workers should stop altogether. So either all of them work or no one works.
My current implementation looks something like this and I don't think it's elegant. Unfortunately I couldn't think of a better way that wouldn't include race conditions and I'm not entirely sure if this implementation actually works as intended:
var queue // from somewhere
const WORKER_COUNT = 3
var done chan struct{}
func work(working chan int) {
absent := make(chan struct{}, 1)
// if x>1 jobs in sequence are popped, send to "absent" channel only 1 struct.
// This implementation also assumes that the select statement will be evaluated "in-order" (channel 2 only if channel 1 yields nothing) - is this actually correct? EDIT: It is, according to the specs.
one := false
for {
select {
case u, ok := <-queue.Pop():
if !ok {
close(absent)
return
}
if !one {
// I have started working (delta + 1)
working <- 1
absent <- struct{}{}
one = true
}
// do work with u (which may lead to queue.Push(urls...))
case <-absent: // no jobs at the moment. consume absent => wait
one = false
working <- -1
}
}
}
func Start() {
working := make(chan int)
for i := 0; i < WORKER_COUNT; i++ {
go work(working)
}
// the amount of actually working workers...
sum := 0
for {
delta := <-working
sum += delta
if sum == 0 {
queue.Close() // close channel -> kill workers.
done <- struct{}{}
return
}
}
}
Is there a better way to tackle this problem?
You can use a sync.WaitGroup (see docs) to control the lifetime of the workers, and use a non-blocking send so workers can't deadlock when they try to queue up more jobs:
package main
import "sync"
const workers = 4
type job struct{}
func (j *job) do(enqueue func(job)) {
// do the job, calling enqueue() for subtasks as needed
}
func main() {
jobs, wg := make(chan job), new(sync.WaitGroup)
var enqueue func(job)
// workers
for i := 0; i < workers; i++ {
go func() {
for j := range jobs {
j.do(enqueue)
wg.Done()
}
}()
}
// how to queue a job
enqueue = func(j job) {
wg.Add(1)
select {
case jobs <- j: // another worker took it
default: // no free worker; do the job now
j.do(enqueue)
wg.Done()
}
}
todo := make([]job, 1000)
for _, j := range todo {
enqueue(j)
}
wg.Wait()
close(jobs)
}
The difficulty with trying to avoid deadlocks with a buffered channel is that you have to allocate a big enough channel up front to definitely hold all pending tasks without blocking. Problematic unless, say, you have a small and known number of URLs to crawl.
When you fall back to doing ordinary recursion in the current thread, you don't have that static buffer-size limit. Of course, there are still limits: you'd probably run out of RAM if too much work were pending, and theoretically you could exhaust the stack with deep recursion (but that's hard!). So you'd need to track pending tasks some more sophisticated way if you were, say, crawling the Web at large.
Finally, as a more complete example, I'm not super proud of this code, but I happened to write a function to kick off a parallel sort that's recursive in the same way your URL fetching is.
I am running the following go code and it doesn't produce output:
package main
import "fmt"
//import "strconv"
import "time"
func Wait(){
time.Sleep(2000 * time.Millisecond)
}
func Print(c chan string){
fmt.Println("Running Print go-routine")
for{
fmt.Println("len(c): ", len(c))
str := <- c
fmt.Println(str)
}
}
func main() {
c := make(chan string, 4)
c <- "0"
c <- "1"
c <- "2"
c <- "3"
Wait()
fmt.Println("Before go Print(c)")
go Print(c)
fmt.Println("After go Print(c)")
}
Is there a deadlock? The Print(c) function is not even called...? Its very strange to me.
The link to it in go playground is: http://play.golang.org/p/tDjEJKwkRJ
There isn't an error, the Print() function is called in a goroutine, but the main program is exiting right after that... so the goroutine is terminated.
Read this talk: Go Concurrency Patterns (or better, view it's video), to understand how channels and goroutines work.
Keep in mind that the program is finished when the main function returns.
When the main function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.
Call your Wait function later, after the go Print(c) statement. For example,
package main
import (
"fmt"
"time"
)
func Wait() {
time.Sleep(2000 * time.Millisecond)
}
func Print(c chan string) {
fmt.Println("Running Print go-routine")
for {
fmt.Println("len(c): ", len(c))
str := <-c
fmt.Println(str)
}
}
func main() {
c := make(chan string, 4)
c <- "0"
c <- "1"
c <- "2"
c <- "3"
fmt.Println("Before go Print(c)")
go Print(c)
Wait()
fmt.Println("After go Print(c)")
}
Output:
Before go Print(c)
Running Print go-routine
len(c): 4
0
len(c): 3
1
len(c): 2
2
len(c): 1
3
len(c): 0
After go Print(c)
The Go Programming Language Specification
Program execution
A complete program is created by linking a single, unimported package
called the main package with all the packages it imports,
transitively. The main package must have package name main and declare
a function main that takes no arguments and returns no value.
func main() { … }
Program execution begins by initializing the main package and then
invoking the function main. When that function invocation returns, the
program exits. It does not wait for other (non-main) goroutines to
complete.
I'm trying to implement a pool of workers in Go. The go-wiki (and Effective Go in the Channels section) feature excellent examples of bounding resource use. Simply make a channel with a buffer that's as large as the worker pool. Then fill that channel with workers, and send them back into the channel when they're done. Receiving from the channel blocks until a worker is available. So the channel and a loop is the entire implementation -- very cool!
Alternatively one could block on sending into the channel, but same idea.
My question is about changing the size of the worker pool while it's running. I don't believe there's a way to change the size of a channel. I have some ideas, but most of them seem way too complicated. This page actually implements a semaphore using a channel and empty structs in much the same way, but it has the same problem (these things come up all the time while Googling for "golang semaphore".
I would do it the other way round. Instead of spawning many goroutines (which still require a considerable amount of memory) and use a channel to block them, I would model the workers as goroutines and use a channel to distribute the work. Something like this:
package main
import (
"fmt"
"sync"
)
type Task string
func worker(tasks <-chan Task, quit <-chan bool, wg *sync.WaitGroup) {
defer wg.Done()
for {
select {
case task, ok := <-tasks:
if !ok {
return
}
fmt.Println("processing task", task)
case <-quit:
return
}
}
}
func main() {
tasks := make(chan Task, 128)
quit := make(chan bool)
var wg sync.WaitGroup
// spawn 5 workers
for i := 0; i < 5; i++ {
wg.Add(1)
go worker(tasks, quit, &wg)
}
// distribute some tasks
tasks <- Task("foo")
tasks <- Task("bar")
// remove two workers
quit <- true
quit <- true
// add three more workers
for i := 0; i < 3; i++ {
wg.Add(1)
go worker(tasks, quit, &wg)
}
// distribute more tasks
for i := 0; i < 20; i++ {
tasks <- Task(fmt.Sprintf("additional_%d", i+1))
}
// end of tasks. the workers should quit afterwards
close(tasks)
// use "close(quit)", if you do not want to wait for the remaining tasks
// wait for all workers to shut down properly
wg.Wait()
}
It might be a good idea to create a separate WorkerPool type with some convenient methods. Also, instead of type Task string it is quite common to use a struct that also contains a done channel that is used to signal that the task had been executed successfully.
Edit: I've played around a bit more and came up with the following: http://play.golang.org/p/VlEirPRk8V. It's basically the same example, with a nicer API.
A simple change that can think is to have a channel that controls how big is the semaphore.
The relevant part is the select statements. If there is more work from the queue process it with the current semaphore. If there is a request to change the size of the semaphore change it and continue processing the req queue with the new semaphore. Note that the old one is going to be garbage collected.
package main
import "time"
import "fmt"
type Request struct{ num int }
var quit chan struct{} = make(chan struct{})
func Serve(queue chan *Request, resize chan int, semsize int) {
for {
sem := make(chan struct{}, semsize)
var req *Request
select {
case semsize = <-resize:
{
sem = make(chan struct{}, semsize)
fmt.Println("changing semaphore size to ", semsize)
}
case req = <-queue:
{
sem <- struct{}{} // Block until there's capacity to process a request.
go handle(req, sem) // Don't wait for handle to finish.
}
case <-quit:
return
}
}
}
func process(r *Request) {
fmt.Println("Handled Request", r.num)
}
func handle(r *Request, sem chan struct{}) {
process(r) // May take a long time & use a lot of memory or CPU
<-sem // Done; enable next request to run.
}
func main() {
workq := make(chan *Request, 1)
ctrlq := make(chan int)
go func() {
for i := 0; i < 20; i += 1 {
<-time.After(100 * time.Millisecond)
workq <- &Request{i}
}
<-time.After(500 * time.Millisecond)
quit <- struct{}{}
}()
go func() {
<-time.After(500 * time.Millisecond)
ctrlq <- 10
}()
Serve(workq, ctrlq, 1)
}
http://play.golang.org/p/AHOLlAv2LH
I'm writing a function where I'm trying to increment a channel. In a much larger program, this is not working and it actually hangs on a line that looks like:
current = <-channel
The go funcs are running, but the program seems to halt on this line.
I tried to write a smaller SSCCE, but now I'm having a different problem. Here it is:
package main
import (
"fmt"
)
func main() {
count := make(chan int)
go func(count chan int) {
current := 0
for {
current = <-count
current++
count <- current
fmt.Println(count)
}
}(count)
}
However, in the above the go func does not actually seem to be called at all. If I put a fmt.Println statement before for {, it does not print out. If I put fmt.Println statements before or after they go func block, they will both print out.
Why does the self-calling block in the above example not seem to run at all?
If it were running, why would it block on current = <-count? How could I properly increment the channel?
I can't answer the first one issue without more info. The code you did show has two issues. First, the program exits after the goroutine is started. The second issue is that the goroutine is waiting for something to be sent to count, if you receive from the count channel it will not deadlock.
Here is an example showing the deadlock (http://play.golang.org/p/cRgjZt7U2A):
package main
import (
"fmt"
)
func main() {
count := make(chan int)
go func() {
current := 0
for {
current = <-count
current++
count <- current
fmt.Println(count)
}
}()
fmt.Println(<-count)
}
Here is an example of it working the way I think you are expecting (http://play.golang.org/p/QQnRpCDODu)
package main
import (
"fmt"
)
func main() {
count := make(chan int)
go func() {
current := 0
for {
current = <-count
current++
count <- current
fmt.Println(count)
}
}()
count <- 1
fmt.Println(<-count)
}
Channel :- Channel is something that can't store the value. It can only buffer the value so the basic usage is it can send and receive the value. So when you declare count := make(chan int) it does not contain any value. So the statement current = <-count will give you error that all go routines are asleep. Basically channel was design to work as communicator for different go routines which are running on different process and your main function is running on different process.
So your answer to first question is:-
1.Why does the self-calling block in the above example not seem to run at all?
Answer- See the main function you are running has one process and the go routine is running on another process so if your main function gets its execution completed before your go-routine than you will never get result form go-routine because your main thread gets dead after the execution is complete. So i am providing you a web example which is related to your example of incrementing the counter. In this example you will create a server and listen on port 8000.First of all run this example and go in your web browser and type localhost:8000 and it will so you the incrementing counter that channel stores in every buffer. This example will provide you an idea of how channel works.
2.If it were running, why would it block on current = <-count? How could I properly increment the channel?
Answer-You are receiving from the channel but channel does not have anything in its buffer so you will get an error "All go-routines are asleep". First you should transfer value into the channel and correspondingly receive it otherwise it will again go to deadlock.
package main
import (
"fmt"
"http"
)
type webCounter struct {
count chan int
}
func NewCounter() *webCounter {
counter := new(webCounter)
counter.count = make(chan int, 1)
go func() {
for i:=1 ;; i++ { counter.count <- i }
}()
return counter
}
func (w *webCounter) ServeHTTP(r http.ResponseWriter, rq *http.Request) {
if rq.URL.Path != "/" {
r.WriteHeader(http.StatusNotFound)
return
}
fmt.Fprintf(r, "You are visitor %d", <-w.count)
}
func main() {
http.ListenAndServe(":8000", NewCounter());
}