Playing around with Go's channels and routines I have come across a peculiar behaviour I was hoping somebody could explain.
Below is a short program that is supposed to print a couple strings to stdout, by sending the strings through a channel to a "listener" (the select statement) running in a separate goroutine.
package main
import (
"fmt"
"time"
)
func main() {
a := make(chan string)
go func() {
for {
select {
case <-a:
fmt.Print(<-a)
}
}
}()
a <- "Hello1\n"
a <- "Hello2\n"
a <- "Hello3\n"
a <- "Hello4\n"
time.Sleep(time.Second)
}
Using
go func() {
for s := range a {
fmt.Print(s)
}
}()
// or even simpler
go func() {
for {
fmt.Print(<-a)
}
}()
works as expected. However, running the uppermost snippet with the select statement produces the following output:
Hello2
Hello4
i.e. only every other statement is printed. What kind of sorcery is this?
In the uppermost snippet, you're pulling two values from the channel for each loop. One in the select statement and one in the print statement.
Change
select {
case <-a:
fmt.Print(<-a)
To
select {
case val := <-a:
fmt.Print(val)
http://play.golang.org/p/KIADcwkoKs
<-a
gets a value from the channel, destructively. So in your code you get two values, one in the select statement, and one to print. The one received in the select statement is not bound to any variable, and is therefore lost.
Try
select {
case val := <-a:
fmt.Print(val)
instead, to get only one value, bind it to variable val, and print it out.
package main
import (
"fmt"
"time"
)
func main() {
a := make(chan string)
go func() {
for {
select {
case v:= <-a:
fmt.Print(v)
}
}
}()
a <- "Hello1\n"
a <- "Hello2\n"
a <- "Hello3\n"
a <- "Hello4\n"
time.Sleep(5*time.Second)
}
Related
I've written a Go application, and all of the packages have full test coverage. I'm in the process of writing my main package - which will handle all of the initial setup for the application in the main() function - this function currently reads in 14 environment variables and then sets the relevant variable in the application. A simple overview of the code is:
func main() {
myStruct1 := privatePackage.myStructType{}
myStruct2 := publicPackage.otherStructType{}
if config1 := os.Getenv("CONFIG_FOO"); config1 != "" {
myStruct1.attribute1 = config1
}
// ....
if config14 := os.Getenv("CONFIG_BAR"); config14 != "" {
myStruct2.attribute5 = config14
}
}
When I test unit env variables/OS args, I typically just set the env variable directly in the test function - so something like:
func TestMyArgument(t *testing.T) {
os.Setenv("CONFIG_BAZ", "apple")
//Invoke function that depends on CONFIG_BAZ
//Assert that expected outcome occurred
}
I pretty much always use table-driven tests, so the above snippet is a simplified example.
The issue is that my main() function takes in 14 (and growing) env variables, and whilst some env variables are essentially enums (so there's a small number of valid options - for example there's a small number of database drivers to choose from), other env variables have virtually unlimited potential values. So how can I effectively cover all of the (or enough of the) permutations of potential configs?
EDIT: When this application is deployed, it's going into a K8s cluster. Some of these variables are secrets that will be pulled in from secure store. Using a JSON file isn't viable because some of the values need to be encrypted/changed easily.
Also, using a JSON file would require me to store this file and share it between hundreds/thousands of running pods - this storage would then act as a point of failure.
To clarify, this question isn't about env vars VS config files; this question is about the best way to approach testing when there's a significant number of configurable variables - with each variables having a vast number of potential values - resulting in thousands of possible configuration permutations. How do I guarantee sufficient test coverage in such a scenario?
#Steven Penny is right: uses json
and use reflect can make the code more simple:
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
"strconv"
)
type MyStructType struct {
Attribute1 string `json:"CONFIG_FOO"`
Attribute2 string `json:"CONFIG_BAZ"`
Attribute3 int `json:"CONFIG_BAR"`
}
func NewMyStructTypeFormEnv() *MyStructType {
myStructType := MyStructType{}
ReflectMyStructType(&myStructType)
fmt.Println("myStructType is now", myStructType)
return &myStructType
}
func NewMyStructTypeFormJson() *MyStructType {
myStructType := MyStructType{}
f, e := os.Open("file.json")
if e != nil {
panic(e)
}
defer f.Close()
json.NewDecoder(f).Decode(&myStructType)
fmt.Println("myStructType is now", myStructType)
return &myStructType
}
func ReflectMyStructType(ptr interface{}){
v := reflect.ValueOf(ptr).Elem()
fmt.Printf("%v\n", v.Type())
for i := 0; i < v.NumField(); i++ {
env_str := v.Type().Field(i).Tag.Get("json")
if(env_str == ""){continue}
if config := os.Getenv(env_str); config != "" {
if v.Field(i).Kind() == reflect.String{
v.Field(i).SetString(config)
}else if v.Field(i).Kind() == reflect.Int{
iConfig,_ := strconv.Atoi(config)
v.Field(i).SetInt(int64(iConfig))
}
}
}
}
func main() {
NewMyStructTypeFormJson()
os.Setenv("CONFIG_FOO", "apple")
os.Setenv("CONFIG_BAZ", "apple")
os.Setenv("CONFIG_BAR", "1")
NewMyStructTypeFormEnv()
}
Beyond one or two, I don't think using environment variables is the right approach, unless it's required (calling something with os/exec). Instead, would be better to read from a config file. Here is an example with JSON:
{
"CONFIG_BAR": "east",
"CONFIG_BAZ": "south",
"CONFIG_FOO": "north"
}
package main
import (
"encoding/json"
"fmt"
"os"
)
func main() {
f, e := os.Open("file.json")
if e != nil {
panic(e)
}
defer f.Close()
var s struct { CONFIG_BAR, CONFIG_BAZ, CONFIG_FOO string }
json.NewDecoder(f).Decode(&s)
// {CONFIG_BAR:east CONFIG_BAZ:south CONFIG_FOO:north}
fmt.Printf("%+v\n", s)
}
TOML would be a good choice as well.
https://golang.org/pkg/encoding/json
https://pkg.go.dev/github.com/pelletier/go-toml
I'm trying to port a simple synchronous bit of PHP to Go, but am having a hard time getting my head around how concurrency works with regards to channels. The PHP script makes a request to get a list of media library sections, then makes requests to get the items within each of these sections. If the section is a list of TV Shows, it then makes a request for each show to get all the seasons and then another to get the episodes within each season.
I've trying writing in pidgeon-go what I expected to work, but I'm not having any luck. I've tried various channel guides online, but normally end up with deadlock warnings. Currently this example warns about item := <-ch used as value and doesn't look like it's waiting on the goroutines to return. Does anyone have any ideas what I can do?
package main
import (
"fmt"
"time"
)
// Get all items for all sections
func main() {
ch := make(chan string)
sections := getSections()
for _, section := range sections {
go getItemsInSection(section, ch)
}
items := make([]string, 0)
for item := <- ch {
items = append(items, item)
}
fmt.Println(items)
}
// Return a list of the various library sections
func getSections() []string {
return []string{"HD Movies", "Movies", "TV Shows"}
}
// Get items within the given section, note that some items may spawn sub-items
func getItemsInSection(name string, ch chan string) {
time.Sleep(1 * time.Second)
switch name {
case "HD Movies":
ch <- "Avatar"
ch <- "Avengers"
case "Movies":
ch <- "Aliens"
ch <- "Abyss"
case "TV Shows":
go getSubItemsForItem("24", ch)
go getSubItemsForItem("Breaking Bad", ch)
}
}
// Get sub-items for a given parent
func getSubItemsForItem(name string, ch chan string) {
time.Sleep(1 * time.Second)
ch <- name + ": S01E01"
ch <- name + ": S01E02"
}
First, that code doesn't compile because for item := <- ch should be for item := range ch
Now the problem is you either have to close the channel or run your loop forever inside a goroutine.
go func() {
for {
item, ok := <-ch
if !ok {
break
}
fmt.Println(item)
items = append(items, item)
}
}()
time.Sleep(time.Second)
fmt.Println(items)
playground
This is doing my head in, I cant figure out how to solve it;
I want to have a fixed number N of goroutines running in parallell
From a never-ending queue I will fetch X msg about jobs to process
I want to let the N goroutines process these X jobs, and as soon as one of the routines have nothing more to do, I want to fetch another X jobs from the neverending queue
The code in the answer below (see url) works brilliantly to process the tasks, but the workers will die once that tasks list is empty, I want them to stay alive and somehow notify the main code that they are out of work so I can fetch more jobs to fill the tasks list with tasks
How would you define a pool of goroutines to be executed at once in Golang?
Using user:Jsor example code from below, I try to create a simple program, but I am confused.
import (
"fmt"
"strconv"
)
//workChan - read only that delivers work
//requestChan - ??? what is this
func Worker(myid string, workChan <- chan string, requestChan chan<- struct{}) {
for {
select {
case work := <-workChan:
fmt.Println("Channel: " + myid + " do some work: " + work)
case requestChan <- struct{}{}:
//hm? how is the requestChan used?
}
}
}
func Logic(){
workChan := make(chan string)
requestChan := make(chan struct{})
//Create the workers
for i:=1; i < 5; i++ {
Worker( strconv.Itoa( i), workChan, requestChan)
}
//Give the workers some work
for i:=100; i < 115; i++ {
workChan<- "workid"+strconv.Itoa( i)
}
}
This is what the select statement is for.
func Worker(workChan chan<- Work, requestChan chan<- struct{}) {
for {
select {
case work := <-workChan:
// Do work
case requestChan <- struct{}{}:
}
}
}
This worker will run forever and ever. If work is available, it will pull it from the worker channel. If there's nothing left it will send a request.
Not that since it runs forever and ever, if you want to be able to kill a worker you need to do something else. One possibility is to always check ok with workChan and if that channel is closed quit the function. Another option is to use an individual quit channel for each worker.
Compared to the other solution you posted, you just need (first) not to close the channel, and just keep feeding items to it.
Then you need to answer the following question: is it absolutely necessary that (a) you fetch the next X items from your queue only once one of the workers has “nothing more to do” (or, what is the same, once the first X items are either fully processed, or assigned to a worker); or (b) is it okay if you keep the second set of X items in memory, and go feeding them to the workers as new work items are needed?
As I understand it, only (a) needs the requestChan you’re wondering about (see below). For (b), something as simple as the following would suffice:
# B version
type WorkItem int
const (
N = 5 // Number of workers
X = 15 // Number of work items to get from the infinite queue at once
)
func Worker(id int, workChan <-chan WorkItem) {
for {
item := <-workChan
doWork(item)
fmt.Printf("Worker %d processes item #%v\n", id, item)
}
}
func Dispatch(workChan chan<- WorkItem) {
for {
items := GetNextFromQueue(X)
for _, item := range items {
workChan <- item
fmt.Printf("Dispatched item #%v\n", item)
}
}
}
func main() {
workChan := make(chan WorkItem) // Shared amongst all workers; could make it buffered if GetNextFromQueue() is slow.
// Start N workers.
for i := 0; i < N; i++ {
go Worker(i, workChan)
}
// Dispatch items to the workers.
go Dispatch(workChan)
time.Sleep(20 * time.Second) // Ensure main(), and our program, finish.
}
(I’ve uploaded to the Playground a full working solution for (b).)
As for (a), the workers change to say: do work, or if there’s no more work, tell the dispatcher to get more via the reqChan communication channel. That “or” is implemented via select. Then, the dispatcher waits on reqChan before making another call to GetNextFromQueue(). It’s more code, but ensures the semantics that you might be interested in. (The previous version is overall simpler, though.)
# A version
func Worker(id int, workChan <-chan WorkItem, reqChan chan<- int) {
for {
select {
case item := <-workChan:
doWork(item)
fmt.Printf("Worker %d processes item #%v\n", id, item)
case reqChan <- id:
fmt.Printf("Worker %d thinks they requested more work\n", id)
}
}
}
func Dispatch(workChan chan<- WorkItem, reqChan <-chan int) {
for {
items := GetNextFromQueue(X)
for _, item := range items {
workChan <- item
fmt.Printf("Dispatched item #%v\n", item)
}
id := <-reqChan
fmt.Printf("Polling the queue in Dispatch() at the request of worker %d\n", id)
}
}
(I’ve also uploaded to the Playground a full working solution for (a).)
I'm writing a function where I'm trying to increment a channel. In a much larger program, this is not working and it actually hangs on a line that looks like:
current = <-channel
The go funcs are running, but the program seems to halt on this line.
I tried to write a smaller SSCCE, but now I'm having a different problem. Here it is:
package main
import (
"fmt"
)
func main() {
count := make(chan int)
go func(count chan int) {
current := 0
for {
current = <-count
current++
count <- current
fmt.Println(count)
}
}(count)
}
However, in the above the go func does not actually seem to be called at all. If I put a fmt.Println statement before for {, it does not print out. If I put fmt.Println statements before or after they go func block, they will both print out.
Why does the self-calling block in the above example not seem to run at all?
If it were running, why would it block on current = <-count? How could I properly increment the channel?
I can't answer the first one issue without more info. The code you did show has two issues. First, the program exits after the goroutine is started. The second issue is that the goroutine is waiting for something to be sent to count, if you receive from the count channel it will not deadlock.
Here is an example showing the deadlock (http://play.golang.org/p/cRgjZt7U2A):
package main
import (
"fmt"
)
func main() {
count := make(chan int)
go func() {
current := 0
for {
current = <-count
current++
count <- current
fmt.Println(count)
}
}()
fmt.Println(<-count)
}
Here is an example of it working the way I think you are expecting (http://play.golang.org/p/QQnRpCDODu)
package main
import (
"fmt"
)
func main() {
count := make(chan int)
go func() {
current := 0
for {
current = <-count
current++
count <- current
fmt.Println(count)
}
}()
count <- 1
fmt.Println(<-count)
}
Channel :- Channel is something that can't store the value. It can only buffer the value so the basic usage is it can send and receive the value. So when you declare count := make(chan int) it does not contain any value. So the statement current = <-count will give you error that all go routines are asleep. Basically channel was design to work as communicator for different go routines which are running on different process and your main function is running on different process.
So your answer to first question is:-
1.Why does the self-calling block in the above example not seem to run at all?
Answer- See the main function you are running has one process and the go routine is running on another process so if your main function gets its execution completed before your go-routine than you will never get result form go-routine because your main thread gets dead after the execution is complete. So i am providing you a web example which is related to your example of incrementing the counter. In this example you will create a server and listen on port 8000.First of all run this example and go in your web browser and type localhost:8000 and it will so you the incrementing counter that channel stores in every buffer. This example will provide you an idea of how channel works.
2.If it were running, why would it block on current = <-count? How could I properly increment the channel?
Answer-You are receiving from the channel but channel does not have anything in its buffer so you will get an error "All go-routines are asleep". First you should transfer value into the channel and correspondingly receive it otherwise it will again go to deadlock.
package main
import (
"fmt"
"http"
)
type webCounter struct {
count chan int
}
func NewCounter() *webCounter {
counter := new(webCounter)
counter.count = make(chan int, 1)
go func() {
for i:=1 ;; i++ { counter.count <- i }
}()
return counter
}
func (w *webCounter) ServeHTTP(r http.ResponseWriter, rq *http.Request) {
if rq.URL.Path != "/" {
r.WriteHeader(http.StatusNotFound)
return
}
fmt.Fprintf(r, "You are visitor %d", <-w.count)
}
func main() {
http.ListenAndServe(":8000", NewCounter());
}
I would like a unit test that verifies a particular command line flag is within an enumeration.
Here is the code I would like to write tests against:
var formatType string
const (
text = "text"
json = "json"
hash = "hash"
)
func init() {
const (
defaultFormat = "text"
formatUsage = "desired output format"
)
flag.StringVar(&formatType, "format", defaultFormat, formatUsage)
flag.StringVar(&formatType, "f", defaultFormat, formatUsage+" (shorthand)")
}
func main() {
flag.Parse()
}
The desired test would pass only if -format equalled one of the const values given above. This value would be available in formatType. An example correct call would be: program -format text
What is the best way to test the desired behaviors?
Note: Perhaps I have phrased this poorly, but the displayed code it not the unit test itself, but the code I want to write unit tests against. This is a simple example from the tool I am writing and wanted to ask if there were a good way to test valid inputs to the tool.
Custom testing and processing of flags can be achieved with the flag.Var function in the flag package.
Flag.Var "defines a flag with the specified name and usage string. The type and value of the flag are represented by the first argument, of type Value, which typically holds a user-defined implementation of Value."
A flag.Value is any type that satisfies the Value interface, defined as:
type Value interface {
String() string
Set(string) error
}
There is a good example in the example_test.go file in the flag package source
For your use case you could use something like:
package main
import (
"errors"
"flag"
"fmt"
)
type formatType string
func (f *formatType) String() string {
return fmt.Sprint(*f)
}
func (f *formatType) Set(value string) error {
if len(*f) > 0 && *f != "text" {
return errors.New("format flag already set")
}
if value != "text" && value != "json" && value != "hash" {
return errors.New("Invalid Format Type")
}
*f = formatType(value)
return nil
}
var typeFlag formatType
func init() {
typeFlag = "text"
usage := `Format type. Must be "text", "json" or "hash". Defaults to "text".`
flag.Var(&typeFlag, "format", usage)
flag.Var(&typeFlag, "f", usage+" (shorthand)")
}
func main() {
flag.Parse()
fmt.Println("Format type is", typeFlag)
}
This is probably overkill for such a simple example, but may be very useful when defining more complex flag types (The linked example converts a comma separated list of intervals into a slice of a custom type based on time.Duration).
EDIT: In answer to how to run unit tests against flags, the most canonical example is flag_test.go in the flag package source. The section related to testing custom flag variables starts at Line 181.
You can do this
func main() {
var name string
var password string
flag.StringVar(&name, "name", "", "")
flag.StringVar(&password, "password", "", "")
flag.Parse()
for _, v := range os.Args {
fmt.Println(v)
}
if len(strings.TrimSpace(name)) == 0 || len(strings.TrimSpace(password)) == 0 {
log.Panicln("no name or no passward")
}
fmt.Printf("name:%s\n", name)
fmt.Printf("password:%s\n", password)
}
func TestMainApp(t *testing.T) {
os.Args = []string{"test", "-name", "Hello", "-password", "World"}
main()
}
You can test main() by:
Making a test that runs a command
Which then calls the app test binary, built from go test, directly
Passing the desired flags you want to test
Passing back the exit code, stdout, and stderr which you can assert on.
NOTE This only works when main exits, so that the test does not run infinitely, or gets caught in a recursive loop.
Given your main.go looks like:
package main
import (
"flag"
"fmt"
"os"
)
var formatType string
const (
text = "text"
json = "json"
hash = "hash"
)
func init() {
const (
defaultFormat = "text"
formatUsage = "desired output format"
)
flag.StringVar(&formatType, "format", defaultFormat, formatUsage)
flag.StringVar(&formatType, "f", defaultFormat, formatUsage+" (shorthand)")
}
func main() {
flag.Parse()
fmt.Printf("format type = %v\n", formatType)
os.Exit(0)
}
Your main_test.go may then look something like:
package main
import (
"fmt"
"os"
"os/exec"
"path"
"runtime"
"strings"
"testing"
)
// This will be used to pass args to app and keep the test framework from looping
const subCmdFlags = "FLAGS_FOR_MAIN"
func TestMain(m *testing.M) {
// Only runs when this environment variable is set.
if os.Getenv(subCmdFlags) != "" {
runAppMain()
}
// Run all tests
exitCode := m.Run()
// Clean up
os.Exit(exitCode)
}
func TestMainForCorrectness(tester *testing.T) {
var tests = []struct {
name string
wantCode int
args []string
}{
{"formatTypeJson", 0, []string{"-format", "json"}},
}
for _, test := range tests {
tester.Run(test.name, func(t *testing.T) {
cmd := getTestBinCmd(test.args)
cmdOut, cmdErr := cmd.CombinedOutput()
got := cmd.ProcessState.ExitCode()
// Debug
showCmdOutput(cmdOut, cmdErr)
if got != test.wantCode {
t.Errorf("unexpected error on exit. want %q, got %q", test.wantCode, got)
}
})
}
}
// private helper methods.
// Used for running the application's main function from other test.
func runAppMain() {
// the test framework has process its flags,
// so now we can remove them and replace them with the flags we want to pass to main.
// we are pulling them out of the environment var we set.
args := strings.Split(os.Getenv(subCmdFlags), " ")
os.Args = append([]string{os.Args[0]}, args...)
// Debug stmt, can be removed
fmt.Printf("\nos args = %v\n", os.Args)
main() // will run and exit, signaling the test framework to stop and return the exit code.
}
// getTestBinCmd return a command to run your app (test) binary directly; `TestMain`, will be run automatically.
func getTestBinCmd(args []string) *exec.Cmd {
// call the generated test binary directly
// Have it the function runAppMain.
cmd := exec.Command(os.Args[0], "-args", strings.Join(args, " "))
// Run in the context of the source directory.
_, filename, _, _ := runtime.Caller(0)
cmd.Dir = path.Dir(filename)
// Set an environment variable
// 1. Only exist for the life of the test that calls this function.
// 2. Passes arguments/flag to your app
// 3. Lets TestMain know when to run the main function.
subEnvVar := subCmdFlags + "=" + strings.Join(args, " ")
cmd.Env = append(os.Environ(), subEnvVar)
return cmd
}
func showCmdOutput(cmdOut []byte, cmdErr error) {
if cmdOut != nil {
fmt.Printf("\nBEGIN sub-command out:\n%v", string(cmdOut))
fmt.Print("END sub-command\n")
}
if cmdErr != nil {
fmt.Printf("\nBEGIN sub-command stderr:\n%v", cmdErr.Error())
fmt.Print("END sub-command\n")
}
}
I'm not sure whether we agree on the term 'unit test'. What you want to achieve seems to me
more like a pretty normal test in a program. You probably want to do something like this:
func main() {
flag.Parse()
if formatType != text || formatType != json || formatType != hash {
flag.Usage()
return
}
// ...
}
Sadly, it is not easily possible to extend the flag Parser with own value verifiers
so you have to stick with this for now.
See Intermernet for a solution which defines a custom format type and its validator.