I'm trying to run several tasks concurrently and get the result or error back.
//data channels
ch := make(chan int)
ch2 := make(chan int)
ch2 := make(chan int)
//error channels
errCh := make(chan error)
errCh2 := make(chan error)
errCh3 := make(chan error)
//functions
go taskF(ch, errCh)
go taskF2(ch2, errCh2)
go taskF3(ch3, errCh3)
Then I start checking on each error. If there is any error we print it otherwise we print the result of each task
err := <-errCh
if err != nil{
print('we have an error ')
return
}
err2 := <-errCh2
if err2 != nil{
print('we have an error 2')
return
}
err3 := <-errCh3
if err3!= nil{
print('we have an error 3')
return
}
Then if there is no error I collect the values returned though each channel
task := <-ch
task2 := <-ch2
task3 := <-ch3
print("task %v task2 %v task3 %v", task, task2, task3)
I'm wondering if I'm doing it right. I'm worried that the code quite verbose. I was thinking to use buffered channels for errors but i can't figure it out how to check all the errors. I think it would also be nice to somehow sync the errors within the goroutines so that if there is an error on one goroutine the other goroutines would stop but I don't know any way to do in an unblocking manner.
Consider using only one channel for synchronization, and having this channel include an error state (see the Result struct). Whenever you receive a Result, make sure the error state is nil. Wrapping each task in a Task struct will allow you to call Stop on each task when one returns an error. Depending on your exact application there may be better ways to handle this, such as WaitGroups (http://golang.org/pkg/sync/#WaitGroup).
type Result struct {
Val int
Err error
}
type Task struct {
stopped bool
}
func (t *Task) Stop() {
t.stopped = true
}
func (t *Task) Run(doneChan chan Result) {
// long-running task here
// periodically check t.stopped
doneChan <- Result{Val: ..., Err: nil}
}
Related
I am covering project with tests and for that purpose I need dummy TCP Server, which could accept connection, write/read data to/from it, close it etc... I have found this question on stack overflow, covering mocking connection, but it doesn't cover what I actually need to test.
My idea relies on this article as starting point, but when I started implementing channel to let server write some data to newly opened connection, I got stuck with undebuggable deadlock in writing to channel.
What I want to achieve is to write some data to server's channel, say sendingQueue chan *[]byte, so later corresponding []byte will be sent to newly established connection.
During these little research I have tried debugging and printing out messages before/after sending data to channel and trying to send / read data from channel in different places of program.
What I found out:
My idea works if I add data directly in handleConnection with
go func() {
f := []byte("test.")
t.sendingQueue <- &f
}()
My idea doesn't work if I push data to channel from TestUtils_TestingTCPServer_WritesRequest in any form, either with func (t *TCPServer) Put(data *[]byte) (err error) or directly with:
go func(queue chan *[]byte, data *[]byte) {
queue <- data
}(t.sendingQueue, &payload)
It doesn't matter if channel is buffered or not.
So, obviously, there is something wrong either with the way I debug my code (I didn't dive into cli dlv, using just IDE debugger), or something that I completely miss about working with go channels, goroutines or net.Conn module.
For convenience public gist with full code is available. Note — there is // INIT part in the TestUtils_TestingTCPServer_WritesRequest which is required to run/debug single test. It should be commented out when running go test in the directory.
utils.go:
// NewServer creates a new Server using given protocol
// and addr.
func NewTestingTCPServer(protocol, addr string) (*TCPServer, error) {
switch strings.ToLower(protocol) {
case "tcp":
return &TCPServer{
addr: addr,
sendingQueue: make(chan *[]byte, 10),
}, nil
case "udp":
}
return nil, errors.New("invalid protocol given")
}
// TCPServer holds the structure of our TCP
// implementation.
type TCPServer struct {
addr string
server net.Listener
sendingQueue chan *[]byte
}
func (t *TCPServer) Run() (err error) {}
func (t *TCPServer) Close() (err error) {}
func (t *TCPServer) Put(data *[]byte) (err error) {}
func (t *TCPServer) handleConnection(conn net.Conn){
// <...>
// Putting data here successfully sends it via freshly established
// Connection:
// go func() {
// f := []byte("test.")
// t.sendingQueue <- &f
// }()
for {
fmt.Printf("Started for loop\n")
select {
case data := <-readerChannel:
fmt.Printf("Read written data\n")
writeBuffer.Write(*data)
writeBuffer.Flush()
case data := <-t.sendingQueue:
fmt.Printf("Read pushed data\n")
writeBuffer.Write(*data)
writeBuffer.Flush()
case <-ticker:
fmt.Printf("Tick\n")
return
}
fmt.Printf("Finished for loop\n")
}
}
utils_test.go
func TestUtils_TestingTCPServer_WritesRequest(t *testing.T) {
payload := []byte("hello world\n")
// <...> In gist here is placed INIT piece, which
// is required to debug single test
fmt.Printf("Putting payload into queue\n")
// This doesn't affect channel
err = utilTestingSrv.Put(&payload)
assert.Nil(t, err)
// This doesn't work either
//go func(queue chan *[]byte, data *[]byte) {
// queue <- data
//}(utilTestingSrv.sendingQueue, &payload)
conn, err := net.Dial("tcp", ":41123")
if !assert.Nil(t, err) {
t.Error("could not connect to server: ", err)
}
defer conn.Close()
out := make([]byte, 1024)
if _, err := conn.Read(out); assert.Nil(t, err) {
// Need to remove trailing byte 0xa from bytes array to make sure bytes array are equal.
if out[len(payload)] == 0xa {
out[len(payload)] = 0x0
}
assert.Equal(t, payload, bytes.Trim(out, "\x00"))
} else {
t.Error("could not read from connection")
}
}
After a help from a colleague and reading the article on how init works, I found a problem.
It was in init function, which was recreating extra server, due to using := assignment. I also updated code to make sure server runs before net.Dial and conn.Read.
I wrote an SSH client in Go and I would like to write some tests. The problem is that I've never really written proper unit tests before, and most tutorials seem to focus on writing tests for a function that adds two numbers or some other toy problem. I've read about mocking, using interfaces, and other techniques, but I'm having trouble applying them. Also, my client is going to be used concurrently to allow fast configuration of multiple devices at a time. Not sure if that would change the way I write my tests or would add additional tests. Any help is appreciated.
Here is my code. Basically, a Device has 4 main functions: Connect, Send, Output/Err and Close for connecting to a device, sending it a set of configuration commands, capturing the output of the session, and closing the client, respectively.
package device
import (
"bufio"
"fmt"
"golang.org/x/crypto/ssh"
"io"
"net"
"time"
)
// A Device represents a remote network device.
type Device struct {
Host string // the device's hostname or IP address
client *ssh.Client // the client connection
session *ssh.Session // the connection to the remote shell
stdin io.WriteCloser // the remote shell's standard input
stdout io.Reader // the remote shell's standard output
stderr io.Reader // the remote shell's standard error
}
// Connect establishes an SSH connection to a device and sets up the session IO.
func (d *Device) Connect(user, password string) error {
// Create a client connection
client, err := ssh.Dial("tcp", net.JoinHostPort(d.Host, "22"), configureClient(user, password))
if err != nil {
return err
}
d.client = client
// Create a session
session, err := client.NewSession()
if err != nil {
return err
}
d.session = session
return nil
}
// configureClient sets up the client configuration for login
func configureClient(user, password string) *ssh.ClientConfig {
var sshConfig ssh.Config
sshConfig.SetDefaults()
sshConfig.Ciphers = append(sshConfig.Ciphers, "aes128-cbc", "aes256-cbc", "3des-cbc", "des-cbc", "aes192-cbc")
config := &ssh.ClientConfig{
Config: sshConfig,
User: user,
Auth: []ssh.AuthMethod{ssh.Password(password)},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Timeout: time.Second * 5,
}
return config
}
// setupIO creates the pipes connected to the remote shell's standard input, output, and error
func (d *Device) setupIO() error {
// Setup standard input pipe
stdin, err := d.session.StdinPipe()
if err != nil {
return err
}
d.stdin = stdin
// Setup standard output pipe
stdout, err := d.session.StdoutPipe()
if err != nil {
return err
}
d.stdout = stdout
// Setup standard error pipe
stderr, err := d.session.StderrPipe()
if err != nil {
return err
}
d.stderr = stderr
return nil
}
// Send sends cmd(s) to the device's standard input. A device only accepts one call
// to Send, as it closes the session and its standard input pipe.
func (d *Device) Send(cmds ...string) error {
if d.session == nil {
return fmt.Errorf("device: session is closed")
}
defer d.session.Close()
// Start the shell
if err := d.startShell(); err != nil {
return err
}
// Send commands
for _, cmd := range cmds {
if _, err := d.stdin.Write([]byte(cmd + "\r")); err != nil {
return err
}
}
defer d.stdin.Close()
// Wait for the commands to exit
d.session.Wait()
return nil
}
// startShell requests a pseudo terminal (VT100) and starts the remote shell.
func (d *Device) startShell() error {
modes := ssh.TerminalModes{
ssh.ECHO: 0, // disable echoing
ssh.OCRNL: 0,
ssh.TTY_OP_ISPEED: 14400,
ssh.TTY_OP_OSPEED: 14400,
}
err := d.session.RequestPty("vt100", 0, 0, modes)
if err != nil {
return err
}
if err := d.session.Shell(); err != nil {
return err
}
return nil
}
// Output returns the remote device's standard output output.
func (d *Device) Output() ([]string, error) {
return readPipe(d.stdout)
}
// Err returns the remote device's standard error output.
func (d *Device) Err() ([]string, error) {
return readPipe(d.stdout)
}
// reapPipe reads an io.Reader line by line
func readPipe(r io.Reader) ([]string, error) {
var lines []string
scanner := bufio.NewScanner(r)
for scanner.Scan() {
lines = append(lines, scanner.Text())
}
if err := scanner.Err(); err != nil {
return nil, err
}
return lines, nil
}
// Close closes the client connection.
func (d *Device) Close() error {
return d.client.Close()
}
// String returns the string representation of a `Device`.
func (d *Device) String() string {
return fmt.Sprintf("%s", d.Host)
}
You make a good point about unit test tutorials nearly always being toy problems (why is it always Fibonacci?), when what we have is databases and http servers. The big realization that helped me is that you can only unit test things where you can control the input and output of the unit. configureClient or readPipe (give it a strings.Reader) would be good candidates. Start there.
Anything that leaves your program by talking directly to the disk, the network, stdout, etc, like the Connect method you would consider part of the external interface of your program. You don't unit test those. You integration test them.
Change Device to be an interface rather than a struct, and make a MockDevice that implements it. The real device is now maybe SSHDevice. You can unit test the rest of your program (which uses Device interface) by inserting a MockDevice, to isolate yourself from the network.
The SSHDevice will get tested in your integration tests. Start a real ssh server (maybe a test one you write in Go using crypto/ssh package, but any sshd would work). Start your program with an SSHDevice, make them talk to each other, and check outputs. You'll be using the os/exec package a lot. Integration tests are even more fun to write than unit tests!
In golang syscall.GetLastError() doesn't return the last error. See the following example
if handle := _OpenSCManager(machineNamePtr, databaseNamePtr, desiredAccess); handle == nil {
if err := syscall.GetLastError(); err != nil {
return InvalidServiceDatabaseHandleHandle, ServiceErrno(err.(syscall.Errno))
}
}
err is always nil. Assume machineNamePtr is a non exsiting machine. Tested the same code with c++ and GetLastError() throws RPC server is not available. So why not on go?
EDIT
_OpenSCManager is generated with go generate.
//sys _OpenSCManager(machineName *uint16, databaseName *uint16, desiredAcces ServiceAccessRight) (handle ServiceDatabaseHandle) = advapi32.OpenSCManagerW
func _OpenSCManager(machineName *uint16, databaseName *uint16, desiredAcces ServiceAccessRight) (handle ServiceDatabaseHandle) {
r0, _, _ := syscall.Syscall(procOpenSCManagerW.Addr(), 3, uintptr(unsafe.Pointer(machineName)), uintptr(unsafe.Pointer(databaseName)), uintptr(desiredAcces))
handle = ServiceDatabaseHandle(r0)
return
}
So finally i've got this working. First i have tried to return an error variable with the following signature //sys _OpenSCManager(machineName *uint16, databaseName *uint16, desiredAcces ServiceAccessRight) (handle ServiceDatabaseHandle, lasterror error) = advapi32.OpenSCManagerW. But go generate throws always Only last windows error is allowed as second return value.... But if you change it to //sys ... (handle ServiceDatabaseHandle, err error) = advapi32.OpenSCManagerW it successfully generates the code. So you explicit have to write err error. Someone knows why? So now the function looks like
func _OpenSCManager(machineName *uint16, databaseName *uint16, desiredAcces ServiceAccessRight) (handle ServiceDatabaseHandle, err error) {
r0, _, e1 := syscall.Syscall(procOpenSCManagerW.Addr(), 3, uintptr(unsafe.Pointer(machineName)), uintptr(unsafe.Pointer(databaseName)), uintptr(desiredAcces))
handle = ServiceDatabaseHandle(r0)
if handle == 0 {
if e1 != 0 {
err = errnoErr(e1)
} else {
err = syscall.EINVAL
}
}
return
}
and it returns an error. So there is no need to call GetLastError().
IIUC, the syscall.Syscall() on Windows automatically and atomically calls GetLastError() after the actual syscall finishes. That should be understandable once you consider that as soon as a goroutine exits from a syscall, the Go runtime scheduler is free to preempt it and run another goroutine on the very thread the just-preempted goroutine had just been running on.
Since GetLastError() accesses per-thread state, if the second goroutine makes another syscall, it would likely thrash that last error value, so in the context of Go, each syscall should be accompanied with the following call to GetLastError() done in the context of a single syscall invocation — as seen from the Go side.
I'm currently looking into creating some unit tests for net.Conn interface in Go, as well as other functions that build up on top of that functionality, and I'm wondering what is the best way to unit test that in Google Go? My code looks like:
conn, _:=net.Dial("tcp", "127.0.0.1:8080")
...
fmt.Fprintf(conn, "test")
...
buffer:=make([]byte, 100)
conn.Read(buffer)
Is the most efficient way of testing this code and the code that uses these functions to spin up a separate goroutine to act like the server, use net.http.httptest package, or something else?
You might be able to do what you need with net.Pipe which basically gives you both ends of a connection (think, after .Accept())
server, client := net.Pipe()
go func() {
// Do some stuff
server.Close()
}()
// Do some stuff
client.Close()
Although it will depend on the implementation details of your particular case, the general approach will be to start a server (in a separate goroutine, as you already hinted), and listen to the incoming connections.
For example, let's spin up a server and verify that the content we are reading from the connection is indeed the one we send over from the client:
func TestConn(t *testing.T) {
message := "Hi there!\n"
go func() {
conn, err := net.Dial("tcp", ":3000")
if err != nil {
t.Fatal(err)
}
defer conn.Close()
if _, err := fmt.Fprintf(conn, message); err != nil {
t.Fatal(err)
}
}()
l, err := net.Listen("tcp", ":3000")
if err != nil {
t.Fatal(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
return
}
defer conn.Close()
buf, err := ioutil.ReadAll(conn)
if err != nil {
t.Fatal(err)
}
fmt.Println(string(buf[:]))
if msg := string(buf[:]); msg != message {
t.Fatalf("Unexpected message:\nGot:\t\t%s\nExpected:\t%s\n", msg, message)
}
return // Done
}
}
Note that here I'm not starting the server in the goroutine, as otherwise the test case is likely to be finished before the listener has run the test.
Another option is the counterfeiter package which lets you create mocks from interfaces and then you can stub out whatever calls you need. I have used it with great success to stub out net.Conn instances where I am testing out a protobuf client for Geode.
For example - https://github.com/gemfire/geode-go-client/blob/master/connector/protobuf_test.go
I have a function that is executed concurrently. It's task is to run a command line method, logging an error if there is one. My main thread of execution does not wait for this method to finish. It simply returns optimistically.
How do I test my function? Assuming that I give it a cmdStr, such as {"sleep", "1"}, that works but doesn't return instantly, how can I get my test to wait for this function to finish?
I want to ensure that the actual program that runs this does not have to wait for it to finish.
func runCmd(cmdStr []string, errChan chan error) {
cmd := exec.Command(cmdStr...)
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Start()
if err != nil {
errChan <- err
return
}
// Command started successfully, so our caller can continue.
errChan <- nil
err = cmd.Wait()
if err != nil {
log.Println(err)
}
}
Use a wait group
wg := sync.WaitGroup{}
errc := make(chan error)
wg.Add(1)
go func() {
runCmd([]string{"sleep", 1}, errc)
wg.Done()
}()
err <- errc
if err != nil {
// handle error
}
wg.Wait()
Simplest method to aware all goroutings done is to add select{} as last main.main() statement. But main() will never return this way and you should kill process explicitly.
It's also more gentle runtime.Goshed() to wait for others, but it can't provide such a strong guarantee.
And the canonical way would be
wg := sync.WaitGroup
...
wg.Add(1)
go runCmd(...)
...
wg.Wait()