I'm learning Go at the moment and this is the code for a web-app I'm writing:
func init() {
db, err := sql.Open("mysql", "master:123456#/shopping_list")
if err != nil {
panic(err.Error())
}
http.HandleFunc("/sql", func(w http.ResponseWriter, r *http.Request) {
sqlHandler(w, r, db)
})
}
sqlHandler() reads a record from a table in shopping_list, edits it and then updates the record.
Now as I understand it, each request runs on a separate goroutine and that mysql locks a record while it is being read or written. So, in this case, does this code need any synchronization?
Short answer: no.
You don't need to explicitly synchronise your code, because calls to any method on db that require a lock will simply block until the lock is released. In other words, synchronisation is taken care of by the package that actually needs the synchronisation.
As a side note, I would suggest to run your code through gofmt, which will make other Go nuts who read your code happy.
Related
I'm learning go and am working on a simple service that ingests some data from a queue and sticks it in the database. It also runs a web server to allow scraping of data. Right now I have two go files (omitted some text for brevity):
func main() {
parseConfig()
s := &Service{ServiceConfig: config}
err := s.Run()
if err != nil {
panic(err)
}
}
And then the definition of the service (again left out some pieces for brevity):
func (s *Service) Run() error {
if err := s.validate(); err != nil {
return err
}
if err := s.initDB(); err != nil {
return err
}
defer s.db.Close()
// Same pattern with health check library (init, start, close)
// Same pattern starting queue consumer (init, start, close)
s.mux = http.NewServeMux()
s.registerHandlers(s.mux)
http.ListenAndServe(":8080", s.mux)
return nil
}
And the struct
type Service struct {
Config // Hold db connection info
db *sql.DB
hc *health
}
I'm able to test the individual pieces fine (like initDB or validate) but I'm not unclear how one would test the Run function because http.ListenAndServe blocks. I eventually time out. Previously, I would use httpTest and make a test server but that was when main would start the server (the application was more basic at first).
Some things I would test:
That I can hit the metrics endpoint once started.
That I can hit the health endpoint once started.
That I can push a message on the queue and it is received once started.
That Run actually starts w/o a panic.
Some notes: I am using docker to spin up a queue and database. The point of testing the Run function is to ensure that the bootstrapping works and the application can run successfully. Eventually I will want to push data through the queue and assert that its been processed correctly.
Question: How should I test this or refactor it so that it is more easily testable end to end?
You can build a test harness using a goroutine to execute Run in your unit test:
func TestRun(t *testing.T) {
service := Service{}
serviceRunning := make(chan struct{})
serviceDone := make(chan struct{})
go func() {
close(serviceRunning)
err := service.Run()
defer close(serviceDone)
}()
// wait until the goroutine started to run (1)
<-serviceRunning
//
// interact with your service to test whatever you want
//
// stop the service (2)
service.Shutdown()
// wait until the service is shutdown (3)
<-serviceDone
}
This is just a basic example to show how it could be done in principle. There are several points that should be improved for use in production:
(0) The most important detail: Do not use http.ListenAndServe in production! Create your own instance of http.Server instead. This saves you a lot of trouble.
s.httpServer := http.Server {
Addr: ":8080",
Handler: s.mux,
}
(1) The indication that the service is running should be moved into your Service type. The initialization part in Run might take a while and the indication channel should be closed right before ListenAndServe is called:
close(s.Running)
s.httpServer.ListenAndServe()
Of course, you need to add the indication channel to your Service type.
(2) Add a Shutdown method to Service that calls s.httpServer.Shutdown(). This will cause the call to s.httpServer.ListenAndServe to return with the error http.ErrServerClosed.
if err := s.httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
return err
}
return nil
It is important to shutdown your service at the end of the test. Otherwise you will not be able to have more than one unit test for your service. And it is good citizenship anyway to clean up the resources.
(3) You need to wait until service.Run returned to make sure that the service is actually shutdown.
The httptest package comes with a test server built for this purpose.
https://pkg.go.dev/net/http/httptest#example-Server
If I have code that works with a net.Conn, how can I write tests for it without actually creating a network connection to localhost?
I've seen no solutions to this online; people seem to either ignore it (no tests), write tests that cannot run in parallel (ie. use an actual network connection, which uses up ports), or use io.Pipe.
However, net.Conn defines SetReadDeadline, SetWriteDeadline; and io.Pipe doesn't. net.Pipe also doesnt, despite superficially claiming to implement the interface, it's simply implemented with:
func (p *pipe) SetDeadline(t time.Time) error {
return &OpError{Op: "set", Net: "pipe", Source: nil, Addr: nil, Err: errors.New("deadline not supported")}
}
func (p *pipe) SetReadDeadline(t time.Time) error {
return &OpError{Op: "set", Net: "pipe", Source: nil, Addr: nil, Err: errors.New("deadline not supported")}
}
func (p *pipe) SetWriteDeadline(t time.Time) error {
return &OpError{Op: "set", Net: "pipe", Source: nil, Addr: nil, Err: errors.New("deadline not supported")}
}
(see: https://golang.org/src/net/pipe.go)
So... is there some other way of doing this?
I'll accept any answer that shows how to use a stream in a test with a working deadline that is not an actual network socket.
(Idly, this cloudflare blogpost covers the motivation for using deadlines, and why blocking forever in a goroutine per connection is not an acceptable solution; but regardless of that argument, particularly in this case I'm looking for a solution for tests where we deliberately want to handle edge cases where a bad connection hangs, etc.)
(NB. this may seem like a duplicate of Simulate a tcp connection in Go, but notice that all the solutions proposed in that question do not implement the Deadline functions, which is specifically what I'm asking about how to test here)
Your question is very open, so it is not possible to give you "the correct answer". But I think I understand the point where you stuck. This answer is also open, but it should bring you back on the right track.
A few days ago I wrote a short article, which shows the principle which you have to use.
Before I make some small examples how you such tests work, we need to fix one important point:
We do not test the net package. We asume, that the package has no bug and does, what the documentation says. That means we do not care how the Go team has implementet SetReadDeadline and SetWriteDeadline. We test only the usage in our programm.
Step 1: Refactoring your code
You did not post any code snippets, so I give you just a simple example. I guess you have a method or function, where you are using the net package.
func myConn(...) error {
// You code is here
c, err := net.Dial("tcp", "12.34.56.78:80")
c.setDeadline(t)
// More code here
}
That you are able to to test you need to refactor your function, so it is just using the net.Conn interface. To do this, the net.Dial() call has to be moved outside of the function. Remember that we don't want to test the net.Dial function.
The new function could look something like that:
func myConn(c, net.Conn, ...) error {
// You code is here
c.setDeadline(t)
// More code here
}
Step 2: Implement the net.Conn interface
For testing you need to implement the net.Conn interface:
type connTester struct {
deadline time.Time
}
func (c *connTester) Read(b []byte) (n int, err error) {
return 0, nil
}
...
func (c *connTester) SetDeadline(t time.Time) error {
c.deadline = t
return nil
}
...
Complete implementation including a small type check:
https://play.golang.org/p/taAmI61vVz
Step 3: Testing
When testing, we don't care about the Dial() Method, we just create a pointer to our testtype, which implements the net.Conn interface and put it into your function. Afterwards we look inside our test cases, if the deadline parameter is set correctly.
func TestMyConn(t *testing.T){
myconnTester = &connTester{}
err := myConn(myconnTester,...)
...
if myconntester.deadline != expectedDeadline{
//Test fails
}
}
So when testing, you should always think about, what feature you want to test. I think it is the really most difficult part to abstract the functionality you really want to write. Inside of simple unit tests you should never test functionalities of the standard library. Hope this examples can help you to bring you back on the right track.
Code that needs to be swapped out, for a controlled version in a unittest, should live behind an abstraction. In this case the abstraction would be the net.Conn interface. The production code would use go std lib net.Conn but the test code would use a test stub that is configured with the exact logic to exercise your function.
Introducing an abstraction is a powerful pattern that should allow swapping out all IO, or timing based code, to allow for controlled execution of code, during a unittest.
#apxp stated the same approach in a comment.
The same approach should work for the deadlines. It could get tricky simulating a deadline that is reached, because you may have to configure your stub with multiple responses. ie. The first response succeeds, but the second simulates a deadline that has been reached, and throws an error for the second request.
I just read the article: Build You Own Web Framework In Go and for sharing values among handlers I picked the context.Context and I'm using it in the following way to share values across handlers and middlewares:
type appContext struct {
db *sql.DB
ctx context.Context
cancel context.CancelFunc
}
func (c *appContext)authHandler(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request {
defer c.cancel() //this feels weird
authToken := r.Header.Get("Authorization") // this fakes a form
c.ctx = getUser(c.ctx, c.db, authToken) // this also feels weird
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
func (c *appContext)adminHandler(w http.ResponseWriter, r *http.Request) {
defer c.cancel()
user := c.ctx.Value(0).(user)
json.NewEncoder(w).Encode(user)
}
func getUser(ctx context.Context, db *sql.DB, token string) context.Context{
//this function mimics a database access
return context.WithValue(ctx, 0, user{Nome:"Default user"})
}
func main() {
db, err := sql.Open("my-driver", "my.db")
if err != nil {
panic(err)
}
ctx, cancel := context.WithCancel(context.Background())
appC := appContext{db, ctx, cancel}
//....
}
Everything is working and handlers are loading faster than using gorilla/context So my questions are:
Is this approach safe?
Is it really necessary to defer the c.cancel() function the way I'm doing it?
Can I use it to implement a custom web framework by using controllers like struct to share values with models?
Note: go 1.7.0-rc2 does clarify a bit how to release resources associated with Contexts (Sameer Ajmani):
Some users don't realize that creating a Context with a CancelFunc attaches a subtree to the parent, and that that subtree is not released until the CancelFunc is called or the parent is canceled.
Make this clear early in the package docs, so that people learning about this package have the right conceptual model.
The documentation now includes:
Incoming requests to a server should create a Context, and outgoing calls to servers should accept a Context.
The chain of function calls between them must propagate the Context, optionally replacing it with a derived Context created using WithCancel, WithDeadline, WithTimeout, or WithValue.
These Context values form a tree: when a Context is canceled, all Contexts derived from it are also canceled.
The WithCancel, WithDeadline, and WithTimeout functions return a derived Context and a CancelFunc.
Calling the CancelFunc cancels the new Context and any Contexts derived from it, removes the Context from the parent's tree, and stops any associated timers.
Failing to call the CancelFunc leaks the associated resources until the parent Context is canceled or the timer fires.
You have a problem with your code because you are storing the user into the app context. With multiple users at the same time, it doesn't work. The context must be related to the request to not be overwrote by other requests. The user must be stored in a request context. In my articles I use the following gorilla function: context.Set(r, "user", user). r is the request.
If you want to use context.Context in your app, you should use their gorilla wrapper (you can find it at the end of this article: https://blog.golang.org/context).
Also, you don't need a context with cancel. context.Background() is okay for the root context.
Two major use cases of Context package are:
For storing request scoped values - Using context.WithValue()
For cancellation - using context.WithCancel(), context.WithTimeout(), context.WithDeadline()
Use of context forms a context tree with context.Background() as root. WithValue() ,context.WithCancel(), WithTimeout(), WithDeadline() are derived context from root context which can be further divide. An example of each can make it clear so ast to what is the correct use context. Came across this guide which provides the use of all discussed above with proper examples.
Source: https://golangbyexample.com/using-context-in-golang-complete-guide/
I'm not very familiar with Go's routines but since I'm working with the router of net/http I saw a few times thatListenAndServe() is wrapped by a go routine.
A server needs to be able to handle multiple requests simultaneously out of the box to be efficient. So why are go routines as 'lightweight threads' used?
Does the concurrency provide any advantages?
Here's an example by OpenShift
package main
import (
"fmt"
"net/http"
)
func helloHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello OpenShift!")
}
func main() {
http.HandleFunc("/", helloHandler)
go func() {
fmt.Println("serving on 8080")
err := http.ListenAndServe(":8080", nil)
if err != nil {
panic("ListenAndServe: " + err.Error())
}
}()
go func() {
fmt.Println("serving on 8888")
err := http.ListenAndServe(":8888", nil)
if err != nil {
panic("ListenAndServe: " + err.Error())
}
}()
select {}
}
http.ListenAndServe is a blocking call. If you want to go one doing more work (like making a second http.ListenAndServe call), you need to move it over to a separate goroutine. That's all they're doing here.
They're using select{} at the end to block the main goroutine, since all their calls to http.ListenAndServe are on other goroutines. If they didn't call select{}, the program would terminate because main() would return.
They could have achieved the same thing by dropping select{}, and removing the go func() wrapper around the last block of code. But I suspect they did it this way so that all the code is consistent.
But this has nothing to do with performance.
In the comments you provided some other examples that are similar. In the first example:
func main() {
http.HandleFunc("/", responsehandler.Handler)
go func() {
http.ListenAndServe(":8888", nil)
}()
fileservice.NewWatcher()
}
This calls http.ListenAndServe and then calls fileservice.NewWatcher() (which blocks). If they hadn't wrapped the call in a goroutine, fileservice.NewWatcher() would never have been called.
The other two examples are a common piece of boilerplate:
func init() {
go func() {
log.Println(http.ListenAndServe("localhost:6060", nil))
}()
}
This turns on the debug profiler web server. Again, it's a goroutine so that calling init returns immediately rather than blocking. This particular case allows the caller to just import _ "profiling" and "magically" get the debug profiler web server.
No it does not have any special benefits beside being "run in the background".
I don't think you need a go routine to start ListenAndServe. According to the Go Documentations.
"ListenAndServe calls Serve". Serve is go routine.
ListenAndServe listens on the TCP network address addr and then calls Serve with handler to handle requests on incoming connections. Accepted connections are configured to enable TCP keep-alives. Handler is typically nil, in which case the DefaultServeMux is used.
https://golang.org/pkg/net/http/#ListenAndServe
func Serve(l net.Listener, handler Handler) error
Serve accepts incoming HTTP connections on the listener l, creating a new service goroutine for each. The service goroutines read requests and then call handler to reply to them. Handler is typically nil, in which case the DefaultServeMux is used.
https://golang.org/pkg/net/http/#Serve
Is there a way to execute test cases in GoLang in a pre-defined order.
P.S: I am writing test cases for life cycle of a event. So I have different api's for all the CURD operations. I want to run these test cases in a particular order as only if an event is created it can be destroyed.
Also can I get some value from one test case and pass it as input to another. (example:- To test the delete event api, I need a event_id which i get when I call create_event test case)
I am new to GoLang, can someone please guide me through.
Thanks in advance
The only way to do it is to encapsulate all your tests into one test function, that calls sub-functions in the right order and with the right context, and pass the testing.T pointer to each so they can fail. The down-side is that they will all appear as one test. But in fact that is the case - tests are stateless as far as the testing framework is concerned, and each function is a separate test case.
Note that although the tests may run in the order they are written in, I found no documentation stating that this is actually a contract of some sort. So even though you can write them in order and keep the state as external global variables - that's not recommended.
The only flexibility the framework gives you since go 1.4 is the TestMain method that lets you run before/after steps, or setup/teardown:
func TestMain(m *testing.M) {
if err := setUp(); err != nil {
panic(err)
}
rc := m.Run()
tearDown()
os.Exit(rc)
}
But that won't give you what you want. The only way to do that safely is to do something like:
// this is the whole stateful sequence of tests - to the testing framework it's just one case
func TestWrapper(t *testing.T) {
// let's say you pass context as some containing struct
ctx := new(context)
test1(t, ctx)
test2(t, ctx)
...
}
// this holds context between methods
type context struct {
eventId string
}
func test1(t *testing.T, c *context) {
// do your thing, and you can manipulate the context
c.eventId = "something"
}
func test2(t *testing.T, c *context) {
// do your thing, and you can manipulate the context
doSomethingWith(c.eventId)
}