Channel race condition - unit-testing

I'm trying to write tests for a simple program that reads events from the inotify bus, filters them, then puts them on a channel. I have seen no issues when actually running the code, but in the last line of the following test, it will occasionally deadlock (about 30% of the time) on reading from that channel, even though as best I can tell, the event is always being put on the channel.
Here is the function:
func eventFilter(watcher *rfsnotify.RWatcher, excludes []string, out chan<- fsnotify.Event) {
for {
select {
case event := <-watcher.Events:
log.Debug(fmt.Sprintf("Got event %v", event))
if isExcluded(event.Name, excludes) {
log.Info("Ignoring excluded file: %v", event)
} else if isRelevantOp(event) {
log.Info(fmt.Sprintf("Handling event %v", event))
out <- event
} else {
log.Info(fmt.Sprintf("Ignoring event %v", event))
}
case err := <-watcher.Errors:
log.Error(fmt.Sprintf("Error: %v", err))
}
}
}
Here is the testing code:
var hook *logrusTest.Hook
var testDir string
var rWatcher *rfsnotify.RWatcher
func TestMain(m *testing.M) {
// initialize bad globals, amongst other things
log.SetLevel(log.DebugLevel)
hook = logrusTest.NewGlobal()
// Create directory to watch
testDir = tempMkdir()
defer os.RemoveAll(testDir)
rWatcher, _ = rfsnotify.NewWatcher()
defer rWatcher.Close()
err := rWatcher.AddRecursive(testDir)
if err != nil {
log.Fatal("Failed to add watcher: ", err)
}
os.Exit(m.Run())
}
func TestEventFilterMove(t *testing.T) {
ch := make(chan fsnotify.Event, 10)
go eventFilter(rWatcher, []string{"test"}, ch)
testFileSrc := filepath.Join(testDir, "TestFsnotifyEvents.testfileSrc")
testFileDest := filepath.Join(testDir, "TestFsnotifyEvents.testfileDest")
f, err := os.OpenFile(testFileSrc, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
t.Fatalf("creating test file failed: %s", err)
}
f.WriteString("data")
f.Sync()
hook.Reset()
err = os.Rename(testFileSrc, testFileDest)
if err != nil {
t.Fatalf("renaming test file failed: %s", err)
}
time.Sleep(100 * time.Millisecond)
expectedLogs := []string{
fmt.Sprintf("Got event \"%v\": CREATE|UPDATE", testFileDest),
fmt.Sprintf("Handling event \"%v\": CREATE|UPDATE", testFileDest),
fmt.Sprintf("Got event \"%v\": RENAME", testFileSrc),
fmt.Sprintf("Ignoring event \"%v\": RENAME", testFileSrc),
}
var actualLogs []string
for _, logEntry := range hook.AllEntries() {
actualLogs = append(actualLogs, logEntry.Message)
}
assert.ElementsMatch(t, actualLogs, expectedLogs)
hook.Reset()
_ = <-ch
assert.Equal(t, 0, len(ch))
}

Related

Golang FormFile unit test

I have the following function:
func CreateByID(w http.ResponseWriter, r *http.Request) {
formFile, _, err := r.FormFile("audio")
if err != nil {
return
}
defer formFile.Close()
defer r.MultipartForm.RemoveAll()
tmpFile, err := os.CreateTemp("/tmp")
if err != nil {
return
}
defer os.Remove(tmpFile.Name())
io.Copy(tmpFile, formFile)
err = validateFile(tmpFile.Name())
if err != nil {
return
}
}
func validateFile(path string) error {
fs, err := os.Stat(path)
if err != nil {
return
}
if fs.Size() < allowedSize {
return fmt.Errorf("file is too small")
}
}
which is basically creating a temp file from the HTTP request and validating it before further processing. This works fine except for the unit test. When I'm testing this function - I'm getting always filesize = 0.
Below you can see the Test case:
func TestCreateByID(t *testing.T) {
pr, pw := io.Pipe()
writer := multipart.NewWriter(pw)
go func() {
defer writer.Close()
_, err := writer.CreateFormFile("audio", "/tmp/audioFile")
if err != nil {
t.Error(err)
}
generateAudioSample("/tmp/audioFile")
}()
request := httptest.NewRequest(http.MethodGet, "/{id}")
request.Header.Add("Content-Type", writer.FormDataContentType())
response := httptest.NewRecorder()
# Calling The Actual Function
CreateByID(response, request)
handler := http.NewServeMux()
handler.ServeHTTP(response, request)
if response.Code != 200 {
t.Errorf("Expected %d, received %d", 200, response.Code)
return
}
}
generateAudioSample function is working fine because I can see the file on the filesystem and its size it bigger than 0 but for some reason this function io.Copy(tmpFile, formFile) doesn't seem to handle FormFile correctly, and is creating just an empty file on the filesystem and obviously fail the test. What else should I add to the test function in order to pass the test?

Unit-testing grpc functions in golang

I have created a function that utilizes the grpc package in golang. I don't know if it is relevant but the purpose is the communication with a GoBGP router over grpc. An example is the following function which prints all the peers (neighbors) of the router:
func (gc *Grpc) Peers(conn *grpc.ClientConn) error {
defer conn.Close()
c := pb.NewGobgpApiClient(conn)
ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
defer cancel()
p := pb.ListPeerRequest{}
peer, err := c.ListPeer(ctx, &p)
if err != nil {
return err
}
for {
res, err := peer.Recv()
if err != nil {
return err
}
fmt.Println(res)
}
return nil
}
Now, I want to create unit tests for the function. To do so, I used google.golang.org/grpc/test/bufconn package, and initialized the following:
type server struct {
pb.UnimplementedGobgpApiServer
}
func (s *server) ListDefinedSet(in *pb.ListDefinedSetRequest, ls pb.GobgpApi_ListDefinedSetServer) error {
return nil
}
var lis *bufconn.Listener
const bufSize = 1024 * 1024
func init() {
lis = bufconn.Listen(bufSize)
s := grpc.NewServer()
pb.RegisterGobgpApiServer(s, &server{})
go func() {
if err := s.Serve(lis); err != nil {
fmt.Println("Server failed!")
}
}()
}
func bufDialer(context.Context, string) (net.Conn, error) {
return lis.Dial()
}
This way, I can run a unit-test creating a connection as follows:
ctx := context.Background()
conn, _ := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
Peers(conn)
However, the problem is that the stream seems to be always empty and thus the peer.Recv()
always returns EOF. Is there any way to populate the stream with dummy data? If you have experience, is my methodology correct?

How to write an unit test for a handler that invokes a function that interacts with db in Golang using pgx driver?

I've been trying to write unit tests for my http handler. The code segment is as below:
func (s *Server) handleCreateTicketOption(w http.ResponseWriter, r *http.Request) {
var t ticket.Ticket
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, er.ErrInternal.Error(), http.StatusInternalServerError)
return
}
err = json.Unmarshal(body, &t)
if err != nil {
http.Error(w, er.ErrInvalidData.Error(), http.StatusBadRequest)
return
}
ticket, err := s.TicketService.CreateTicketOption(r.Context(), t)
if err != nil {
http.Error(w, er.ErrInternal.Error(), http.StatusInternalServerError)
return
}
res, err := json.Marshal(ticket)
if err != nil {
http.Error(w, er.ErrInternal.Error(), http.StatusInternalServerError)
return
}
log.Printf("%v tickets allocated with name %v\n", t.Allocation, t.Name)
s.sendResponse(w, res, http.StatusOK)
}
Actual logic that interacts with DB. This code segment is invoked by the handler as you can see in the code above. ticket, err := s.TicketService.CreateTicketOption(r.Context(), t)
func (t *TicketService) CreateTicketOption(ctx context.Context, ticket ticket.Ticket) (*ticket.Ticket, error) {
tx, err := t.db.dbPool.Begin(ctx)
if err != nil {
return nil, er.ErrInternal
}
defer tx.Rollback(ctx)
var id int
err = tx.QueryRow(ctx, `INSERT INTO ticket (name, description, allocation) VALUES ($1, $2, $3) RETURNING id`, ticket.Name, ticket.Description, ticket.Allocation).Scan(&id)
if err != nil {
return nil, er.ErrInternal
}
ticket.Id = id
return &ticket, tx.Commit(ctx)
}
And that is my unit test for the handler.
func TestCreateTicketOptionHandler(t *testing.T) {
caseExpected, _ := json.Marshal(&ticket.Ticket{Id: 1, Name: "baris", Description: "test-desc", Allocation: 10})
srv := NewServer()
// expected := [][]byte{
// _, _ = json.Marshal(&ticket.Ticket{Id: 1, Name: "baris", Description: "test-desc", Allocation: 20}),
// // json.Marshal(&ticket.Ticket{Id: 1, Name: "baris", Description: "test-desc", Allocation: 20})
// }
tt := []struct {
name string
entry *ticket.Ticket
want []byte
code int
}{
{
"valid",
&ticket.Ticket{Name: "baris", Description: "test-desc", Allocation: 10},
caseExpected,
http.StatusOK,
},
}
var buf bytes.Buffer
for _, tc := range tt {
t.Run(tc.name, func(t *testing.T) {
json.NewEncoder(&buf).Encode(tc.entry)
req, err := http.NewRequest(http.MethodPost, "/ticket_options", &buf)
log.Println("1")
if err != nil {
log.Println("2")
t.Fatalf("could not create request: %v", err)
}
log.Println("3")
rec := httptest.NewRecorder()
log.Println("4")
srv.handleCreateTicketOption(rec, req)
log.Println("5")
if rec.Code != tc.code {
t.Fatalf("got status %d, want %v", rec.Code, tc.code)
}
log.Println("6")
if reflect.DeepEqual(rec.Body.Bytes(), tc.want) {
log.Println("7")
t.Fatalf("NAME:%v, got %v, want %v", tc.name, rec.Body.Bytes(), tc.want)
}
})
}
}
I did research about mocking pgx about most of them were testing the logic part not through the handler. I want to write unit test for both handler and logic itself seperately. However, the unit test I've written for the handler panics as below
github.com/bariis/gowit-case-study/psql.(*TicketService).CreateTicketOption(0xc000061348, {0x1485058, 0xc0000260c0}, {0x0, {0xc000026dd0, 0x5}, {0xc000026dd5, 0x9}, 0xa})
/Users/barisertas/workspace/gowit-case-study/psql/ticket.go:24 +0x125
github.com/bariis/gowit-case-study/http.(*Server).handleCreateTicketOption(0xc000061340, {0x1484bf0, 0xc000153280}, 0xc00018e000)
/Users/barisertas/workspace/gowit-case-study/http/ticket.go:77 +0x10b
github.com/bariis/gowit-case-study/http.TestCreateTicketOptionHandler.func2(0xc000119860)
/Users/barisertas/workspace/gowit-case-study/http/ticket_test.go:80 +0x305
psql/ticket.go:24: tx, err := t.db.dbPool.Begin(ctx)
http/ticket.go:77: ticket, err := s.TicketService.CreateTicketOption(r.Context(), t)
http/ticket_test.go:80: srv.handleCreateTicketOption(rec, req)
How can I mock this type of code?
Create an interface which has the required DB functions
Your DB handler implements this interface. You use the handler in actual execution
Create a mock handler using testify/mock and use this in place of DB handler in test cases
From what I can read, you have the following structure:
type Server struct {
TicketService ticket.Service
}
type TicketService struct {
db *sql.Db // ..or similar
}
func (ts *TicketService) CreateTicketOption(...)
The trick to mock this is by ensuring ticket.Service is an interface instead of a struct.
Like this:
type TicketService interface {
CreateTicketOption(ctx context.Context, ticket ticket.Ticket) (*ticket.Ticket, error) {
}
By doing this, your Server expects a TicketService interface.
Then you could do this:
type postgresTicketService struct {
db *sql.Db
}
func (pst *postgresTicketService) CreateTicketOption(...)...
Which means that the postgresTicketService satisfies the requirements to be passed as a ticket.Service to the Server.
This also means that you can do this:
type mockTicketService struct {
}
func (mts *mockTicketService) CreateTicketOption(...)...
This way you decouple the Server from the actual implementation, and you could just init the Server with the mockTicketService when testing and postgresTicketService when deploying.

How to solve this syncing issue in Unit testing golang?

Here when I'm printing the activity it is printing them in the order they are getting created but at the time of assertion it is picking up activities in random order and also it is picking expected values in random order. The api that I'm calling mastercontroller have some goroutines and could take time maybe that is the reason but not sure.
for i, param := range params {
gin.SetMode(gin.TestMode)
w := httptest.NewRecorder()
ctx, _ := gin.CreateTestContext(w)
ctx.Request = &http.Request{
URL: &url.URL{},
Header: make(http.Header),
}
MockJsonPost(ctx, param)
MasterController(ctx)
time.Sleep(3 * time.Second)
fmt.Println("response body", string(w.Body.Bytes()))
fmt.Println("status", w.Code)
// var activity *activity.Activity
activity, err := activityController.GetLastActivity(nil)
//tx.Raw("select * from activity order by id desc limit 1").Find(&activity)
if err != nil {
fmt.Println("No activity found")
}
activityJson, err := activity.ToJsonTest()
if err != nil {
fmt.Println("error converting in json")
}
fmt.Printf("reponse activity %+v", string(activityJson))
assert.EqualValues(t, string(expected[i]), string(activityJson))
}
func MockJsonPost(c *gin.Context, content interface{}) {
c.Request.Method = "POST" // or PUT
c.Request.Header.Set("Content-Type", "application/json")
jsonbytes, err := json.Marshal(content)
if err != nil {
panic(err)
}
// the request body must be an io.ReadCloser
// the bytes buffer though doesn't implement io.Closer,
// so you wrap it in a no-op closer
c.Request.Body = io.NopCloser(bytes.NewBuffer(jsonbytes))
}

Testing with Gomock returns error: Expected call has already been called the max number of times

I am using Gomock https://godoc.org/github.com/golang/mock and mockgen
The Source code for this test is:
package sqs
import (
"fmt"
"log"
"os"
"runtime"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/sqs"
"github.com/aws/aws-sdk-go/service/sqs/sqsiface"
)
var sess *session.Session
var svc *sqs.SQS
var queueURL string
func init() {
// Setting the runtime to run with max CPUs available
runtime.GOMAXPROCS(runtime.NumCPU())
sess = session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
svc = sqs.New(sess)
queueURL = os.Getenv("QUEUE_URL")
}
type Poller interface {
Poll(chan bool)
}
// NewPoller is a factory to create a Poller object
func NewPoller(msgr Messenger) Poller {
p := &poller{
m: msgr,
}
return p
}
type poller struct {
m Messenger
}
func (p *poller) Poll(done chan bool) {
sqsMsgCh := make(chan *sqs.Message, 100)
for {
messages, err := p.m.GetMessage()
if err != nil {
log.Printf("error when getting message")
if len(messages) == 0 {
// Stop the system
log.Printf("I am here")
done <- true
}
}
for _, msg := range messages {
sqsMsgCh <- msg
}
}
}
type Messenger interface {
GetMessage() ([]*sqs.Message, error)
}
func NewMessenger() Messenger {
return &messenger{
s: svc,
}
}
type messenger struct {
s sqsiface.SQSAPI
}
func (m *messenger) GetMessage() ([]*sqs.Message, error) {
result, err := m.s.ReceiveMessage(&sqs.ReceiveMessageInput{
AttributeNames: []*string{
aws.String(sqs.MessageSystemAttributeNameSentTimestamp),
},
MessageAttributeNames: []*string{
aws.String(sqs.QueueAttributeNameAll),
},
QueueUrl: aws.String(queueURL),
MaxNumberOfMessages: aws.Int64(10),
VisibilityTimeout: aws.Int64(36000), // 10 hours
WaitTimeSeconds: aws.Int64(0),
})
if err != nil {
fmt.Println("Error", err)
return nil, err
}
msgs := result.Messages
if len(msgs) == 0 {
fmt.Println("Received no messages")
return msgs, err
}
return msgs, nil
}
The test case for this Source file is here:
package sqs
import (
"errors"
"testing"
"path_to_the_mocks_package/mocks"
"github.com/golang/mock/gomock"
"github.com/aws/aws-sdk-go/service/sqs"
)
func TestPollWhenNoMessageOnQueue(t *testing.T) {
mockCtrl := gomock.NewController(t)
defer mockCtrl.Finish()
msgr := mocks.NewMockMessenger(mockCtrl)
mq := make([]*sqs.Message, 1)
err := errors.New("Mock Error")
// msgr.EXPECT().GetMessage().Return(mq, err) //.Times(1)
// msgr.GetMessage().Return(mq, err) //.Times(1)
msgr.EXPECT().GetMessage().Return(mq, err)
p := NewPoller(msgr)
done := make(chan bool)
go p.Poll(done)
<-done
t.Logf("Successfully done: %v", done)
}
When I run the tests I am getting the following error:
sqs\controller.go:150: Unexpected call to
*mocks.MockMessenger.GetMessage([]) at path_to_mocks_package/mocks/mock_messenger.go:38 because: Expected
call at path_to_sqs_package/sqs/sqs_test.go:35 has already been called
the max number of times. FAIL
If I write my own mock as follows the test case executes successfully:
type mockMessenger struct {
mock.Mock
}
func (m *mockMessenger) GetMessage() ([]*sqs.Message, error) {
msgs := make([]*sqs.Message, 0)
err := errors.New("Error")
return msgs, err
}
You are implicitly telling gomock that you only expect a single call.
msgr.EXPECT().GetMessage().Return(mq, err)
Adding a number of Times to the mock, allows you to return those values more than once.
msgr.EXPECT().GetMessage().Return(mq, err).AnyTimes()
For more details please read the gomock's AnyTimes documentation.