When I use tikv api, and I found it has an option in TxnKV client, then I test it, but I can not find what the difference between optimistic and pessimistic is in tikv?
The test code is this:
func begin() kv.Transaction{
transaction, err := store.Begin()
if err!=nil{
panic(err)
}
return transaction
}
func main() {
pdAddr := os.Getenv("PD_ADDR")
if pdAddr != "" {
os.Args = append(os.Args, "-pd", pdAddr)
}
flag.Parse()
initStore()
k2 := []byte("key2")
v22 := []byte("value22")
v23 := []byte("value22")
testTxn(k2,v22,v23)
}
func testTxn(k2 []byte, v22 []byte, v23 []byte) {
txn1, txn2 := begin(), begin()
txn1.SetOption(kv.Pessimistic, true)
fmt.Println("txn1 after:", txn1.IsPessimistic())
txn2.SetOption(kv.Pessimistic, true)
fmt.Println("txn2 after:", txn2.IsPessimistic())
err := txn1.Set(k2, v22)
if err != nil {
panic(err)
}
err = txn2.Set(k2, v23)
if err != nil {
panic(err)
}
err = txn1.Commit(context.Background())
if err != nil {
panic(err)
}
fmt.Println(get(k2))
err = txn2.Commit(context.Background())
if err != nil {
panic(err)
}
}
No matter whether I set txn1.SetOption(kv.Pessimistic, true) and txn2.SetOption(kv.Pessimistic, true) or not, I haven't found the difference between them.
But in tidb or mysql, modify the same records with a pessimistic transaction, it will block.
Such as transaction A:
begin;
updata t1 set col ="value1" where id=1;
transaction B:
begin;
updata t1 set col ="value2" where id=1; //it will block until transaction A commit or rollback
I have two question :
What's the difference between optimistic and pessimistic in tikv?
What's the difference between tikv's pessimistic-lock and mysql/tidb's pessimistic-lock?
If anyone has any idea, please share it with me, thanks
What's the difference between optimistic and pessimistic in tikv?
TiKV adopts Google’s Percolator transaction model to implement the optimistic transaction model. Clients don't write data until the transaction commits.
The pessimistic transaction model is built on the optimistic transaction model which can lock keys before committing the transaction to prevent concurrent modifications.
What's the difference between tikv's pessimistic-lock and mysql/tidb's pessimistic-lock?
TiDB follows the protocol of TiKV's pessimistic transaction model to implement the pessimistic-lock. The different between MySQL and TiDB is documented here.
The kv.Pessimistic option just indicates the transaction is a pessimistic one. Doesn't affect its behavior.
Pessimistic transaction in client-go is not an out-of-the-box feature for now(2021.06). It just provides the basic toolkit and users have to follow the undocumented pessimistic transaction protocol to use it which is error-prone. AFAIK, the only user is TiDB, and it's recommended to use the optimistic transaction model instead.
Related
I'm trying to mock a method written with sqlboiler but I'm having massive trouble building the mock-query.
The model I'm trying to mock looks like this:
type Course struct {
ID int, Name string, Description null.String, EnrollKey string, ForumID int,
CreatedAt null.Time, UpdatedAt null.Time, DeletedAt null.Time,
R *courseR, L courseL
}
For simplicity I want to test the GetCourse-method
func (p *PublicController) GetCourse(id int) (*models.Course, error) {
c, err := models.FindCourse(context.Background(), p.Database, id)
if err != nil {
return nil, err
}
return c, nil
}
with this test
func TestGetCourse(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("an error '%s' was not expected", err)
}
oldDB := boil.GetDB()
defer func() {
db.Close()
boil.SetDB(oldDB)
}()
boil.SetDB(db)
ctrl := &PublicController{db}
rows := sqlmock.NewRows([]string{"ID", "Name", "Description", "EnrollKey", "ForumID"}).AddRow(42, "Testkurs", "12345", 33)
query := regexp.QuoteMeta("SELECT ID, Name, Description, EnrollKey, ForumID FROM courses WHERE ID = ?")
//mockQuery := regexp.QuoteMeta("SELECT * FROM `courses` WHERE (`course AND (`courses`.deleted_at is null) LIMIT 1;")
mock.ExpectQuery(query).WithArgs(42).WillReturnRows(rows)
course, err := ctrl.GetCourse(42)
assert.NotNil(t, course)
assert.NoError(t, err)
}
But running this test only returns
Query: could not match actual sql: "select * from course where id=? and deleted_at is null" with expected regexp "SELECT ID, Name, Description, EnrollKey, ForumID FROM courses WHERE ID = ?"
bind failed to execute query
And I can't really find out how to construct it correctly.
How do I correctly mock the sqlboiler-query for running unit tests?
UPDATE
I managed to solve this by using different parameters in AddRow()
.AddRow(c.ID, c.Name, null.String{}, c.EnrollKey, c.ForumID)
and building the query differently
query := regexp.QuoteMeta("select * from `course` where `id`=? and `deleted_at` is null")
Now my issue is that in contrast to this method the others have a very large complexity in comparison with a large amount of complex queries (mainly insert-operations). From my understanding, sqlboiler-tests needs to mimic every single interaction made with the database.
How do I extract the necessary queries for the large amount of database interactions? I solved my problem by just using the "actual sql-query" instead of the previously used one but I'm afraid this procedure is the opposite of efficient testing.
How do I fill os.Stdin in my test for a function that reads from it using a scanner?
I request a user command line input via a scanner using following function:
func userInput() error {
scanner := bufio.NewScanner(os.Stdin)
println("What is your name?")
scanner.Scan()
username = scanner.Text()
/* ... */
}
Now how do I test this case and simulate a user input?
Following example does not work. Stdin is still empty.
func TestUserInput(t *testing.T) {
var file *os.File
file.Write([]byte("Tom"))
os.Stdin = file
err := userInput()
/* ... */
}
Mocking os.Stdin
You're on the right track that os.Stdin is a variable (of type *os.File) which you can modify, you can assign a new value to it in tests.
Simplest is to create a temporary file with the content you want to simulate as the input on os.Stdin. To create a temp file, use ioutil.TempFile(). Then write the content into it, and seek back to the beginning of the file. Now you can set it as os.Stdin and perform your tests. Don't forget to cleanup the temp file.
I modified your userInput() to this:
func userInput() error {
scanner := bufio.NewScanner(os.Stdin)
fmt.Println("What is your name?")
var username string
if scanner.Scan() {
username = scanner.Text()
}
if err := scanner.Err(); err != nil {
return err
}
fmt.Println("Entered:", username)
return nil
}
And this is how you can test it:
func TestUserInput(t *testing.T) {
content := []byte("Tom")
tmpfile, err := ioutil.TempFile("", "example")
if err != nil {
log.Fatal(err)
}
defer os.Remove(tmpfile.Name()) // clean up
if _, err := tmpfile.Write(content); err != nil {
log.Fatal(err)
}
if _, err := tmpfile.Seek(0, 0); err != nil {
log.Fatal(err)
}
oldStdin := os.Stdin
defer func() { os.Stdin = oldStdin }() // Restore original Stdin
os.Stdin = tmpfile
if err := userInput(); err != nil {
t.Errorf("userInput failed: %v", err)
}
if err := tmpfile.Close(); err != nil {
log.Fatal(err)
}
}
Running the test, we see an output:
What is your name?
Entered: Tom
PASS
Also see related question about mocking the file system: Example code for testing the filesystem in Golang
The easy, preferred way
Also note that you can refactor userInput() to not read from os.Stdin, but instead it could receive an io.Reader to read from. This would make it more robust and a lot easier to test.
In your app you can simply pass os.Stdin to it, and in tests you can pass any io.Reader to it created / prepared in the tests, e.g. using strings.NewReader(), bytes.NewBuffer() or bytes.NewBufferString().
os.Pipe()
Instead of messing with the actual file system and doing writes and reads to and from real files on a storage device, the simplest solution is using os.Pipe().
Example
The code of your userInput() does have to be adjusted, and #icza's solution would indeed do for that purpose. But the test itself should be something more like this:
func Test_userInput(t *testing.T) {
input := []byte("Alice")
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
_, err = w.Write(input)
if err != nil {
t.Error(err)
}
w.Close()
// Restore stdin right after the test.
defer func(v *os.File) { os.Stdin = v }(os.Stdin)
os.Stdin = r
if err = userInput(); err != nil {
t.Fatalf("userInput: %v", err)
}
}
Details
There are several important points about this code:
Always close your w stream when you're done writing. Many utilities rely on an io.EOF returned by a Read() call to know that no more data is coming, and the bufio.Scanner is no exception. If you don't close the stream, your scanner.Scan() call will never return, but keep looping internally and waiting for more input until the program is terminated forcefully (as when the test times out).
The pipe buffer capacity varies from system to system, as discussed at length in a post in the Unix & Linux Stack Exchange, so if the size of your simulated input could exceed that, you should wrap your write(s) in a goroutine like so:
//...
go func() {
_, err = w.Write(input)
if err != nil {
t.Error(err)
}
w.Close()
}()
//...
This prevents a deadlock when the pipe is full and writes have to wait for it to start emptying, but the code that's supposed to be reading from and emptying the pipe (userInput() in this case) is not starting, because of writing not being over yet.
A test should also verify that errors are handled properly, in this case, returned by userInput(). This means you'd have to figure out a way to make the scanner.Err() call return an error in a test. One approach could be closing the r stream it was supposed to be reading, before it has had the chance.
Such a test would look almost identical to the nominal case, only you don't write anything at the w end of the pipe, just close the r end, and you actually expect and want userInput() to return an error. And when you have two or more tests of the same function that are almost identical, it is often a good time to implement them as a single table driven test. See Go playground for an example.
io.Reader
The example of userInput() is trivial enough that you could (and should) refactor it and similar cases to read from an io.Reader, just like #icza suggests (see the playground).
You should always strive to rely on some form of dependency injection instead of global state (os.Stdin, in this case, is a global variable in the os package), as that gives more control to the calling code to determine how a called piece of code behaves, which is essential to unit testing, and facilitates better code reuse in general.
Return of os.Pipe()
There may also be cases when you can't really alter a function to take injected dependencies, as when you have to test the main() function of a Go executable. Altering the global state in the test (and hoping that you can properly restore it by the end not to affect subsequent tests) is your only option then. This is where we come back to os.Pipe()
When testing main(), do use os.Pipe() to simulate input to stdin (unless you already hava a file prepared for the purpose) and to capture the output of stdout and stderr (see the playground for an example of the latter).
Implementation of #icza's easy, preferred way:
Also note that you can refactor userInput() to not read from os.Stdin,
but instead it could receive an io.Reader to read from. This would
make it more robust and a lot easier to test.
In your app you can simply pass os.Stdin to it, and in tests you can
pass any io.Reader to it created / prepared in the tests, e.g. using
strings.NewReader(), bytes.NewBuffer() or bytes.NewBufferString().
hello.go
package main
import (
"bufio"
"fmt"
"os"
"io"
)
func userInput(reader io.Reader) error {
scanner := bufio.NewScanner(reader)
var username string
fmt.Println("What is your name?")
if scanner.Scan() {
username = scanner.Text()
}
if scanner.Err() != nil {
return scanner.Err()
}
fmt.Println("Hello", username)
return nil
}
func main() {
userInput(os.Stdin)
}
hello_test.go
package main
import (
"bytes"
"io"
"strings"
"testing"
)
func TestUserInputWithStringsNewReader(t *testing.T) {
input := "Tom"
var reader io.Reader = strings.NewReader(input)
err := userInput(reader)
if err != nil {
t.Errorf("Failed to read from strings.NewReader: %w", err)
}
}
func TestUserInputWithBytesNewBuffer(t *testing.T) {
input := "Tom"
var reader io.Reader = bytes.NewBuffer([]byte(input))
err := userInput(reader)
if err != nil {
t.Errorf("Failed to read from bytes.NewBuffer: %w", err)
}
}
func TestUserInputWithBytesNewBufferString(t *testing.T) {
input := "Tom"
var reader io.Reader = bytes.NewBufferString(input)
err := userInput(reader)
if err != nil {
t.Errorf("Failed to read from bytes.NewBufferString: %w", err)
}
}
Running the program:
go run hello.go
What is your name?
Tom
Hello Tom
Running the test:
go test hello_test.go hello.go -v
=== RUN TestUserInputWithStringsNewReader
What is your name?
Hello Tom
--- PASS: TestUserInputWithStringsNewReader (0.00s)
=== RUN TestUserInputWithBytesNewBuffer
What is your name?
Hello Tom
--- PASS: TestUserInputWithBytesNewBuffer (0.00s)
=== RUN TestUserInputWithBytesNewBufferString
What is your name?
Hello Tom
--- PASS: TestUserInputWithBytesNewBufferString (0.00s)
PASS
ok command-line-arguments 0.141s
You can use *bufio.Scanner to abstract io.Stdin and io.Writer to abstract io.Stdout while passing them as dependencies to your struct, see
Gist: https://gist.github.com/antonzhukov/2a6749f780b24f38b08c9916caa96663 and
Playground: https://play.golang.org/p/BZMqpACupSc
How do I fill os.Stdin in my test for a function that reads from it using a scanner?
I request a user command line input via a scanner using following function:
func userInput() error {
scanner := bufio.NewScanner(os.Stdin)
println("What is your name?")
scanner.Scan()
username = scanner.Text()
/* ... */
}
Now how do I test this case and simulate a user input?
Following example does not work. Stdin is still empty.
func TestUserInput(t *testing.T) {
var file *os.File
file.Write([]byte("Tom"))
os.Stdin = file
err := userInput()
/* ... */
}
Mocking os.Stdin
You're on the right track that os.Stdin is a variable (of type *os.File) which you can modify, you can assign a new value to it in tests.
Simplest is to create a temporary file with the content you want to simulate as the input on os.Stdin. To create a temp file, use ioutil.TempFile(). Then write the content into it, and seek back to the beginning of the file. Now you can set it as os.Stdin and perform your tests. Don't forget to cleanup the temp file.
I modified your userInput() to this:
func userInput() error {
scanner := bufio.NewScanner(os.Stdin)
fmt.Println("What is your name?")
var username string
if scanner.Scan() {
username = scanner.Text()
}
if err := scanner.Err(); err != nil {
return err
}
fmt.Println("Entered:", username)
return nil
}
And this is how you can test it:
func TestUserInput(t *testing.T) {
content := []byte("Tom")
tmpfile, err := ioutil.TempFile("", "example")
if err != nil {
log.Fatal(err)
}
defer os.Remove(tmpfile.Name()) // clean up
if _, err := tmpfile.Write(content); err != nil {
log.Fatal(err)
}
if _, err := tmpfile.Seek(0, 0); err != nil {
log.Fatal(err)
}
oldStdin := os.Stdin
defer func() { os.Stdin = oldStdin }() // Restore original Stdin
os.Stdin = tmpfile
if err := userInput(); err != nil {
t.Errorf("userInput failed: %v", err)
}
if err := tmpfile.Close(); err != nil {
log.Fatal(err)
}
}
Running the test, we see an output:
What is your name?
Entered: Tom
PASS
Also see related question about mocking the file system: Example code for testing the filesystem in Golang
The easy, preferred way
Also note that you can refactor userInput() to not read from os.Stdin, but instead it could receive an io.Reader to read from. This would make it more robust and a lot easier to test.
In your app you can simply pass os.Stdin to it, and in tests you can pass any io.Reader to it created / prepared in the tests, e.g. using strings.NewReader(), bytes.NewBuffer() or bytes.NewBufferString().
os.Pipe()
Instead of messing with the actual file system and doing writes and reads to and from real files on a storage device, the simplest solution is using os.Pipe().
Example
The code of your userInput() does have to be adjusted, and #icza's solution would indeed do for that purpose. But the test itself should be something more like this:
func Test_userInput(t *testing.T) {
input := []byte("Alice")
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
_, err = w.Write(input)
if err != nil {
t.Error(err)
}
w.Close()
// Restore stdin right after the test.
defer func(v *os.File) { os.Stdin = v }(os.Stdin)
os.Stdin = r
if err = userInput(); err != nil {
t.Fatalf("userInput: %v", err)
}
}
Details
There are several important points about this code:
Always close your w stream when you're done writing. Many utilities rely on an io.EOF returned by a Read() call to know that no more data is coming, and the bufio.Scanner is no exception. If you don't close the stream, your scanner.Scan() call will never return, but keep looping internally and waiting for more input until the program is terminated forcefully (as when the test times out).
The pipe buffer capacity varies from system to system, as discussed at length in a post in the Unix & Linux Stack Exchange, so if the size of your simulated input could exceed that, you should wrap your write(s) in a goroutine like so:
//...
go func() {
_, err = w.Write(input)
if err != nil {
t.Error(err)
}
w.Close()
}()
//...
This prevents a deadlock when the pipe is full and writes have to wait for it to start emptying, but the code that's supposed to be reading from and emptying the pipe (userInput() in this case) is not starting, because of writing not being over yet.
A test should also verify that errors are handled properly, in this case, returned by userInput(). This means you'd have to figure out a way to make the scanner.Err() call return an error in a test. One approach could be closing the r stream it was supposed to be reading, before it has had the chance.
Such a test would look almost identical to the nominal case, only you don't write anything at the w end of the pipe, just close the r end, and you actually expect and want userInput() to return an error. And when you have two or more tests of the same function that are almost identical, it is often a good time to implement them as a single table driven test. See Go playground for an example.
io.Reader
The example of userInput() is trivial enough that you could (and should) refactor it and similar cases to read from an io.Reader, just like #icza suggests (see the playground).
You should always strive to rely on some form of dependency injection instead of global state (os.Stdin, in this case, is a global variable in the os package), as that gives more control to the calling code to determine how a called piece of code behaves, which is essential to unit testing, and facilitates better code reuse in general.
Return of os.Pipe()
There may also be cases when you can't really alter a function to take injected dependencies, as when you have to test the main() function of a Go executable. Altering the global state in the test (and hoping that you can properly restore it by the end not to affect subsequent tests) is your only option then. This is where we come back to os.Pipe()
When testing main(), do use os.Pipe() to simulate input to stdin (unless you already hava a file prepared for the purpose) and to capture the output of stdout and stderr (see the playground for an example of the latter).
Implementation of #icza's easy, preferred way:
Also note that you can refactor userInput() to not read from os.Stdin,
but instead it could receive an io.Reader to read from. This would
make it more robust and a lot easier to test.
In your app you can simply pass os.Stdin to it, and in tests you can
pass any io.Reader to it created / prepared in the tests, e.g. using
strings.NewReader(), bytes.NewBuffer() or bytes.NewBufferString().
hello.go
package main
import (
"bufio"
"fmt"
"os"
"io"
)
func userInput(reader io.Reader) error {
scanner := bufio.NewScanner(reader)
var username string
fmt.Println("What is your name?")
if scanner.Scan() {
username = scanner.Text()
}
if scanner.Err() != nil {
return scanner.Err()
}
fmt.Println("Hello", username)
return nil
}
func main() {
userInput(os.Stdin)
}
hello_test.go
package main
import (
"bytes"
"io"
"strings"
"testing"
)
func TestUserInputWithStringsNewReader(t *testing.T) {
input := "Tom"
var reader io.Reader = strings.NewReader(input)
err := userInput(reader)
if err != nil {
t.Errorf("Failed to read from strings.NewReader: %w", err)
}
}
func TestUserInputWithBytesNewBuffer(t *testing.T) {
input := "Tom"
var reader io.Reader = bytes.NewBuffer([]byte(input))
err := userInput(reader)
if err != nil {
t.Errorf("Failed to read from bytes.NewBuffer: %w", err)
}
}
func TestUserInputWithBytesNewBufferString(t *testing.T) {
input := "Tom"
var reader io.Reader = bytes.NewBufferString(input)
err := userInput(reader)
if err != nil {
t.Errorf("Failed to read from bytes.NewBufferString: %w", err)
}
}
Running the program:
go run hello.go
What is your name?
Tom
Hello Tom
Running the test:
go test hello_test.go hello.go -v
=== RUN TestUserInputWithStringsNewReader
What is your name?
Hello Tom
--- PASS: TestUserInputWithStringsNewReader (0.00s)
=== RUN TestUserInputWithBytesNewBuffer
What is your name?
Hello Tom
--- PASS: TestUserInputWithBytesNewBuffer (0.00s)
=== RUN TestUserInputWithBytesNewBufferString
What is your name?
Hello Tom
--- PASS: TestUserInputWithBytesNewBufferString (0.00s)
PASS
ok command-line-arguments 0.141s
You can use *bufio.Scanner to abstract io.Stdin and io.Writer to abstract io.Stdout while passing them as dependencies to your struct, see
Gist: https://gist.github.com/antonzhukov/2a6749f780b24f38b08c9916caa96663 and
Playground: https://play.golang.org/p/BZMqpACupSc
I'm currently looking into creating some unit tests for my service in Go, as well as other functions that build up on top of that functionality, and I'm wondering what is the best way to unit test that in Go? My code looks like:
type BBPeripheral struct {
client *http.Client
endpoint string
}
type BBQuery struct {
Name string `json:"name"`
}
type BBResponse struct {
Brand string `json:"brand"`
Model string `json:"model"`
...
}
type Peripheral struct {
Brand string
Model string
...
}
type Service interface {
Get(name string) (*Peripheral, error)
}
func NewBBPeripheral(config *peripheralConfig) (*BBPeripheral, error) {
transport, err := setTransport(config)
if err != nil {
return nil, err
}
BB := &BBPeripheral{
client: &http.Client{Transport: transport},
endpoint: config.Endpoint[0],
}
return BB, nil
}
func (this *BBPeripheral) Get(name string) (*Peripheral, error) {
data, err := json.Marshal(BBQuery{Name: name})
if err != nil {
return nil, fmt.Errorf("BBPeripheral.Get Marshal: %s", err)
}
resp, err := this.client.Post(this.endpoint, "application/json", bytes.NewBuffer(data))
if resp != nil {
defer resp.Body.Close()
}
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf(resp.StatusCode)
}
var BBResponse BBResponse
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &BBResponse)
if err != nil {
return nil, err
}
peripheral := &Peripheral{}
peripheral.Model = BBResponse.Model
if peripheral.Model == "" {
peripheral.Model = NA
}
peripheral.Brand = BBResponse.Brand
if peripheral.Brand == "" {
peripheral.Brand = NA
}
return peripheral, nil
}
Is the most efficient way of testing this code and the code that uses these functions to spin up a separate goroutine to act like the server, use http.httptest package, or something else? that's the first time that i try to write a test then i don't realy know how.
It really completely depends. Go provides pretty much all the tools you need to test your application at every single level.
Unit Tests
Design is important because there aren't many tricks to dynamically provide mock/stub objects. You can override variables for tests, but it unlocks all sorts of problems with cleanup. I would focus on IO free unit tests to check that your specific logic works.
For example, you could test BBPeripheral.Get method by making client an interface, requiring it during instantiation, and providing a stub one for the test.
func Test_BBPeripheral_Get_Success(*testing.T) {
bb := BBPeripheral{client: &StubSuccessClient, ...}
p, err := bb.Get(...)
if err != nil {
t.Fail()
}
}
Then you could create a stub error client that exercises error handling in the Get method:
func Test_BBPeripheral_Get_Success(*testing.T) {
bb := BBPeripheral{client: &StubErrClient, ...}
_, err := bb.Get(...)
if err == nil {
t.Fail()
}
}
Component/Integration Tests
These tests can help exercise that each individual unit in your package can work together in unison. Since your code talks over http, Go provides the httptest package that could be used.
To do this the test could create an httptest server with a handler registered to provide the response that this.endpoint expects. You could then exercise your code using its public interface by requesting a NewBBPeripheral, passing in this.endpoint corresponding to the Server.URL property.
This allows you to simulate your code talking to a real server.
Go Routine Tests
Go makes it so easy to write concurrent code, and makes it just as easy to test it. Testing the top level code that spawns a go routine that exercises NewBBPeripheral could look very much like the test above. In addition to starting up a test server your test will have to wait your your asynchronous code to complete. If you don't have a service wide way to cancel/shutdown/signal complete then one may be required to test it using go routines.
RaceCondition/Load Testing
Using go's built in bechmark test combined with -race flag, you can easily exercise your code, and profile it for race conditions, leveraging the tests you wrote above.
One thing to keep in mind, if the implementation of your application is still in flux, writing unit tests may cost a large amount of time. Creating a couple tests, which exercise the public interface of your code, should allow you to easily verify that your application is working, while allowing the implementation to change.
I reimplement my project with golang recently. The project was implemented with C++. When I finished the code and have a performance test. I'm shocked by the result. When I query the database with C++, I can get the 130 million rows result in 5 mins. But with golang, it's almost 45 mins. But when I separate the code from the project and build the code snippet, it's finished in 2mins. Why does they have so much huge difference performance result?
My code snippet :
https://gist.github.com/pyanfield/2651d23311901b33c5723b7de2364148
package main
import (
"database/sql"
"fmt"
"runtime"
"strconv"
"time"
_ "github.com/go-sql-driver/mysql"
)
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
// defer profile.Start(profile.CPUProfile, profile.ProfilePath(".")).Stop()
dbRead, err := connectDB("test:test#tcp(127.0.0.1:3306)/test_oltp?charset=utf8&readTimeout=600s&writeTimeout=600s")
if err != nil {
fmt.Printf("Error happend when connecting to DB. %s\n", err.Error())
return
}
defer dbRead.Close()
dbRead.SetMaxIdleConns(0)
dbRead.SetMaxOpenConns(100)
query := fmt.Sprintf("WHERE company_id in (11,22,33,44,55,66,77,88,99,00,111,222,333,4444,555,666,777,888,999)")
relations := getRelations(dbRead, query)
}
func connectDB(addr string) (*sql.DB, error) {
db, err := sql.Open("mysql", addr)
if err != nil {
return nil, err
}
if err = db.Ping(); err != nil {
return nil, err
}
return db, nil
}
type Relation struct {
childId int64
parentId int64
}
func getRelations(db *sql.DB, where string)[]Relation {
begin := time.Now()
var err error
var rows *sql.Rows
query := fmt.Sprintf("SELECT `child_id`, `parent_id` FROM `test_relations` %s", where)
rows, err = db.Query(query)
if err != nil {
fmt.Println("query error:", err.Error())
return nil
}
defer rows.Close()
columns, err := rows.Columns()
buffer := make([]sql.RawBytes, len(columns))
scanArgs := make([]interface{}, len(buffer))
for i := range scanArgs {
scanArgs[i] = &buffer[i]
}
relations := []Relation{}
relation := Relation{}
for rows.Next() {
if err = rows.Scan(scanArgs...); err != nil {
fmt.Println("scan:", err.Error())
return nil
}
relation.parentId, _ = strconv.ParseInt(string(buffer[1]), 10, 64)
relation.childId, _ = strconv.ParseInt(string(buffer[0]), 10, 64)
relations = append(relations, relation)
}
if err = rows.Err(); err != nil {
fmt.Println("next error:", err.Error())
return nil
}
fmt.Printf(">>> getRelations cost: %s\n", time.Since(begin).String())
// output :>>> getRelations cost:1m45.791047s
return relations
// len(relations): 131123541
}
Update:
My go version is 1.6. The cpu profile I got are as below:
The Code Snippet profile top20:
75.67s of 96.82s total (78.16%)
Dropped 109 nodes (cum <= 0.48s)
Showing top 20 nodes out of 82 (cum >= 12.04s)
flat flat% sum% cum cum%
11.85s 12.24% 12.24% 11.85s 12.24% runtime.memmove
10.28s 10.62% 22.86% 20.01s 20.67% runtime.mallocgc
5.82s 6.01% 28.87% 5.82s 6.01% strconv.ParseUint
5.79s 5.98% 34.85% 5.79s 5.98% runtime.futex
3.42s 3.53% 38.38% 10.28s 10.62% github.com/go-sql-driver/mysql.(*buffer).readNext
3.42s 3.53% 41.91% 6.38s 6.59% runtime.scang
3.37s 3.48% 45.39% 36.97s 38.18% github.com/go-sql-driver/mysql.(*textRows).readRow
3.37s 3.48% 48.87% 3.37s 3.48% runtime.memclr
3.20s 3.31% 52.18% 3.20s 3.31% runtime.heapBitsSetType
3.02s 3.12% 55.30% 7.36s 7.60% database/sql.convertAssign
2.96s 3.06% 58.36% 3.02s 3.12% runtime.(*mspan).sweep.func1
2.53s 2.61% 60.97% 2.53s 2.61% runtime._ExternalCode
2.39s 2.47% 63.44% 2.96s 3.06% runtime.readgstatus
2.24s 2.31% 65.75% 8.06s 8.32% strconv.ParseInt
2.21s 2.28% 68.03% 5.24s 5.41% runtime.heapBitsSweepSpan
2.15s 2.22% 70.25% 7.68s 7.93% runtime.rawstring
2.06s 2.13% 72.38% 3.18s 3.28% github.com/go-sql-driver/mysql.readLengthEncodedString
1.95s 2.01% 74.40% 12.23s 12.63% github.com/go-sql-driver/mysql.(*mysqlConn).readPacket
1.83s 1.89% 76.29% 79.42s 82.03% main.Relations
1.81s 1.87% 78.16% 12.04s 12.44% runtime.slicebytetostring
The project cpu profile top20:
(pprof) top20
38.71mins of 42.82mins total (90.40%)
Dropped 334 nodes (cum <= 0.21mins)
Showing top 20 nodes out of 76 (cum >= 1.35mins)
flat flat% sum% cum cum%
12.02mins 28.07% 28.07% 12.48mins 29.15% runtime.addspecial
5.95mins 13.89% 41.96% 15.08mins 35.21% runtime.pcvalue
5.26mins 12.29% 54.25% 5.26mins 12.29% runtime.readvarint
2.60mins 6.08% 60.32% 7.87mins 18.37% runtime.step
1.98mins 4.62% 64.94% 19.45mins 45.43% runtime.gentraceback
1.65mins 3.86% 68.80% 1.65mins 3.86% runtime/internal/atomic.Xchg
1.57mins 3.66% 72.46% 2.93mins 6.84% runtime.(*mspan).sweep
1.52mins 3.54% 76.01% 1.78mins 4.15% runtime.findfunc
1.41mins 3.30% 79.31% 1.42mins 3.31% runtime.markrootSpans
1.13mins 2.64% 81.95% 1.13mins 2.64% runtime.(*fixalloc).alloc
0.64mins 1.50% 83.45% 0.64mins 1.50% runtime.duffcopy
0.46mins 1.08% 84.53% 0.46mins 1.08% runtime.findmoduledatap
0.44mins 1.02% 85.55% 0.44mins 1.02% runtime.fastrand1
0.42mins 0.97% 86.52% 15.49mins 36.18% runtime.funcspdelta
0.38mins 0.89% 87.41% 36.02mins 84.13% runtime.mallocgc
0.30mins 0.7% 88.12% 0.78mins 1.83% runtime.scanobject
0.26mins 0.6% 88.72% 0.32mins 0.74% runtime.stkbucket
0.26mins 0.6% 89.32% 0.26mins 0.6% runtime.memmove
0.23mins 0.55% 89.86% 0.23mins 0.55% runtime.heapBitsForObject
0.23mins 0.53% 90.40% 1.35mins 3.15% runtime.lock
I got my answer and want to share it. This is caused by my mistake. Sometimes ago, I tried to add memory profile and set runtime. MemProfileRate=1 in my init method. But I forgot to reset it to a reasonable value. I ignored this method when I checked my code every time. After removing this setting from my project, it returns to normal, and spend almost 5~6mins to query these 130M datas. The speed is pretty close to the C++ version. My advise is that please carefully when you set runtime.MemProfileRate=1 unless you make sure you want to do that, and remember to reset it back.
Golang is likely running the DB query processing more in parallel for the snippet alone. Your complete application is almost certainly using some of those cores for other things.
The loop where you process all 130M rows seems the likely culprit.
Try setting the max procs to 1 in the snippet if you want to test this theory.