Data streaming fails with BigQuery
It took 1,000 daunting inserts at the streaming insert, and it suddenly failed today. Why do not you know why you failed?
In the log, "[1 row insertion failed]"
That's all.
It looks like you're doing streaming inserts in Go using the package cloud.google.com/go/bigquery.
The method Inserter.Put() returns a PutMultiError when something goes wrong. PutMultiError is responsible for the message you saw. If you want to see more detail about the failure, you have to look at each sub-error separately. Something like this:
err := uploader.Put(ctx, rows)
switch e := err.(type) {
case bigquery.PutMultiError:
rowInsertionError := e[0] // the first failed row
for _, err := range rowInsertionError.Errors {
log.Printf("err = %v", err)
}
}
Related
I have following code
func (s *MyRepo) InsertOrder(ctx context.Context, orderID string) error {
query := `INSERT INTO orders (orderID) VALUES (?)`
stmt, err := s.db.RawDatabase().PrepareContext(ctx, query)
if err != nil {
return err
}
defer stmt.Close()
_, err = stmt.ExecContext(ctx, orderID)
if err != nil {
//log err
}
return err
}
And the corresponding test case is
func TestMyRepo_InsertOrder_Success(t *testing.T) {
orderID := "orderID"
mockDB, repo := getDBStore()
query := `[INSERT INTO orders (orderID) VALUES (?)]`
mockDB.ExpectPrepare(query).
ExpectExec().
WithArgs(orderID).
WillReturnResult(sqlmock.NewResult(1, 1)).
WillReturnError(nil)
err := repo.InsertOrder(context.Background(), orderID)
assert.Nil(t, err)
}
But this doesn't test if defer stmt.Close() has been called or not (which gets called once the function ends). How can I test this?
It looks like you are making use of data-dog's sqlmock package, so you ought to be able to use ExpectClose() to register the expectation that the database will be closed, and ExpectationsWereMet() to collect this information.
If you're using some other package, feel free to link it; there's probably something similar available, and worst-case you can write your own wrapper around their wrapper. Making sure that a particular method of a particular dependency was called is a fairly common desire when developers write tests using mocks, so most of the better mock packages will go out of their way to provide some sort of API to check that.
As noted in the comments on this question, tests of this nature are often of somewhat questionable value and can seem like they exist more to increase a dubious metric like % code coverage than to increase code reliability or maintainability.
I'm using the "cloud.google.com/go/datastore" library in my Go application (library version below), and am running into an error datastore: invalid entity type when I use the Ancestor() query.
Here's my method call:
ctx := context.Background()
client, err := datastore.NewClient(ctx, "MyProjectId", option.WithCredentialsFile(myJsonFile))
// ... err check ...
myId := 112233
myKey := datastore.IDKey("MyKind", myId, nil)
query := datastore.NewQuery("").Ancestor(myKey)
it := client.Run(*ctx, query)
c = &struct{ ... }
key, err := it.Next(&c)
// ... err check ...
However, when I check the value of err, it's non-nil and has the message: datastore: invalid entity type. When I go into the cloud console and try what I think would be the equivalent GQL (e.g. SELECT * WHERE __key__ HAS ANCESTOR KEY(MyKind, 112233)), it does indeed return results in the cloud console.
I also didn't find a convenient way to print out debugging information in the "cloud.google.com/go/datastore" library to see what it's actually sending, so any help would be much appreciated there.
Any ideas on how I might debug this, or is it an issue in the Go library? Thank you!
P.S. Here's the "cloud.google.com/go/datastore" version I'm using:
commit f0afaf5fad3c46ae392ebab6b7553d37d65d07ac (HEAD -> master, origin/master, origin/HEAD)
Author: [redacted]
Date: Thu May 31 13:15:30 2018 -0700
oslogin: generate v1 client
Change-Id: I515527be0e8acc1fe55cab969df9f76fd8d93b26
There's a bug in the code above which is passing a double reference to a struct.
c = &struct{ ... }
key, err := it.Next(&c)
Should actually be:
c = &struct{ ... }
key, err := it.Next(c)
Unfortunately the "Next" function returns a datastore: invalid entity type rather than something a little more... helpful ;)
I am creating a web application that lists applicants and their position on a waiting list.
We need to be able to add new applicants to this list and remove applicants from the list. There will be under 10k applicants in the list.
Specifics:
I plan to write the app in Golang.
The list needs to be safe, I the program shuts down, it should be recoverable.
The app should contain this data for every applicant: Name, Student ID, position.
Questions:
How do I secure the list (lock?) so it is updated correctly for both if two updates to it is made at the same time?
Should I save the data in a database or use a file?
I need your help!
UPDATE:
Mockup code:
package main
import (
"log"
"sync"
"time"
"github.com/boltdb/bolt"
)
type applicant struct {
FirstName string
LastName string
StudentID string
Position int
}
type priorityList struct {
sync.Mutex
applicants []applicant
}
func (l *priorityList) newApplicant(fn string, ln string, sid string) error {
// add applicant to priorityList
return nil
}
func (l *priorityList) removeApplicant(sid string) error {
// remove applicant from priorityList
return nil
}
func (l *priorityList) editApplicant(sid string) error {
// edit applicant in priorityList
return nil
}
func main() {
// Database
db, err := bolt.Open("priorityList.db", 0600, &bolt.Options{Timeout: 1 * time.Second})
if err != nil {
log.Fatal(err)
}
defer db.Close()
}
If you use a file, you could use a Mutex to block concurrent writes.
Otherwise a database would be fine. For example BoltDB could be suitable. It is pure go and runs withing your program.
There are many approaches. You can use file and protect it with Go mutex or system lock. You can memory map the file for performance. You either can use BoltDB which is nice peace of software and provide needed machinery and can work in-process. If you write rare and mostly read, then constant DB https://github.com/colinmarc/cdb also looks interesting.
But, classic SQL DB has some advantages
You can use third party store for data and easely migrate when needed
You can access your data from third party app or just plain SQL
request
You can think about data schema and code logic separately
First I apologice for my English and I advice I'm new to Go.
I'm trying to get a list of all stackId's on my opsworks stacks,
or to give the stackname like I did with the ruby sdk and get the stackid so I can use with another service calls, but for now im trying to get them all to get familiar with the sdk.
func main() {
svc := opsworks.New(session.New(&aws.Config{
Region: aws.String("us-east-1"),
Credentials: credentials.NewSharedCredentials("", "development"),
}))
resp, err := svc.DescribeStacks()
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(resp)
}
the reason I omitted "params" is because of this:
http://docs.aws.amazon.com/sdk-for-go/api/service/opsworks.html#type-DescribeStacksInput
that says: "An array of stack IDs that specify the stacks to be described. If you omit this parameter, DescribeStacks returns a description of every stack."
O keep getting or "not enough arguments in call to svc.DescribeStacks"
or "alidationException: Please provide no arguments or, one or more stack IDs" or "!v(PANIC=reflect: call of reflect.Value.Interface on zero Value)"
So I'm trying many ways.. not just the one I pasted here.. thats why diff results... changing to () to (nil) etc.. to omit the parameters so I can get a list of all the stacks.
Im sure is very silly, any idea? I have search in google etc but all examples are for ec2 or s3...
got a response that worked in the main github issues site.
I see thanks for clarifying. The example you listed in 180498284 is actually the best way to get a list of all of your stack Ids.
To retrieve the individual stack Ids from the response you use the fields of opsworks.DescribeStacksOutput struct. This struct has a field Stacks which is a list of opsworks.Stack structs. The StackId is a field on this struct.
resp, err := svc.DescribeStacks(nil)
if err != nil {
return err
}
for _, stack := range resp.Stacks {
stackName := aws.StringValue(stack.Name)
if stackName == owstackname {
fmt.Printf("found %s stackid\n", stackName)
fmt.Println(aws.StringValue(stack.StackId))
}
}
The aws.StringValue functions are helpers to convert the string pointer types of the SDK's input/output structs to values. It also protects against the value being nil where it will return empty string.
I am currently trying to test a piece of my code that runs a query on the datastore before putting in a new entity to ensure that duplicates are not created. The code I wrote works fine in the context of the app, but the tests I wrote for that methods are failing. It seems that I cannot access data put into the datastore through queries in the context of the testing package.
One possibility might lie in the output from goapp test which reads: Applying all pending transactions and saving the datastore. This line prints out after both the get and put methods are called (I verified this with log statements).
I tried closing the context and creating a new one for the different operations, but unfortunately that didn't help either. Below is a simple test case that Puts in an object and then runs a query on it. Any help would be appreciated.
type Entity struct {
Value string
}
func TestEntityQuery(t *testing.T) {
c, err := aetest.NewContext(nil)
if err != nil {
t.Fatal(err)
}
defer c.Close()
key := datastore.NewIncompleteKey(c, "Entity", nil)
key, err = datastore.Put(c, key, &Entity{Value: "test"})
if err != nil {
t.Fatal(err)
}
q := datastore.NewQuery("Entity").Filter("Value =", "test")
var entities []Entity
keys, err := q.GetAll(c, &entities)
if err != nil {
t.Fatal(err)
}
if len(keys) == 0 {
t.Error("No keys found in query")
}
if len(entities) == 0 {
t.Error("No entities found in query")
}
}
There is nothing wrong with your test code. The issue lies in the Datastore itself. Most queries in the HR Datastore are not "immediately consistent" but eventually consistent. You can read more about this in the Datastore documentation.
So basically what happens is that you put an entity into the Datastore, and the SDK's Datastore "simulates" the latency that you can observe in production, so if you run a query right after that (which is not an ancestor query), the query result will not include the new entity you just saved.
If you put a few seconds sleep between the datastore.Put() and q.GetAll(), you will see the test passes. Try it. In my test it was enough to sleep just 100ms, and the test always passed. But when writing tests for such cases, use the StronglyConsistentDatastore: true option as can be seen in JonhGB's answer.
You would also see the test pass without sleep if you'd use Ancestor queries because they are strongly consistent.
The way to do this is to force the datastore to be strongly consistent by setting up the context like this:
c, err := aetest.NewContext(&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
Now the datastore won't need any sleep to work, which is faster, and better practice in general.
Update: This only works with the old aetest package which was imported via appengine/aetest. It does not work with the newer aetest package which is imported with google.golang.org/appengine/aetest. App Engine has changed from using an appengine.Context to using a context.Context, and consequently the way that the test package now works is quite different.
To compliment #JohnGB's answer in the latest version of aetest, there are more steps to get a context with strong consistency. First create an instance, then create a request from that instance, which you can use to produce a context.
inst, err := aetest.NewInstance(
&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
t.Fatal(err)
}
ctx := appengine.NewContext(req)