I want to simulate bytes.ErrTooLarge panic error on bytes.Buffer.Write method and test panic handling. I have tried to write unlimited data amount to exceed memory but then whole test crashed. What are the other options?
Sounds like a job for a mock object. Use this (badBuffer) in place of your bytes.Buffer during your test.
type badBuffer bytes.Buffer
func (b *badBuffer) Write(p []byte) (n int, err error) {
panic(bytes.ErrTooLarge)
}
Related
As we know, the advantage of BigQuery Storage Write API, one month ago, we replace insertAll with managedwriter API on our server. It seems to work well for one month, however, we met the following errors recently
rpc error: code = Unavailable desc = closing transport due to: connection error:
desc = "error reading from server: EOF", received prior goaway: code: NO_ERROR,
debug data: "server_shutting_down"
The version of managedwriter API are:
cloud.google.com/go/bigquery v1.25.0
google.golang.org/protobuf v1.27.1
There is a piece of retrying logic for storage write API that detects error messages on our server-side. We notice the response time of storage write API becomes longer after retrying, as a result, OOM is happening on our server. We also tried to increase the request timeout to 30 seconds, and most of those requests could not be completed within it.
How to handle the error server_shutting_down correctly?
Update 02/08/2022
The default stream of managedwrite API is used in our server. And server_shutting_down error comes up periodically. And this issue happened on 02/04/2022 12:00 PM UTC and the default stream of managedwrite API works well for over one month.
Here is one wrapper function of appendRow and we log the cost time of this function.
func (cl *GBOutput) appendRows(ctx context.Context,datas [][]byte, schema *gbSchema) error {
var result *managedwriter.AppendResult
var err error
if cl.schema != schema {
cl.schema = schema
result, err = cl.managedStream.AppendRows(ctx, datas, managedwriter.UpdateSchemaDescriptor(schema.descriptorProto))
} else {
result, err = cl.managedStream.AppendRows(ctx, datas)
}
if err != nil {
return err
}
_, err = result.GetResult(ctx)
return err
}
When the error server_shutting_down comes up, the cost time of this function could be several hundred seconds. It is so weird, and it seems to there is no way to handle the timeout of appendRow.
Are you using the "raw" v1 storage API, or the managedwriter? I ask because managedwriter should handle stream reconnection automatically. Are you simply observing connection closes periodically, or something about your retry traffic induces the closes?
The interesting question is how to deal with in-flight appends for which you haven't yet received an acknowledgement back (or the ack ended in failure). If you're using offsets, you should be able to re-send the append without risk of duplication.
Per the GCP support guy,
The issue is hit once 10MB has been sent over the connection, regardless of how long it takes or how much is inflight at that time. The BigQuery Engineering team has identified the root cause and the fix would be rolled out by Friday, Feb 11th, 2022.
I'm using Go on AWS lambda and I'm writing the following code:
func HanldeLambdaFunction(ctx context.Context, request events.APIGatewayProxyRequest) (Response, error) {
limitString := request.QueryStringParameters["limit"]
fmt.Println("limitString", limitString) //nothing is written
}
I'm testing the lambda function using the api gateway and providing params:
limit=2
The lambda function is being executed successfully but the limit is not being read.
How can I access the params?
The problem with the code you posted is, that it should not even compile. The biggest problem being, that there is no Response struct. It probably should be events.APIGatewayProxyResponse.
Furthermore, the code does not return anything, even though you define that it should return Response and error.
I took your code and fixed all of this and it works for me. The fixed code looks like this:
package main
import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
)
func HanldeLambdaFunction(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
limitString := request.QueryStringParameters["limit"]
fmt.Println("limitString", limitString) //nothing is written
return events.APIGatewayProxyResponse{StatusCode: 200}, nil
}
func main() {
lambda.Start(HanldeLambdaFunction)
}
The output is:
START RequestId: 0c63f94f-b0aa-49de-ba6d-b1150d711b8a Version: $LATEST
limitString 2
END RequestId: 0c63f94f-b0aa-49de-ba6d-b1150d711b8a
REPORT RequestId: 0c63f94f-b0aa-49de-ba6d-b1150d711b8a Duration: 0.56 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 34 MB
If I had to guess, then your code does not even run. If it would run but could not read the limit parameter, it should at least print limitString.
Remember, if you compile a go binary for AWS Lambda, you need to compile it for linux and amd64 like so:
$ GOOS=linux GOARCH=amd64 go build
I asked this in a previous question, but some people felt that my original question was not detailed enough ("Why would you ever want a timed condition wait??") so here is a more specific one.
I have a goroutine running, call it server. It is already started, will execute for some amount of time, and do its thing. Then, it will exit since it is done.
During its execution some large number of other goroutines start. Call them "client" threads if you like. They run step A, and step B. Then, they must wait for the "server" goroutine to finish for a specified amount of time, and exit with status if "server is not finished, and say run step C if it finishes.
(Please do not tell me how to restructure this workflow. It is hypothetical and a given. It cannot be changed.)
A normal, sane way to do this is to have the server thread signal a condition variable with a selectAll or Broadcast function, and have the other threads in a timed wait state monitoring the condition variable.
func (s *Server) Join(timeMillis int) error {
s.mux.Lock()
defer s.mux.Unlock()
while !s.isFinished {
err = s.cond.Wait(timeMillis)
if err != nil {
stepC()
}
}
return err
}
Where the server will enter a state where isFinished becomes true and broadcast signal the condition variable atomically with respect to the mutex. Except this is impoosible, since Go does not support timed condition waits. (But there is a Broadcast())
So, what is the "Go-centric" way to do this? I've reall all of the Go blogs and documentation, and this pattern or its equivalent, despite its obviousness, never comes up, nor any equivalent "reframing" of the basic problem - which is that IPC style channels are between one routine and one other routine. Yes, there is fan-in/fan-out, but remember these threads are constantly appearing and vanishing. This should be simple - and crucially /not leave thousands of "wait-state" goroutines hanging around waiting for the server to die when the other "branch" of the mux channel (the timer) has signalled/.
Note that some of the "client" above might be started before the server goroutine has started (this is when channel is usually created), some might appear during, and some might appear after... in all cases they should run stepC if and only if the server has run and exited after timeMillis milliseconds post entering the Join() function...
In general the channels facility seems sorely lacking when there's more than one consumer. "First build a registry of channels to which listeners are mapped" and "there's this really nifty recursive data structure which sends itself over a channel it holds as field" are so.not.ok as replacements to the nice, reliable, friendly, obvious: wait(forSomeTime)
I think what you want can be done by selecting on a single shared channel, and then having the server close it when it's done.
Say we create a global "Exit channel", that's shared across all goroutines. It can be created before the "server" goroutine is created. The important part is that the server goroutine never sends anything down the channel, but simply closes it.
Now the client goroutines, simply do this:
select {
case <- ch:
fmt.Println("Channel closed, server is done!")
case <-time.After(time.Second):
fmt.Println("Timed out. do recovery stuff")
}
and the server goroutine just does:
close(ch)
More complete example:
package main
import(
"fmt"
"time"
)
func waiter(ch chan struct{}) {
fmt.Println("Doing stuff")
fmt.Println("Waiting...")
select {
case <- ch:
fmt.Println("Channel closed")
case <-time.After(time.Second):
fmt.Println("Timed out. do recovery stuff")
}
}
func main(){
ch := make(chan struct{})
go waiter(ch)
go waiter(ch)
time.Sleep(100*time.Millisecond)
fmt.Println("Closing channel")
close(ch)
time.Sleep(time.Second)
}
This can be abstracted as the following utility API:
type TimedCondition struct {
ch chan struct{}
}
func NewTimedCondition()*TimedCondition {
return &TimedCondition {
ch: make(chan struct{}),
}
}
func (c *TimedCondition)Broadcast() {
close(c.ch)
}
func (c *TimedCondition)Wait(t time.Duration) error {
select {
// channel closed, meaning broadcast was called
case <- c.ch:
return nil
case <-time.After(t):
return errors.New("Time out")
}
}
I am building a server that uses websockets.
Currently every connected client uses two goroutines. One for reading and one for writing.
The writing goroutine basically listens to a channel for messages it should send and then tries to deliver them.
type User struct{
send chan []byte
...
}
func (u *User) Send(msg []byte){
u.send <- msg
}
Problem is, that a read from client A may cause a write to client B.
Suppose the connection to B has some problems however (e.g. very slow) and it's send channel is already full. Current behaviour is, that trying to add a message to the channel now starts blocking, until something is removed from the channel.
This means, that now A waits until B's buffer is no longer full.
I want to solve it somewhat like this:
func (u *User) Send(msg []byte) err{
u.send, err <- msg
if err != nil{
//The channels buffer is full.
//Writing currently not possible.
//Needs appropriate error handling.
return err
}
return nil
}
Basically instead of blocking I want error handling in case the buffer is full.
How do I achieve that best?
As ThunderCat pointed out in his comment the solution is
func (u *User) Send(msg []byte){
select{
case u.send <- msg:
default: //Error handling here
}
}
How can other goroutines keep executing whilst invoking a syscall? (when using GOMAXPROCS=1)
As far as I'm aware of, when invoking a syscall the thread gives up control until the syscall returns.
How can Go achieve this concurrency without creating a system thread per blocking-on-syscall goroutine?
From the documentation:
Goroutines
They're called goroutines because the existing terms—threads,
coroutines, processes, and so on—convey inaccurate connotations. A
goroutine has a simple model: it is a function executing concurrently
with other goroutines in the same address space. It is lightweight,
costing little more than the allocation of stack space. And the stacks
start small, so they are cheap, and grow by allocating (and freeing)
heap storage as required.
Goroutines are multiplexed onto multiple OS threads so if one should
block, such as while waiting for I/O, others continue to run. Their
design hides many of the complexities of thread creation and
management.
If a goroutine is blocking, the runtime will start a new OS thread to handle the other goroutines until the blocking one stops blocking.
Reference : https://groups.google.com/forum/#!topic/golang-nuts/2IdA34yR8gQ
Ok, so here's what I've learned:
When you're doing raw syscalls, Go indeed creates a thread per blocking goroutine. For example, consider the following code:
package main
import (
"fmt"
"syscall"
)
func block(c chan bool) {
fmt.Println("block() enter")
buf := make([]byte, 1024)
_, _ = syscall.Read(0, buf) // block on doing an unbuffered read on STDIN
fmt.Println("block() exit")
c <- true // main() we're done
}
func main() {
c := make(chan bool)
for i := 0; i < 1000; i++ {
go block(c)
}
for i := 0; i < 1000; i++ {
_ = <-c
}
}
When running it, Ubuntu 12.04 reported 1004 threads for that process.
On the other hand, when utilizing Go's HTTP server and opening 1000 sockets to it, only 4 operating system threads were created:
package main
import (
"fmt"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
So, it's a mix between an IOLoop and a thread per blocking system call.
It can't. There's only 1 goroutine that can be running at a time when GOMAXPROCS=1, whether that one goroutine is doing a system call or something else.
However, most blocking system calls, such as socket I/O, waiting for a timer are not blocked on a system call when performed from Go. They're multiplexed by the Go runtime onto epoll, kqueue or similar facilities the OS provides for multiplexing I/O.
For other kinds of blocking system calls that cannot be multiplexed with something like epoll, Go does spawn a new OS thread, regardless of the GOMAXPROCS setting (albeit that was the state in Go 1.1, I'm not sure if the situation is changed)
You have to differentiate processor number and thread number: you can have more threads than physical processors, so a multi-thread process can still execute on a single core processor.
As the documentation you quoted explain, a goroutine isn't a thread: it's merely a function executed in a thread that is dedicated a chunk of stack space. If your process have more than one thread, this function can be executed by either thread. So a goroutine that is blocking for a reason or another (syscall, I/O, synchronization) can be let in its thread while other routines can be executed by another.
I’ve written an article explaining how they work with examples and diagrams. Please feel free to take a look at it here: https://osmh.dev/posts/goroutines-under-the-hood
From documentation of runtime:
The GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the GOMAXPROCS limit.
so when one OS thread is blocked for syscall, another thread can be started - and GOMAXPROCS = 1 is still satisfied. The two threads are NOT running simultaneously.
Hope this example of sum numbers helps you.
package main
import "fmt"
func put(number chan<- int, count int) {
i := 0
for ; i <= (5 * count); i++ {
number <- i
}
number <- -1
}
func subs(number chan<- int) {
i := 10
for ; i <= 19; i++ {
number <- i
}
}
func main() {
channel1 := make(chan int)
channel2 := make(chan int)
done := 0
sum := 0
//go subs(channel2)
go put(channel1, 1)
go put(channel1, 2)
go put(channel1, 3)
go put(channel1, 4)
go put(channel1, 5)
for done != 5 {
select {
case elem := <-channel1:
if elem < 0 {
done++
} else {
sum += elem
fmt.Println(sum)
}
case sub := <-channel2:
sum -= sub
fmt.Printf("atimta : %d\n", sub)
fmt.Println(sum)
}
}
close(channel1)
close(channel2)
}