I am writing a web application. Incoming requests will pass through middlewares first. At the moment, I've added two middlwares session and security.
After walked through middlwares, it will execute the handler for the request.
Session and security middlwares will execute in their own goroutine.
When I am testing I've got a lot of data race, specially in middlware section
WARNING: DATA RACE
Write by goroutine 18:
runtime.mapassign1()
c:/go/src/runtime/hashmap.go:383 +0x0
net/textproto.MIMEHeader.Set()
c:/go/src/net/textproto/header.go:22 +0xf4
net/http.Header.Set()
c:/go/src/net/http/header.go:31 +0x64
project/middlewares/session.(*ctrl).setHttpHeader()
D:/gocode/src/project/middlewares/session/ctrl.go:76 +0x9d
project/middlewares/session.(*ctrl).evaluateJwt()
D:/gocode/src/project/middlewares/session/ctrl.go:56 +0x31a
project/middlewares/session.(*ctrl).serveHttp()
D:/gocode/src/project/middlewares/session/ctrl.go:94 +0x8a
project/middlewares/session.func┬À006()
D:/gocode/src/project/middlewares/session/serve_http.go:23 +0x
Previous write by goroutine 17:
runtime.mapassign1()
c:/go/src/runtime/hashmap.go:383 +0x0
net/textproto.MIMEHeader.Add()
c:/go/src/net/textproto/header.go:15 +0x212
net/http.Header.Add()
c:/go/src/net/http/header.go:24 +0x64
github.com/unrolled/secure.(*Secure).Process()
D:/gocode/src/github.com/unrolled/secure/secure.go:177 +0xe5b
project/middlewares/security.func┬À001()
D:/gocode/src/project/middlewares/security/serve_http.go:33 +0
Goroutine 18 (running) created at:
project/middlewares/session.ServeHttp()
D:/gocode/src/project/middlewares/session/serve_http.go:29 +0x
project/middlewares.New()
D:/gocode/src/project/middlewares/ctrl.go:12 +0x99
github.com/codegangsta/negroni.HandlerFunc.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:24 +0x5f
github.com/codegangsta/negroni.middleware.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x114
github.com/codegangsta/negroni.middleware.ServeHTTP┬Àfm()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x84
github.com/codegangsta/negroni.(*Static).ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/static.go:30 +0xb71
github.com/codegangsta/negroni.middleware.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x114
github.com/codegangsta/negroni.middleware.ServeHTTP┬Àfm()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x84
github.com/codegangsta/negroni.(*Logger).ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/logger.go:25 +0x249
github.com/codegangsta/negroni.middleware.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x114
github.com/codegangsta/negroni.middleware.ServeHTTP┬Àfm()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x84
github.com/codegangsta/negroni.(*Recovery).ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/recovery.go:45 +0xd9
github.com/codegangsta/negroni.middleware.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x114
github.com/codegangsta/negroni.(*Negroni).ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:73 +0x1c7
net/http/httptest.(*waitGroupHandler).ServeHTTP()
c:/go/src/net/http/httptest/server.go:200 +0xfe
net/http.serverHandler.ServeHTTP()
c:/go/src/net/http/server.go:1703 +0x1fd
net/http.(*conn).serve()
c:/go/src/net/http/server.go:1204 +0x108e
Goroutine 17 (finished) created at:
project/middlewares/security.ServeHttp()
D:/gocode/src/project/middlewares/security/serve_http.go:37 +0
project/middlewares.New()
D:/gocode/src/project/middlewares/ctrl.go:12 +0x64
github.com/codegangsta/negroni.HandlerFunc.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:24 +0x5f
github.com/codegangsta/negroni.middleware.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x114
github.com/codegangsta/negroni.middleware.ServeHTTP┬Àfm()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x84
github.com/codegangsta/negroni.(*Static).ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/static.go:30 +0xb71
github.com/codegangsta/negroni.middleware.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x114
github.com/codegangsta/negroni.middleware.ServeHTTP┬Àfm()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x84
github.com/codegangsta/negroni.(*Logger).ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/logger.go:25 +0x249
github.com/codegangsta/negroni.middleware.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x114
github.com/codegangsta/negroni.middleware.ServeHTTP┬Àfm()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x84
github.com/codegangsta/negroni.(*Recovery).ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/recovery.go:45 +0xd9
github.com/codegangsta/negroni.middleware.ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:33 +0x114
github.com/codegangsta/negroni.(*Negroni).ServeHTTP()
D:/gocode/src/github.com/codegangsta/negroni/negroni.go:73 +0x1c7
net/http/httptest.(*waitGroupHandler).ServeHTTP()
c:/go/src/net/http/httptest/server.go:200 +0xfe
net/http.serverHandler.ServeHTTP()
c:/go/src/net/http/server.go:1703 +0x1fd
net/http.(*conn).serve()
c:/go/src/net/http/server.go:1204 +0x108e
==================
2015/02/03 15:05:31 ctrl.go:17: End of process middlewares
2015/02/03 15:05:31 funcs.go:10: Create new user
2015/02/03 15:05:31 validate.go:14: Validate email thompson#example.com
2015/02/03 15:05:31 validate.go:100: Validate password Test!1234
2015/02/03 15:05:31 validate.go:52: Validate name: ValidName
2015/02/03 15:05:31 create.go:120: Done
[negroni] Completed 0 in 197ms
PASS
ok project/testing/account_test 0.664s
PS D:\gocode\src\project\testing\account_test> go test -race
2015/02/03 15:08:10 vs.go:50: Connect to neo4j db.
[negroni] Started POST /user
==================
WARNING: DATA RACE
Write by goroutine 18:
net/http.(*response).Header()
c:/go/src/net/http/server.go:615 +0x11a
github.com/codegangsta/negroni.(*responseWriter).Header()
<autogenerated>:42 +0x78
project/middlewares/session.(*ctrl).setHttpHeader()
D:/gocode/src/project/middlewares/session/ctrl.go:76 +0x68
project/middlewares/session.(*ctrl).evaluateJwt()
D:/gocode/src/project/middlewares/session/ctrl.go:56 +0x31a
project/middlewares/session.(*ctrl).serveHttp()
D:/gocode/src/project/middlewares/session/ctrl.go:94 +0x8a
project/middlewares/session.func┬À006()
D:/gocode/src/project/middlewares/session/serve_http.go:23 +0x
Previous write by goroutine 17:
net/http.(*response).Header()
c:/go/src/net/http/server.go:615 +0x11a
github.com/codegangsta/negroni.(*responseWriter).Header()
<autogenerated>:42 +0x78
github.com/unrolled/secure.(*Secure).Process()
D:/gocode/src/github.com/unrolled/secure/secure.go:177 +0xe24
project/middlewares/security.func┬À001()
D:/gocode/src/project/middlewares/security/serve_http.go:33 +0
I use negroni to process middlwares works.
The way, how I process middlwares
func New(res http.ResponseWriter, req *http.Request, next http.HandlerFunc) {
if err := process(security.ServeHttp(res, req), session.ServeHttp(res, req)); err != nil {
res.WriteHeader(http.StatusInternalServerError)
return
}
log.Println("End of process middlewares")
next(res, req)
}
// Process all middlewares
func process(chErrs ...<-chan error) error {
for _, chErr := range chErrs {
// Will abort the loop, when error occurs
if err := <-chErr; err != nil {
return err
}
}
return nil
}
As you can see, every middlware have they own channel. The for statement will loop until the error channel is closed or error send.
The server configuration
func Config() *negroni.Negroni {
n := negroni.Classic()
n.Use(negroni.HandlerFunc(middlewares.New))
n.UseHandler(routes.Set())
return n
}
My question is, are the middlewares the reason, why I've got the data race?
It's the fact that you're running the middleware within a goroutine that is the problem.
If you want to run your middleware in parallel you will have to set up a mutex around the memory that is written to by any middleware executed within a goroutine.
In your particular case you are both writing and reading from the ResponseWriter's Header
Related
I am collecting data that I push to Microsoft IoT Hub thru MQTT.
My code runs on an ESP32-based custom PCB.
Everything sounds good (I get the data on the cloud), but after some duration (usually after 5 to 15 minutes), I get the following error:
E (200965) TRANS_SSL: ssl_poll_write select error 113, errno = Software caused connection abort, fd = 57
As per my understanding, this tells me it was not able to get an acknowledgment from the recipient, right?
My logs are as follows:
b6611|—|📦 IoT: 🟢 MQTT: event MQTT_EVENT_PUBLISHED (activity: 24)
b6611|—|📦 IoT: 🟢 MQTT: event MQTT_EVENT_PUBLISHED (activity: 25)
b6611|—|📦 IoT: ==MQTT==: connected
(2)b6611|—|📦 IoT: 🟢 WiFi connected
b6611|—|📦 IoT: 🟢 SAS token not expired
b6611|—|📦 IoT: 🟢 MQTT connected
b6611|—|📦 IoT: 🟢 MQTT: event MQTT_EVENT_PUBLISHED (activity: 26)
E (200965) TRANS_SSL: ssl_poll_write select error 113, errno = Software caused connection abort, fd = 57
b6611|—|📦 IoT: 🔴 MQTT: event MQTT_EVENT_ERROR
E (200970) MQTT_CLIENT: Error write data or timeout, written len = -1, errno=0
E (200977) MQTT_CLIENT: Error to resend data
b6611|—|📦 IoT: 🟡 MQTT: event MQTT_EVENT_DISCONNECTED
b6611|—|📦 IoT: 🔴 ==MQTT==: disconnected!
(0)b6611|—|📦 IoT: 🔴 WiFi no longer connected, establishing connection…
My code's as follows:
int Iot :: initializeMqttClient() {
stateMqttClientOk = false;
if (!isConnectedToIp()) return 2;
log("Init MQTT client");
#if SUPPORT_IOT_DPS
#else
if (sasToken->Generate(SAS_TOKEN_DURATION_IN_MINUTES) != 0) {
error("Failed generating SAS token");
return 1;
}
else {
success("Successfully generated SAS token from %s", sasToken->moduleName());
}
#endif
mqttConnectionState = IOT_MQTT_CONNECTING;
// == If a client is already there, destroy it before creating a new one
if (mqtt_client) {
log("MQTT client already exists, remove it for a brand new one");
(void)esp_mqtt_client_destroy(mqtt_client);
}
// == Create a new mqtt client
esp_mqtt_client_config_t mqtt_config;
memset(&mqtt_config, 0, sizeof(mqtt_config));
mqtt_config.uri = mqtt_broker_uri;
mqtt_config.port = mqtt_port;
mqtt_config.client_id = mqtt_client_id;
mqtt_config.username = mqtt_username;
az_span t = sasToken->get();
#if LOG_LEVEL >= LOG_LEVEL_DEBUG
log("SAS token: sasToken is %p", sasToken);
log("SAS token: sasToken->get() is %p", t);
#endif
mqtt_config.password = (const char*)az_span_ptr(t);
// == Removed 30s period heartbeat down to 120s (default)
//mqtt_config.keepalive = 30;
mqtt_config.disable_clean_session = 0;
mqtt_config.disable_auto_reconnect = false;
mqtt_config.event_handle = mqtt_event_handler;
mqtt_config.user_context = NULL;
//== Removed the certificate as it makes MQTT crash with the following:
// E (21414) TRANS_SSL: ssl_poll_read select error 113, errno = Software caused connection abort, fd = 54
// E (21416) MQTT_CLIENT: Poll read error: 0, aborting connection
//mqtt_config.cert_pem = (const char*)ca_pem;
mqtt_client = esp_mqtt_client_init(&mqtt_config);
if (mqtt_client == NULL) {
error("Failed creating mqtt client");
return 1;
}
else {
success("Successfully create MQTT client, now let's start it");
}
I tried to comment out the two lines without any noticeable improvement, as follows:
// == Removed this by Sdl on Thursday October 13th 2022 3:15pm
//mqtt_config.disable_clean_session = 0;
//mqtt_config.disable_auto_reconnect = false;
There is probably something wrong with my config code. Any idea?
Also, it would probably be interesting to unmount the MQTT and mount it back again in order to continue doing the collection. But how can this be done?
[UPDATE]
The SAS is checked on a regular basis (and will be updated if expired). But in my case it did not expire (You can see the line the SAS did not expire in the provided logs).
What I should mention is that after a lost of connection (typically between 5 to 15 minutes), MQTT some time manages to reconnect and then succeeds to send a few messages before losing the connection again.
I am covering project with tests and for that purpose I need dummy TCP Server, which could accept connection, write/read data to/from it, close it etc... I have found this question on stack overflow, covering mocking connection, but it doesn't cover what I actually need to test.
My idea relies on this article as starting point, but when I started implementing channel to let server write some data to newly opened connection, I got stuck with undebuggable deadlock in writing to channel.
What I want to achieve is to write some data to server's channel, say sendingQueue chan *[]byte, so later corresponding []byte will be sent to newly established connection.
During these little research I have tried debugging and printing out messages before/after sending data to channel and trying to send / read data from channel in different places of program.
What I found out:
My idea works if I add data directly in handleConnection with
go func() {
f := []byte("test.")
t.sendingQueue <- &f
}()
My idea doesn't work if I push data to channel from TestUtils_TestingTCPServer_WritesRequest in any form, either with func (t *TCPServer) Put(data *[]byte) (err error) or directly with:
go func(queue chan *[]byte, data *[]byte) {
queue <- data
}(t.sendingQueue, &payload)
It doesn't matter if channel is buffered or not.
So, obviously, there is something wrong either with the way I debug my code (I didn't dive into cli dlv, using just IDE debugger), or something that I completely miss about working with go channels, goroutines or net.Conn module.
For convenience public gist with full code is available. Note — there is // INIT part in the TestUtils_TestingTCPServer_WritesRequest which is required to run/debug single test. It should be commented out when running go test in the directory.
utils.go:
// NewServer creates a new Server using given protocol
// and addr.
func NewTestingTCPServer(protocol, addr string) (*TCPServer, error) {
switch strings.ToLower(protocol) {
case "tcp":
return &TCPServer{
addr: addr,
sendingQueue: make(chan *[]byte, 10),
}, nil
case "udp":
}
return nil, errors.New("invalid protocol given")
}
// TCPServer holds the structure of our TCP
// implementation.
type TCPServer struct {
addr string
server net.Listener
sendingQueue chan *[]byte
}
func (t *TCPServer) Run() (err error) {}
func (t *TCPServer) Close() (err error) {}
func (t *TCPServer) Put(data *[]byte) (err error) {}
func (t *TCPServer) handleConnection(conn net.Conn){
// <...>
// Putting data here successfully sends it via freshly established
// Connection:
// go func() {
// f := []byte("test.")
// t.sendingQueue <- &f
// }()
for {
fmt.Printf("Started for loop\n")
select {
case data := <-readerChannel:
fmt.Printf("Read written data\n")
writeBuffer.Write(*data)
writeBuffer.Flush()
case data := <-t.sendingQueue:
fmt.Printf("Read pushed data\n")
writeBuffer.Write(*data)
writeBuffer.Flush()
case <-ticker:
fmt.Printf("Tick\n")
return
}
fmt.Printf("Finished for loop\n")
}
}
utils_test.go
func TestUtils_TestingTCPServer_WritesRequest(t *testing.T) {
payload := []byte("hello world\n")
// <...> In gist here is placed INIT piece, which
// is required to debug single test
fmt.Printf("Putting payload into queue\n")
// This doesn't affect channel
err = utilTestingSrv.Put(&payload)
assert.Nil(t, err)
// This doesn't work either
//go func(queue chan *[]byte, data *[]byte) {
// queue <- data
//}(utilTestingSrv.sendingQueue, &payload)
conn, err := net.Dial("tcp", ":41123")
if !assert.Nil(t, err) {
t.Error("could not connect to server: ", err)
}
defer conn.Close()
out := make([]byte, 1024)
if _, err := conn.Read(out); assert.Nil(t, err) {
// Need to remove trailing byte 0xa from bytes array to make sure bytes array are equal.
if out[len(payload)] == 0xa {
out[len(payload)] = 0x0
}
assert.Equal(t, payload, bytes.Trim(out, "\x00"))
} else {
t.Error("could not read from connection")
}
}
After a help from a colleague and reading the article on how init works, I found a problem.
It was in init function, which was recreating extra server, due to using := assignment. I also updated code to make sure server runs before net.Dial and conn.Read.
In golang syscall.GetLastError() doesn't return the last error. See the following example
if handle := _OpenSCManager(machineNamePtr, databaseNamePtr, desiredAccess); handle == nil {
if err := syscall.GetLastError(); err != nil {
return InvalidServiceDatabaseHandleHandle, ServiceErrno(err.(syscall.Errno))
}
}
err is always nil. Assume machineNamePtr is a non exsiting machine. Tested the same code with c++ and GetLastError() throws RPC server is not available. So why not on go?
EDIT
_OpenSCManager is generated with go generate.
//sys _OpenSCManager(machineName *uint16, databaseName *uint16, desiredAcces ServiceAccessRight) (handle ServiceDatabaseHandle) = advapi32.OpenSCManagerW
func _OpenSCManager(machineName *uint16, databaseName *uint16, desiredAcces ServiceAccessRight) (handle ServiceDatabaseHandle) {
r0, _, _ := syscall.Syscall(procOpenSCManagerW.Addr(), 3, uintptr(unsafe.Pointer(machineName)), uintptr(unsafe.Pointer(databaseName)), uintptr(desiredAcces))
handle = ServiceDatabaseHandle(r0)
return
}
So finally i've got this working. First i have tried to return an error variable with the following signature //sys _OpenSCManager(machineName *uint16, databaseName *uint16, desiredAcces ServiceAccessRight) (handle ServiceDatabaseHandle, lasterror error) = advapi32.OpenSCManagerW. But go generate throws always Only last windows error is allowed as second return value.... But if you change it to //sys ... (handle ServiceDatabaseHandle, err error) = advapi32.OpenSCManagerW it successfully generates the code. So you explicit have to write err error. Someone knows why? So now the function looks like
func _OpenSCManager(machineName *uint16, databaseName *uint16, desiredAcces ServiceAccessRight) (handle ServiceDatabaseHandle, err error) {
r0, _, e1 := syscall.Syscall(procOpenSCManagerW.Addr(), 3, uintptr(unsafe.Pointer(machineName)), uintptr(unsafe.Pointer(databaseName)), uintptr(desiredAcces))
handle = ServiceDatabaseHandle(r0)
if handle == 0 {
if e1 != 0 {
err = errnoErr(e1)
} else {
err = syscall.EINVAL
}
}
return
}
and it returns an error. So there is no need to call GetLastError().
IIUC, the syscall.Syscall() on Windows automatically and atomically calls GetLastError() after the actual syscall finishes. That should be understandable once you consider that as soon as a goroutine exits from a syscall, the Go runtime scheduler is free to preempt it and run another goroutine on the very thread the just-preempted goroutine had just been running on.
Since GetLastError() accesses per-thread state, if the second goroutine makes another syscall, it would likely thrash that last error value, so in the context of Go, each syscall should be accompanied with the following call to GetLastError() done in the context of a single syscall invocation — as seen from the Go side.
I have a strange problem.
void MySocket::OnReceive( int nErrorCode )
{
static CMutex mutex;
static int depth=0;
static int counter=0;
CSingleLock lock(&mutex, true);
Receive(pBuff, iBuffSize-1);
counter++;
depth++; //<-- Breakpoint
log("Onreceive: enter %d %d %d", GetCurrentThreadId(), counter, depth);
.....
Code handling data
depth--;
log("Onreceive: exit %d %d %d", GetCurrentThreadId(), counter, depth);
}
Results in this log statement:
02/19/2014 08:33:14:982 [DEBUG] Onreceive Enter: 3200 1 2
02/19/2014 08:34:13:726 [DEBUG] Onreceive Exit : 3200 2 1
02/19/2014 08:32:34:193 [DEBUG] Onreceive Enter: 3200 0 1 <- Log statement was created but interrupted before it was written to disk
02/19/2014 08:34:13:736 [DEBUG] Onreceive Exit : 3200 2 0
Now what happens:
I start the program and the debugger stops at the breakpoint
Step through into the log
Some where in the log the debugger jumps back to the break point
This is the second entry into the OnReceive
Second call completes
First call continues
My questions:
How is it possible to get two concurrent calls to OnReceive
Why does the Mutex not work (Due to the same threadid?)
And how can I have two executing paths with the same ThreadID???
And ofcourse, how can I fix this??
Note that this only happens if I send a lot small messages (<50Bytes) until the Send blocks. In total it's around 500KB/s. If I put a Sleep(1) after each send it doesn't happen.. but that ofcourse kills my transfer speed.
Ok, I found the root cause. In the Log statement a Win32 Mutex is used and the following Wait:
DWORD dwResult = MsgWaitForMultipleObjects(nNoOfHandle, handle, FALSE, dwTimeout, QS_POSTMESSAGE|QS_ALLPOSTMESSAGE|QS_SENDMESSAGE|QS_TIMER);
if (dwResult == WAIT_OBJECT_0 + nNoOfHandle) // new message is in the queue, let's clear
{
MSG Msg;
while (PeekMessage(&Msg, NULL, 0, 0, PM_REMOVE))
{
::TranslateMessage(&Msg);
::DispatchMessage(&Msg);
}
}
This waits for the Mutex to be cleared OR for a message to be posted. CSocket posts messages to the thread when it receives data and that will call the OnReceive. So, this code produced the problem that while waiting for the mutex it would handle incoming messages and effectively call the OnReceive again.
One way of solving this was to prevent the CSocket from posting more notifications like this:
void MySocket::OnReceive(int nErrorCode)
{
/* Remove FD_READ notifications */
VERIFY(AsyncSelect(FD_WRITE | FD_OOB | FD_ACCEPT | FD_CONNECT | FD_CLOSE));
OldOnReceive(nErrorCode);
/* Restore default notifications */
VERIFY(AsyncSelect());
}
The play application which we are using, has to be kept always alive. This application is basically a RabbitMQ listener which has an infinite while loop, which keeps listening to capture the messages from Message Queue.
The following code is placed in controller, and this play application has to be kept alive at all times
public class Application extends Controller {
public static Result wanHLPT() {
ckmsg()
return ok();
}
public static Result ckmsg() {
try{
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
// business logic
process()
}
}
}
}
But the play application throws akka exception [AskTimeOut Exception: Timed Out],
Please provide any pointers on how to handle this.
Here is the exception:
! #6f977jo6m - Internal server error, for (GET) [/myapp] ->
play.api.Application$$anon$1: Execution exception[[AskTimeoutException: Timed out]]
at play.api.Application$class.handleError(Application.scala:289) ~ [play_2.10.jar:2.1.0]
at play.api.DefaultApplication.handleError(Application.scala:383) ~[play_2.10.jar:2.1.0]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anon$2$$anonfun$handle$1.apply(PlayDefaultUpstreamHandler.scala:132) ~[play_2.10.jar:2.1.0]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anon$2$$anonfun$handle$1.apply(PlayDefaultUpstreamHandler.scala:128) ~[play_2.10.jar:2.1.0]
at play.api.libs.concurrent.PlayPromise$$anonfun$extend1$1.apply(Promise.scala:113) ~[play_2.10.jar:2.1.0]
at play.api.libs.concurrent.PlayPromise$$anonfun$extend1$1.apply(Promise.scala:113) ~[play_2.10.jar:2.1.0]
at play.api.libs.concurrent.PlayPromise$$anonfun$extend$1$$anonfun$apply$1.apply(Promise.scala:104) ~[play_2.10.jar:2.1.0]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[scala-library.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[scala-library.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0]
at java.lang.Thread.run(Thread.java:781) ~[na:1.7.0]
akka.pattern.AskTimeoutException: Timed out
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:310) ~[akka-actor_2.10.jar:na]
at akka.actor.DefaultScheduler$$anon$8.run(Scheduler.scala:193) ~[akka-actor_2.10.jar:na]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:137) ~[akka-actor_2.10.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1417) ~[scala-library.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262) ~[scala-library.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) ~[scala-library.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478) ~[scala-library.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) ~[scala-library.jar:na]