We use TestKit for testing our application and there are seems random timeout failures during running tests
timeout (10016 milliseconds) during fishForMessage
testKit.fishForMessage(duration, "", events);
I tried to increase duration, but it did not help.
Any ideas of fixing such problem ?
Related
I am getting very big response(50k lines). I am getting 200 status and my test code written in Test tab starts getting executed before whole response loaded in postman.
In other words I and to add delay between success response and test starts.
You can wait until get response by setting the request timeout is zero. It means infinity waiting.
unit is milli seconds.
You can find it from gear the menu of Postman
You can make timeout error with this URL by 10 msec
https://raw.githubusercontent.com/json-iterator/test-data/master/large-file.json
In case of 10 msec,
In case of 0 msec,
I have a failing test because of the timeout. Here's what I see in log output:
2018-05-15 10:47:56.152 WARN com.datastax.driver.core.NettyUtil [UserDataServiceSpec-cassandra-plugin-default-dispatcher-27] [] [] - Found Netty's native epoll transport, but not running on linux-based operating system. Using NIO instead.
2018-05-15 10:48:38.616 ERROR n.f.c.indexing.UniquelyIndexingActor [UserDataServiceSpec-akka.actor.default-dispatcher-39] [UserDataServiceSpec-akka.actor.default-dispatcher-29] [akka.tcp://UserDataServiceSpec#127.0.0.1:51627/user/$c/user-email-indexer] - Persistence failure when replaying events for persistenceId [user-email-indexer]. Last known sequence number [0]
akka.persistence.RecoveryTimedOut: Recovery timed out, didn't get snapshot within 30000 milliseconds
2018-05-15 10:48:38.617 ERROR a.c.sharding.PersistentShardCoordinator [UserDataServiceSpec-akka.actor.default-dispatcher-39] [UserDataServiceSpec-akka.actor.default-dispatcher-30] [akka.tcp://UserDataServiceSpec#127.0.0.1:51627/system/sharding/userdataCoordinator/singleton/coordinator] - Persistence failure when replaying events for persistenceId [/sharding/userdataCoordinator]. Last known sequence number [0]
akka.persistence.RecoveryTimedOut: Recovery timed out, didn't get snapshot within 30000 milliseconds
2018-05-15 10:48:38.618 INFO akka.actor.LocalActorRef [UserDataServiceSpec-akka.actor.default-dispatcher-39] [UserDataServiceSpec-akka.actor.default-dispatcher-35] [akka://UserDataServiceSpec/user/$c/user-email-indexer] - Message [akka.persistence.SnapshotProtocol$LoadSnapshotFailed] from Actor[akka://UserDataServiceSpec/system/cassandra-snapshot-store#-750137778] to Actor[akka://UserDataServiceSpec/user/$c/user-email-indexer#1201357728] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2018-05-15 10:48:38.619 INFO akka.actor.LocalActorRef [UserDataServiceSpec-akka.actor.default-dispatcher-37] [UserDataServiceSpec-akka.actor.default-dispatcher-39] [akka://UserDataServiceSpec/system/sharding/userdataCoordinator/singleton/coordinator] - Message [akka.cluster.sharding.ShardCoordinator$RebalanceTick$] from Actor[akka://UserDataServiceSpec/system/sharding/userdataCoordinator/singleton/coordinator#-75387958] to Actor[akka://UserDataServiceSpec/system/sharding/userdataCoordinator/singleton/coordinator#-75387958] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
(Don't ask my why we are not using in memory storage to test persistent actors. It's not relevant to the problem right now.)
I am not experienced in Akka and JVM, but the messages I see are just the equivalence of "You screwed up, man". There is no hint in them how to fix the problem or why this RecoveryTimedOut occurs.
If someone could give me a valuable advice how to diagnose the problem, it would be nice.
UniquelyIndexingActor is created as cluster singleton.
Try adding this to your config:
akka {
persistence {
journal-plugin-fallback {
recovery-event-timeout = 60s
}
}
}
It solved the problem for me. I found a reference to it in https://github.com/akka/akka/blob/master/akka-persistence/src/main/resources/reference.conf
How would you replay a web service request every ten seconds for ten times until it answers?
I've tried RecoverWithRetries and InitialDelay, but the first recovery immediately replays the web service call:
FromThirdOfContract().RecoverWithRetries(e =>
{
return Source.FromTask(_third.GetThird(message.ContractIdLegacy)).InitialDelay(TimeSpan.FromSeconds(secondsbetween));
}, retry);
The first retry happens immediately instead of ten seconds later. In Akka, there's a RestartSource class; we don't have it in Akka.NET. Any ideas?
I finally played with Source.Lazily() with my source. It's working, it's not evaluated before the initial delay call. But I'm listening for any other ideas
Our system processes messages delivered from a messaging system. If no message is received after 10 seconds, an error should be raised (inactivity timeout).
I was thinking of using a ScheduledExecutorService (with 1 Thread). Each time a message is received, I cancel the previous timeout task and submit a new one:
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
Callable timeoutTask = new Callable() {...};
...
synchronized {
timeout.cancel();
timeout = executor.schedule( timeoutTask, 10, TimeUnit.SECONDS);
}
In normal case, we process ~ 1000 / sec. Would this approach scale?
If you share the threadpool and keep the running time of timeoutTask low, it'd be most probably fine. If you'd have one threadpool per ~ 1000 / sec tasks then that would not work.
If you are still worried, you can have a look at HashedWheelTimer from the Netty project (link). This thing is very efficient in scheduling timeouts. Note that you MUST share instance of HashedWheelTimer as well.
I wrote two equal projects in Golang+Martini and Play Framework 2.2.x to compare it's performance. Both have 1 action that render 10K HTML View. Tested it with ab -n 10000 -c 1000 and monitored results via ab output and htop. Both uses production confs and compiled views. I wonder about results:
Play: ~17000 req/sec + constant 100% usage of all cores of my i7 = ~0.059 msec/req
Martini: ~4000 req/sec + constant 70% usage of all cores of my i7 = ~0.25 msec/req
...as I understand martini is not bloated, so why it 4.5 times slower? Any way to speedup?
Update: Added benchmark results
Golang + Martini:
./wrk -c1000 -t10 -d10 http://localhost:9875/
Running 10s test # http://localhost:9875/
10 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 241.70ms 164.61ms 1.16s 71.06%
Req/Sec 393.42 75.79 716.00 83.26%
38554 requests in 10.00s, 91.33MB read
Socket errors: connect 0, read 0, write 0, timeout 108
Requests/sec: 3854.79
Transfer/sec: 9.13MB
Play!Framework 2:
./wrk -c1000 -t10 -d10 http://localhost:9000/
Running 10s test # http://localhost:9000/
10 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 32.99ms 37.75ms 965.76ms 85.95%
Req/Sec 2.91k 657.65 7.61k 76.64%
276501 requests in 10.00s, 1.39GB read
Socket errors: connect 0, read 0, write 0, timeout 230
Requests/sec: 27645.91
Transfer/sec: 142.14MB
Martini running with runtime.GOMAXPROCS(runtime.NumCPU())
I want to use golang in production, but after this benchmark I don't know how can I make such decision...
Any way to speedup?
I made a simple martini app with rendering 1 html file and it's quite fast:
✗ wrk -t10 -c1000 -d10s http://localhost:3000/
Running 10s test # http://localhost:3000/
10 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 31.94ms 38.80ms 109.56ms 83.25%
Req/Sec 3.97k 3.63k 11.03k 49.28%
235155 requests in 10.01s, 31.17MB read
Socket errors: connect 774, read 0, write 0, timeout 3496
Requests/sec: 23497.82
Transfer/sec: 3.11MB
Macbook pro i7.
This is app https://gist.github.com/zishe/9947025
It helps if you show your code, perhaps you didn't disable logs or missed something.
But timeout 3496 seems bad.
#Kr0e, right! I figured out that heavy use of reflection in DI of martini makes it perform slowly.
I moved to gorilla mux and wrote some martini-style helpers and got a wanted performance.
#Cory LaNou: I can't accept yours comment) Now I agree with you, no framework in prod is good idea. Thanks
#user3353963: See my question: Both uses production confs and compiled views
add this code:
func init(){
martini.Env = martini.Prod
}
sorry, your code have done.