I'm trying to test some actors using scala specs. I run the test in IDEA or Maven (as junit) and it does not exit. Looking at the code, my test finished, but some internal threads (scheduler) are hanging around. How can I make the test finish?
Currently this is only possible by causing the actor framework's scheduler to forcibly shut down:
scala.actors.Scheduler.impl.shutdown
However, the underlying implementation of the scheduler has been changing in patch-releases lately, so this may be different, or not quite work with the version you are on. In 2.7.7 the default scheduler appears to be an instance of scala.actors.FJTaskScheduler2 for which this approach should work, however if you end up with a SingleThreadedScheduler it will not, as the shutdown method is a no-op
This will only work if your actors are not waiting on a react at that time
Related
I have a class that uses asio::generic::stream_protocol::socket to connect to domain (asio::local::stream_protocol::endpoint) and TCP sockets (asio::ip::tcp::endpoint).
To test that class I have a series of unit tests in a single file using the Catch framework.
I've suddenly come across a problem: when running tests they will get stuck. Passing -DASIO_ENABLE_HANDLER_TRACKING to the compiler flags I can see that it gets stuck on async_connect. This does not happen if I comment all tests but one, no matter which. If I have two tests, no matter if they connect to domain or tcp sockets, or one of each, I get a blockage.
The output of Asio changes but this is an example:
$ tests/unit_tests
#asio|1478248907.301230|0*1|deadline_timer#0x7f96f1c07ad8.async_wait
#asio|1478248907.301276|0*2|resolver#0x7f96f1c07ac0.async_resolve
#asio|1478248907.301322|>1|ec=system:0
#asio|1478248907.301328|<1|
#asio|1478248907.302052|>2|ec=system:0,...
#asio|1478248907.302186|2*3|socket#0x7f96f1c07a20.async_connect
#asio|1478248907.302302|<2|
#asio|1478248907.302468|>3|ec=system:0
#asio|1478248907.302481|<3|
#asio|1478248907.302551|0*4|socket#0x7f96f1c07a20.async_send
#asio|1478248907.302611|>4|ec=system:0,bytes_transferred=23
#asio|1478248907.302617|<4|
#asio|1478248907.302621|0*5|socket#0x7f96f1c07a20.async_receive(null_buffers)
#asio|1478248907.356478|>5|ec=system:0,bytes_transferred=0
#asio|1478248907.356547|<5|
#asio|1478248907.356622|0|socket#0x7f96f1c07a20.close
#asio|1478248907.372967|0|deadline_timer#0x7f96f1c07ad8.cancel
#asio|1478248907.372981|0|resolver#0x7f96f1c07ac0.cancel
#asio|1478248907.373509|0*6|deadline_timer#0x7f96f1d00468.async_wait
#asio|1478248907.373526|0*7|resolver#0x7f96f1d00450.async_resolve
#asio|1478248907.374910|>7|ec=system:0,...
#asio|1478248907.374946|7*8|socket#0x7f96f1d003b0.async_connect
#asio|1478248907.375014|<7|
#asio|1478248907.375127|>8|ec=system:0
#asio|1478248907.375135|<8|
My question is: what is the problem with running unit tests that open and close connections? If this is a no-no, how do you write unit tests that use async_open?
io_service has run, run_one, poll and poll_one methods which actually execute the completion handlers. Boost asio may have its own threads, but the thread state of those may not be correct to call your handlers. Hence, even in a unit test you must figure out which thread is going to call completion handlers.
Secondly, run runs to completion, and then returns. From your description (first test succeeds, second fails) it sounds like you did call run but did not reset and re-runthe io_service.
The problem seemed to be related to the way I was iterating through the output of a tcp::resolver.
The following is a programming problem I am trying to solve...
I am creating a (C++ 11) server application that manages operations being performed on other servers. Thus far, I have designed it to do the following:
start a thread for each remote server (with operations on it that are being managed), that polls the server to make sure it's still there/working.
start a thread for each operation that it is managing, that polls the remote server to get info about the current state of the operation.
Where I am running into issues is... The calls to the remote server to verify that it is there/working, and the calls to get the status of the operations being managed on the remote server are "waited" operations. If something stops working (either a remote server stops responding, or I am not able to get the state of an operation running on a remote server, I WAS attempting to just "kill" the thread that monitors that server or operation, and flag it (the server or operation) as dead/non-responsive. Unfortunately, there does not seem to be any clean way to kill a thread.
Note that I was trying to "kill" threads because many of the calls that I am performing are "waited" operations, and I can not have the application hang because it is waiting for completion of an operation (that in some cases may never complete). I have done some research on ways to "kill" an active thread (stop it from a manager thread). One of the only C++ thread management libraries that I could find that even supported a C++ method to "kill" a thread is the "ZThread" library, but using it's kill method doesn't seem to work consistently.
I need to find a way to start executing a segment of code (ie: a thread), that will be performing some waited operation, and kill execution of that code when needed (not wait for the operation to complete.
Suggestions???
I have read the Akka docs on fault tolerance & supervision, and I think I totally get them, with one big exception (no pun intended).
Why would you ever want/need to stop a child actor???
The only clue in the docs is:
Closer to the Erlang way is the strategy to just stop children when they fail and then take corrective action in the supervisor...
But to me, stopping a child is the same as saying "don't execute this code any longer", which to me, is effectively the same as deploying new changes to the code which has that actor removed entirely:
Every Actor plays some critical role in the actor system
To simply stop the actor means that actor currently doesn't have a role any longer, and presumes the system can now somehow (magically) work without it
So again, to me, this is no different than refactoring the code to not even have the actor any more, and then deploying those changes
I'm sure I'm just not seeing the forest through the trees on this one, but I just don't see any use cases where I'd have this big complex actor system, where each actor does critical work and then hands it off to the next critical actor, but then I stop an actor, and magically the whole system keeps on working perfectly.
In short: stopping an actor (to me) is like ripping the transmission out of a moving vehicle. How can this ever be a good/desirable thing?!?
The essence of the "error kernel" pattern is to delegate risky operations and protect essential state, it is common to spawn child-actors for one-off operations, and when that operation is completed and its result send off somewhere else, the child-actor or the parent-actor needs to stop it. (otherwise the child-actor will remain active/leak)
If the child actor is doing a longer process that could be terminated safely, such as video coding, or some kind of file transformation and you have to deploy a new build, in that case a terminate sign would be useful to stop running processes gracefully.
Every Actor plays some critical role in the actor system
This is where you are running into trouble, I can create a child actor to do a job, for example execute a query against a database or maintain the state of a connected user and this is its only purpose.
Once the database query is complete or the user has gracefully disconnected the child actor no longer has any role to play and should be stopped so that it will release any resources it holds.
To simply stop the actor means that actor currently doesn't have a role any >longer, and presumes the system can now somehow (magically) work without it
The system is able to continue because I can create new child actors if/when they are needed.
I have a simple c++ application that generates reports on the back end of my web app (simple LAMP setup). The problem is the back end loads a data file that takes about 1.5GB in memory. This won't scale very well if multiple users are running it simultaneously, so my thought is to split into several programs :
Program A is the main executable that is always running on the server, and always has the data loaded, and can actually run reports.
Program B is spawned from php, and makes a simple request to program A to get the info it needs, and returns the data.
So my questions are these:
What is a good mechanism for B to ask A to do something?
How should it work when A has nothing to do? I don't really want to be polling for tasks or otherwise spinning my tires.
Use a named mutex/event, basically what this does is allows one thread (process A in your case) to sit there hanging out waiting. Then process B comes along, needing something done, and signals the mutex/event this wakes up process A, and you proceed.
If you are on Microsoft :
Mutex, Event
Ipc on linux works differently, but has the same capability:
Linux Stuff
Or alternatively, for the c++ portion you can use one of the boost IPC libraries, which are multi-platform. I'm not sure what PHP has available, but it will no doubt have something equivalent.
Use TCP sockets running on localhost.
Make the C++ application a daemon.
The PHP front-end creates a persistent connection to the daemon. pfsockopen
When a request is made, the PHP sends a request to the daemon which then processes and sends it all back. PHP Sockets C++ Sockets
EDIT
Added some links for reference. I might have some really bad C code that uses sockets of interprocess communication somewhere, but nothing handy.
IPC is easy on C++, just call the POSIX C API.
But what you're asking would be much better served by a queue manager. Make the background daemon wait for a message on the queue, and the frontend PHP just add there the specifications of the task it wants processed. Some queue managers allow the result of the task to be added to the same object, or you can define a new queue for the finish messages.
One of the best known high-performance queue manager is RabbitMQ. Another one very easy to use is MemcacheQ.
Or, you could just add a table to MySQL for tasks, the background process just queries periodically for unfinished ones. This works and can be very reliable (sometimes called Ghetto queues), but break down at high tasks/second.
I'm building my first web application after many years of desktop application development (I'm using Django/Python but maybe this is a completely generic question, I'm not sure). So please beware - this may be an ultra-newbie question...
One of my user processes involves heavy processing in the server (i.e. user inputs something, server needs ~10 minutes to process it). On a desktop application, what I would do it throw the user input into a queue protected by a mutex, and have a dedicated background thread running in low priority blocking on the queue using that mutex.
However in the web application everything seems to be oriented towards synchronization with the HTTP requests.
Assuming I will use the database as my queue, what is best practice architecture for running a background process?
There are two schools of thought on this (at least).
Throw the work on a queue and have something else outside your web-stack handle it.
Throw the work on a queue and have something else in your web-stack handle it.
In either case, you create work units in a queue somewhere (e.g. a database table) and let some process take care of them.
I typically work with number 1 where I have a dedicated windows service that takes care of these things. You could also do this with SQL jobs or something similar.
The advantage to item 2 is that you can more easily keep all your code in one place--in the web tier. You'd still need something that triggers the execution (e.g. loading the web page that processes work units with a sufficiently high timeout), but that could be easily accomplished with various mechanisms.
Since:
1) This is a common problem,
2) You're new to your platform
-- I suggest that you look in the contributed libraries for your platform to find a solution to handle the task. In addition to queuing and processing the jobs, you'll also want to consider:
1) status communications between the worker and the web-stack. This will enable web pages that show the percentage complete number for the job, assure the human that the job is progressing, etc.
2) How to ensure that the worker process does not die.
3) If a job has an error, will the worker process automatically retry it periodically?
Will you or an operations person be notified if a job fails?
4) As the number of jobs increase, can additional workers be added to gain parallelism?
Or, even better, can workers be added on other servers?
If you can't find a good solution in Django/Python, you can also consider porting a solution from another platform to yours. I use delayed_job for Ruby on Rails. The worker process is managed by runit.
Regards,
Larry
Speaking generally, I'd look at running background processes on a different server, especially if your web server has any kind of load.
Running long processes in Django: http://iraniweb.com/blog/?p=56