I was thinking of creating a web service that does a long running process. What would be the best way to design these to work with a load balancer? I can't think of any way of doing it besides writing a custom queue.
That is exactly what you should do. You typically want your web service calls to be a quick request/response. So make a call to the web service, have the web service queue the work then have worker processes pick up the messages from the queue and process them.
This is the way to go, queuing the long running processes allows your system to scale, allows you to add recovery logic if a process fails, allows you to scale quickly by adding additional workers to process the queue, and best of all does not tie up the client waiting for a response.
REDIS (http://redis.io/) has been my choice over the past few years, if you are using Azure or AWS they have messaging services as well.
You can also use websockets to notify the client when processes are completed to keep the UI state in the loop.
Related
I have a quite heavy batch process (a python script called "run_simulation.py") on which I have very little control, it can be launched by a single user through a web api but it read and writes from disk so it wouldn't handle parallel requests.
Now, I'd like to have one docker container instanciated per request so that all requests can be handled in parallel, what would be the way to do this ? Is this even doable with Docker ? What would be the module responsible to instanciate the container and pass the http request to it ?
Generally you don’t do this. There are two good reasons for that: if you unconditionally launch a container per request it becomes very easy to swamp your system with these background jobs to the point where none can progress; and the setup that would allow you to launch more Docker containers would also give you unlimited root-level access to the host, which you don’t want in a process that accepts network requests.
A better approach is to set up a job queue system. RabbitMQ is popular and open-source, but by no means the only option. When you receive a request that needs background work, you add a job to the queue and return immediately. Meanwhile, you have some number of worker processes which accept jobs from the queue and do the work.
This gives you a couple of benefits. You control how much work can be done in parallel (by controlling the number of worker containers). If you need to do more work by setting up a second server (or even more), they can all connect back to the same queue server, without requiring a complex multi-host container setup. If your workers crash (or get OOM-killed) their jobs will be returned to the queue and can be picked up and retried by other workers. If you decide Docker doesn’t work for you, or that you need a different orchestrator (Nomad, Kubernetes) you can run this exact same setup without making any code changes, just changing the deployment configuration.
I'm designing a web log analytic.
And I found an architect with Django(Back-end & front-end)+ kafka + spark.
I also found some same system from this link:http://thevivekpandey.github.io/posts/2017-09-19-high-velocity-data-ingestion.html with below architect
But I confuse about the role of kafka-consumer. It will is a service, independent to Django, right?
So If I want to plot real-time data to front-end chart, how to I attached to Django.
It will too ridiculous if I place both kafka-consumer & producer in Django. Request from sdk come to Django by pass to kafa topic (producer) and return Django (consumer) for process. Why we don't go directly. It looks simple and better.
Please help me to understand the role of kafka consumer, where it should belong? and how to connect to my front-end.
Thanks & best Regards,
Jame
The article mentions about the use case without Kafka:
We saw that in times of peak load, data ingestion was not working properly: it was taking too long to connect to MongoDB and requests were timing out. This was leading to data loss.
So the main point of introducing Kafka and Kafka Consumer is to avoid too much load on DB layer and handle it gracefully with a messaging layer in between. To be honest, any message queue can be used in this case, not only Kafka.
Kafka Consumer can be a part of the web layer. It wouldn't be optimal, because you want the separation of concerns (which makes the system more reliable in case of failures) and ability to scale things independently.
It's better to implement the Kafka Consumer as a separate service if the concerns mentioned above really matter (scalability and reliability) and it's easy for you to do operationally (because you need to deploy, monitor, etc. a new service now). In the end it's a classic monolith vs. microservices dilemma.
I have never used celery before and I'm also a django newbie so I'm not sure if I should use celery in my project.
Brief description of my project:
There is an API for sending (via SSH) jobs to scientific computation clusters. The API is an abstraction to the different scientific job queue vendors out there. http://saga-project.github.io/saga-python/
My project is basically about doing a web GUI for this API with django.
So, my concern is that, if I use celery, I would have a queue in the local web server and another one in each of the remote clusters. I'm afraid this might complicate the implementation needlessly.
The API is still in development and some of the features aren't fully finished. There is a function for checking the state of the remote job execution (running, finished, etc.) but the callback support for state changes is not ready. Here is where I think celery might be appropriate. I would have one or several periodic task(s) monitoring the job states.
Any advice on how to proceed please? No celery at all? celery for everything? celery just for the job states?
I use celery for similar purpose and it works well. Basically I have one node running celery workers that manage the entire cluster. These workers generate input data for the cluster nodes, assign tasks, process the results for reporting or generating dependent tasks.
Each cluster node is running a very small python server which takes a db id of it's assigned job. It then calls into the main (http) server to request the data it needs and finally posts the data back when complete. In my case, the individual nodes don't need to message each other and run time of each task is very long (hours). This makes the delays introduced by central management and polling insignificant.
It would be possible to run a celery worker on each node taking tasks directly from the message queue. That approach is appealing. However, I have complex dependencies that are easier to work out from a centralized control. Also, I sometimes need to segment the cluster and centralized control makes this possible to do on the fly.
Celery isn't good at managing priorities or recovering lost tasks (more reasons for central control).
Thanks for calling my attention to SAGA. I'm looking at it now to see if it's useful to me.
Celery is useful for execution of tasks which are too expensive to be executed in the handler of HTTP request (i.e. Django view). Consider making an HTTP request from Django view to some remote web server and think about latencies, possible timeouts, time for data transfer, etc. It also makes sense to queue computation intensive tasks taking much time for background execution with Celery.
We can only guess what web GUI for API should do. However Celery fits very well for queuing requests to scientific computation clusters. It also allows to track the state of background task and their results.
I do not understand your concern about having many queues on different servers. You can have Django, Celery broker (implementing queues for tasks) and worker processes (consuming queues and executing Celery tasks) all on the same server.
I am wanting to write some web services using WCF.
I would like to have a "thread pool" in my web service.
For example, I have nearly 6gb of data I need to manipulate.
I would like the client to call an operation on the webservice and have a new task or thread created. The client is able to call a ListRunningTasks(); and have the webservice return a list of tasks. The client should be able to forcefully kill a task if it is taking too long e.g. KillTask(int taskID); or something. I have previously done some threading, but not inside WCF or a service that doesn't have state. Is this possible? If so, how would one go about implementing such a thing? Any reading, links or suggestions would be great.
Thanks, Mike.
One possible solution:
Implement explicit queues for your outstanding tasks taking into consideration that they take that long (20-30mins as you wrote).
Build a custom component to manage those queues e.g. you might even want capabilities to persist them, resume work when you restart the service etc.
Have explicitly created worker threads that pickup work from those queues.
Implement a WCF service to make your queue manager available to external systems.
Thread pools are more designed to process a high volume of short-running tasks.
You should consider using Windows Workflow Foundation to create such services. A state machine workflow can be exposed as a service in such a way that when method A is called, it will start the workflow (task), after which methods can be called to stop, suspend, or query the running task. WF will handle the state transitions, preventing illegal state changes, and making sure that new tasks are only spun up as necessary.
Note that WF will handle the threading issues for you in an almost transparent manner.
What would be a more standard way of notifying a web service consumer of a data change?
Having the consumer periodically calling the web service to pull change notification.
Consumer setting up a call back web service that can be invoked to forward notification about the change.
Other?
Both of these are options. There is also something called "comet" which is like setting up a stream between between the consumer and producer - messages can then be passed back and forth between the two. Wikipedia is probably the best place to start investigating to see if it will work for you project: http://en.wikipedia.org/wiki/Comet_(programming)
Depends on the scenario. If you're working in a closed environment with only a few consumers of your service, you could switch to a COMET style service which allows a callback from the service to the client. More about that here:
Wikipedia - COMET
From what I've read, that method doesn't scale well in larger environments so I'd be careful.
The more traditional method is your first option of polling the service for changes. As long as your service performs well and you have the appropriate hardware to serve up the requests, it's probably your best bet for a public facing web service.
In case you weren't aware of it, and in case it helps: WCF can work with a Duplex contract that in effect creates a callback service contract on the client. It's fairly transparent.