I am working for private video network where I have to schedule the
task based on following parameter.There is client Portal, Server and Gateway.
Through portal a user can request Streaming the video.
User can also Schedule Streaming for some future time.Each each task is having a task ID.
Task is scheduled based on following date time parameter.
start time
end time
Repeat (every day,just once, a particular day)
start date
end date
Now at the gateway I need to add logic to Implement schedule task.
I am exploring Waitable Timer Objects and CreateWaitableTimerEe.
I am bit confused whether it is possible to implement the feature using this.
I am using C++, MFC and can't use third party library.
I need Suggestion how to implement this.
There are dozens of ways to design this. It all depends on what you want to do and what the specific requirements are.
In a basic design I'd create an additional field called "next run time" which will be calculated by using start time, frequency and previous (if any) end time. Then I'd dump all the tasks in a queue sorted using this field.
The main scheduling will pick up the first queue item and create a suspended thread for that specific task. Now just calculate the time difference to the first item's 'next run time' and sleep for that time period. When you wake up just resume the thread and pick the next queue item and repeat.
I would just create a timer thread callback loop that checks the time every minute and executes your task on the specified schedule.
Related
There is a task which creates database record {R) when it runs for the first time. When task is started second time it should read database record, perform some calculations and call external API. First and second start happens in a loop
In case of single start of the task there are no problems, but in the case of loops (at each loop's iteration the new task is created and starts at certain time) there is a problem. In the task queue (for it we use a flower) we have crashed task on every second iteration.
If we add, at the and of the loop time.sleep(1) sometimes the tasks work properly, but sometimes - not. How to avoid this problem? We afraid that task for different combination of two users started at the same time also will be crashed.
Is there some problem with running tasks in Celery simultaneously? Or something we should consider, tasks are for scheduled payments so they have to work rock solid
I am looking for a good cloud solution to handle below scenario, where I need to wait for future events within a specific time interval to know whether to process current event. Its kindoff like Debounce (“group” multiple sequential calls within a time period in a single one) but little more complex as the timer needs to be reset when next event is received.
Eg:
I get a request of Event A at X time for a particular User(U1).
a. If I get a similar Event A from same User within 5mins of X time, I need to reset the timer and keep watching again.
b. If 5 mins have passed by, I need to process Event A.
I have nearly 1000 items in my DB. I have to run the same operation on each item. The issue is that this is a third party service that has a 1 second rate limit for each operation. Until now, I was able to do the entire thing inside a lambda function. It is now getting close to the 15 minute (900 second) timeout limit.
I was wondering what the best way for splitting this would be. Can I dump each item (or batches of items) into SQS and have a lambda function process them sequentially? But from what I understood, this isn't the recommended way to do this as I can't delay invocations sufficiently long. Or I would have to call lambda within a lambda, which also sounds weird.
Is AWS Step Functions the way to go here? I haven't used that service yet, so I was wondering if there are other options too. I am also using the serverless framework for doing this if it is of any significance.
Both methods you mentioned are options that would work. Within lambda you could add a delay (sleep) after one item has been processed and then trigger another lambda invocation following the delay. You'll be paying for that dead time, of course, if you use this approach, so step functions may be a more elegant solution. One lambda can certainly invoke another--even invoking itself. If you invoke the next lambda asynchronously, then the initial function will finish while the newly-invoked function starts to run. This article on Asynchronous invocation will be useful for that approach. Essentially, each lambda invocation would be responsible for processing one item, delaying sufficiently to accommodate the service limit, and then invoking the next item.
If anything goes wrong you'd want to build appropriate exception handling so a problem with one item either halts the rest or allows the rest of the chain to continue, depending on what is appropriate for your use case.
Step Functions would also work well to handle this use case. With options like Wait and using a loop you could achieve the same result. For example, your step function flow could invoke one lambda that processes an item and returns the next item, then it could next run a wait step, then process the next item and so on until you reach the end. You could use a Map that runs a lambda task and a wait task:
The Map state ("Type": "Map") can be used to run a set of steps for
each element of an input array. While the Parallel state executes
multiple branches of steps using the same input, a Map state will
execute the same steps for multiple entries of an array in the state
input.
This article on Iterating a Loop Using Lambda is also useful.
If you want the messages to be processed serially and are happy to dump the messages to sqs, set both the concurency of the lambda and the batchsize property of the sqs event that triggers the function to 1
Make it a FIFO queue so that messages dont potentially get processed more than once if that is important.
My goal is to have an workflow which periodically (every 30 seconds) add a same activity (doing nothing but sleep for 1 minute) to the taskList. Also I have multiple machines hosting activity workers to poll the taskList simultaneously. When the activity got scheduled, one of the workers can poll it and execute.
I tried to use a cron decorator to create a DynamicActivityClient and use the DynamicActivityClient.scheduleActivity() to schedule the activity periodically. However, it seems the the activity will not be scheduled until the last activity is finished. In my case, the activity got scheduled every 1 minute rather than 30 seconds which I set in the cron pattern.
The package structure is almost the same as aws sdk sample code: cron
Is there any other structure recommended to achieve this? I am very much new to SWF.Any suggestion is highly appreciated.
You may do so by writing a much simpler workflow code and using workflow clock and timer. Refer to the example in the link below.
http://docs.aws.amazon.com/amazonswf/latest/awsflowguide/executioncontext.html
Also remember one thing. The maximum number of events allowed in a workflow execution is 25000. So the cron job will not run for ever but you will have to write code to start a new workflow execution after some time. Refer to continuous workflow example provided at link below
http://docs.aws.amazon.com/amazonswf/latest/awsflowguide/continuous.html
The cron decorator internally relies on AsyncScheduledExecutor which is by design written to wait for all asynchronous code in the invoked method to complete before calling the cron again. So the behavior you are witnessing is expected. The workaround is to not invoke activity from the code under cron, but from the code in the different scope. Something like:
// This is a field
Settable<Void> invokeNextActivity = new Settable<>();
void executeCron() {
scheduledExecutor.execute(new AsyncRunnable() {
#Override
public void run() throws Throwable {
// Instead of executing activity here just unblock
// its execution in a different scope.
invokeNextActivity.set(null);
}
});
// Recursive loop with each activity invocation
// gated on invokeNextActivity
executeActivityLoop(invokeNextActivity);
}
#Asynchronous
void executeActivityLoop(Promise waitFor) {
activityClient.executeMyActivityOnce();
ivnokeNextActivity = new Settable<>();
executeActivityLoop(ivnokeNextActivity);
}
I recommend reading TryCatchFinally documentation to get understanding of error handling and scopes.
Another option is to rewrite AsyncScheduledExecutor to invoke invoked.set(lastInvocationTime) not from the doFinally but immediately after calling the command.run()
I have number of "sites" (m) that each has to process an event (chunks of data. all available on the get go). Each event (n of them) is sent to each site for processing. So you may think that I have nxm tasks. The order of processing is not important, only that one site may not process more than one event at a time (so Task(m,x) cannot run in parallel to Task(m,y))
Currently it's implemented using "OMP parallel for" on the sites, nested in a regular for loop on the events
for(...event...)
#pragma omp parallel for
for(...site...)
site.process(event)
This is working fine, however not all sites have the same complexity for each event. i.e all sites have to wait for the slowest site before moving on to the next event. I guesstimate that if I allow workers to move on to the next event I can save a factor of two.
What is the best way to implement this? I'm using C++
I'm looking into TBB Flow Graph, or multiple pipe lines...
One more consideration is that each "event" has to be read from disk, and takes up a bit of memory. Although not critical yet, I would like to have as few event in the system at a time (or limit them). In the current implementation I have only one (plus a couple being prepared in the background)
Thanks
I would use a manager process that keeps track of the processed events of each side (m*n bool matrix ) and events that are currently "in use".
Every site starts with a "random" event.
The manager cycles through the sites to check if they finished the current event and assigns a new one if possible.
C++11 provides std::async for such tasks.
Each process call is done async and you can cycle through the corresponding future elements to check if they finished ( wait_for ).