I know in Jenkins you can avoid duplicate builds from running on top of each other by leaving the "Execute concurrent builds if necessary" checkbox unchecked.
However, the way it determines a concurrent build is just whether the job itself is running. I want it to determine concurrent build based on whether all the parameters that are passed into the job are the same.
So if I have a job in the queue with parameters foo and bar, then queuing one with foo and baz should actually queue, but queuing another with parameters foo and bar should just drop it.
Is this possible?
I believe that Jenkins will automatically kill off a queued job if the PREVIOUSLY queued job has the exact same parameters. If a job is queued with parameters foo and bar, then the same job is queued with foo and bar again, the second instance is dropped.
In your example (first job with foo and bar, second job with foo and baz, third job with foo and bar), I believe that all jobs will be queued.
Related
My goal is to have an workflow which periodically (every 30 seconds) add a same activity (doing nothing but sleep for 1 minute) to the taskList. Also I have multiple machines hosting activity workers to poll the taskList simultaneously. When the activity got scheduled, one of the workers can poll it and execute.
I tried to use a cron decorator to create a DynamicActivityClient and use the DynamicActivityClient.scheduleActivity() to schedule the activity periodically. However, it seems the the activity will not be scheduled until the last activity is finished. In my case, the activity got scheduled every 1 minute rather than 30 seconds which I set in the cron pattern.
The package structure is almost the same as aws sdk sample code: cron
Is there any other structure recommended to achieve this? I am very much new to SWF.Any suggestion is highly appreciated.
You may do so by writing a much simpler workflow code and using workflow clock and timer. Refer to the example in the link below.
http://docs.aws.amazon.com/amazonswf/latest/awsflowguide/executioncontext.html
Also remember one thing. The maximum number of events allowed in a workflow execution is 25000. So the cron job will not run for ever but you will have to write code to start a new workflow execution after some time. Refer to continuous workflow example provided at link below
http://docs.aws.amazon.com/amazonswf/latest/awsflowguide/continuous.html
The cron decorator internally relies on AsyncScheduledExecutor which is by design written to wait for all asynchronous code in the invoked method to complete before calling the cron again. So the behavior you are witnessing is expected. The workaround is to not invoke activity from the code under cron, but from the code in the different scope. Something like:
// This is a field
Settable<Void> invokeNextActivity = new Settable<>();
void executeCron() {
scheduledExecutor.execute(new AsyncRunnable() {
#Override
public void run() throws Throwable {
// Instead of executing activity here just unblock
// its execution in a different scope.
invokeNextActivity.set(null);
}
});
// Recursive loop with each activity invocation
// gated on invokeNextActivity
executeActivityLoop(invokeNextActivity);
}
#Asynchronous
void executeActivityLoop(Promise waitFor) {
activityClient.executeMyActivityOnce();
ivnokeNextActivity = new Settable<>();
executeActivityLoop(ivnokeNextActivity);
}
I recommend reading TryCatchFinally documentation to get understanding of error handling and scopes.
Another option is to rewrite AsyncScheduledExecutor to invoke invoked.set(lastInvocationTime) not from the doFinally but immediately after calling the command.run()
I've read about graceful shutdowns here using the WEBJOBS_SHUTDOWN_FILE and here using Cancellation Tokens, so I understand the premise of graceful shutdowns, however I'm not sure how they will affect WebJobs that are in the middle of processing a queue message.
So here's the scenario:
I have a WebJob with functions listening to queues.
Message is added to Queue and job begins processing.
While processing, someone pushes to develop, triggering a redeploy.
Assuming I have my WebJobs hooked up to deploy on git pushes, this deploy will also trigger the WebJobs to be updated, which (as far as I understand) will kick off some sort of shutdown workflow in the jobs. So I have a few questions stemming from that.
Will jobs in the middle of processing a queue message finish processing the message before the job quits? Or is any shutdown notification essentially treated as "this bitch is about to shutdown. If you don't have anything to handle it, you're SOL."
If we are SOL, is our best option for handling shutdowns essentially to wrap anything you're doing in the equivalent of DB transactions and implement your shutdown handler in such a way that all changes are rolled back on shutdown?
If a queue message is in the middle of being processed and the WebJob shuts down, will that message be requeued? If not, does that mean that my shutdown handler needs to handle requeuing that message?
Is it possible for functions listening to queues to grab any more queue messages after the Job has been notified that it needs to shutdown?
Any guidance here is greatly appreciated! Also, if anyone has any other useful links on how to handle job shutdowns besides the ones I mentioned, it would be great if you could share those.
After no small amount of testing, I think I've found the answers to my questions and I hope someone else can gain some insight from my experience.
NOTE: All of these scenarios were tested using .NET Console Apps and Azure queues, so I'm not sure how blobs or table storage, or different types of Job file types, would handle these different scenarios.
After a Job has been marked to exit, the triggered functions that are running will have the configured amount of time (grace period) (5 seconds by default, but I think that is configurable by using a settings.job file) to finish before they are exited. If they do not finish in the grace period, the function quits. Main() (or whichever file you declared host.RunAndBlock() in), however, will finish running any code after host.RunAndBlock() for up to the amount of time remaining in the grace period (I'm not sure how that would work if you used an infinite loop instead of RunAndBlock). As far as handling the quit in your functions, you can essentially "listen" to the CancellationToken that you can pass in to your triggered functions for IsCancellationRequired and then handle it accordingly. Also, you are not SOL if you don't handle the quits yourself. Huzzah! See point #3.
While you are not SOL if you don't handle the quit (see point #3), I do think it is a good idea to wrap all of your jobs in transactions that you won't commit until you're absolutely sure the job has ran its course. This way if your function exits mid-process, you'll be less likely to have to worry about corrupted data. I can think of a couple scenarios where you might want to commit transactions as they pass (batch jobs, for instance), however you would need to structure your data or logic so that previously processed entities aren't reprocessed after the job restarts.
You are not in trouble if you don't handle job quits yourself. My understanding of what's going on under the covers is virtually non-existent, however I am quite sure of the results. If a function is in the middle of processing a queue message and is forced to quit before it can finish, HAVE NO FEAR! When the job grabs the message to process, it will essentially hide it on the queue for a certain amount of time. If your function quits while processing the message, that message will "become visible" again after x amount of time, and it will be re-grabbed and ran against the potentially updated code that was just deployed.
So I have about 90% confidence in my findings for #4. And I say that because to attempt to test it involved quick-switching between windows while not actually being totally sure what was going on with certain pieces. But here's what I found: on the off chance that a queue has a new message added to it in the grace period b4 a job quits, I THINK one of two things can happen: If the function doesn't poll that queue before the job quits, then the message will stay on the queue and it will be grabbed when the job restarts. However if the function DOES grab the message, it will be treated the same as any other message that was interrupted: it will "become visible" on the queue again and be reran upon the restart of the job.
That pretty much sums it up. I hope other people will find this useful. Let me know if you want any of this expounded on and I'll be happy to try. Or if I'm full of it and you have lots of corrections, those are probably more welcome!
I'm writing a simple networking framework for Clojure using Java's new I/O package. It manages a pool of "selector agents", each of which holds a Selector.
I defined a dispatch action to for the selector agent. This action blocks on a call to selector.select(). When that returns, the selector agent iterates over the selected keys and performs I/O. When I/O is completed, the selector agent send's itself the dispatch action using send-off, effectively looping on calls to selector.select().
When I want to add a new channel or change a channel's interest ops, I send the selector agent the appropriate action and then unblock the selector (it's blocked on select(), remember?). This ensures that (send-off selector-agent dispatch) in the selector agent is executed after (send selector-agent add-channel channel).
I thought this would be bullet-proof since the call to send-off is performed before the selector waking up, and thus, before the selector agent send itself the dispatch action. However, this yields inconsistent behavior. Sometimes, the dispatch action occurs first and sometimes it doesn't.
My understanding is that it's not guaranteed that agents execute actions in the exact order they were sent when they come from multiple threads (i.e. send and send-off are not synchronous as far as queuing actions is concerned).
Is this correct?
send and send-off guarantee that actions will be placed on the Agent's queue in the order they are sent, within a single thread. Updating the Agent's queue happens synchronously.
I expect you have a simple race condition, although I can't identify it from the description.
You are absolutely right. Actions coming from the same thread will be executed in the same order, as they were submitted. But you cannot make any assumptions about execution order of actions, that come from different threads.
send and send off are built for asynchronous state changes.
if you need synchronous updates then atoms are likely your best tool.
since you need to preserve the order of requests you may have to use another data structure within a concurrency object (atom) can be syncrounoustly updated. It may work to put a persistent-queue inside an atom and have all your threads synchronousness add to that queue while your consumers synchronously pull entries from it.
here is the super brief decission chart:
more than one and synchronous: use a ref
asynchronous one: use an agent
asynchronous more than one: agents within a dosync
synchronous and only one: use an agent.
In the context of Windows Job Objects, how can I get the job object for the current process (in case it is in a job object)? The IsProcessInJob function lets me test whether a given process (e.g. the current one) is in a given (or any) job - but it doesn't yield the handle of the matching job.
If you just want to find out what quotas/limits you are running under, or enumerate all the other processes in the job, you don't need to get the Job object for the current process.
You can call QueryInformationJobObject with NULL, which will be the Job object of the current process.
QueryInformationJobObject: http://msdn.microsoft.com/en-us/library/ms684925(VS.85).aspx
Job Objects: http://msdn.microsoft.com/en-us/library/ms684161(VS.85).aspx
Set job attributes using SetInformationJobObject
To answer the specific question, call IsProcessInJob find out if you are in a job.
You can find out everything about the Job by passing NULL to QueryInformationJobObject
Your child processes will inherit your job automatically, unless you pass CREATE_BREAKAWAY_FROM_JOB and the job has JOB_OBJECT_LIMIT_BREAKAWAY_OK or JOB_OBJECT_LIMIT_SILENT_BREAKAWAY_OK is set. In these cases you can assign the process to a new job if you wish.
So without knowing the handle, you can find out all about your current Job, and assign child processes within the current job, or if you have permission, without the current job. I.e. you can do almost everything the handle would allow you to do.
The only exception is duplicate it to another sibling process. If you need to do that you will have to have the parent process communicate the handle value to you somehow.
We have a windows service in C++/ MFC which has to carry out a number of tasks on the host workstation some of which may be long running and may fail a few times before they are completed. Each task will only need to be completed once and sequentially.
I was of thinking of some form of callback initially to retry the failed task but each function has totally different parameters and the code has already been written and tested and just needs a re-queuing method.
I thought we could write the failed task to the registry, sleep() for a while and then restart the service. Is there a better approach?
TIA..
I'm doing quite the same thing in my professional project. My server component is getting runnable objects from different sources and execute them sequentially in a separated thread. All my runnable objects are using different parameters but they all have one function run(void* pUserParam). the void* parameters is a special object that contains a collection of field with different type (double, string, etc...).
My component is queuing the runnable object and launch a new one each time the thread is freed. Of course my component is sleeping when queue is empty and wake up when an object arrives. In your case when a task fail you just need to re-queue it and it will automatically retry the task later.
To achieve these you need:
a Pool mechanism that manage a queue
of tasks,
a task object that contains all information about the runnable object to launch and the parameters,
a runnable object that contains your action to execute.
How it works:
Your service is listening for demands,
When a demand arrives, it give it to the Pool mechanism,
The Pool take the runnable object and its parameter(s) and create a task. This task is queued,
(2b. If the queue was empty, the pool wakes up the execution thread,)
The Thread pick up one task from the queue and execute it calling the Run() function of the runnable object and passing to it the parameters previously stored in the task,
(3b. If the task failed, the thread re-queue a task with the runnable object and its parameter(s),)
The thread picks up a new task or sleeps if queue is empty.
This is my approach and I know this works fine. I know with this method you need to rewrite a part of your application but then the only thing to modify when adding a sort of task is to create a new runnable object (one sort of task => one runnable object that inherit from the abstract one).
Hope this will help you