I am writing in AWS CDK a Step Function which run two tasks in parallel. I would like to access from one of the tasks , a value of the second tasks , which runs in parallel (for example, I would like to know in task 1, which is the time started task 2, or maybe id from task 2).
Here an screenshot of the state machine definition in Step Function.
In the example of the screenshot, I would like to use the Id of the GlueStartRunJob (1) in GlueStartRunJob.
I was thinking about using the Context Object for that purpose. Nevertheless, I am not sure if this is the right approach...
The Context Object is read-only and allows a give state to access contextual information about it's self, not about other states from elsewhere in the workflow.
I'm not 100% clear what you are aiming to accomplish here, but I can see a couple of possible approaches.
First, you might just want to order these Glue Jobs to run sequentially so the output from the first can be used in the second.
Second, if you need the workflow to take action after the Glue Jobs have started but before they have completed, you'd need to take an approach that does not use the .sync integration pattern. With this integration pattern, Step Functions puts a synchronous facade over an asynchronous interaction, taking care of the steps to track completion and return you the results. You could instead use the default RequestResponse pattern to start the jobs in your parallel state, then do whatever you needed to after. You'd need to then include your own polling logic if you wanted the workflow to wait for completion of the jobs and return data on them or take action on completion. You can see an example of such polling for Glue Crawlers in this blog post (for which you can find sample code here).
Related
I am using GCP Workflow Beta to check if I can build some of my workflows. The documentation mentions how we can conditionally execute steps with switch case and next for jumps. However can we have a flows where
A step waits for two or more previous steps to complete
Multiple steps triggered the same time.
As you can see, what I am implying is conditional parallel execution of steps. Is there a way to do this ?
Also, I see we have some basic functions like len, string etc in the examples. Can you please guide me where I can find a list of all such functions that are available ? I was looking for something to manipulate JSON.
You can't wait a step because you can't run multiple step in parallel for now.
In my company, we also expect, and wait, a lot about parallelization and that's why I had a meeting with the PM yesterday to share these expectation with him. It's one of the top priority in the roadmap but I don't know when it will be released (something like Q1 or Q2 2021 I guess).
The functions sounds like python code but it's not really clear in the documentation. I will share this with the PM.
I'm trying to improve our queue manager, and what I'd like to do is this:
There are two types triggers that can start a job (in this case regular, and upstream). If in the queue, there is ever a regular build and an upstream build, the upstream job will always execute, and we cancel the regular build. And if there are ever multiple instances with the same trigger (for the same job), we always take the first one, and cancel the rest, we don't want duplicate jobs in the queue.
These are triggers for the same job, and has nothing to do with concurrency of other jobs!
How can I achieve this? Using groovy, how can I get a list of triggers for the job and apply the logic I described above? Is there a plugin that'll solve my problem?
new to groovy, and jenkins, so maybe I'm trying to re-invent the wheel here
It might not do exactly what you want, but take a look at the Accelerated Build Now plugin in combination with the Priority Sorter plugin
Our Jenkins instance has a job for our main application. It builds all git branches in the one job, and so can sometimes get pretty far behind. However, the Build Queue on the lefthand side only ever shows the next job, not all the others. Is there a way to see all the queued executions of a single job? Ideally it'd even show the branch as well.
I'm aware of solutions like creating a new job for each branch, but this really clutters up the already horrible interface, and I'd rather avoid that.
For a single job, with same parameters, Jenkins doesn't place a build in the queue if it already contained in the queue. You can use a simple trick to add an unused parameter and set some random value to this parameter every time you run the job. Now you can have multiple jobs in the queue for the same job.
I'm writing PHP for fairly simple workflow for Amazon SWF. I've found myself starting to write a library to check if certain actions have been started or completed. Essentially looping over the event list to check how things have progressed, and then starting an appropriate activity if its needed. This can be a bit faffy at times as the activity type and input information isn't in every event, it seems to be in the ActivityTaskScheduled event. This sort of thing I've discovered along the way, and I'm concerned that I could be missing subtle things about event lists.
It makes me suspect that someone must have already written some sort of generic library for finding the current state of various activities. Maybe even some sort of more declarative way of coding up the flowcharts that are associated with SWF. Does anything like this exist for PHP?
(Googling hasn't come up with anything)
I'm not aware of anything out there that does what you want, but you are doing it right. What you're talking about is coding up the decider, which necessarily has to look at the entire execution state (basically loop through the event list) and decide what to do next.
Here's an example written in python
( Using Amazon SWF To communicate between servers )
that looks for events of type 'ActivityTaskCompleted' to then decide what to do next, and then, yes, looks at the previous 'ActivityTaskScheduled' entry to figure out what the attributes for the previous task were.
If you write a php framework that specifies the workflow in a declarative way then a generic decider that implements it, please consider sharing it :)
I've since found https://github.com/cbalan/aws-swf-fluent-php which looks promising, but not really used it, so can't speak to the whether it works or not.
I've forked it and started a bit of very light refactoring to allow some testing, available at https://github.com/michalc/aws-swf-fluent-php
We have a quatz job that does a lot of calculations and is taking a while to complete. In order to speed it up we want to split the primary job to start multiple smaller jobs that do the calculations and return the result. After all the small jobs complete we need a final job that will pull the subtotals together.
Currently the idea is each small job will write to a store, and when creating the final job we pass in all small job names to it with MapData. The final job will look for these jobs and reschedule if any are found, else it will run the totals.
Is there a better way to accomplish this in quartz?
This isn't necessarily answering the question, but I'm afraid I don't think Quartz is the tool for the job here. It's a scheduler, not a mechanism for load balancing. You could look at using Quartz in combination with NServiceBus or MassTransit. The job could fire multiple messages for the small jobs, maybe even using the same message type and a Distributor and use a Saga to pull everything back together.