I have a use case where I schedule a task 24h into the future after an event occurs. This task represents some sort of "deadline" for other things to happen.
The scheduled task triggers a creation of a report. If not all of the above mentioned "other things" have completed by this time, then the triggered report creation process creates it anyways with the information it has at the time.
If, on the other hand, all other things do complete before these 24h, then ideally I'd like to re-use the same Google Cloud Task to trigger the same process (as it's identical as the previous case but will contain all of the information possible).
I would imagine the easiest way to achieve the above is to:
schedule a task 24h into the future
if all information arrives: run the task early before it's scheduled time
However, reading through the Google Cloud Tasks documentation I don't see the option to run the task early. However, that feature does exist on the Cloud Tasks console, so I was wondering if it is available in the documentation and client libraries.
Thanks!
This is probably what you're looking for
https://cloud.google.com/tasks/docs/reference/rest/v2/projects.locations.queues.tasks/run
NOTE: It does say however that "This command is meant to be used for manual debugging"
Related
I'm using AWS-DMS to migrate existing data only from a Postgres db as source to aws-S3 as target. I have created a migration task for this, and I'm able to do the aforementioned.
However, I wanted to know how much time it took for a task to complete. I couldn't find a time completion metric in either the metrics corresponding to the task or the metrics corresponding to the replication-instance.
How do I find out the time taken for the full load?
Using the AWS CLI you can try using the describe-replication-tasks function.
This will provide you with both the Start and Stop times, as well as the time elapsed.
actually the following steps to my data:
new objects in GCS bucket trigger a Google Cloud function that create a BigQuery Job to load this data to BigQuery.
I need low cost solution to know when this Big Query Job is finished and trigger a Dataflow Pipeline only after the job is completed.
Obs:
I know about BigQuery alpha trigger for Google Cloud Function but i
dont know if is a good idea,from what I saw this trigger uses the job
id, which from what I saw can not be fixed and whenever running a job
apparently would have to deploy the function again. And of course
it's an alpha solution.
I read about a Stackdriver Logging->Pub/Sub -> Google cloud function -> Dataflow solution, but i didn't find any log that
indicates that the job finished.
My files are large so isn't a good idea to use a Google Cloud Function to wait until the job finish.
Despite your mention about Stackdriver logging, you can use it with this filter
resource.type="bigquery_resource"
protoPayload.serviceData.jobCompletedEvent.job.jobStatus.state="DONE"
severity="INFO"
You can add dataset filter in addition if needed.
Then create a sink into Function on this advanced filter and run your dataflow job.
If this doesn't match your expectation, can you detail why?
You can look at Cloud Composer which is managed Apache Airflow for orchestrating jobs in a sequential fashion. Composer creates a DAG and executes each node of the DAG and also checks for dependencies to ensure that things either run in parallel or sequentially based on the conditions that you have defined.
You can take a look at the example mentioned here - https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/cloud-composer-examples/composer_dataflow_examples
I noticed some strange behavior by AWS data pipeline.
The Execution start time is before the scheduled start time. Please refer to the screenshot below.
Am I missing something here ?
Is this acceptable behavior for AWS data pipline ? What are the recommended way to avoid this ?
Data Pipeline here is recording creation of instance as Execution start time. But it does not start execution (Running state) before Scheduled start time. You can verify this by clicking on the instance, view all fields, it has additional info.
This is definitely misleading. Data Pipeline needs to fix the recording of timestamps.
Any ideas on how to reliably trigger an URL (web service) at a specific time? With the precision in seconds? For example, the script will be set so that it will be able to trigger a web service at 2015-05-27 12:34:55. In my scenario, the user will be able to select at what time, down to seconds a trade should execute. The web service must be then triggered at a specific time
AWS Lambda is not able to run at specific times.
Cron jobs won't work as it does not run every second
An SQS might work but coding it up to be reliable could be hard.
Thanks!
"at" command does what you need: https://calomel.org/cron_at.html
An addtitional tool one can use is called "at" and is used to execute a job only once. "at" is very useful, for example if you want run a backup job starting at 8pm and you expect to be leaving at 5:30pm.
The question might sound very silly but here it is,
Are the scheduled jobs in play framework triggered automatically or do they need to be called from some place, say may be from application start up or so?
Or does it need any kind of configuration to be done except the scheduled time which in my case is
"#On("0 40 12 ? * MON-FRI")"
Please revert.
Jobs are triggered automatically. You can also invoke them manually, for example from another controller, by invoking the now() method on an Job instance. Have a look at http://playframework.com/documentation/1.2.7/jobs#scheduling for more details.