I have created AWS App flow using cloud formation template and I want to schedule an app flow using Trigger Config.
How can we pass date ScheduleStartTime using cloud formation?
Error I'm getting
AWS::AppFlow::FlowCreate Flow request failed:
[Schedule start time cannot be in the past. Please update the schedule
start time to a value in future.
The snippet I'm using in cloud formation,
"TriggerConfig": {
"TriggerType": "Scheduled",
"TriggerProperties": {
"DataPullMode": "Incremental",
"ScheduleExpression": "rate(5minutes)",
"TimeZone": "America/New_York",
"ScheduleStartTime" : 4.05
}
}
TriggerConfig:
TriggerType: Scheduled
TriggerProperties:
DataPullMode: Incremental
ScheduleExpression: rate(1days)
ScheduleStartTime: 1652970600
Use Unix timestamp for start time.
Please refer to the below link for conversion.
https://www.epochconverter.com/
try ScheduleInterval parameter instead of ScheduleExpression
Related
I want to trigger airflow dags in postman.
recently I am hitting
http://localhost:8080/api/v1/dags/a_pipline/dagRuns/run_id
It's asking for run_id.
How can I specify run id in dags
I have found answers after a lot of struggle. Its working on postman restAPI
POST: http://localhost:8080/api/v1/dags/a_pipline/dagRuns
body: select json formate
{
"dag_run_id": "manual__2022-10-22T11:05:59.461142+00:00"
}
Coming to Airflow from 15 years of using LSF, the terminology trips me up.
A dag_id is its "name"
The above is a "dag_run_id", which is the id of an instance of a dag having run, or in the process of running. What LSF would call it's flow_id.
In LSF these are purely numerical, issued sequentially across all flow instances, 1,2,3 and so on.
In Airflow, these are datetimestamps, which theoretically are not unique across the whole system, but should be within a dag.
In the airflow CLI - you trigger a dag as follows:
airflow dags trigger name_of_the_dag
(glossing over authentication)
I'm just getting to grips with scripts doing this via the REST API...
Error message:
Error: There was an error deploying functions
firebease-debug.log holds this:
[debug] [2021-11-16T12:12:16.165Z] Error: Failed to upsert schedule function lab2 in region europe-west3
function code:
exports.lab2 =
functions
.region('europe-west3')
.pubsub.schedule('*/10 * * * *')
.onRun(lab);
What can I do? Google support leads to stackoverflow, so I post it here. Are there better ways to deal with the Google Cloud problems?
When you are using scheduled functions in Firebase Functions, an App Engine instance is created that is needed for Cloud Scheduler to work. You can read about it here. During its setup you're prompted to select your project's default Google Cloud Platform (GCP) resource location (if it wasn't already selected when setting up another service).
You are getting that error because there is a difference between the default GCP resource location you specified and the region of your scheduled Cloud Function. If you click on the cogwheel next to project-overview in Firebase you can see where your resources are located. Setting the default GCP resource location same as the scheduler function region, solves the issue.
The reason our schedule function deployment failed was because of the cron job formula. For some reason firebase couldn't understand 0 0 3 1/1 * ? *, but could understand every 5 minutes.
It's a shame that firebase doesn't provide a better error message. Error: Failed to upsert schedule function xxx in region xxx is way too generic.
I got the same problem of deploying my schedule functions, and I solved it with a few steps as below:
Step 1: Rename the schedule function (for example: newScheduleABC), then only deploy it:
$ Firebase deploy --only functions:newScheduleABC
NOTE: as #Nicholas mentioned, the Unix Cron Format The GCP accepts has only 5 fields: schedule(* * * * *)
Step 2: Delete old (schedule) functions by going to your Firebase Console : https://console.firebase.google.com/.../functions , you will see all of your functions there. click on the vertical ellipsis at the end of a function and click delete.
That's it. There should be no problem from now.
You can read more from Manage functions
I have two service accounts with exact same roles under the same project and one can run the Flex template without any issue but the other fails to do so and returns:
Timeout in polling result file: <LOGGING_BUCKET>. Service account: <SERVICE_ACCOUNT> Image URL: <IMAGE_URL> Troubleshooting guide at https://cloud.google.com/dataflow/docs/guides/common-errors#timeout-polling
The SA that fails to run doesn't write the logs to GCS bucket, making it really difficult to debug. The graph doesn't get created and seems to get stuck at queue stage. The roles of both SAs are:
BigQuery Admin
Bigtable User
Dataflow Developer
Editor
Storage Object Viewer
Sorry if is it obvious...but
Have you checked the google doc from the error? (https://cloud.google.com/dataflow/docs/guides/common-errors#timeout-polling).
Both SAs have the same roles?
Let's say that SA1 can run Flex1, and SA2 can't run Flex2. Have you tried to assign SA1 into Flex2?
What could be any possible difference between both SAs?
If you create SA3 with the same roles as SA2 and assign it to Flex2, does it work?
Good luck
So I have a very simple python script that writes a txt-file to my google storage bucket.
I just want to set this job to run each hour i.e not based on a trigger. It seems like that when using SDK, it needs to have a --triger- flag, but I only want it to be "triggered" by the scheduler.
Is that possible?
You can create a Cloud Function with Pub/Sub trigger and then create a Cloud Scheduler job targeting the topic which triggers the function.
I did it by following these steps:
Create a Cloud Function with Pub/Sub trigger
Select your topic or create a new one
This is the default code I am using:
exports.helloPubSub = (event, context) => {
const message = event.data
? Buffer.from(event.data, 'base64').toString()
: 'Hello, World';
console.log(message);
};
Create a Scheduler job targeting with the same Pub/Sub topic
Check it is working.
I tried it with the frequency ***** (every minute) and it works for me, I can see the logs from the Cloud Function.
Currently in order to execute a Cloud Function it needs to be triggered because once it stops the execution the only way to execute it again is through the trigger.
You can also follow the same steps I indicated to you in this page where you can find some images for further help.
We need a way to automatically create a Pub/Sub trigger on new compute images (preferably triggered on a specific image family). Alternatively, we know that Pub/Sub on GCS buckets, but we have not found a way to automate transferring images to a GCS bucket.
For some background: we are automating image baking through packer and we need this piece to trigger a terraform creation. We know that a cron job can be created to simply poll images when they are created, but we are wondering if there is already support for such a trigger in GCP.
You can have a Stackdriver Logging export sink that publishes to Pub/Sub and is triggered by a specific filter (docs). For example:
resource.type="gce_image"
jsonPayload.event_subtype="compute.images.insert"
jsonPayload.event_type="GCE_OPERATION_DONE"
To trigger it only for a specific family you can use this other filter below but protoPayload.request.family is only present when the API request is received and not when it is actually fulfilled (maybe you could add some delay in your processing function if needed)
resource.type="gce_image"
protoPayload.request."#type"="type.googleapis.com/compute.images.insert"
protoPayload.request.family="FAMILY-NAME"
Another solution would be to create a cloud function with --trigger-topic={your pub sub topic} and then filter only the images that you want to act on based on some environment variables on the cloud function
Psuedo code
1. create a pub sub topic for images being inserted in the GCR
gcloud pubsub topics create projects/<project_id>/topics/gcr
This will now publish all messages corresponding to all images being inserted/modified/deleted in the repo
Create a cloud function that has the function signature thus
// contents of index.js
// use the Storage function from google-coud node js api to work on storages
// https://www.npmjs.com/package/#google-cloud/storage
const Storage = require(#google-cloud/storage).Storage;
function moveToStorageBucket(pubSubEvents, context, callback) {
/* this is how the pubsub comes from GCR
{"data":{"#type":"... .v1.PuSubMessage", "attribute":null, "data": "<base 64 encoded>"},
"context":{..other details}}
data that is base 64 encoded in in this format
{ "action":"INSERT","digest":"<image name>","tag":<"tag name>"}
*/
const data = JSON.parse(Buffer.from(pubSubEvents.data, 'base64').toString())
// get image name from the environment variable passed
const IMAGE_NAME = process.env.IMAGE_NAME;
if (data.digest.indexOf(IMAGE_NAME) !== -1) {
// your action here...
}
}
module.exports.moveToStorageBucket = moveToStorageBucket;
deploy the cloud function
gcloud functions deploy <function_name> --region <region> --runtime=nodejs8 --trigger-topic=<topic created> --entry-point=moveToStorageBucket --set-env-vars=^--^IMAGE_NAME=<your image name>
Hope that helps