So I have a very simple python script that writes a txt-file to my google storage bucket.
I just want to set this job to run each hour i.e not based on a trigger. It seems like that when using SDK, it needs to have a --triger- flag, but I only want it to be "triggered" by the scheduler.
Is that possible?
You can create a Cloud Function with Pub/Sub trigger and then create a Cloud Scheduler job targeting the topic which triggers the function.
I did it by following these steps:
Create a Cloud Function with Pub/Sub trigger
Select your topic or create a new one
This is the default code I am using:
exports.helloPubSub = (event, context) => {
const message = event.data
? Buffer.from(event.data, 'base64').toString()
: 'Hello, World';
console.log(message);
};
Create a Scheduler job targeting with the same Pub/Sub topic
Check it is working.
I tried it with the frequency ***** (every minute) and it works for me, I can see the logs from the Cloud Function.
Currently in order to execute a Cloud Function it needs to be triggered because once it stops the execution the only way to execute it again is through the trigger.
You can also follow the same steps I indicated to you in this page where you can find some images for further help.
Related
I am developing a solution where a cloud function calls BigQuery procedure and upon successful completion of this stored proc trigger another cloud function. For this I am using Audit Logs "jobservice.jobcompleted" method. Problem with this approach is it will trigger cloud function on every job that are completed in BigQuery irrespective of dataset and procedure.
Is there any way to add Path Pattern to the filter so that it triggers only for specific query completion and not for all?
My query starts something like: CALL storedProc() ...
Also, as I tried to create a 2nd Gen function from console, I tried Eventarc trigger. But to my surprise BigQuery Event provider doesn't have Event for jobCompleted
Now I'm wondering if it's possible to trigger based on job complete event.
Update:I changed my logic now to use google.cloud.bigquery.v2.TableService.InsertTable method to make sure after inserting a record to a table it will add AuditLog message so that I can trigger the next service. This insert statement is present as the last statement in BigQuery procedure.
After running the procedure, the insert statement is inserting the data but resource name is coming as projects/<project_name>/jobs
I was expecting something like projects/<project_name>/tables/<table_name> so that I can apply path pattern on resource name.
Do I need to use different protoPayload.method?
Try to create a Log Sink for job completed with unique principal-email sv account and use pubsub with the sink.
Get pubsub published event to run destination service.
I have created AWS App flow using cloud formation template and I want to schedule an app flow using Trigger Config.
How can we pass date ScheduleStartTime using cloud formation?
Error I'm getting
AWS::AppFlow::FlowCreate Flow request failed:
[Schedule start time cannot be in the past. Please update the schedule
start time to a value in future.
The snippet I'm using in cloud formation,
"TriggerConfig": {
"TriggerType": "Scheduled",
"TriggerProperties": {
"DataPullMode": "Incremental",
"ScheduleExpression": "rate(5minutes)",
"TimeZone": "America/New_York",
"ScheduleStartTime" : 4.05
}
}
TriggerConfig:
TriggerType: Scheduled
TriggerProperties:
DataPullMode: Incremental
ScheduleExpression: rate(1days)
ScheduleStartTime: 1652970600
Use Unix timestamp for start time.
Please refer to the below link for conversion.
https://www.epochconverter.com/
try ScheduleInterval parameter instead of ScheduleExpression
I'm new to GCP and Python. I have got a task to import JSON file into google firestore using google cloud functions via Python.
Kindly assist please.
I could achieve this system setup using below code. Posting for your reference:-
CLOUD FUNCTIONS CODE
REQUIREMENTS.TXT (Dependencies)
`google-api-python-client==1.7.11
google-auth==1.6.3
google-auth-httplib2==0.0.3
google-cloud-storage==1.19.1
google-cloud-firestore==1.6.2`
MAIN.PY
from google.cloud import storage
from google.cloud import firestore
import json
client = storage.Client()``
def hello_gcs_generic(data, context):
print('Bucket: {}'.format(data['bucket']))
print('File: {}'.format(data['name']))
bucketValue = data['bucket']
filename = data['name']
print('bucketValue : ',bucketValue)
print('filename : ',filename)
testFile = client.get_bucket(data['bucket']).blob(data['name'])
dataValue = json.loads(testFile.download_as_string(client=None))
print(dataValue)
db = firestore.Client()
doc_ref = db.collection(u'collectionName').document(u'documentName')
doc_ref.set(dataValue)
Cloud functinos are server less functions provided by google. The beauty of cloud function is it will destroy it will invoke any trigger happens and itself once the execution is complete. Cloud functions are single purpose functions. Not only python, you can also use NodeJS and Go to write cloud functions. You can create a cloud function very easily by visiting quick start of cloud functions (https://cloud.google.com/functions/docs/quickstart-console).
Your task is to import a JSON file into google firestore. This part you can do using Firestore python connector like any normal python program and add into cloud function console or upload via gcloud. Still the trigger part is missing here. As I mentioned cloud function is serverless. It will execute when any event happens in the attached trigger. You haven't mentioned any trigger here (when you want to trigger the function). Once you give information about the trigger I can give more insights on resolution.
Ishank Aggarwal, You can add the above code snippet as a part of cloud function by following steps:
https://console.cloud.google.com/functions/
Create function using function name, yours requirements, choose run time as python and choose trigger as your gcs bucket.
Once you create it, if any change happens in your bucket, the function will trigger and execute your code
We need a way to automatically create a Pub/Sub trigger on new compute images (preferably triggered on a specific image family). Alternatively, we know that Pub/Sub on GCS buckets, but we have not found a way to automate transferring images to a GCS bucket.
For some background: we are automating image baking through packer and we need this piece to trigger a terraform creation. We know that a cron job can be created to simply poll images when they are created, but we are wondering if there is already support for such a trigger in GCP.
You can have a Stackdriver Logging export sink that publishes to Pub/Sub and is triggered by a specific filter (docs). For example:
resource.type="gce_image"
jsonPayload.event_subtype="compute.images.insert"
jsonPayload.event_type="GCE_OPERATION_DONE"
To trigger it only for a specific family you can use this other filter below but protoPayload.request.family is only present when the API request is received and not when it is actually fulfilled (maybe you could add some delay in your processing function if needed)
resource.type="gce_image"
protoPayload.request."#type"="type.googleapis.com/compute.images.insert"
protoPayload.request.family="FAMILY-NAME"
Another solution would be to create a cloud function with --trigger-topic={your pub sub topic} and then filter only the images that you want to act on based on some environment variables on the cloud function
Psuedo code
1. create a pub sub topic for images being inserted in the GCR
gcloud pubsub topics create projects/<project_id>/topics/gcr
This will now publish all messages corresponding to all images being inserted/modified/deleted in the repo
Create a cloud function that has the function signature thus
// contents of index.js
// use the Storage function from google-coud node js api to work on storages
// https://www.npmjs.com/package/#google-cloud/storage
const Storage = require(#google-cloud/storage).Storage;
function moveToStorageBucket(pubSubEvents, context, callback) {
/* this is how the pubsub comes from GCR
{"data":{"#type":"... .v1.PuSubMessage", "attribute":null, "data": "<base 64 encoded>"},
"context":{..other details}}
data that is base 64 encoded in in this format
{ "action":"INSERT","digest":"<image name>","tag":<"tag name>"}
*/
const data = JSON.parse(Buffer.from(pubSubEvents.data, 'base64').toString())
// get image name from the environment variable passed
const IMAGE_NAME = process.env.IMAGE_NAME;
if (data.digest.indexOf(IMAGE_NAME) !== -1) {
// your action here...
}
}
module.exports.moveToStorageBucket = moveToStorageBucket;
deploy the cloud function
gcloud functions deploy <function_name> --region <region> --runtime=nodejs8 --trigger-topic=<topic created> --entry-point=moveToStorageBucket --set-env-vars=^--^IMAGE_NAME=<your image name>
Hope that helps
Referring to item: Watching for new files matching a filepattern in Apache Beam
Can you use this for simple use cases? My use case is that I have user uploads data to Cloud Storage -> Pipeline (Process csv to json) -> Big Query. I know Cloud Storage is bounded collection so it represents Batch Dataflow.
What I would like is to do is keep pipeline running in streaming mode and as soon as a file is uploaded to Cloud Storage, it will be processed through pipeline. Is this possible with watchfornewfiles?
I wrote my code as follows:
p.apply(TextIO.read().from("<bucketname>")
.watchForNewFiles(
// Check for new files every 30 seconds
Duration.standardSeconds(30),
// Never stop checking for new files
Watch.Growth.<String>never()));
None of the contents is being forwarded to Big Query, but the pipeline shows that it is streaming.
You may use Google Cloud Storage Triggers here :
https://cloud.google.com/functions/docs/calling/storage#functions-calling-storage-python
These triggers uses Cloud Functions similar to Cloud Pub/Sub which gets triggered on objects if they were: created/ deleted/archived/ or metadata change.
These event are sent using Pub/Sub notifications from Cloud Storage, but pay attention not to set many functions over the same bucket as there is some notification limits.
Also, at the end of the document there is a link to a sample implementation.