I'm getting started on Dialogflow and I would like to integrate it with BigQuery.
I have some tables in BigQuery with difference data, for instance a record of alarms that a wind turbine showed during time.
In one of my test cases, let's say I want my chatbot to tell me what alarms were raised in the wind turbine number 5 of my farm, on the 25th of October.
I have already created a chatbot in Dialogflow that asks for all the necessary parameters of the enquiry, such as the wind farm name, the wind turbine number, the date, and the name of the alarm.
My doubt now is how I can send those parameters to BigQuery in order to dig into my tables, extract the required information, and print it in Dialogflow.
I have been looking for documentation or tutorials but nothing came out that could fit my case...
Thanks in advance!
You need to implement a fulfillment. It triggers a webhook, for example a Cloud Functions or a Cloud Run service.
This webhook call contains the value gather by your intent and parameters. You have to extract them and perform your process, for example a call to BigQuery. Then format the response and display it on Dialogflow.
Related
I'm trying to establish email alerts at a project level to send email alerts for when a certain number of query/job concurrency is reached e.g. 5 concurrent queries. We have a flat-rate pricing model.
I want to do a similar email notification when total slot Usage exceeds a certain threshold as well e.g. slot usage reaching 1000 slots
As a next step I would like to throttle new incoming queries based on the above mentioned thresholds. Meaning if there are already for example 5 queries actively running the 6th one will be put on hold until one of the 5 running earlier have completed.
You may create an Alert Policy in which you can set your desired metric type (eg. slots) and then configure your desired threshold similar to below.
In creating an Alert Policy you may also configure the notification channel to email notification which is also included on the same documentation.
For the available metric types for SLOTS in BigQuery, you may refer to this Google Cloud Metrics for BigQuery documentation.
For your next step, you may code (python, node.js, etc) using BigQuery API to count queries actively running (through JOB ID) and when the count hits 5, you may print "query queue is full" and then wait for the total JOBS to hit below 5 before running the next query. You may refer to this BigQuery Managing Jobs API Documentation.
I have a table in google bigquery in which I calculate a column (think of it as anomaly detection column),
is there a way, within GCP, to send a rule based alert (e.g. once the value in the column is 1)
if not how would you recommend dealing with this issue.
Thanks
A solution could be to use eventarc triggers: when data is inserted into your table, the job is written into cloud audit logs and you can trigger a Cloud Run like this.
With this Cloud Run it's possible to inspect the column you mention and send notifications accordingly.
Here is a good reference on how to proceed.
Because you run a request to calculate the line and the alert to send, the solution is to get the result of this request and to use it.
You can export data to a file (csv for example) and then to trigger a cloud functions on the file creation on Cloud Storage. The cloud functions will read the file and for each line trigger an alert, or send only one alert with the summary of the file, or to send the file in attachement.
You can also fetch all the line of the request result and publish a PubSub message for each line. Like that, you can process in parallel all the messages with Cloud Function or Cloud Run (this time it's not possible to have only 1 alert with a summary of all the lines, the message are unitary)
I am trying to get my application(where hosted in google cloud) uptime history to a my own page. Is there any api so something on google cloud? I only need to get date and the up/down percentage or time.
I am already configure the uptime checks on google console. But I need to integrate this into my application.
Yes, you can but it's not obvious and it may be easier to use something other than Cloud Monitoring to export uptime data to a non-GCP site :-)
If you do want to use Cloud Monitoring to source this data into an off-GCP page, one of the Cloud Monitoring SDKs may be best. You can create a URL too (see below) but you'll need to authenticate this URL and that may make it too complex.
By way of an example, here's an Uptime check I created against my blog:
I recommend Google APIs Explorer as it's an excellent way to understand Google's services (via the REST APIs) and to test an approach.
First: List|Get Uptime Check(s)
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.uptimeCheckConfigs/list
Plug in to the form on the right hand side parent, the value of projects/${PROJECT}
If your Project ID is freddie-210224-66311747 then you'd type project/freddie-210224-66311747.
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.uptimeCheckConfigs/get
For this one, you need to provide name, the value of projects/${PROJECT}/uptimeCheckConfigs/${UPTIME_CHECK}
If your Uptime check is called test, then you'd type projects/freddie-210224-66311747/uptimeCheckConfigs/test
NOTE In my case, I used an Uptime check name that included periods (my.blog.com) and this was converted (to my-blog-com). So, you may want to list first to check the name.
Click "Execute" (You don't need to have API Key checked but it makes no difference).
What I learned is that Uptime checks are Metrics like all others. I confirmed this by watching the Chrome Dev Tools while I was watching Uptime checks.
Ensure that you use the correct metric name. You can use Monitoring's Metrics Explorer to confirm this:
The Resource Type is Uptime Check URL (uptime_url)
One (!) of the Metrics you may use is Request Latency (monitoring.googleapis.com/uptime_check/request_latency)
If you populate the Metrics Explorer, you should see the same data plotted as with the Uptime Check page.
Click Query Editor to get your Uptime Metric represented as Cloud Monitoring Query Language (MQL), remove any line-feeds. You can use:
fetch uptime_url | metric 'monitoring.googleapis.com/uptime_check/request_latency' | group_by 1m, [value_request_latency_mean: mean(value.request_latency)] | every 1m
So, now we want to query Montioring Metric Time-series
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/query
The value for name is projects/${PROJECT}
For query, paste in the MQL from above retain the quotes, i.e. "fetch uptime_url ..."
Hit EXECUTE
You should receive a snapshot of the time-series data underlying your Uptime URL. You can revise the MQL to reflect exactly the subset that you need. At 2021-02-24T20:55:38 the latency was 20.869:
So, to get e.g. request latencies for your uptime checks, you can use the Monitoring API's TimeSeries Query method and, with a suitable Query, this will yield JSON data including an array of Point (values). These values could then be transformed and surfaced into your external page.
We are working in one of the customer module. With the help of this module, we are calculating price for google cloud components like compute engine with attributes images, boot disk, region, snapshot etc.
But we found that GCP is revising the prices and JSON is not modifying immediately. Newly revised pricing values comes in JSON after few days.
So the price according to JSON and price in google cosole/calculator are different in this case.
Is there any exchangeable format to get revised pricing immediately?
The JSON file isn't exactly the ideal way to obtain pricing information anymore. In fact, the link to the JSON file was removed from the Pricing Calculator page very recently precisely because they want people to start using the Catalogue API. That one has the prices up-to-date for sure, so I'd recommend you to use that instead.
I am working on Amazon Matillion for Redshift and we have multiple jobs running daily triggered by SQS messages. Now I am checking the possibility of creating a UI dashboard for stakeholders which will monitor live progress of jobs and will show report of previous jobs, like Job name, tables impacted, job status/reason for failure etc. Does Matillion maintain this kind of information implicitly? Or I will have to maintain this information for each job.
Matillion has an API which you can use to obtain details of all task history. Information on the tasks API is here:
https://redshiftsupport.matillion.com/customer/en/portal/articles/2720083-loading-task-information?b_id=8915
You can use this to pull data on either currently running jobs or completed jobs down to component level including name of job, name of component, how long it took to run, whether it ran successfully or not and any applicable error message.
This information can be pulled into a Redshift table using the Matillion API profile which comes built into the product and the API Query component. You could then build your dashboard on top of this table. For further information I suggest you reach out to Matillion via their Support Center.
The API is helpful, but you can only pass a date as a parameter (this is for Matillion for Snowflake, assume it's the same for Redshift). I've requested the ability to pass a datetime so we can run the jobs throughout the day and not pull back the same set of records every time our API call runs.