I am not using my Cloud SQL since last 7 days and there is no query I am firing. Then why this graph shows I am making 10 queries per second. What does this graph signify ?
That graph signifies the number of statements executed by the server. This variable includes statements executed within stored programs.
To have a wider vision of what queries you Cloud SQL server is executing you can:
1. Enable Cloud SQL flag general_log=ON and log_output=FILE[1]
2. Go to [2], select your Cloud SQL instance and All logs=cloudsql.googleapis.com/mysql-general.log
You will see in the stackdriver logs the queries your server is executing behind.
You can further look up these queries in the mysql reference manual.
If you have Cloud Audit logs enabled then the latter could be the reason why you are seeing these results in the graph. To disable Audits logs, you need to go to[3] and uncheck the Cloud SQL row.
Important:
1. When you set, remove, or modify a flag for a database instance, the database might be restarted. The flag value is then persisted for the instance until you remove it. If the instance is the source of a replica, the replica will also restart to align with the current configuration of the instance[1].
After looking into the cloudsql.googleapis.com/mysql-general.log, you may want to disable general_log and output_file to stop further charges due to the log size. I would recommend you also visit the tips for general log flags[4].
[1] https://cloud.google.com/sql/docs/mysql/flags#config
[2] https://console.cloud.google.com/logs/viewer
[3] https://console.cloud.google.com/iam-admin/audit
[4] https://cloud.google.com/sql/docs/mysql/flags#tips
Related
I am trying to get my application(where hosted in google cloud) uptime history to a my own page. Is there any api so something on google cloud? I only need to get date and the up/down percentage or time.
I am already configure the uptime checks on google console. But I need to integrate this into my application.
Yes, you can but it's not obvious and it may be easier to use something other than Cloud Monitoring to export uptime data to a non-GCP site :-)
If you do want to use Cloud Monitoring to source this data into an off-GCP page, one of the Cloud Monitoring SDKs may be best. You can create a URL too (see below) but you'll need to authenticate this URL and that may make it too complex.
By way of an example, here's an Uptime check I created against my blog:
I recommend Google APIs Explorer as it's an excellent way to understand Google's services (via the REST APIs) and to test an approach.
First: List|Get Uptime Check(s)
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.uptimeCheckConfigs/list
Plug in to the form on the right hand side parent, the value of projects/${PROJECT}
If your Project ID is freddie-210224-66311747 then you'd type project/freddie-210224-66311747.
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.uptimeCheckConfigs/get
For this one, you need to provide name, the value of projects/${PROJECT}/uptimeCheckConfigs/${UPTIME_CHECK}
If your Uptime check is called test, then you'd type projects/freddie-210224-66311747/uptimeCheckConfigs/test
NOTE In my case, I used an Uptime check name that included periods (my.blog.com) and this was converted (to my-blog-com). So, you may want to list first to check the name.
Click "Execute" (You don't need to have API Key checked but it makes no difference).
What I learned is that Uptime checks are Metrics like all others. I confirmed this by watching the Chrome Dev Tools while I was watching Uptime checks.
Ensure that you use the correct metric name. You can use Monitoring's Metrics Explorer to confirm this:
The Resource Type is Uptime Check URL (uptime_url)
One (!) of the Metrics you may use is Request Latency (monitoring.googleapis.com/uptime_check/request_latency)
If you populate the Metrics Explorer, you should see the same data plotted as with the Uptime Check page.
Click Query Editor to get your Uptime Metric represented as Cloud Monitoring Query Language (MQL), remove any line-feeds. You can use:
fetch uptime_url | metric 'monitoring.googleapis.com/uptime_check/request_latency' | group_by 1m, [value_request_latency_mean: mean(value.request_latency)] | every 1m
So, now we want to query Montioring Metric Time-series
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/query
The value for name is projects/${PROJECT}
For query, paste in the MQL from above retain the quotes, i.e. "fetch uptime_url ..."
Hit EXECUTE
You should receive a snapshot of the time-series data underlying your Uptime URL. You can revise the MQL to reflect exactly the subset that you need. At 2021-02-24T20:55:38 the latency was 20.869:
So, to get e.g. request latencies for your uptime checks, you can use the Monitoring API's TimeSeries Query method and, with a suitable Query, this will yield JSON data including an array of Point (values). These values could then be transformed and surfaced into your external page.
I need to log all the queries on my sql instances, filter those that seem to fetch too many rows and archive it for a period of time.
Doc says
(2) All SQL queries executed on the database instance
about "Data Access audit logs".
So I've set out to enable access audit logs for my cloud sql server.
I have checked data read in the audit logs api page as said in here, but I cannot seem to find it anywhere in the log explorer. I have aleady checked log names for "projects/PROJECT_NAME/logs/cloudaudit.googleapis.com%2Fdata_access" to no use.
What am I missing? what am I doing wrong?
To view the logs in Cloud Logging you need to activate the flag general_log in the MySQL flags feature page.
For context, we would like to visualize our data in google data studio - this dataset receives more entries each week. I have tried hosting our data sets in google drive, but it seems that they're too large and this slows down google data studio (the file is only 50 mb, am I doing something wrong?).
I have loaded our data into google cloud storage --> google bigquery, and connected my google data studio to my bigquery table. This has allowed me to use the google data studio dashboard much quicker!
I'm not sure what is the best way to update our data weekly in google cloud/bigquery. I have found a slow way to do this by uploading the new weekly data to google cloud, then appending the data to my table manually in bigquery, but I'm wondering if there's a better way to do this (or at least a more automated way)?
I'm open to any suggestions, and if you think that bigquery/google cloud storage is not the answer for me, please let me know!
If I understand your question correctly, you want to automate the query that populate your table, which is connected to Data Studio.
If this is the case, then you can use Scheduled Query from BigQuery. Scheduled query allow you to define a query which results can be inserted in a new table. Particularly you can specify different rules for repetition (minimum each 15 minutes) and execution, as well as destination writing options (destination table, writing mode: append, truncate).
In order to use Scheduled Queries your account must have the right permissions. You can have a look at the following documentation to better understand how to use Scheduled Query [1].
Also, please note that at the front end the updated data in the BigQuery table will be seen updated in Datastudio at each refresh (click on refresh button in Datastudio). To automatically refresh the front-end visualization you can use the following plugin [2] or automate the click on the refresh button through Browser console commands.
[1] https://cloud.google.com/bigquery/docs/scheduling-queries
[2] https://chrome.google.com/webstore/detail/data-studio-auto-refresh/inkgahcdacjcejipadnndepfllmbgoag?hl=en
We use lots of components in Google Cloud, for example a job may start on App Engine, then do some work in Apache Airflow, then do some Dataflow work which will run a BigQuery insert.
Is there any way we can track the status of a job across all components using stack driver. For example tell stackdriver somehow a custom job id and query for it.
You can use advanced logs filters [1] to include log entries from various products. In the logging page search for your BigQuery Job ID. Click to the Job ID and select show matching entries. This will open advanced filter text box with the proper syntax. Then you can add more queries with an OR in between.
We have sysdig running on our WSO2 API gateway machine and we notice that it fires a large number of SQL queries to the database for a minute, than waits a minute and repeats.
The query looks like this:
Every minute it goes wild, waits for a minute and goes wild again with a request of the following format:
SELECT REG_PATH, REG_USER_ID, REG_LOGGED_TIME, REG_ACTION, REG_ACTION_DATA
FROM REG_LOG
WHERE REG_LOGGED_TIME>'2016-02-29 09:57:54'
AND REG_LOGGED_TIME<'2016-03-02 11:43:59.959' AND REG_TENANT_ID=-1234
There is no load on the server. What is causing this? What can we do to avoid this?
screen shot sysdig api gateway process
This particular query is the result of the registry indexing task that runs in the background. The REG_LOG table is being queried periodically to retrieve the latest registry actions. The indexing task cannot be stopped. However, one can configure the frequency of the indexing task through the following parameter that is in the registry.xml. See [1] for more information.
indexingFrequencyInSeconds
If this table is filled up, one can clean the data using a simple SQL query. However, when deleting the records, one must be careful not to delete all the data. The latest records of each resource path should be left in the REG_LOG table since reindexing of data requires at least one reference of each resource path.
Also, if required, before clearing up the REG_LOG table, you can take a dump of the data in case you do not want to loose old records. Hope this answer provides information you require.
[1] - https://docs.wso2.com/display/Governance510/Configuration+for+Indexing