I am currently learning about AWS Logs Insights, and I was wondering if the following is possible.
Let's say I gather logs from Route53. So I have an event for each query that reaches the AWS DNS servers of course. Now, I know I can count the number of queries per resolverIp for example as such:
stats count(*) by resolverIp
I also know that I can count the number of the queries, per resolverIp, that returned the NXDOMAIN responseCode, as such:
filter responseCode="NXDOMAIN" | stats count(*) by resolverIp
My question is, is there a way to get a percentage of the later (number of queries that returned NXDOMAIN per resolverIp) from the former (number of queries per resolverIp)?
This query gives you the percentage sorted by resolver IP
stats
sum(strcontains(responseCode,"NXDOMAIN")) / count(*)
* 100 by resolverIp
Related
I am trying to find the amount of data queried per statement from AWS Lambda on Redshift, but all I can find is amount of data queried per query ID. There are multiple lambdas which I am running but I can't seem to relate the lambdas to the query ID.
I tried to look up the documentation on AWS Redshift system views, but there doesn't seem to be any tables which contain these values.
So there are a few ways to do this. First off the Lambda can find its session id with PG_BACKEND_PID(). This can be reported out / logged from the Lambda to report all statements from this session. Or you can add a unique comment to to all the queries coming from Lambda and you can search on this in svl_statementtext. Or you can do both. Once you have the query id and session id you look at the query statistics (SVL_QUERY_REPORT or other catalog tables).
Be aware that query ids and session ids repeat over time so also check the date to make sure you are not seeing a query from some time ago.
Is there a way to measure the impact to Kusto Cluster when we run a Query from Power BI. This is because the Query I use in Power BI might get large data even if it is for a limited time range. I am aware of setting - limit Query result record ,but I would like to measure the impact to Cluster for specific queries .
Do I need to use the metrics under - Data explorer monitoring. Is there a best way to do it and any specific metrics . Thanks.
You can use .show queries or Query diagnostics logs - these can show you the resources utilization per query (e.g. Total CPU time & memory peak), and you can filter to a specific user or application name (e.g. PowerBI).
Recently we faced an outage due to 403 rateLimitExceeded error. We are trying to setup an alert using gcp metric for this error. However the metric for bigquery.googleapis.com/query/count or bigquery.googleapis.com/job/num_in_flight is not showing the number of queries running correctly. We believe we crossed the threshold of 100 concurrent queries several times over the past few days but the metric explorer shows only a maximum of 5 only on few occasions. Do these metrics need any other configs to show the right number or we should use some other way to create an alert that shows that we have crossed 80% of concurrent query no.
I am running a sequence of queries in BigQuery that take data from a table, enrich/transform them and load into other tables within the same project.
Very high level query structure looks as such:
WITH query_1 as (
SELECT columns from Table_A
WHERE some_condition = some_value
),
query_2 as (
SELECT processing_function(columns)
from A)
SELECT * from query_2
I am calling this query through Python and specifying the destination table in the query job config.
The Table_A mentioned above has about 2 TB of data for a day and I am looping this operation for 10 days. After processing just 1 day of data, BigQuery gave the following error
403 Quota exceeded: Your usage exceeded quota for ExtractBytesPerDay. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors
I checked the quota and it is pushing the 10TB quota limit I have but I'm not able to figure out what ExtractBytesPerDay quota really is. I can increase the quota limit, but I would like to evaluate how much I would need to increase it.
So any direction on which operations account for this quota would be helpful. Anybody know what ExtractBytesPerDay quota means??
You quota seems to be related to Exporting Data from BQ that extract data to a destination like Cloud Storage.
You should know that there is a new API called BigQuery Storage API that can help to mitigate your quota issue.
In case that don't help you, you might want to contact the Google Cloud Platform Support to get more insights and know how to resolve your error.
I have the following Dynamo DB table structure:
item_type (string) --> Partition Key
item_id (number) --> Secondary Index
The table has around 5 million records and auto scaling is enabled with default read capacity of 5. I need to fetch the item_id given certain item type. We have around 500000 item_types and each item type will be associated with multiple item ids. I see a response of around 4 seconds for popular item_types. I am testing this on AWS Lambda, I start the timer when we make the query and end it once we get the response. Both Lambda and Dynamo DB are in the same region.
This is the query I am using:
response = items_table.query(
KeyConditionExpression=Key('item_type').eq(search_term),
ProjectionExpression='item_id'
)
Following are some of the observations:
It takes more time to fetch popular items
As the number of records increase, the response time increases
I have tried Dynamo DB Cache but the Python SDK is not up to the mark and it has certain limitations.
Given these details following are the questions:
Why is the response time so high? Is it because I am querying on a string not a number.
Increasing the read capacity also did not help but why?
Is there any other aws service which is faster than Dynamo DB for such type of queries.
I have seen seminars where they claim to get sub millisecond response times on billions of records with multiple users accessing the table. Any pointers towards achieving sub second response time will be helpful. Thanks.