My stream analytics job is getting data for last 24 hours
There is a lot of data to look at here, and whilst this worked for a while, its now stopped generating output events
This prevents data being sent to power bi
I only want the last 24 hours of data to be shown in Power BI
How can I do this?
I have tried to reduce the time window, but I dont want to do that as a fix.
SELECT [Timestamp], Broker, Price, COUNT(*)
INTO [powerbi2]
FROM [eventhubinput] TIMESTAMP BY [TimeStamp]
GROUP BY [TimeStamp], Broker, Price, TUMBLINGWINDOW(hh,3)
The query looks correct. There are a couple things that could be happening here:
Your PowerBI account is being throttled (see here for limits on data ingress). If this is occurring, there should be warnings in your job's Activity Log. If this is the case, you may have to decrease the rate of your job's egress and/or upgrade your PowerBI account.
Your job is falling behind your Event hub due to the high rate of ingress. You can check this by looking at the Input Events Backlogged metric in the Portal. If this is the case, scaling your job may help.
If neither of these suggestions help, I'd recommend reaching out to Azure support so the team can take a closer look.
Related
Is there a way to see up-to-date DynamoDB total read and write capacity usage, maybe by day, and ideally by table? I know basically how to calculate it, and I can pull the value on responses, but I’m just barely starting to try the service out to see if it’s feasible to use, and I’d like to just throw a bunch of data and a bunch of queries at it, then see overall how much that’s going to cost, without waiting for the monthly bill.
Is this possible, or would I just need to keep track of all of my individual requests and add the results up?
CloudWatch can show you the ConsumedReadCapacityUnits and ConsumedWriteCapacityUnits metrics per table, and a granularity of up to 1 hour.
Using the console, go to the CloudWatch service, and choose 'Metrics' in the left hand menu.
From there you can choose 'DynamoDB' -> 'Table Metrics' -> and then choose which table you want capacity usage of.
Good Day
A client I am working with wants to use a PowerBI dashboard to display in their call centre with stats pulled from an Azure SQL Database.
Their specific requirement is that the dashboard automaticly refresh every minute between their operating hours (8am - 5pm).
I have been researching this a bit but can't find a definitive answer.
Is it possible for PowerBI to automaticly refresh every 1min?
Is it dependant on the type of license and/or the type of connection (DIRECTQUERY vs IMPORT)
You can set a report to refresh on a direct query source, using the automatic report refresh feature.
https://learn.microsoft.com/en-us/power-bi/create-reports/desktop-automatic-page-refresh
This will allow you to refresh the report every 1 minute or other defined interval. This is report only, not dashboards as it is desktop only.
When publishing to the service you will be limited to a minimum refresh of 30 mins, unless you have a dedicated capacity. You could add an A1 Power BI Embedded SKU and turn it on then off during business hours to reduce the cost. Which would work out around £200 per month.
Other options for importing data would be to set a Logic App or Power Automate task to refresh the dataset using an API call, for a lower level of frequency, say 5 mins. It would be best to optimise your query to return a small amount of pre aggregated data to the dataset.
You can use Power Automate to schedule refresh your dataset more than 48 times a day. You can refresh it every minute with Power Automate, it looks like. I can also see that you may be able to refresh your dataset more frequently than that with other tools.
Refreshing the data with 1 min frequency is not possible in PowerBI. If you are not using powerBI premium than you can schedule upto 8 times in a day, with the minimum gap of 15 minutes. If in case you are using PowerBI premium you are allowed to schedule 48 slots.
If you are not able to compromise with the above restrictions, then it might be worth to look into powerBI reports for streaming datasets. But then again there are some cons to that as well, like they work only with DirectQuery etc etc.
I deployed a Google Cloud Run service running in a docker container. Out of the box, it looks like I get insight into some metrics on the Metrics tab of the service page such as Request count, Request latencies and more. Although it sounds like request count would answer my question, what I am really looking for is insight into adoption so that I can answer "How many visits to my application were there in the past week" or something like that. Is there a way to get insight like that out of the box?
Currently, the Request count metric reports responses/second, so I can see blips that look like "0.05/s", which can give me some insight but it's hard to aggregate.
I've tried using the Monitoring > Metrics explorer as well, but I'm not seeing any data for the metrics I select. I'm considering hooking into Google Analytics from within my application if that seems like the suggested solution. Thank you!
I've realized it's quite difficult to have Metrics Explorer give you a straight answer on "how many requests I received this month". However, it's possible:
Go to Metrics Explorer as you said, choose resource type "Cloud Run Revision" (cloud_run_revision) and you'll see "Request Count" (run.googleapis.com/request_count) metric:
Description: Number of requests reaching the revision. Excludes requests that are not reaching your container instances (e.g. unauthorized requests or when maximum number of instances is reached).
Resource type: cloud_run_revision
Unit: number Kind: Delta Value type: Int64
Then, choose Aggregator: None, and click Show Advanced Options. In the form, choose Aligner: sum (instead of default "Rate" default). You now should be able to see total request count per minute:
Now if you change "Alignment Period" to "10 minutes", you'll see one data point for every 10m (sadly, there seems to be a bug that says X req/s, but that's more like X reqs/10m in this case):
If you collect enough data, you can change "Alignment Period" to "Custom" and set 30 days, then update your timeframe on the top to 1 year and see monthly request count.
This does not show sums of all Alignment Periods (I think that part is up to you to do manually, maybe possible via the API), but it lets you see requests per month. For example, here's a service I've been running for some months and I set alignment period to 7 days, viewing past 6 weeks, so I get 6 data points on weekly request count. Hope this helps.
I have been trying to register an alert on spike of some metrics using Stackdriver. Here's the query and details:
If there a sudden spike and 500s cross 20
If the total number of requests (200s or others) cross 3000 over 5 mins
To achieve [1], I put the aggregation as mean, aligner as mean (sum over aligner doesn't seem to work - I dont understand why). This query works if the average of requests over 5 mins is over 20 (which is the expected behavior). But I am not able to register any single spike which is the requirement.
Again, for [2] the average over a certain duration works but the summation of requests doesn't seem to work.
If there a way of achieving either or both of the requirements.
PS: Please let me know if you need more data or snippets of the dashboard to understand what I have done till now. I will go ahead and add some accordingly.
I do not believe there is aggregation when trying to set up an alert. As an example for [1], please go to
Stackdriver Monitoring
Alerting
Create a policy and add your conditions
Select your Resource Type
Select your metric, condition and threshold = 20
Response_code_class = 500
Save condition
The alerting UI has changed since the previous answer was written. You can now specify aggregations when creating alerting policies. That said, I don't think you want mean; that's going to smooth out your curve which will defeat your intended use case. A simple threshold alert with a short duration (even zero) ought to do it, I think.
For your second case, you ought to be able to compute a five-minute sum and alert on that. If you still can't get it to work, respond here or file a support ticket and we'll see how we can help you.
Aaron Sher, Stackdriver engineer
I am sending data from Stream Analytics to powerbi.
Most of the time it works fine but from time to time I upload large amounts of data for test purposes. I notice that my SU utilization is 100% and get error messages in Stream Analytics saying that I might encounter strange behaviour because of that.
And in most of these cases when I goto powerbi I see my datasets but when I try to explore them I get nothing. The table page is not opened.
Best to connect with the Stream Analytics folks on this. If you've maximized your SA jobs, you might try scaling up your SA implementation. You should also double check how much data you're sending to Power BI and how often. You want to have a tumbling windows of 1 second or greater for most purposes (Power BI updates the dashboards every second) and choose an appropriate amount of data so you don't get throttled on the Power BI side (see restrictions: https://powerbi.microsoft.com/pricing).
HTH,
-Lukasz
PowerBI is currently in preview, and there are still some silent bugs here and there.
Two silent bugs that come to mind are:
PowerBI currently does not support embedded JSON objects. If a single entry goes into a PowerBI that has an embedded object, it will break the entire table.
Table schemas are set in stone. If later you decided you wanted to change your schema (by changing the select statement in Stream Analytics), you have to output to an entirely different table. If two different schemas go into the same table, the table will break and fail to load as you have mentioned.