I need help with improving refresh times on a Power BI dashboard with about 20M rows of data and 80 columns pulling from SQL Server. I cannot use Power BI Service in any capacity, this has to load into Power BI Desktop.
My refresh times on the raw data (virtually no transformations in Power Query) are taking about 3-4 hours.
Microsoft recommends incremental refresh to archive my historical data and only refresh the latest changes, but that requires Service and I 100% cannot use it.
Is there any other way to significantly improve my refresh times beyond Service's incremental refresh? If it was under an hour I'd be happy.
What I've tried:
Native Query to leverage the server
reducing column selections
removing all transformations
Splitting tables in Power Query and selectively turning off refresh in the historical tables - as soon as they get stacked/appended Power Query triggers a refresh on all stacked tables regardless of which ones have refresh turned off.
Looked into Power Query PQFL/M code to activate refresh of tables - can't find any method/property to control this in M code.
optimizing the SQL, haven't gotten any significant improvements.
20 million rows should not take that long, especially with no transformations. Something else is going wrong but without access to your data and hardware, it is impossible to say.
One possibility is do an initial data load and then turn off refresh on that query. Add a new query for just the new data (which should be quick) but load the new query to a completely new table. In PBI, you will then have two tables. Create a calculated table in DAX which is a union of your old, non-refreshed data and your new data. Refreshes should be very quick after your first load but obviously you need to think about how it scales as your data grows.
Related
There is a dataset that I have established with a connection with ODBC. I'm transferring this dataset to the desktop with the import mode. But since I won't be using most of the columns, I deleted the ones I didn't use in the power query editor. Will those I delete still be updated at scheduled refresh times? So can I see any performance improvement for deleting these columns?
My opinion on this is that the power bi still keeps the data in some place and refreshes it. Because I can undo the operations I have done from the power query. But I am not sure that is just my opinion. Can anyone provide any helpful links?
I'm really confused about how i can refresh data that i use in my powerBi.
The data come from a server in SQL, at first time i choosed to use DirectQuery as importation mode, after a litle bit of time i observed that its took time when i want to change things like measure or representations.
So i opted to change directquery to importation in order to gain time.
I would like to know how can i get the newest data from my sql server automaticly, because i have line which is added every day, would i use accrementiel refresh?
Just set up a daily refresh schedule in the Power BI Service. If you have big data with a long history you might also benefit from incremental refresh, but that's alread an optimization step.
Are you aware of power bi gate way ?
Keep your dashboards and reports up to date by connecting to your on-premises data sources without the need to move the data. (https://powerbi.microsoft.com/en-us/gateway/)
You just need to install power bi gateway to your server.
Then you can visit power bi service -> settings -> manage gateway . You can setup new gateway here.
Once you create connection you can schedule refresh time with your UTC time.
Note: you need to buy power bi pro or premium licence for that. Pro licence has 8 times refreshes per day and premium has about 48 times per day(premium is very expensive) .
And their is feature called incremental refresh, you have to setup it prom power bi dashboard.
It may take 10-20 minutes to refresh data set.(mentioned in power bi site)
Hi do you using Power BI desktop?
click on your page not visual, on the visualization you will see,option called page refresh. set this On.
set Refresh type to Auto page refresh.
set Refresh this page every the value you want(seconds,minutes,hours,day).
this picture
When going through DAX overview documentation on the following https://learn.microsoft.com/en-us/dax/dax-overview,
under the Calculated Column heading, I couldn't understand the highlighted information in the following statement:
"(calculated) Column values are only recalculated if the table or any related table is processed (refresh) or the model is unloaded from memory and then reloaded, like when closing and reopening a Power BI Desktop file."
Does it mean, that whenever we close the Power BI Desktop file, the model is unloaded from memory? does the memory is used like Cache?
Would anyone please help me understand what actually happens in practical world? Thank you for giving your valuable time and support!
The main point of this remark is to distinguish the difference between measures and calculated columns since the distinction is often not clear to report builders.
A measure is dynamically calculated on-demand when the end-user interacts with the report via slicers or any other sort of filtering and interaction.
A calculated column is only calculated when the model is refreshed. As a result, calculated columns cannot be dynamically responsive to the end-user since they have already been computed before the user can interact with the report.
Power BI Desktop is excel on steraloids. It is an application and as with every application when you close it, it unloads from memory. Its great and you can do a lot..
It has the option to refresh (a button) when you press it, it recalculates your calculated columns.
If you want dashboards and reports served, to be shown in other (web) applications. You want to go with Power BI server. It runs in the cloud and you can refresh the data (daily, on the hour or direct) you can set this all up scheduled in the admin tools.
It is logical that only on a refresh, your calculated columns get recalculated. Only in this moment of time your raw data is changing what will have effect on your calculated columns.
I am looking at retrieving data from an API in Power BI.
I would like to avoid processing all of the data again when I run a refresh. Is there a way to add the new data to the existing data in the dashboard as a delta upload instead of a full refresh every time?
Thanks for any help.
You need to buy Premium for this. Power BI Premium supports Incremental refresh (see announcement and documentation).
With Incremental refresh you define two parameters, RangeStart and RangeEnd, and policies how to refresh the deltas. The values for these parameters are managed by the service. Then first refresh will be slow, because it will load all the data, but next refreshes will only load the data for the period after the last refresh.
I'm using Power BI Desktop to get data from a local SQL Server instance. I have 60+ queries in the Queries pane. Most of them have 1 million+ rows. For reasons that I don't fully understand I'm frequently prompted to "apply query changes". Each time Power BI decides to do this it starts updating all queries, which is simply time unfeasible. I don't know how many hours it would take because it's so slow I have to cancel the process. I've tried changing several Data Load settings but without results. Is there a workaround for this problem? For example an option to do it incrementally, one query at a time? Prompting to apply query changes to all queries at once as Power BI does seems unworkable.