There is a dataset that I have established with a connection with ODBC. I'm transferring this dataset to the desktop with the import mode. But since I won't be using most of the columns, I deleted the ones I didn't use in the power query editor. Will those I delete still be updated at scheduled refresh times? So can I see any performance improvement for deleting these columns?
My opinion on this is that the power bi still keeps the data in some place and refreshes it. Because I can undo the operations I have done from the power query. But I am not sure that is just my opinion. Can anyone provide any helpful links?
Related
I need help with improving refresh times on a Power BI dashboard with about 20M rows of data and 80 columns pulling from SQL Server. I cannot use Power BI Service in any capacity, this has to load into Power BI Desktop.
My refresh times on the raw data (virtually no transformations in Power Query) are taking about 3-4 hours.
Microsoft recommends incremental refresh to archive my historical data and only refresh the latest changes, but that requires Service and I 100% cannot use it.
Is there any other way to significantly improve my refresh times beyond Service's incremental refresh? If it was under an hour I'd be happy.
What I've tried:
Native Query to leverage the server
reducing column selections
removing all transformations
Splitting tables in Power Query and selectively turning off refresh in the historical tables - as soon as they get stacked/appended Power Query triggers a refresh on all stacked tables regardless of which ones have refresh turned off.
Looked into Power Query PQFL/M code to activate refresh of tables - can't find any method/property to control this in M code.
optimizing the SQL, haven't gotten any significant improvements.
20 million rows should not take that long, especially with no transformations. Something else is going wrong but without access to your data and hardware, it is impossible to say.
One possibility is do an initial data load and then turn off refresh on that query. Add a new query for just the new data (which should be quick) but load the new query to a completely new table. In PBI, you will then have two tables. Create a calculated table in DAX which is a union of your old, non-refreshed data and your new data. Refreshes should be very quick after your first load but obviously you need to think about how it scales as your data grows.
When going through DAX overview documentation on the following https://learn.microsoft.com/en-us/dax/dax-overview,
under the Calculated Column heading, I couldn't understand the highlighted information in the following statement:
"(calculated) Column values are only recalculated if the table or any related table is processed (refresh) or the model is unloaded from memory and then reloaded, like when closing and reopening a Power BI Desktop file."
Does it mean, that whenever we close the Power BI Desktop file, the model is unloaded from memory? does the memory is used like Cache?
Would anyone please help me understand what actually happens in practical world? Thank you for giving your valuable time and support!
The main point of this remark is to distinguish the difference between measures and calculated columns since the distinction is often not clear to report builders.
A measure is dynamically calculated on-demand when the end-user interacts with the report via slicers or any other sort of filtering and interaction.
A calculated column is only calculated when the model is refreshed. As a result, calculated columns cannot be dynamically responsive to the end-user since they have already been computed before the user can interact with the report.
Power BI Desktop is excel on steraloids. It is an application and as with every application when you close it, it unloads from memory. Its great and you can do a lot..
It has the option to refresh (a button) when you press it, it recalculates your calculated columns.
If you want dashboards and reports served, to be shown in other (web) applications. You want to go with Power BI server. It runs in the cloud and you can refresh the data (daily, on the hour or direct) you can set this all up scheduled in the admin tools.
It is logical that only on a refresh, your calculated columns get recalculated. Only in this moment of time your raw data is changing what will have effect on your calculated columns.
I have a power BI dataset that takes its data from a software made by the IT team in my organization.
I was wondering if it was possible for me to "freeze" all the data in the PBI dataset (like, taking a picture of the data for exemple today) and use this dataset for further analysis (I have another power BI file linked to that Power BI dataset). I know the data won't refresh, but it's not important for what I need to do, as I only need to have the past info.
The reason why I need to know if that's possible is that I'm going oversea for one month and won't have access to the original dataset. Downloading all the data into one excel is impossible as it is way to big.
thanks
It sounds like you're after some sort of snapshotting functionality
If you just wanted to keep the file as is, then you can download the pbix and just not refresh it provided its in import mode.
However one approach you could take if you want to continue doing development without worrying about accidentally refreshing is to use a power bi dataflow
You could copy your power query queries to a dataflow. Refresh them all as at today. Then don't refresh the dataflow anymore
You can then point your power bi dataset to your dataflow
https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-create
That way if you wanted to do further transformation of data, you wouldnt be getting new data from the data source (so long as you dont refresh the dataflow)
I’m pretty new to Power BI and am still at the point of assessing whether it will meet our needs.
I’ve got as far as realising that when creating a new report I can either Import tables (I’m using SQL Server) and use a Direct Query.
The particular report I’m trying to report is quite resource intensive. To create the report in TSql requires iterating through hundreds of thousands of rows in multiple tables in a cursor and then storing some data in a temp table which is the output of the query. I’m very concerned about using the Direct Query option for this because of potential performance degradation on the server.
Is it possible in Power BI Desktop to Import the 5 tables that are used in my query, and then somehow write my query against these tables? That way (in-theory) the query wouldn’t be sent directly to our server each time someone views the report.
My question is based on my lack of knowledge of Power Bi so I may be asking something that is completely impossible!!
Thanks in advance for your help
Regards
Dotdev
That's exactly what Import option does. It imports the tables only once (unless you refresh or change your query). The viewer would be looking at the data that was extracted upon import and packaged into the PBIX file rather than a direct connection to the database.
I'm using Power BI Desktop to get data from a local SQL Server instance. I have 60+ queries in the Queries pane. Most of them have 1 million+ rows. For reasons that I don't fully understand I'm frequently prompted to "apply query changes". Each time Power BI decides to do this it starts updating all queries, which is simply time unfeasible. I don't know how many hours it would take because it's so slow I have to cancel the process. I've tried changing several Data Load settings but without results. Is there a workaround for this problem? For example an option to do it incrementally, one query at a time? Prompting to apply query changes to all queries at once as Power BI does seems unworkable.