I'm struggling with a difficult problem about calculate exchange rates. I've two tables, the first one has data about date, currency and exchange rates, the second one has data about date, currency, value in local currency. I can't use RELATED funtion, because I can connect my data only by many to many relationship, so I wanted to use LOOKUPVALUE.
My code is below:
LOOKUPVALUE('ExchangeRates'[ExchangeRate],
'ExchangeRates'[Currency],Data[Currency],
'ExchangeRates'[Date],
LASTDATE(
FILTER(
ALL('ExchangeRates'[Date]),
'ExchangeRates'[Date] <= Data[Date])
)
)
Yes, It works, but LOOKUPVALUE is heavy and very bad for the report'perfomance. Do you know better solutions? I have heard something about TREATAS, but TREATAS is also a performance killer for large amount of data (about 12kk rows)
Best Regards
Related
Should we use the group by function in Power Query and create a new table, or is it better to create as many measures as we need ? (one measure for each column) ?
Which one is more powerful?
Thank you !
It depends on your purpose. If you have a granular fact table that you want to aggregate first before creating the data model, you can do that through Power Query before feeding the model. Even then, I would recommend doing it on the server-side if you are bringing a SQL table; so that you can perform a native SQL group by rather than having to do it through Power Query syntax solely. Power Query has some performance lagging and each nth step in PQ is evaluated from 1st step internally and it requires a full refresh of the table.
However, if you only want to perform group by to be utilized in an analysis, it is always a good idea to use DAX measures and refrain from using PQ. Also, you can't resort to PQ for different analysis scenarios. DAX is built for those scenarios and it is extremely powerful. DAX measures are the most powerful concept of Power BI. Also, they get evaluated in filter context/slicers; i.e. respond to the selection of values in slicers and / or whatever is present in the Axis (business case)
There are tons of supports for DAX measure optimization, such as SQLBI, Stack, Power BI community. If optimized correctly, DAX measures enhance report performance tremendously without creating any lagging in the report at all.
Few resources to look into
1
2
3
When you are creating a new table in power query, it means results are pre calculated and there will be some performance gain if we consider report usage. But, it will increase your Data Model size. Where as Measure will calculate things on the fly. This will keep your model size same but add some slowness in the presentation part. As a whole, there is no specific answer for your question as per my knowledge as it depends on so many other things like-
Your data size
How many measure you wants to create
How complex your logic inside measure's
How often you need reload your data
and so on...
I receive every day 100 rows from an application. Good practices in my company suggest to partition every table by day. I dont think is good to do this on the new table that I will create to daily insert a hundred of rows. I want to partition the data by year, is it good?
How many rows by partition are required for the best performances?
It really also depends on the queries that you are going to execute on this table that is what kind of date filters are going to use and joins on what columns. Refer to below answer which will really help you to decide on this.
Answer1
Answer2
Keep in mind that the number of partitions is limited (to 4000). Therefore partitioning is great for low cardinality. Per day, is perfect (about 11 years -> 4000 days).
If you have higher cardinality, customer ID for example (and I hope you have more than 4000 customers!), clustering is the solution to speed up the request.
When you partition, and cluster, your data, you create small bag. Lesser the data to process (load, read, store in cache (...)) you have, faster will be your query! Of course, on only 100 rows, you won't see any differences
What is the best way to vertically combine two large tables of the same structure. Each table is about 2 mln rows. Is there any performance advantage to do it in M, not in DAX?
M approach
BigTable_M = Table.Combine( {Table1, Table2} )
DAX approach
BigTable_DAX = UNION ( 'Table1', 'Table2' )
I have a feeling that M way loads the tables two times. Separately each primary source (Table1 and Table2) and then again both tables while loading rows to BigTable_M. Is there any reason to suffer this double load for better performance later?
By this article it seems that M is faster.
https://www.sqlbi.com/articles/comparing-dax-calculated-columns-with-power-query-computed-columns/
Best practice would be to do it in M/Power Query first, before loading the data to the data model. You always want the data model to be quick and responsive with as little overhead in calculations. I always recommend working from the lowest level,for example, if you can do it in the source do it there, then if you can't do it there do it in Power Query, and as a last resort do it in the Dax/Power Pivot part.
This works well if you are working with a database, as you let the technology designed to do the heavy lifting/shifting of data, rather then do it all in Power BI.
If you are working with files, then it would be best to do it in the Power Query part were possible, and again let the Power Pivot engine be as quick as possible.
When consulting with clients data models, both Power BI and Analysis services, most of the trouble comes from doing stuff in the data model, rather than doing it before then. For example, data type transformations, string replacement, iterative calculations, ranking etc that would be best placed to do long before it hits the model.
Doing it in the query editor, you can choose to only load the combined table into your data model while having Table1 and Table2 exist merely as staging tables. That should address your concern about loading the tables twice.
I'd expect combining them in M would result in better compression (though the difference might not be very much). Combining in M would also allow for query folding in some situations.
Let's say you had a table.
OrderNumber, OrderDate, City, & Sales
The Sales field is given to you. No need to calculate it.
When you bring in this data into Power BI, say you want to analyze Sales by City (in a table format).
You can just straight away drag the two fields into the table.
No need to create a measure.
So now, suppose you created a measure, though.
Total Sales = Sum(Sales).
Is there any advantage to it, in this scenario?
Is it more efficient to use: City, Total Sales
than it is to use: City, Sales
Both display the same information.
When you drag the field into the table, what Power BI does is create an implicit measure automatically based on its best guess of what aggregation (e.g. sum, max, count) it thinks you want.
So in this case, using an explicitly defined measure or an implicitly generated measure should perform the same since it is doing the same thing in the background, i.e., SUM(TableName[Sales]).
It's generally considered best practice to use explicit measures.
You may be interested in this video discussing the differences.
I was told that it is good to always create explicit measures, and that measures are more efficient. Weather right or wrong, I don't know, but from perspective of policy, it is a good idea, since measures do protect you from column name changes. In general, I think I can just make a rule of thumb to always define any measures that you want to report on explicitly.... BUT the answer above could also be correct... stack exchange doesn't let you choose multiple answers....
I'm trying to create a matrix table in Power BI to display the monthly rent projections for a number of properties. I thought I could simply create a measure that summed the rent from one table and then displayed it by month based on start and end date conditions, but it's been a while since I created any measures and I had forgotten that there needs to be a relationship between columns, among other things.
Data Model
A site can have more than one lease associated with it and a lease can have both car-parks and floors associated with it, of which there can be multiple.
In addition to the tables in the linked image, once I had sorted out what I thought would be the easy step I was going to add another table which includes the estimated percentage rent increase and the period in which the increase will occur.
I started out by trying to create a measure along the lines of the following:
Matrix Test =
IF (
HASONEVALUE ( Period[Month] ),
IF (
Period[Month] >= Leases[Custom Start Date],
SUM ( Floor_Rent[Annual Rent] ) / 12,
0
),
0
)
This would need to be expanded upon because the end date of a lease would also need to be taken into consideration.
As well as forgetting about the relationship requirements, I've forgotten how to deal with the issue of narrowing down to a single value within a column.
The result is supposed to be something that looks like this:
The blanks indicate a lease that starts in the future or ends within the time-frame displayed.
When I try linking the Leases table and the Period table on Leases[Start month for current term] and Period[Month] all I can get to is a table that shows the rent amount in the month the lease starts.
Is what I'm trying to achieve possible? If so, how do I accomplish the desired result?
Link to .pbix file
Solution
The direct answer to the title question is probably 'no', but while trying to figure out how I could use Pratik Bhavsar's LOOKUPVALUE suggestion I had a thought and performed a clumsy google search - power bi create table for each value in column - and found this post. By meddling with some of the DAX in said post I was able to come up with the following:
Test Table =
GENERATE(
SELECTCOLUMNS(
VALUES(Leases[Lease ID]),"Lease ID",[Lease ID]
),
SELECTCOLUMNS(
VALUES(Period[Month]),"Month",[Month]
)
)
The result is a table with each Lease ID mapped against each Month. I can't claim to understand exactly how the functions work, and it's not the outcome I thought I needed, but it allows me to achieve exactly what I set out to do.
I've accepted Pratik Bhavsar's answer because it effectively accomplishes the same thing as the work around I implemented. Pratik's solution might be better than what I eventually landed on, but I need to have a closer look at how the two compare.
The following DAX will give you a table with all buildings mapped against all rows in the period table, eliminating the requirement of a relationship.
SiteToPeriod =
CROSSJOIN(
SELECTCOLUMNS(Sites1, "Building name/label", Sites1[Building name/label]),
Period
)