I have a bit complicated PowerQuery query which has many steps. Within these steps I have a date, team and conditions as below (not actual data)
So the challenge is that I want to count the Pass number for each team for each day and then create another count for Pass and Fail and then it will be used in so many calculations which I can handle later.
I have tried many options, like for example grouping, but it was so confusing because as I mentioned before, the query has so many columns and calculations now. I could successfully solve the issue by creating DAX measure, but the issue here that I need to calculate the average outcome which is not possible to because I couldn't also average the measure of the outcome. So I have no other option but to make the countif though PowerQuery.
Appreciate your help and ideas.
Raw data as text is here in google sheets
I am assuming that your dates showing the year 0203 is a typo and should be 2023
Not sure exactly what you want for output, but you should be able to adapt the below.
The solution seems to be a simple grouping with a count of the number of passes and/or fails.
The below code generates a separate column for passes and fails per team and date.
It is not sorted in the original order, but that could be added if necessary.
#"Grouped Rows" = Table.Group(#"Previous Step", {"Date", "Team"}, {
{"Pass", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Pass")), Int64.Type},
{"Fail", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Fail")), Int64.Type}
})
Using your data table from the Google sheet (after correcting the year):
let
Source = Excel.CurrentWorkbook(){[Name="Table12"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}, {"Team", type text}, {"PASS/FAIL", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Date", "Team"}, {
{"Pass", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Pass")), Int64.Type},
{"Fail", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Fail")), Int64.Type}
})
in
#"Grouped Rows"
Note If you require that all teams show on all dates even if they didn't play, one merely creates a new table containing all dates and teams; and then Joins that to the original table, as in the code below:
let
Source = Excel.CurrentWorkbook(){[Name="Table12"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}, {"Team", type text}, {"PASS/FAIL", type text}}),
//If you need to show all teams on all dates, even if they didn't play on a date
//we merely create a blank table (dates and teams), and execute an outer join.
//then remove the original date/team columns before the grouping.
#"Date List" = List.Distinct(#"Changed Type"[Date]),
#"All Teams" = List.Distinct(#"Changed Type"[Team]),
Blank = Table.FromColumns(
{List.Combine(List.Transform(#"Date List", each List.Repeat({_}, List.Count(#"All Teams")))),
List.Repeat(#"All Teams", List.Count(#"Date List"))},
type table[dates=date, teams=text]),
join = Table.Join(Blank,{"dates","teams"}, #"Changed Type",{"Date","Team"}, JoinKind.LeftOuter),
#"Removed Columns" = Table.RemoveColumns(join,{"Date", "Team"}),
#"Grouped Rows" = Table.Group(#"Removed Columns", {"dates", "teams"}, {
{"Pass", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Pass")), Int64.Type},
{"Fail", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Fail")), Int64.Type}
}),
#"Sorted Rows" = Table.Sort(#"Grouped Rows",{{"dates", Order.Ascending}, {"teams", Order.Ascending}})
in
#"Sorted Rows"
If you need the zeroes in your final output, you'll need to do a cross join to bring in combinations not present in the original.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMtQ31DcyMDJW0lFyDAISAYnFxUqxOigSrn44JFxcgYRbYmYOuoSfD7KEkb4RilEkSeA0Cr8E3LlIEmDnEpYw1jfWNzAywDSKIgmcduCUQI0PJAnU+CDGKNSwotwOiFGxAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Date = _t, Team = _t, #"PASS/FAIL" = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type text}, {"Team", type text}, {"PASS/FAIL", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Date", "Team"}, {{"Count", each Table.RowCount(_), Int64.Type}}),
Custom1 = List.Distinct( #"Grouped Rows"[Date]),
#"Converted to Table" = Table.FromList(Custom1, Splitter.SplitByNothing(), {"Date"}, null, ExtraValues.Error),
#"Added Custom" = Table.AddColumn(#"Converted to Table", "Team", each List.Distinct( #"Grouped Rows"[Team])),
#"Expanded Team" = Table.ExpandListColumn(#"Added Custom", "Team"),
#"Merged Queries" = Table.NestedJoin(#"Expanded Team", {"Date", "Team"}, #"Grouped Rows", {"Date", "Team"}, "Expanded Team", JoinKind.LeftOuter),
#"Expanded Expanded Team" = Table.ExpandTableColumn(#"Merged Queries", "Expanded Team", {"Count"}, {"Count"}),
#"Replaced Value" = Table.ReplaceValue(#"Expanded Expanded Team",null,0,Replacer.ReplaceValue,{"Count"})
in
#"Replaced Value"
Stepping through the code:
Group by date and team and create a count:
Get a distinct list of dates
Convert to table
Add column for a cross join with all teams (to get zero values later)
Expand
Merge back to the grouped step to pull in the previous grouped values.
Replace nulls with 0
You can amend this for your other question by simply filtering for pass before you do the grouping.
Just add column, custom column
= 1
then click select Pass/Fail column, transform .. pivot .. and choose the new column as values column
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each 1),
#"Pivoted Column" = Table.Pivot(#"Added Custom", List.Distinct(#"Added Custom"[PassFail]), "PassFail", "Custom", List.Sum)
in #"Pivoted Column"
Related
I am trying to use fill down function available in power query to replace black cells with previous values.
Below is the sample of data I am working on;
The goal is to repeat values in column Status for respective IDs. Using Fill down would be easy except for the coloured instances as there is no value against those IDs and I would want them blank as there is no value for them.
The desired output is as follows;
Is there is DAX formula which I can use to justify the need?
Truly appreciate your help.
Start point:
Select status column and replace blank with null.
Click ID column and then Group By using following options.
Add a custom column as follows:
Remove first two columns.
Click expand arrows on top right.
Just group before you fill down.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjQyMTVT0lEKLk0qTi7KLCjJzM9T8EstV4rVQUgS4JiYWZhjmOGfk4IiSQrH1MQCIQEUNyGSZ2ZigcsrMEkIx9QcQyHYvbEA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, Status = _t]),
#"Replaced Value" = Table.ReplaceValue(Source,"",null,Replacer.ReplaceValue,{"Status"}),
#"Changed Type" = Table.TransformColumnTypes(#"Replaced Value",{{"ID", Int64.Type}, {"Status", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"ID"}, {{"All", each _, type table [ID=nullable number, Status=nullable text]}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom", each Table.FillDown([All], {"Status"})),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"ID", "All"}),
#"Expanded Custom" = Table.ExpandTableColumn(#"Removed Columns", "Custom", {"ID", "Status"}, {"Custom.ID", "Custom.Status"})
in
#"Expanded Custom"
Hi need to add 1000+ calculated columns in power bi which provide the count per entry, for example.*means calculated columns
ID
RankCode
Count_RankCode*
RankAdvance
Count_RanAdvance*
1000
AAA
2
XYZ
2
1001
AAA
2
XYA
1
1002
AAB
1
XYZ
2
found the right way to count in power BI DAX:
COUNTROWS(FILTER('24Jun_1973',[rankCode]=earlier([rankCode])))
Requirement:
add 1000 columns that count rows in probably in one code using DAX
or create the 1000 count cloumn in power query M language (need it to be fast since raw date is 60gb).
As suggested by #smpa01, I was able to complete this task using the tabular editor. Just used the DAX script in tabular editor, put my all my measures in there since I was able to create all expressio in excels as it is just repeating then voila, 1000 measures added.
example:
Measure '24Jun_1973'[measure]=calculate(COUNTROWS(FILTER('28Jun_1973',[rankcode]='28Jun_1973'[rankcode])))
Measure '24Jun_1973'[measure2]=calculate(COUNTROWS(FILTER('28Jun_1973',[rankAdvance]='28Jun_1973'[rankAdvance])))
I have no idea why you would want another 1000 columns.
If you really want to though, in powerquery, you could unpivot, group and count, append the results to original data, then re-pivot. I don't know how fast it would be. I suspect not very.
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ID", Int64.Type}, {"RankCode", type text}, {"RankAdvance", type text}}),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type", {"ID"}, "Attribute", "Value"),
// group and count
#"Grouped Rows" = Table.Group(#"Unpivoted Other Columns", {"Attribute", "Value"}, {{"Count", each Table.RowCount(_), type number}}),
#"Duplicated Column" = Table.DuplicateColumn(#"Grouped Rows", "Attribute", "Attribute - Copy"),
#"Change column name" = Table.TransformColumns(#"Duplicated Column",{{"Attribute - Copy", each "count_" & _, type text}}),
// append back to original table, then repivot
#"Merged Queries" = Table.NestedJoin(#"Unpivoted Other Columns",{"Attribute", "Value"},#"Change column name",{"Attribute", "Value"},"Table2",JoinKind.LeftOuter),
#"Expanded Table1" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"Count", "Attribute - Copy"}, {"Count", "Attribute - Copy"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Table1",{"Attribute", "Value"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"Count", "Value"}, {"Attribute - Copy", "Attribute"}}),
combined = #"Unpivoted Other Columns" & #"Renamed Columns",
#"Pivoted Column" = Table.Pivot(combined, List.Distinct(combined[Attribute]), "Attribute", "Value")
in #"Pivoted Column"
I want to move the data from one column which has distinct values in rows but the other column has same values in rows as shown in the pitcure.
I am new to power BI, can someone please help on this?
Thank you in advance
Req
You can create the following two measures - using CONCATENATEX()
Col_1 = CONCATENATEX(Your_Table,Your_Table[Col1]&";")
Col_2 = CONCATENATEX(Your_Table,Your_Table[Col2]&";")
In Power BI, you can perform some Transformation in Power Query Editor to achieve your required output.
Let, your table structure and sample data looks as below-
Now, go to Advanced Editor for your table and replace the code with this below code-
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("PcdJDoAgEETRu/TaSyCoKDjPEu5/Dbs0qcVP/ktJjBTitENy8avSTqrWLqrRbsprz6dSr9WMITuwJANoyQg6sgcrcgDrj1Z3BBtyAj05gy25gB25goHcwEjuYC85vw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [col1 = _t, col2 = _t, col3 = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"col1", type text}, {"col2", type text}, {"col3", type text}}),
//--New Transformation starts from here
#"Grouped Rows" = Table.Group(#"Changed Type", {"col1"}, {{"grouped", each _, type table [col1=nullable text, col2=nullable text, col3=nullable text]}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom", each [grouped][col2]),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Custom.1", each [grouped][col3]),
#"Extracted Values" = Table.TransformColumns(#"Added Custom1", {"Custom", each Text.Combine(List.Transform(_, Text.From), ";"), type text}),
#"Extracted Values1" = Table.TransformColumns(#"Extracted Values", {"Custom.1", each Text.Combine(List.Transform(_, Text.From), ";"), type text}),
#"Removed Columns" = Table.RemoveColumns(#"Extracted Values1",{"grouped"})
in
#"Removed Columns"
Here is the final output-
I have a performance monitoring table in which users are listed vertically and then months are listed across the header. These months are dynamically generated on a 12-month rolling window. At the beginning of each month, one month falls off the back of the query and another appears at the front. After beginning of month, I get the following error until I manually re-run the report:
The '<#MONTH>' column does not exist in the rowset.
Where '<#MONTH>' is the month that gets dropped off, e.g. if it is Sept 2019, it would be 'August 2018'.
I tried adding a window to the query that moved the start of the query ahead a day to try to eliminate the perceived race condition. This did not work.
Here is the M query I have currently:
let
Source = US_TOTALS,
#"Appended Query" = Table.SelectRows(Table.Combine({Source, CA_TOTALS}), each [DATELASTFULFILLMENT] >= #"This Year"),
#"Reordered Columns" = Table.ReorderColumns(#"Appended Query",{"SALESMAN", "RegionName", "DATELASTFULFILLMENT", "Total Sales", "Customer", "GROUP"}),
#"Inserted Start of Month" = Table.AddColumn(#"Reordered Columns", "StartOfMonth", each Date.StartOfMonth([DATELASTFULFILLMENT]), type date),
#"Reordered Columns1" = Table.ReorderColumns(#"Inserted Start of Month",{"SALESMAN", "RegionName", "DATELASTFULFILLMENT", "Total Sales", "StartOfMonth", "GROUP", "Customer"}),
#"Removed Columns" = Table.RemoveColumns(#"Reordered Columns1",{"Customer"}),
#"Date One" = Date.AddMonths(Date.StartOfMonth(Date.AddDays(DateTime.Date(DateTime.LocalNow()),1)),1),
#"Date Two" = Date.AddYears(Date.AddMonths(Date.StartOfMonth(Date.AddDays(DateTime.Date(DateTime.LocalNow()),1)),0),-1),
#"Filtered Rows" = Table.SelectRows(#"Removed Columns", each [DATELASTFULFILLMENT] < Date.AddMonths(Date.StartOfMonth(DateTime.Date(DateTime.LocalNow())),1) and [DATELASTFULFILLMENT] >= Date.AddYears(Date.AddMonths(Date.StartOfMonth(DateTime.Date(DateTime.LocalNow())),0),-1)),
//#"Filtered Rows" = Table.SelectRows(#"Removed Columns", each Date.From([DATELASTFULFILLMENT]) < #"Date One" and Date.From([DATELASTFULFILLMENT]) >= #"Date Two"),
#"Grouped Rows" = Table.Group(#"Filtered Rows", {"SALESMAN", "RegionName", "StartOfMonth", "GROUP"}, {{"Total", each List.Sum([Total Sales]), type number}}),
#"Reordered Columns2" = Table.ReorderColumns(#"Grouped Rows",{"SALESMAN", "RegionName", "GROUP", "StartOfMonth", "Total"}),
#"Inserted Month Name" = Table.AddColumn(#"Reordered Columns2", "Month Name", each Date.MonthName([StartOfMonth]), type text),
#"Inserted Year" = Table.AddColumn(#"Inserted Month Name", "Year", each Date.Year([StartOfMonth]), type number),
#"Merged Columns" = Table.CombineColumns(Table.TransformColumnTypes(#"Inserted Year", {{"Year", type text}}, "en-US"),{"Month Name", "Year"},Combiner.CombineTextByDelimiter(" ", QuoteStyle.None),"Month"),
#"Removed Columns2" = Table.RemoveColumns(#"Merged Columns",{"StartOfMonth", "GROUP"}),
#"Grouped Rows1" = Table.Group(#"Removed Columns2", {"SALESMAN", "Month", "RegionName"}, {{"Total", each List.Sum([Total]), type number}}),
#"Sorted Rows" = Table.Sort(#"Grouped Rows1",{{"SALESMAN", Order.Ascending}}),
#"Pivoted Columns" = Table.Pivot(#"Sorted Rows", List.Distinct(#"Sorted Rows"[Month]), "Month", "Total", List.Sum),
Columns = List.RemoveFirstN(Table.ColumnNames(#"Pivoted Columns"),2),
#"Replaced Value" = Table.ReplaceValue(#"Pivoted Columns",null,0,Replacer.ReplaceValue,Columns),
#"Changed Type" = Table.TransformColumnTypes(#"Replaced Value",List.Transform(Columns, each {_, Currency.Type })),
#"Merged Queries" = Table.NestedJoin(#"Changed Type",{"SALESMAN"},USER_MAPPING_COMBINED,{"USERNAME"},"USER_MAPPING_COMBINED",JoinKind.LeftOuter),
#"Expanded USER_MAPPING" = Table.ExpandTableColumn(#"Merged Queries", "USER_MAPPING_COMBINED", {"NAME"}, {"NAME"})
in
#"Expanded USER_MAPPING"
Expected Results:
The query refreshes as normal
Actual Results:
The query errors with The '<#MONTH>' column does not exist in the rowset.
Where '<#MONTH>' is the month that gets dropped off, e.g. if it is Sept 2019, it would be 'August 2018'.
screenshot for reference:
Ok, I'm going to take a second swing at this. If I'm way off base, I apologize. But let me bring out an example to simulate your error.
let
Seed = Number.Mod(Number.Round(Time.Second(DateTime.LocalNow())), 7) + 1,
Source = List.Generate(()=> Seed, each _ < Seed + 6, each _ + 1),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), {"MonthNumbers"}, null, ExtraValues.Error),
#"Added Custom" = Table.AddColumn(#"Converted to Table", "MonthNames", each Date.MonthName(#date(2019, [MonthNumbers], 1))),
#"Pivoted Column" = Table.Pivot(#"Added Custom", List.Distinct(#"Added Custom"[MonthNames]), "MonthNames", "MonthNumbers", List.Sum)
in
#"Pivoted Column"
So, this creates a table that looks like so:
But every single second, the frame shifts by a month. So if you refresh the preview every second, you'll see the frame slide Jan-Jun, Feb-Jul, Mar-Aug... when December is the last month in the window, it skips back to the Jan-Jun. The point is, the columns are changing.
Now you try to load this model. It does not work!
From the time you create the model with one column set, to the time it takes to load the column set, those columns have been changed and so one of the columns isn't there any more. This is like when your month changes. When you go in and load manually, you'll update your model and things will work fine until the next change in columns. But when you're doing it with the scheduled load, that doesn't update the model, it just tries to load the data and runs into this column mismatch.
So, how do we fix it without losing this dynamic naming? Let's look at that pivot... what if we don't do it and leave our power query looking like this?
Now the column names won't change when we load it into the model. We create a matrix visualization like so, and do some refreshes:
No errors, nice dynamic headers.
So, that's the approach that I think you need. I hope it helps.
Edit: Per comment, this answer shows how to deal with a source that comes with changing column names over time, which is not the problem the asker has.
#RyanB is correct. The right approach here is to do your crosstab layout in reports rather than in data model. The right way, in general, to deal with things that change is to reify these as data, rather than as schema.
Original post below
You're looking for the 'Unpivot other columns' transform:
Select the columns whose names do not change.
Use 'Unpivot other columns' transform
Rename columns
Deal with months as a single month column
Make sure this comes before any steps that depend on the changing column names.
Here are two sample queries that are identical in code except for the source, which has differently named columns:
// query1
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("RU85FsQgCL2LdQoWUTiLL8VkJve/QliSTKF8/oK4VsO2tY8fpLyEe1aWKm3fVgvpiF7SBNLZZulQQLrD9LK339I6mAOYpoJBo5tmUOgUYNr7YwfTiUwShA/VyL/wnufhyMRqv1oHRswTJANNBshGAWN63edNkf41j4GJvzseMn67Xw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [ID = _t, Somethingstatic = _t, #"May-2019" = _t, #"Jun-2019" = _t, #"Jul-2019" = _t, #"Aug-2019" = _t]),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"ID", "Somethingstatic"}, "Attribute", "Value"),
#"Renamed Columns" = Table.RenameColumns(#"Unpivoted Other Columns",{{"Attribute", "Month"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"Value", Int64.Type}, {"ID", Int64.Type}})
in
#"Changed Type"
// query 2
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("RU85FsQgCL2LdQoWUTiLL8VkJve/QliSTKF8/oK4VsO2tY8fpLyEe1aWKm3fVgvpiF7SBNLZZulQQLrD9LK339I6mAOYpoJBo5tmUOgUYNr7YwfTiUwShA/VyL/wnufhyMRqv1oHRswTJANNBshGAWN63edNkf41j4GJvzseMn67Xw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [ID = _t, Somethingstatic = _t, #"Jun-2019" = _t, #"Jul-2019" = _t, #"Aug-2019" = _t, #"Sep-2019" = _t]),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"ID", "Somethingstatic"}, "Attribute", "Value"),
#"Renamed Columns" = Table.RenameColumns(#"Unpivoted Other Columns",{{"Attribute", "Month"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"Value", Int64.Type}, {"ID", Int64.Type}})
in
#"Changed Type"
My question is, is there a way to get the first/earliest date per grouping in a table and then filter the table to only include rows within the first x number of months of that first date, per grouping. Probably easiest to ask with example. Say I have the following table, and want to keep data for first 6 months of each Group:
The resulting table would look like:
Is there a way to accomplish this with DAX or M?
This seems to work:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Group", type text}, {"Date", type date}, {"Quantity", Int64.Type}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Group"}, {{"AllData", each _, type table}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom", each let DateThisRow = [AllData][Date] in Table.SelectRows([AllData],each [Date] <= Date.AddMonths(List.Min(DateThisRow),6))),
#"Removed Other Columns" = Table.SelectColumns(#"Added Custom",{"Custom"}),
#"Expanded Custom" = Table.ExpandTableColumn(#"Removed Other Columns", "Custom", {"Group", "Date", "Quantity"}, {"Group", "Date", "Quantity"})
in
#"Expanded Custom"
Starting from your original table setup, named as Table1 in Excel:
...it gives me this as an end result:
The number 6 near the end of the #"Added Custom" line in the M code above is the number of months.