Related
I have a bit complicated PowerQuery query which has many steps. Within these steps I have a date, team and conditions as below (not actual data)
So the challenge is that I want to count the Pass number for each team for each day and then create another count for Pass and Fail and then it will be used in so many calculations which I can handle later.
I have tried many options, like for example grouping, but it was so confusing because as I mentioned before, the query has so many columns and calculations now. I could successfully solve the issue by creating DAX measure, but the issue here that I need to calculate the average outcome which is not possible to because I couldn't also average the measure of the outcome. So I have no other option but to make the countif though PowerQuery.
Appreciate your help and ideas.
Raw data as text is here in google sheets
I am assuming that your dates showing the year 0203 is a typo and should be 2023
Not sure exactly what you want for output, but you should be able to adapt the below.
The solution seems to be a simple grouping with a count of the number of passes and/or fails.
The below code generates a separate column for passes and fails per team and date.
It is not sorted in the original order, but that could be added if necessary.
#"Grouped Rows" = Table.Group(#"Previous Step", {"Date", "Team"}, {
{"Pass", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Pass")), Int64.Type},
{"Fail", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Fail")), Int64.Type}
})
Using your data table from the Google sheet (after correcting the year):
let
Source = Excel.CurrentWorkbook(){[Name="Table12"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}, {"Team", type text}, {"PASS/FAIL", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Date", "Team"}, {
{"Pass", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Pass")), Int64.Type},
{"Fail", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Fail")), Int64.Type}
})
in
#"Grouped Rows"
Note If you require that all teams show on all dates even if they didn't play, one merely creates a new table containing all dates and teams; and then Joins that to the original table, as in the code below:
let
Source = Excel.CurrentWorkbook(){[Name="Table12"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}, {"Team", type text}, {"PASS/FAIL", type text}}),
//If you need to show all teams on all dates, even if they didn't play on a date
//we merely create a blank table (dates and teams), and execute an outer join.
//then remove the original date/team columns before the grouping.
#"Date List" = List.Distinct(#"Changed Type"[Date]),
#"All Teams" = List.Distinct(#"Changed Type"[Team]),
Blank = Table.FromColumns(
{List.Combine(List.Transform(#"Date List", each List.Repeat({_}, List.Count(#"All Teams")))),
List.Repeat(#"All Teams", List.Count(#"Date List"))},
type table[dates=date, teams=text]),
join = Table.Join(Blank,{"dates","teams"}, #"Changed Type",{"Date","Team"}, JoinKind.LeftOuter),
#"Removed Columns" = Table.RemoveColumns(join,{"Date", "Team"}),
#"Grouped Rows" = Table.Group(#"Removed Columns", {"dates", "teams"}, {
{"Pass", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Pass")), Int64.Type},
{"Fail", (t)=>List.Count(List.Select(t[#"PASS/FAIL"], each _ = "Fail")), Int64.Type}
}),
#"Sorted Rows" = Table.Sort(#"Grouped Rows",{{"dates", Order.Ascending}, {"teams", Order.Ascending}})
in
#"Sorted Rows"
If you need the zeroes in your final output, you'll need to do a cross join to bring in combinations not present in the original.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMtQ31DcyMDJW0lFyDAISAYnFxUqxOigSrn44JFxcgYRbYmYOuoSfD7KEkb4RilEkSeA0Cr8E3LlIEmDnEpYw1jfWNzAywDSKIgmcduCUQI0PJAnU+CDGKNSwotwOiFGxAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Date = _t, Team = _t, #"PASS/FAIL" = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type text}, {"Team", type text}, {"PASS/FAIL", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Date", "Team"}, {{"Count", each Table.RowCount(_), Int64.Type}}),
Custom1 = List.Distinct( #"Grouped Rows"[Date]),
#"Converted to Table" = Table.FromList(Custom1, Splitter.SplitByNothing(), {"Date"}, null, ExtraValues.Error),
#"Added Custom" = Table.AddColumn(#"Converted to Table", "Team", each List.Distinct( #"Grouped Rows"[Team])),
#"Expanded Team" = Table.ExpandListColumn(#"Added Custom", "Team"),
#"Merged Queries" = Table.NestedJoin(#"Expanded Team", {"Date", "Team"}, #"Grouped Rows", {"Date", "Team"}, "Expanded Team", JoinKind.LeftOuter),
#"Expanded Expanded Team" = Table.ExpandTableColumn(#"Merged Queries", "Expanded Team", {"Count"}, {"Count"}),
#"Replaced Value" = Table.ReplaceValue(#"Expanded Expanded Team",null,0,Replacer.ReplaceValue,{"Count"})
in
#"Replaced Value"
Stepping through the code:
Group by date and team and create a count:
Get a distinct list of dates
Convert to table
Add column for a cross join with all teams (to get zero values later)
Expand
Merge back to the grouped step to pull in the previous grouped values.
Replace nulls with 0
You can amend this for your other question by simply filtering for pass before you do the grouping.
Just add column, custom column
= 1
then click select Pass/Fail column, transform .. pivot .. and choose the new column as values column
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each 1),
#"Pivoted Column" = Table.Pivot(#"Added Custom", List.Distinct(#"Added Custom"[PassFail]), "PassFail", "Custom", List.Sum)
in #"Pivoted Column"
I am trying to use fill down function available in power query to replace black cells with previous values.
Below is the sample of data I am working on;
The goal is to repeat values in column Status for respective IDs. Using Fill down would be easy except for the coloured instances as there is no value against those IDs and I would want them blank as there is no value for them.
The desired output is as follows;
Is there is DAX formula which I can use to justify the need?
Truly appreciate your help.
Start point:
Select status column and replace blank with null.
Click ID column and then Group By using following options.
Add a custom column as follows:
Remove first two columns.
Click expand arrows on top right.
Just group before you fill down.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjQyMTVT0lEKLk0qTi7KLCjJzM9T8EstV4rVQUgS4JiYWZhjmOGfk4IiSQrH1MQCIQEUNyGSZ2ZigcsrMEkIx9QcQyHYvbEA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, Status = _t]),
#"Replaced Value" = Table.ReplaceValue(Source,"",null,Replacer.ReplaceValue,{"Status"}),
#"Changed Type" = Table.TransformColumnTypes(#"Replaced Value",{{"ID", Int64.Type}, {"Status", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"ID"}, {{"All", each _, type table [ID=nullable number, Status=nullable text]}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom", each Table.FillDown([All], {"Status"})),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"ID", "All"}),
#"Expanded Custom" = Table.ExpandTableColumn(#"Removed Columns", "Custom", {"ID", "Status"}, {"Custom.ID", "Custom.Status"})
in
#"Expanded Custom"
I'm trying to do web scraping with power bi where I'm using the data from the following site:
https://pt.wikipedia.org/wiki/Jogo_do_bicho
After passing the site URL, the data came organized in the following format:
![Screenshot 1][1]
[1]: https://i.stack.imgur.com/HPjE7.png
where the number is an index related to the animal that has its specific thousand, how do I put everything organized in a column with all indices?
I have an example attached:
![Screenshot 2][2]
[2]: https://i.stack.imgur.com/cxWbU.png
I'll try to add detail later but I think this will work:
let
Source = Web.Page(Web.Contents("https://pt.wikipedia.org/wiki/Jogo_do_bicho")){0}[Data],
ToLists = List.Skip(Table.ToColumns(Source),1),
#"Converted to Table" = Table.FromList(ToLists, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandListColumn(#"Converted to Table", "Column1"),
#"Added Custom" = Table.AddColumn(#"Expanded Column1", "Pivot", each if Text.Length([Column1]) = 2 then "Group" else "Animal"),
#"Added Index" = Table.AddIndexColumn(#"Added Custom", "Index", 0, 1),
#"Integer-Divided Column" = Table.TransformColumns(#"Added Index", {{"Index", each Number.IntegerDivide(_, 2), Int64.Type}}),
#"Pivoted Column" = Table.Pivot(#"Integer-Divided Column", List.Distinct(#"Integer-Divided Column"[Pivot]), "Pivot", "Column1"),
#"Split Column by Delimiter" = Table.SplitColumn(#"Pivoted Column", "Animal", Splitter.SplitTextByDelimiter("#(lf)#(cr)", QuoteStyle.Csv), {"Animal", "Values"}),
#"Trimmed Text" = Table.TransformColumns(#"Split Column by Delimiter",{{"Animal", Text.Trim, type text}, {"Values", Text.Trim, type text}}),
#"Changed Type" = Table.TransformColumnTypes(#"Trimmed Text",{{"Group", Int64.Type}}),
#"Removed Columns" = Table.RemoveColumns(#"Changed Type",{"Index"}),
#"Sorted Rows" = Table.Sort(#"Removed Columns",{{"Group", Order.Ascending}})
in
#"Sorted Rows"
Edit: The key here is to convert the table into a list of columns using Table.ToColumns. This turns it into a list of lists that we can convert into a table and expand into one long column.
Once all of the columns are stacked into one single column, we want to separate the group id from the details, which we can do in this case by checking the length of the text and defining a custom column that labels each row with a different data category.
With that categorization of the rows in place, we want to pivot that new custom column but we want an index column so it knows what stays together. Add an index column and integer divide by two so you get 0,0,1,1,2,2,3,3,... so that each pair gets its own unique ID. Now we can finally pivot.
Once pivoted, do any cleanup you feel like, e.g., splitting columns, trimming whitespace, changing column types, removing unneeded columns, and sorting.
I have a performance monitoring table in which users are listed vertically and then months are listed across the header. These months are dynamically generated on a 12-month rolling window. At the beginning of each month, one month falls off the back of the query and another appears at the front. After beginning of month, I get the following error until I manually re-run the report:
The '<#MONTH>' column does not exist in the rowset.
Where '<#MONTH>' is the month that gets dropped off, e.g. if it is Sept 2019, it would be 'August 2018'.
I tried adding a window to the query that moved the start of the query ahead a day to try to eliminate the perceived race condition. This did not work.
Here is the M query I have currently:
let
Source = US_TOTALS,
#"Appended Query" = Table.SelectRows(Table.Combine({Source, CA_TOTALS}), each [DATELASTFULFILLMENT] >= #"This Year"),
#"Reordered Columns" = Table.ReorderColumns(#"Appended Query",{"SALESMAN", "RegionName", "DATELASTFULFILLMENT", "Total Sales", "Customer", "GROUP"}),
#"Inserted Start of Month" = Table.AddColumn(#"Reordered Columns", "StartOfMonth", each Date.StartOfMonth([DATELASTFULFILLMENT]), type date),
#"Reordered Columns1" = Table.ReorderColumns(#"Inserted Start of Month",{"SALESMAN", "RegionName", "DATELASTFULFILLMENT", "Total Sales", "StartOfMonth", "GROUP", "Customer"}),
#"Removed Columns" = Table.RemoveColumns(#"Reordered Columns1",{"Customer"}),
#"Date One" = Date.AddMonths(Date.StartOfMonth(Date.AddDays(DateTime.Date(DateTime.LocalNow()),1)),1),
#"Date Two" = Date.AddYears(Date.AddMonths(Date.StartOfMonth(Date.AddDays(DateTime.Date(DateTime.LocalNow()),1)),0),-1),
#"Filtered Rows" = Table.SelectRows(#"Removed Columns", each [DATELASTFULFILLMENT] < Date.AddMonths(Date.StartOfMonth(DateTime.Date(DateTime.LocalNow())),1) and [DATELASTFULFILLMENT] >= Date.AddYears(Date.AddMonths(Date.StartOfMonth(DateTime.Date(DateTime.LocalNow())),0),-1)),
//#"Filtered Rows" = Table.SelectRows(#"Removed Columns", each Date.From([DATELASTFULFILLMENT]) < #"Date One" and Date.From([DATELASTFULFILLMENT]) >= #"Date Two"),
#"Grouped Rows" = Table.Group(#"Filtered Rows", {"SALESMAN", "RegionName", "StartOfMonth", "GROUP"}, {{"Total", each List.Sum([Total Sales]), type number}}),
#"Reordered Columns2" = Table.ReorderColumns(#"Grouped Rows",{"SALESMAN", "RegionName", "GROUP", "StartOfMonth", "Total"}),
#"Inserted Month Name" = Table.AddColumn(#"Reordered Columns2", "Month Name", each Date.MonthName([StartOfMonth]), type text),
#"Inserted Year" = Table.AddColumn(#"Inserted Month Name", "Year", each Date.Year([StartOfMonth]), type number),
#"Merged Columns" = Table.CombineColumns(Table.TransformColumnTypes(#"Inserted Year", {{"Year", type text}}, "en-US"),{"Month Name", "Year"},Combiner.CombineTextByDelimiter(" ", QuoteStyle.None),"Month"),
#"Removed Columns2" = Table.RemoveColumns(#"Merged Columns",{"StartOfMonth", "GROUP"}),
#"Grouped Rows1" = Table.Group(#"Removed Columns2", {"SALESMAN", "Month", "RegionName"}, {{"Total", each List.Sum([Total]), type number}}),
#"Sorted Rows" = Table.Sort(#"Grouped Rows1",{{"SALESMAN", Order.Ascending}}),
#"Pivoted Columns" = Table.Pivot(#"Sorted Rows", List.Distinct(#"Sorted Rows"[Month]), "Month", "Total", List.Sum),
Columns = List.RemoveFirstN(Table.ColumnNames(#"Pivoted Columns"),2),
#"Replaced Value" = Table.ReplaceValue(#"Pivoted Columns",null,0,Replacer.ReplaceValue,Columns),
#"Changed Type" = Table.TransformColumnTypes(#"Replaced Value",List.Transform(Columns, each {_, Currency.Type })),
#"Merged Queries" = Table.NestedJoin(#"Changed Type",{"SALESMAN"},USER_MAPPING_COMBINED,{"USERNAME"},"USER_MAPPING_COMBINED",JoinKind.LeftOuter),
#"Expanded USER_MAPPING" = Table.ExpandTableColumn(#"Merged Queries", "USER_MAPPING_COMBINED", {"NAME"}, {"NAME"})
in
#"Expanded USER_MAPPING"
Expected Results:
The query refreshes as normal
Actual Results:
The query errors with The '<#MONTH>' column does not exist in the rowset.
Where '<#MONTH>' is the month that gets dropped off, e.g. if it is Sept 2019, it would be 'August 2018'.
screenshot for reference:
Ok, I'm going to take a second swing at this. If I'm way off base, I apologize. But let me bring out an example to simulate your error.
let
Seed = Number.Mod(Number.Round(Time.Second(DateTime.LocalNow())), 7) + 1,
Source = List.Generate(()=> Seed, each _ < Seed + 6, each _ + 1),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), {"MonthNumbers"}, null, ExtraValues.Error),
#"Added Custom" = Table.AddColumn(#"Converted to Table", "MonthNames", each Date.MonthName(#date(2019, [MonthNumbers], 1))),
#"Pivoted Column" = Table.Pivot(#"Added Custom", List.Distinct(#"Added Custom"[MonthNames]), "MonthNames", "MonthNumbers", List.Sum)
in
#"Pivoted Column"
So, this creates a table that looks like so:
But every single second, the frame shifts by a month. So if you refresh the preview every second, you'll see the frame slide Jan-Jun, Feb-Jul, Mar-Aug... when December is the last month in the window, it skips back to the Jan-Jun. The point is, the columns are changing.
Now you try to load this model. It does not work!
From the time you create the model with one column set, to the time it takes to load the column set, those columns have been changed and so one of the columns isn't there any more. This is like when your month changes. When you go in and load manually, you'll update your model and things will work fine until the next change in columns. But when you're doing it with the scheduled load, that doesn't update the model, it just tries to load the data and runs into this column mismatch.
So, how do we fix it without losing this dynamic naming? Let's look at that pivot... what if we don't do it and leave our power query looking like this?
Now the column names won't change when we load it into the model. We create a matrix visualization like so, and do some refreshes:
No errors, nice dynamic headers.
So, that's the approach that I think you need. I hope it helps.
Edit: Per comment, this answer shows how to deal with a source that comes with changing column names over time, which is not the problem the asker has.
#RyanB is correct. The right approach here is to do your crosstab layout in reports rather than in data model. The right way, in general, to deal with things that change is to reify these as data, rather than as schema.
Original post below
You're looking for the 'Unpivot other columns' transform:
Select the columns whose names do not change.
Use 'Unpivot other columns' transform
Rename columns
Deal with months as a single month column
Make sure this comes before any steps that depend on the changing column names.
Here are two sample queries that are identical in code except for the source, which has differently named columns:
// query1
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("RU85FsQgCL2LdQoWUTiLL8VkJve/QliSTKF8/oK4VsO2tY8fpLyEe1aWKm3fVgvpiF7SBNLZZulQQLrD9LK339I6mAOYpoJBo5tmUOgUYNr7YwfTiUwShA/VyL/wnufhyMRqv1oHRswTJANNBshGAWN63edNkf41j4GJvzseMn67Xw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [ID = _t, Somethingstatic = _t, #"May-2019" = _t, #"Jun-2019" = _t, #"Jul-2019" = _t, #"Aug-2019" = _t]),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"ID", "Somethingstatic"}, "Attribute", "Value"),
#"Renamed Columns" = Table.RenameColumns(#"Unpivoted Other Columns",{{"Attribute", "Month"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"Value", Int64.Type}, {"ID", Int64.Type}})
in
#"Changed Type"
// query 2
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("RU85FsQgCL2LdQoWUTiLL8VkJve/QliSTKF8/oK4VsO2tY8fpLyEe1aWKm3fVgvpiF7SBNLZZulQQLrD9LK339I6mAOYpoJBo5tmUOgUYNr7YwfTiUwShA/VyL/wnufhyMRqv1oHRswTJANNBshGAWN63edNkf41j4GJvzseMn67Xw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [ID = _t, Somethingstatic = _t, #"Jun-2019" = _t, #"Jul-2019" = _t, #"Aug-2019" = _t, #"Sep-2019" = _t]),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"ID", "Somethingstatic"}, "Attribute", "Value"),
#"Renamed Columns" = Table.RenameColumns(#"Unpivoted Other Columns",{{"Attribute", "Month"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"Value", Int64.Type}, {"ID", Int64.Type}})
in
#"Changed Type"
I'm pretty new using queries and I still need to get on the topic properly.. :)
I'm editing a query in Power BI, and I created a new column that shouls show the division of each cell within a column by the total of another column, as %.
I wrote the following: = Table.AddColumn(#"Renamed Columns", "CONTACTED (CVR)", each [CONTACTED]/[LEADS]), but it divide each cell of the column Contacted by each cell of the column Leads.
Is there any way to do it?
Thanks in advance.
Cheers,
AS
In Power Query:
= Table.AddColumn(#"Renamed Columns", "CONTACTED (CVR)", each [CONTACTED]/List.Sum(#"Renamed Columns"[LEADS]))
This should do the trick, or get you going in the right direction.
You're looking for the Table.Group function aka Transform->Group By in the GUI. Detailed info here: https://msdn.microsoft.com/en-us/library/mt260774.aspx .
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Country", type text}, {"Leads", Int64.Type}, {"Contacted", Int64.Type}, {"User", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Country"}, {{"Leads_Total", each List.Sum([Leads]), type number}, {"Contacted_Total", each List.Sum([Contacted]), type number}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Percent", each [Contacted_Total]/[Leads_Total]),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Leads_Total", "Contacted_Total"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"Country", "Country2"}}),
#"Result" = Table.Join( Source , "Country" , #"Renamed Columns" , "Country2" , JoinKind.LeftOuter ),
#"Removed Columns1" = Table.RemoveColumns(Result,{"Country2"})
in
#"Removed Columns1"
Hope it helps.
I think this is what you are asking; you need to make your question a MWE next time.
Drag the value (my value is called "Place") into the toolbar on the right hand side twice.
Right click the second value and select "Quick calc"
Make sure the "Summerize Value By" is Sum.
then select percent of grand total under "Show Value As".
that should do it.
Considering your condition per each country how many contacted out of leads,
that'll do:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
Types = Table.TransformColumnTypes(Source,{{"Country", type text}, {"Leads", Int64.Type}, {"Contacted", Int64.Type}, {"User", type text}}),
AddColumn = Table.AddColumn(Types, "Contacted (CVR)", (x) => x[Contacted]/List.Sum(Table.SelectRows(Types, (y)=> x[Country] = y[Country])[Leads]), Percentage.Type)
in
AddColumn