Powerquery: how to iterate/loop parameters list? - list

How to substitute code below with compact loop?
let
ParametersList = {"CustomerID","FirstName","LastName"},
Source1 = fnCheckId(srcTbl , ParametersList{0}),
Source2 = fnCheckId(Source1, ParametersList{1}),
Source3 = fnCheckId(Source2, ParametersList{2}),
Result = Source3
in
Result
Looping Problems:
It should loop ParametersList
current loop output table should work as input Table for next loop
SIMPLIFIED EXAMPLE DETAILS (Source File):
fnCheckId function example (in real business case much more complex):
(tbl as table, clm as text)=>
let
//tbl = srcTbl, clm = "FirstName",
#"Added Custom" = Table.AddColumn(tbl,"QA "&clm, each if Text.Length(Record.Field(_, clm))>3 then "Ok" else "Nok")
in
#"Added Custom"
Source Table:
Table.FromRows(
{
{1, "Bob", "Smith", "123-4567"},
{2, "Jim", "Brown", "987-6543"},
{3, "Paul", "Wick", "543-7890"}
},
{"CustomerID", "FirstName", "LastName", "Phone"}
)
Estimated Result Table:

With original function, how about
let srcTbl = Table.FromRows(
{
{1, "Bob", "Smith", "123-4567"},
{2, "Jim", "Brown", "987-6543"},
{3, "Paul", "Wick", "543-7890"}
},
{"CustomerID", "FirstName", "LastName", "Phone"}
),
List = {"CustomerID", "FirstName", "LastName"},
#"Unpivoted Only Selected Columns" = Table.Unpivot(srcTbl, List, "Attribute", "Value"),
Source1 = fnCheckId(#"Unpivoted Only Selected Columns","Value"),
#"Removed Columns" = Table.RemoveColumns(Source1,{"Value"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"QA Value", "Value"}}),
#"Change Title" = Table.TransformColumns(#"Renamed Columns",{{"Attribute",each "QA" & _, type text}}),
combined = #"Unpivoted Only Selected Columns" & #"Change Title",
#"Changed Type" = Table.TransformColumnTypes(combined,{{"Attribute", type text}, {"Value", type text}}) ,
#"Pivoted Column" = Table.Pivot(#"Changed Type", List.Distinct(#"Changed Type"[Attribute]), "Attribute", "Value")
in #"Pivoted Column"
or with changed function as below,
let srcTbl = Table.FromRows(
{
{1, "Bob", "Smith", "123-4567"},
{2, "Jim", "Brown", "987-6543"},
{3, "Paul", "Wick", "543-7890"}
},
{"CustomerID", "FirstName", "LastName", "Phone"}
),
List = {"CustomerID", "FirstName", "LastName"},
#"Unpivoted Only Selected Columns" = Table.Unpivot(srcTbl, List, "Attribute", "Value"),
#"Changed Type1" = Table.TransformColumnTypes(#"Unpivoted Only Selected Columns",{{"Value", type text}}),
#"Processed" = Table.TransformColumns(#"Changed Type1",{{"Value",each fnCheckId2(_), type text}}),
Namechange = Table.TransformColumns(Processed,{{"Attribute",each "QA "&_, type text}}),
combined = #"Changed Type1" & Namechange,
#"Pivoted Column" = Table.Pivot(combined, List.Distinct(combined[Attribute]), "Attribute", "Value")
in #"Pivoted Column"
with fnCheckId2
( clm as text)=>
let
z = if Text.Length(clm )>3 then "Ok" else "Nok"
in z

Please try this loop example:
loops ParametersList list
Use fnCheckId function output table as input for next loop
Do NOT require knowledge about looping function fnCheckId
Recursive function Loop_fnCheckId:
(Loop as number, inTbl as table, inPrm as list )=>
try Loop_fnCheckId(Loop-1, fnCheckId(inTbl, inPrm{Loop-1}),inPrm)
otherwise inTbl
code with loop:
let
ParametersList = {"CustomerID","FirstName","LastName"},
cntLoop = List.Count(ParametersList),
Result = Loop_fnCheckId(cntLoop,srcTbl, ParametersList)
in Result
P.S. The question: is it possible to rewrite it using each _ syntax and avoid additional looping function Loop_fnCheckId

Related

How to split string by multiple pairs of delimiters

Is there a way to split string by delimiters if delimiters are pairs enclosing required text?
E.g. If source string is:
[Text1] [Text2] [Text3] ...
I want to have result as
Text1
Text2
Text3
Function Text.BetweenDelimiters picks up only first pair of [].
I guess I need somehow to create a loop. In e.g. Python it is a few lines of code. I cannot figure out how to do it in M.
If something like that s possible do in DAX, that would go too.
You can do it like this:
replace "] [" with ";"
remove "[" and "]"
split by ";"
unpivot columns
let
Source = Table.FromList(
{
"[Text1] [Text2] [Text3]",
"[Text4] [Text5] [Text6]"
},
null, {"Text"}
),
#"Replaced Value" = Table.ReplaceValue(
Source,"] [",";",Replacer.ReplaceText,{"Text"}),
#"Replaced Value1" = Table.ReplaceValue(
#"Replaced Value","[","",Replacer.ReplaceText,{"Text"}),
#"Replaced Value2" = Table.ReplaceValue(
#"Replaced Value1","]","",Replacer.ReplaceText,{"Text"}),
#"Split Column by Delimiter" = Table.SplitColumn(
#"Replaced Value2", "Text",
Splitter.SplitTextByDelimiter(";", QuoteStyle.Csv),
{"Text.1", "Text.2", "Text.3"}
),
#"Changed Type" = Table.TransformColumnTypes(
#"Split Column by Delimiter",
{
{"Text.1", type text},
{"Text.2", type text},
{"Text.3", type text}
}
),
#"Unpivoted Columns" = Table.UnpivotOtherColumns(
#"Changed Type", {}, "Attribute", "Value"),
#"Removed Columns" = Table.RemoveColumns(
#"Unpivoted Columns",{"Attribute"})
in
#"Removed Columns"
The way you put your "sample data" you can even omit step 1 and split by " ".
One way .. split on ] [ then replace the two odd-men odd:
right click .. split column .. by delimeter. Use ] [ and split into rows
right click and replace [ with nothing
right click and replace ] with nothing
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Split Column by Delimiter" = Table.ExpandListColumn(Table.TransformColumns(Source, {{"Column1", Splitter.SplitTextByDelimiter("] [", QuoteStyle.Csv), let itemType = (type nullable text) meta [Serialized.Text = true] in type {itemType}}}), "Column1"),
#"Replaced Value" = Table.ReplaceValue(#"Split Column by Delimiter","[","",Replacer.ReplaceText,{"Column1"}),
#"Replaced Value1" = Table.ReplaceValue(#"Replaced Value","]","",Replacer.ReplaceText,{"Column1"})
in #"Replaced Value1"

How can I populate a column in Power Query (PowerBi) with values from another query based on conditions?

This is the extracted data after some transformation.
This is the final presentation I need.
To illustrate my problem with an example:
In Image 1:
{[Date: 1/11/22, Plant_1: Car, Quantity_1: 2, Plant_2: Boat, Quantity_2: 2, Plant_3: Plane, Quantity_3: 2],
[Date: 1/11/22, Plant_1: Car, Quantity_1: 2, Plant_2: Boat, Quantity_2: 2, Plant_3: Plane, Quantity_3: 2],
[Date: 1/11/22, Plant_1: Car, Quantity_1: 2, Plant_2: Boat, Quantity_2: 2, Plant_3: Plane, Quantity_3: 2],
[Date: 2/11/22, Plant_1: Boat, Quantity_1: 1, Plant_2: Car, Quantity_2: 1, Plant_3: Plane, Quantity_3: 1],
[Date: 2/11/22, Plant_1: Boat, Quantity_1: 1, Plant_2: Car, Quantity_2: 1, Plant_3: Plane, Quantity_3: 1],
[Date: 2/11/22, Plant_1: Boat, Quantity_1: 1, Plant_2: Car, Quantity_2: 1, Plant_3: Plane, Quantity_3: 1]
}
I need to transform this into
{[Date: 1/11/22, Car: 6,Boat: 6, Plane: 6],
[Date: 2/11/22, Car: 3,Boat: 3, Plane: 3],}
I am wondering if there is a way to let cells in Image 2 take the SUM of values in cells of Image 1 under the condition: If the Date is same AND name of Plant is same.
You can see if this works for you in powerquery
method 1: stack the columns using list.accumulate:
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
base_columns=1, groupsof=2,
Combo = List.Transform(List.Split(List.Skip(Table.ColumnNames(Source),base_columns),groupsof), each List.FirstN(Table.ColumnNames(Source),base_columns) & _),
#"Added Custom" =List.Accumulate(
Combo,
#table({"Column1"}, {}),
(state,current)=> state & Table.Skip(Table.DemoteHeaders(Table.SelectColumns(Source, current)),1)
),
#"Changed Type" = Table.TransformColumnTypes(#"Added Custom",{{"Column1", type date}, {"Column3", type number}, {"Column2", type text}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([Column2] <> null)),
#"Pivoted Column" = Table.Pivot(#"Filtered Rows", List.Distinct(#"Filtered Rows"[Column2]), "Column2", "Column3", List.Sum)
in #"Pivoted Column"
method 2: use two pivots in place of the list.accumulate
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
base_columns=1, groupsof=2,
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, List.FirstN(Table.ColumnNames(Source),base_columns), "Attribute", "Value"),
#"Added Index" = Table.AddIndexColumn(#"Unpivoted Other Columns", "Index", 0, 1, Int64.Type),
#"Inserted Integer-Division" = Table.AddColumn(#"Added Index", "Integer-Division", each Number.IntegerDivide([Index], groupsof), Int64.Type),
#"Inserted Modulo" = Table.AddColumn(#"Inserted Integer-Division", "Modulo", each Number.Mod([Index], groupsof), type number),
#"Removed Columns" = Table.RemoveColumns(#"Inserted Modulo",{"Index", "Attribute"}),
#"Pivoted Column" = Table.Pivot(Table.TransformColumnTypes(#"Removed Columns", {{"Modulo", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Removed Columns", {{"Modulo", type text}}, "en-US")[Modulo]), "Modulo", "Value"),
#"Removed Columns1" = Table.RemoveColumns(#"Pivoted Column",{"Integer-Division"}),
#"Pivoted Column1" = Table.Pivot(#"Removed Columns1", List.Distinct(#"Removed Columns1"[#"0"]), "0", "1", List.Sum),
#"Changed Type" = Table.TransformColumnTypes(#"Pivoted Column1",{{"Date", type date}})
in #"Changed Type"
method 3: grouping. Didn't bother to do that one

Power Query. How can I dynamically transform a list of records to columns?

I'm returning some JSON data from an API. There's an ID, a bunch of other fields (not included in this), and most importantly a List. The List contains a number of records (the same number and structure for each row)
I'm trying to map the records to columns rather than having to "Expand to New Rows". Each record in the list contains 3 fields (ID, Value & Text).
This is the current structure:
I would like to transform the list of records to look something like this:
The number of records within the List can change. So today I have 2 records in a List for each ID, but tomorrow there could be 4 records. So I need something dynamic that will add a new column into the table based on each record available in the list.
Any help would be much appreciated
See if this works. Built in sample.
let Source = Table.AddColumn(#table({"ID"}, {{"111"}, {"222"},{"333"}}), "Custom", each List.Repeat({[ID="Field", Value="BOB",Text="ASDSD"]},3)),
#"Added Custom2" = Table.AddColumn(Source, "Custom.2", each Table.FromRecords([Custom])),
#"Added Index" = Table.AddIndexColumn(#"Added Custom2", "Index", 0, 1, Int64.Type),
Names= Table.ColumnNames ( Table.Combine ( #"Added Index"[Custom.2] ) ),
NewNames=List.Transform(Names, each "extracted"&_),
Expand= Table.ExpandTableColumn ( #"Added Index", "Custom.2", Names,NewNames ),
#"Removed Columns" = Table.RemoveColumns(Expand,{"Custom"}),
Unpivot=Table.Unpivot(#"Removed Columns",NewNames,"attribute","value"),
#"Grouped Rows" = Table.Group(Unpivot, {"Index"}, {{"data", each
Table.TransformColumns(
Table.AddIndexColumn(_, "Index2", 1, 1, Int64.Type)
,{{"Index2", each Number.RoundUp(_/ List.Count(NewNames),0), type number}})
, type table }}),
#"Removed Other Columns" = Table.SelectColumns(#"Grouped Rows",{"data"}),
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(Table.Column(#"Removed Other Columns", "data"), each if _ is table then Table.ColumnNames(_) else {}))),
#"Expanded Part2" = Table.ExpandTableColumn(#"Removed Other Columns", "data",ColumnsToExpand ,ColumnsToExpand ),
#"Merged Columns1" = Table.CombineColumns(Table.TransformColumnTypes(#"Expanded Part2", {{"Index2", type text}}, "en-US"),{"Index2", "attribute"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged"),
#"Pivoted Column" = Table.Pivot(#"Merged Columns1", List.Distinct(#"Merged Columns1"[Merged]), "Merged", "value")
in #"Pivoted Column"

PowerQuery: Adding multiple columns with calculation

I have a data model with two tables sharing the same columns.
I have merged the tables using the prefixes "Old." and "New."
I'd like to add a calculated column for each column that shows if the values are different with the name like "Column_IsDifferent" and a boolean value of true or false.
I have already found out that you can add multiple columns by using List.Accumulate. But for some reason my code seems not to work as expected:
= List.Accumulate(List.Select(Table.ColumnNames(#"Extend joined table"), each Text.StartsWith(_, "New")), #"Extend joined table", (state, current) => Table.AddColumn(state, Text.RemoveRange(current, 0, 4) & "_IsDifferent", each Table.Column(state, current) <> Table.Column(state, "Old." & Text.RemoveRange(current, 0, 4)), type logical))
Basically, it takes forever to load data and I don't get an error message...
I suspect there is something wrong with this part:
Table.Column(state, current) <> Table.Column(state, "Old." & Text.RemoveRange(current, 0, 4))
You can try this in powerquery which adds a 3rd column to each 2 column pair showing if they match
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1, Int64.Type),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Added Index", {"Index"}, "Attribute", "Value"),
#"Duplicated Column" = Table.DuplicateColumn(#"Unpivoted Other Columns", "Attribute", "Attribute - Copy"),
#"Split Column by Delimiter" = Table.SplitColumn(#"Duplicated Column", "Attribute - Copy", Splitter.SplitTextByDelimiter(".", QuoteStyle.Csv), {"A1", "A2"}),
#"Grouped Rows" = Table.Group(#"Split Column by Delimiter", {"Index", "A2"}, {{"data", each
let compare = if _{0}[Value] = _{1}[Value] then "match" else "nomatch"
in Table.InsertRows( _,1,{[Index = _{0}[Index], Attribute = "Delta."&_{0}[A2], Value=compare, A1="Delta", A2=_{0}[A2]]})
, type table }}),
#"Expanded data" = Table.ExpandTableColumn(#"Grouped Rows", "data", {"Attribute", "Value" }, {"Attribute", "Value"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded data",{"A2"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns", List.Distinct(#"Removed Columns"[Attribute]), "Attribute", "Value"),
#"Removed Columns1" = Table.RemoveColumns(#"Pivoted Column",{"Index"})
in #"Removed Columns1"
Just stumbled upon something, what do you think about this change:
each Table.Column(state, current) <> Table.Column(state, "Old." & Text.RemoveRange(current, 0, 4))
into:
each Record.Field(_, current) <> Record.Field(_, "Old." & Text.RemoveRange(current, 0, 4))

Transform data in power query grouping data

I have the following table in power query:
I want to group by animals in two columns, one accepted and another rejected, separated by commas:
I tried concatenating the columns animals - status and then using the function group by, obtaining the following result:
The power query code that tried:
let
Origen = Excel.Workbook(File.Contents("C:\Users\user\Desktop\example_animals_status.xlsx"), null, true),
Example_Sheet = Origen{[Item="Example",Kind="Sheet"]}[Data],
#"Encabezados promovidos" = Table.PromoteHeaders(Example_Sheet, [PromoteAllScalars=true]),
#"Tipo cambiado" = Table.TransformColumnTypes(#"Encabezados promovidos",{{"code number", Int64.Type}, {"animal", type text}, {"status", type text}}),
#"Columnas combinadas" = Table.CombineColumns(#"Tipo cambiado",{"animal", "status"},Combiner.CombineTextByDelimiter("-", QuoteStyle.None),"combined"),
#"Tipo cambiado1" = Table.TransformColumnTypes(#"Columnas combinadas",{{"code number", type text}}),
#"Filas agrupadas" = Table.Group(#"Tipo cambiado1", {"code number"}, {{"Recuento", each Text.Combine([combined],";"), type text}}) in #"Filas agrupadas"
But I don't know how to split and create the two columns with the rejected and accepted animals. Thank you in advance
First off, this is Power Query Language (M), not DAX.
Now, you can group by code and status, and combine the distinct animals for each combination;
#"Grouped Rows" = Table.Group(#"Tipo cambiado", {"code number", "status"}, {{"animals", each Text.Combine(List.Sort(List.Distinct(_[animal]),Order.Ascending),", "), type text}}),
Then you can pivot the table on status;
#"Pivoted Status" = Table.Pivot(#"Grouped Rows", List.Distinct(#"Grouped Rows"[status]), "status", "animals")
So your new query becomes:
let
Origen = Excel.Workbook(File.Contents("C:\Users\user\Desktop\example_animals_status.xlsx"), null, true),
Example_Sheet = Origen{[Item="Example",Kind="Sheet"]}[Data],
#"Encabezados promovidos" = Table.PromoteHeaders(Example_Sheet, [PromoteAllScalars=true]),
#"Tipo cambiado" = Table.TransformColumnTypes(#"Encabezados promovidos",{{"code number", Int64.Type}, {"animal", type text}, {"status", type text}}),
#"Grouped Rows" = Table.Group(#"Tipo cambiado", {"code number", "status"}, {{"animals", each Text.Combine(List.Sort(List.Distinct(_[animal]),Order.Ascending),", "), type text}}),
#"Pivoted Status" = Table.Pivot(#"Grouped Rows", List.Distinct(#"Grouped Rows"[status]), "status", "animals")
in
#"Pivoted Status"
Output looks like this: