Convert a table into a function that can act like a Table.SelectRows condition - powerbi

I have a table of Project:
that I would like to filter by the FIELD, OPERATOR, and VALUE columns contained in the Project Group table:
The Power Query M to apply this filter would be:
let
Source = #"Project",
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Projectid", Int64.Type}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each [Projectid] >= 100000 and [Projectid] <= 500000)
in
#"Filtered Rows"
Results (need to remove the error row):
How do I convert the FIELD, OPERATOR, and VALUE columns into a function that can be used as a condition for the SelectRows function?

If you need to do comparisons, might be best to first change the types of the columns (in both tables) that are being compared. Preferably to type number.
The code below assumes that:
the OPERATOR column of Project Group table can only contain: > or < and that these values should be interpreted as >= and <= respectively.
the column in Project table (that needs to be compared) can change and its name will be in the FIELD column of the Project Group. It's assumed that the name matches exactly. If this is not the case, you might need to standardise things (or at least perform a case-insensitive search) to ensure values can be mapped to column names correctly.
Based on the assumptions above, here's one approach:
let
// Dummy table for example purposes
project = Table.FromColumns({
{0..10},
{5..15}
}, type table [projectId = number, name = number]),
// Dummy table for example purposes
projectGroup = Table.FromColumns({
{"projectId", "projectId"},
{">", "<"},
{5, 7}
}, type table [FIELD = text, OPERATOR = text, VALUE = number]),
// Should take in a row from "Project" table and return a boolean
// representing whether said row matches the criteria contained
// within "Project Group" table.
selectorFunc = (projectRow as record) as logical =>
let
shouldKeepProjectRow = Table.MatchesAllRows(projectGroup, (projectGroupRow as record) =>
let
fieldNameToCheck = projectGroupRow[FIELD],
valueFromProjectRow = Record.Field(projectRow, fieldNameToCheck),
compared = if projectGroupRow[OPERATOR] = ">" then
valueFromProjectRow >= projectGroupRow[VALUE]
else
valueFromProjectRow <= projectGroupRow[VALUE]
in compared
)
in shouldKeepProjectRow,
selectedRows = Table.SelectRows(project, selectorFunc)
in
selectedRows
The main function used is Table.MatchesAllRows (https://learn.microsoft.com/en-us/powerquery-m/table-matchesallrows).
Another approach could potentially be: Expression.Evaluate: https://learn.microsoft.com/en-us/powerquery-m/expression-evaluate. However, I've not used it, so I'm not sure whether there are any "gotchas"/implications to be aware of.

Related

Power Query: recursive function to append elements in a table

I am trying to use a recursive function to append values from a list to a table, however the code below only show me the first and second results:
let
Source = {"second", "third", "forth", "fith", "seventh", "eighth"},
Count = List.Count(Source),
Table = Table.FromRecords({[sequence = "first"]}, type table[sequence = text]),
appendTbl = (x as list, n as number, tbl as table) =>
let
appTable = Table.InsertRows(Table, n, {[sequence = Source{n}]}),
Check = if n = (Count-1) then #appendTbl(x, n+1, appTable) else appTable
in
Check,
Result = appendTbl(Source, 0, Table)
in
Result
Can anyone please give me a help? Thanks !
Its kind of hard to tell if you are using the number to designate the spot in the table you want to insert, or the number of times you want to duplicate the array before inserting it into the table
That said, you can combine tables with Table.Combine() after converting the list to a table with Table.FromList(). If you need to append it multiple times then just use List.Repeat on the list. If you need to use the Count variable in your function, you have to send it there appendTbl = (x as list, n as number, tbl as table, count as number) =>
some sample codes that probably don't do exactly what you want
let Source = {"second", "third", "forth", "fith", "seventh", "eighth"},
AppendCount=2, //# times to append the list onto the table
#"Converted to Table" = Table.FromList(List.Repeat(Source,AppendCount), Splitter.SplitByNothing(), null),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table",{{"Column1", "sequence"}}),
Table = Table.FromRecords({[sequence = "first"]}, type table[sequence = text]),
combined= Table.Combine({Table, #"Renamed Columns"})
in combined
or
let Source = {"second", "third", "forth", "fith", "seventh", "eighth"},
Table = Table.FromRecords({[sequence = "first"]}, type table[sequence = text]),
appendTbl = (x as list, n as number, tbl as table) => // append list x to table tbl, n times on column sequence
let #"Converted to Table" = Table.FromList(List.Repeat(x,n), Splitter.SplitByNothing(), null),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table",{{"Column1", "sequence"}}),
combined= Table.Combine({tbl, #"Renamed Columns"})
in combined,
Result = appendTbl(Source, 2, Table) // append Source to Table, 2 times
in Result

Power Query: Table.Group with a dynamic list of columns specifying column type

I have
ttOKLostTypes=Table.Group(#"Pivoted Column", {"Index"}, List.Transform(columnList2, each {_, (grp) => List.Max(Table.Column(grp, _)) })),
However this resets column types. How can I specify column types in the above transformation as here:
#"Grouped Rows" = Table.Group(#"Pivoted Column", {"Index"}, {{"InvoiceDate", each List.Max([InvoiceDate]),type nullable date},....
I know I can find out column types by using
schema=Table.Schema(#"Pivoted Column"),
but I cannot figure out how can I build a proper List with column types to be used in the Table.Group()
You can build a dynamic list of all the aggregations to include the data type, using List.Transform, by just adding the data type to your transformation.
Assuming the data types are all the same:
For example, if your grouping column is "Column1", then
maxCols = List.RemoveItems(Table.ColumnNames(#"Changed Type"),{"Column1"}),
colAggregations =
List.Transform(
maxCols,
(c)=> {c, each List.Max(Table.Column(_,c)),Int64.Type}
),
group = Table.Group(#"Changed Type","Column1", colAggregations)
EDIT
To include the types of the original columns, dynamically, is more difficult. Table.Schema will return the column types as text so they have to be transformed into a Type.
One way to do this is with a custom function.
Custom Function
name it: fnTextToType
I only included a few types. The Field name is a name returned by Table.Schema for a particular type, and the field value is the type. It is hopefully obvious how to extend this function to account for other types
(txt as text) =>
let
typeRecord =
Record.Field(
[Number.Type = Number.Type,
Int64.Type = Int64.Type,
DateTime.Type = DateTime.Type],
txt
)
in
typeRecord
Then you can use it in code like this:
#"Changed Type" = Table.TransformColumnTypes(rem,{{"Column1", Int64.Type}, {"Column2", type number}, {"Column3", Int64.Type}}),
//get list of column types in column order
//note these are returned as text strings and not as "types"
colTypes = Table.Schema(#"Changed Type")[TypeName],
//create list of columns upon which to execute the aggregation (List.Max in this case)
maxCols = List.RemoveItems(Table.ColumnNames(#"Changed Type"),{"Column1"}),
//create list of aggregations
colAggregations =
List.Transform(maxCols,(c)=> {c, each List.Max(Table.Column(_,c)),
fnTextToType(colTypes{List.PositionOf(Table.ColumnNames(#"Changed Type"),c)})}),
//now group them
group = Table.Group(#"Changed Type","Column1", colAggregations)
in
group
You can see how the types were maintained in the screenshots below.
Changed Type
group
Thanks #Ron Rosenfeld. Your answer which works suggested me to find another way using Expression.Evaluate. Evaluate without #shared does not work. See https://blog.crossjoin.co.uk/2015/02/06/expression-evaluate-in-power-querym/
columnList = Table.ColumnNames(#"Pivoted Column"),
columnList2 = List.RemoveItems(columnList,{"Index"}),
ColListWithTypes = List.Transform(columnList2,(colName)=> {colName,Table.SelectRows(schema,each [Name]=colName)[TypeName]{0}}),
ttTestWithTypes=Table.Group(#"Pivoted Column", {"Index"}, List.Transform(ColListWithTypes, each {_{0}, (grp) => List.Max(Table.Column(grp, _{0})),Expression.Evaluate(_{1},#shared)})),

EXPAND MULTIPLE COLUMNS POWER BI

I´ve been struggling with this:
My table shows 3 records but when expanding there are like 100 columns. I used this code:
#"Expanded Data" = Table.ExpandTableColumn(#"Source", "Document", List.Union(List.Transform(#"Source"[Document]), each Table.ColumnNames(_))),
but it's not working. How can I expand simultaneously all columns? Also, inside those columns there are even more, for example I expand the first time end then those new columns have more records inside.
What could I do? Thanks in advance!
Try this ExpandAllRecords function - it recursively expands every Record-type column:
https://gist.github.com/Mike-Honey/0a252edf66c3c486b69b
This should work for Records Columns.
let
ExpandIt = (TableToExpand as table, optional ColumnName as text) =>
let
ListAllColumns = Table.ColumnNames(TableToExpand),
ColumnsTotal = Table.ColumnCount(TableToExpand),
CurrentColumnIndex = if (ColumnName = null) then 0 else List.PositionOf(ListAllColumns, ColumnName),
CurrentColumnName = ListAllColumns{CurrentColumnIndex},
CurrentColumnContent = Table.Column(TableToExpand, CurrentColumnName),
IsExpandable = if List.IsEmpty(List.Distinct(List.Select(CurrentColumnContent, each _ is record))) then false else true,
FieldsToExpand = if IsExpandable then Record.FieldNames(List.First(List.Select(CurrentColumnContent, each _ is record))) else {},
ColumnNewNames = List.Transform(FieldsToExpand, each CurrentColumnName &"."& _),
ExpandedTable = if IsExpandable then Table.ExpandRecordColumn(TableToExpand, CurrentColumnName, FieldsToExpand, ColumnNewNames) else TableToExpand,
NextColumnIndex = CurrentColumnIndex+1,
NextColumnName = ListAllColumns{NextColumnIndex},
OutputTable = if NextColumnIndex > ColumnsTotal-1 then ExpandedTable else #fx_ExpandIt(ExpandedTable, NextColumnName)
in
OutputTable
in
ExpandIt
This basically takes Table to Transform as the main argument,and then one by one checks if the Column Record is expandable (if column has "records" in it, it will expand it, otherwise move to next column and checks it again).
Then it returns the Output table once everything is expanded.
This function is calling the function from inside for each iteration.

Multiple row into one row

I'm new in powerbi and i'm looking some help with a transformation.
What i'm trying to do with powerquery :
first i want to group the following columns : call_key ivr_agent cli dnis lang_id
and after i need to copy the other infos into one row only : all other info need to go on one row.
the second row with same call_key (and others) need to go on a new column.
In few words:
I need that all rows with same call_key are on one row only
File excel test : https://1drv.ms/x/s!AqE6W5akVSvUh59KfGmUiCSnZH6OVg
Thank you so much for your help,
Phil
I couldn't understand exactly if you needed the rows in new columns or just merge them in a single one.
For merging in a single one, try this query:
let
Origen = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
RemoveBlanks = Table.SelectRows(Origen, each [call_key] <> null and [call_key] <> ""),
CombineCols = Table.AddColumn(RemoveBlanks, "MergedCol", each Text.Combine({Text.From([action_time], "es-CO"), [ivr_module], [action_location], [action_type], [action], [action_data1_desc], Text.From([action_data1_value], "es-CO"), [action_data2_desc], [action_data2_value], [action_data3_desc], Text.From([action_data3_value], "es-CO")}, "|"), type text),
RemoveCols = Table.SelectColumns(CombineCols,{"call_key", "ivr_agent", "cli", "dnis", "lang_id", "MergedCol"}),
GroupAndMerge = Table.Group(RemoveCols, {"call_key", "ivr_agent", "cli", "dnis", "lang_id"}, {{"New", each Text.Combine([MergedCol], "#(lf)"), type text}})
in
GroupAndMerge
EDIT: You may split it again, like this:
let
Origen = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
RemoveBlanks = Table.SelectRows(Origen, each [call_key] <> null and [call_key] <> ""),
CombineCols = Table.AddColumn(RemoveBlanks, "MergedCol", each Text.Combine({Text.From([action_time], "es-CO"), [ivr_module], [action_location], [action_type], [action], [action_data1_desc], Text.From([action_data1_value], "es-CO"), [action_data2_desc], [action_data2_value], [action_data3_desc], Text.From([action_data3_value], "es-CO")}, "|"), type text),
RemoveCols = Table.SelectColumns(CombineCols,{"call_key", "ivr_agent", "cli", "dnis", "lang_id", "MergedCol"}),
GroupAndMerge = Table.Group(RemoveCols, {"call_key", "ivr_agent", "cli", "dnis", "lang_id"}, {{"New", each Text.Combine([MergedCol], "#(lf)"), type text}}),
SplitColumn = Table.SplitColumn(GroupAndMerge, "New", Splitter.SplitTextByDelimiter("|", QuoteStyle.Csv), {"New.1", "New.2", "New.3", "New.4", "New.5", "New.6", "New.7", "New.8", "New.9", "New.10", "New.11", "New.12", "New.13", "New.14", "New.15", "New.16", "New.17", "New.18", "New.19", "New.20", "New.21", "New.22", "New.23", "New.24", "New.25", "New.26", "New.27", "New.28", "New.29", "New.30", "New.31", "New.32", "New.33", "New.34", "New.35", "New.36", "New.37", "New.38", "New.39", "New.40", "New.41", "New.42", "New.43", "New.44", "New.45", "New.46", "New.47", "New.48", "New.49", "New.50", "New.51", "New.52", "New.53", "New.54", "New.55", "New.56", "New.57", "New.58", "New.59"})
in
SplitColumn

Power Query conditional sumif

I need to add a column that Sums the value column of all columns that have a common id. However, any id = null is not summed, but equal to the value column.
The above example should result in:
TopPaymendId JournalLineNetAmount TopAmount
fcbcd407-ca26-4ea0-839a-c39767d05403 -3623.98 -7061.23
fcbcd407-ca26-4ea0-839a-c39767d05403 -3437.25 -7061.23
ce77faac-1638-40e9-ad62-be1813ce9031 -88.68 -88.68
531d9bde-3f52-47f3-a9cf-6f3566733af2 -152.23 -152.23
8266dfef-dd14-4654-a6d2-091729defde7 229.42 229.42
f8b97a47-15ef-427d-95e0-ce23cc8efb1f -777 -777
null -3.01 -3.01
null -2.94 -2.94
null 3312.5 3312.5
This code should work:
let
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
group = Table.Group(Source, {"TopPaymendId"}, {"TopAmount", each List.Sum([JournalLineNetAmount])}),
join = Table.Join(Source,{"TopPaymendId"},group,{"TopPaymendId"}),
replace = Table.ReplaceValue(join,each [TopAmount],each if [TopPaymendId] = null
then [JournalLineNetAmount] else [TopAmount],Replacer.ReplaceValue,{"TopAmount"})
in
replace