Power Query Transform data within Group/Buckets - grouping

I want to transform a column in Power query. Only the transformation should be applied within the group based on a condition. This is my data.
Here in the above table, I just want to transform Office based column to all 1 if any Office-based column is set to 1 on the particular ID group. But all the Office based column value is 0 on the particular ID group, it should not transform the column.
My expected result would be,
It would be fine, If an additional column can have the transformed column.

try this
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ID", type text}, {"Item", type text}, {"Home Based", Int64.Type}, {"Office Based", Int64.Type}, {"Amount", Int64.Type}}),
// find all IDs with 1 in Office Based
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([Office Based] = 1)),
#"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"ID"}),
#"Removed Duplicates" = Table.Distinct(#"Removed Other Columns"),
//merge that back in
#"Merged Queries" = Table.NestedJoin(#"Changed Type", {"ID"}, #"Removed Duplicates", {"ID"}, "Table2", JoinKind.LeftOuter),
#"Expanded Table2" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"ID"}, {"ID.1"}),
// if there was a match convert to 1 otherwise take original number
#"Added Custom" = Table.AddColumn(#"Expanded Table2", "OfficeBased2", each try if Text.Length([ID.1])>0 then 1 else [Office Based] otherwise [Office Based]),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Office Based", "ID.1"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"OfficeBased2", "OfficeBased"}})
in #"Renamed Columns"
or the more compact version:
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Custom" = Table.AddColumn(Source,"Custom",(i)=>Table.SelectRows(Source, each [ID]=i[ID]) [Office Based]),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Office Based2", each if List.Contains([Custom],1) then 1 else [Office Based]),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom1",{"Custom", "Office Based"})
in #"Removed Columns"
The first method probably works best for large data sets

Here's another method:
Group by ID
Apply the Table.TransformColumns operation to each subtable
Then re-expand
let
Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ID", type text}, {"Item", type text}, {"Home Based", Int64.Type}, {"Office Based", Int64.Type}, {"Amount", Int64.Type}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"ID"}, {
{"Xform", (t)=>Table.TransformColumns(t, {
{"Office Based", each if List.ContainsAny(t[Office Based],{1}) then 1 else 0}}) ,
type table
[ID=nullable text, Item=nullable text, Home Based=nullable number, Office Based=nullable number, Amount=nullable number]}
}),
#"Expanded Xform" = Table.ExpandTableColumn(#"Grouped Rows", "Xform",
{"Item", "Home Based", "Office Based", "Amount"},
{"Item", "Home Based", "Office Based", "Amount"})
in
#"Expanded Xform"

Related

PBI - Renamed columns - Error 'We cannot convert a Value of type table to type list'

Code
Function
let
FindTables = List.Buffer(TablesAndViews),
Source = Queries,
#"Removed other columns" = Table.SelectColumns(Source, {"Index", "Command Text inc SP"}),
#"Added custom" = Table.AddColumn(#"Removed other columns", "Custom", each Text_ExtractAll([Command Text inc SP], FindTables), type list),
#"Removed columns" = Table.RemoveColumns(#"Added custom", {"Command Text inc SP"}),
#"Expanded Custom" = Table.ExpandListColumn(#"Removed columns", "Custom"),
#"Changed column type" = Table.TransformColumnTypes(#"Expanded Custom", {{"Custom", type text}}),
#"Renamed columns" = Table.RenameColumns(#"Changed column type", {"Custom", "Table Name"},MissingField.Ignore)
in
#"Renamed columns"
Original Code
#"Renamed columns" = Table.RenameColumns(#"Changed column type", {{"Custom", "Table Name"}})
Tried
#"Renamed columns" = Table.RenameColumns(#"Changed column type", {"Custom", "Table Name"},MissingField.Ignore)
Based on
https://learn.microsoft.com/en-us/powerquery-m/table-renamecolumns

Adding random rows within a group in power bi

I would like to know how to create the last column, Audit. To determine whether it is Included or Not Included the sum of any rows in "Average meeting per Day" within the same Employee/Day should match the "Actual Meeting" the employee attended.
I've taken this as an Example dataset -
Further, use Group By and Conditional column to have the Audit added.
Group by -
Conditional Column -
Result -
Code -
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45W8s3PS0msVNJRMgRiAz1TMGkEJBOVYnVwSBtjShuBJQzhZBJYOqQ0tRhZuwlcHqg9=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Day = _t, Emp_ID = _t, #"Actual Meeting" = _t, #"Average Meeting/Day" = _t, Emp_Name = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Day", type text}, {"Emp_ID", Int64.Type}, {"Actual Meeting", type number}, {"Average Meeting/Day", type number}, {"Emp_Name", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Day", "Emp_ID"}, {{"Sum of avg", each List.Sum([#"Average Meeting/Day"]), type nullable number}, {"Actual Meeting", each List.Min([Actual Meeting]), type nullable number}}),
#"Added Conditional Column" = Table.AddColumn(#"Grouped Rows", "Audit", each if [Actual Meeting] = [Sum of avg] then "Included" else "Not Included")
in
#"Added Conditional Column"
Not sure this is going to help, but here goes. Note: not set up to work with duplicate values among the potential items to analyze. No idea if it will work with so many decimals, so that the sum of the decimals might be 0.0000001 off from total and thus not be recognized. Works for me with whole numbers
let
Process=(x as table) as table =>
// Bill Szysz 2017, all combinations of items in list, blows up with too many items to process due to large number of combinations
let ComboList=x[AMD],
find=x[AM]{0},
Source=Table.FromList(List.Transform(ComboList, each Text.From(_))),
AddIndex = Table.AddIndexColumn(Source, "Index", 0, 1),
ReverseIndeks = Table.AddIndexColumn(AddIndex, "RevIdx", Table.RowCount(AddIndex), -1),
Lists = Table.AddColumn(ReverseIndeks, "lists", each
List.Repeat(
List.Combine({
List.Repeat({[Column1]}, Number.Power(2,[RevIdx]-1)),
List.Repeat( {null}, Number.Power(2,[RevIdx]-1))
})
, Number.Power(2, [Index]))
),
ResultTable = Table.FromColumns(Lists[lists]),
#"Replaced Value" = Table.ReplaceValue(ResultTable,null,"0",Replacer.ReplaceValue,Table.ColumnNames(ResultTable )),
#"Added Index" = Table.AddIndexColumn(#"Replaced Value", "Index", 0, 1, Int64.Type),
totals = Table.AddColumn(#"Added Index", "Custom", each List.Sum(List.Transform(Record.ToList( Table.SelectColumns(#"Added Index",Table.ColumnNames(ResultTable )){[Index]}) , each Number.From(_)))),
#"Filtered Rows" = Table.SelectRows(totals, each ([Custom] = find)),
#"Kept First Rows" = Table.FirstN(#"Filtered Rows",1),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Kept First Rows", {"Custom", "Index"}, "Attribute", "Value"),
#"Filtered Rows1" = Table.SelectRows(#"Unpivoted Other Columns", each ([Value] <> "0")),
#"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows1",{"Value"}),
FoundList = Table.TransformColumnTypes(#"Removed Other Columns",{{"Value", type number}}),
#"Merged Queries" = Table.NestedJoin(x, {"AMD"}, FoundList, {"Value"}, "output1", JoinKind.LeftOuter),
#"Expanded output1" = Table.ExpandTableColumn(#"Merged Queries", "output1", {"Value"}, {"Value"}),
#"Calculated Absolute Value" = Table.TransformColumns(#"Expanded output1",{{"Value", each if _=null then "Not Included" else "Included", type text}})
in #"Calculated Absolute Value",
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Day", type text}, {"Employee_ID", Int64.Type}, {"AM", Int64.Type}, {"AMD", Int64.Type}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Day", "Employee_ID"}, {{"data", each _, type table [Day=nullable text, Employee_ID=nullable number, AM=nullable number, AMD=nullable number]}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom", each Process([data])),
#"Removed Other Columns" = Table.SelectColumns(#"Added Custom",{"Custom"}),
#"Expanded Custom" = Table.ExpandTableColumn(#"Removed Other Columns", "Custom", {"Day", "Employee_ID", "AM", "AMD", "Value"}, {"Day", "Employee_ID", "AM", "AMD", "Value"})
in
#"Expanded Custom"

Getting this error: "Expression.Error: The column 't_count' of the table wasn't found. Details: t_count" while executing cummulative sum

Hi I am trying to create a Cummulative sum for "t_count" field in Power BI but everytime I try to run the this formula: List.Sum(List.Range(#"Added Index"[t_count],0,[Hire_Count])) I am getting error that t_count field was not found in the table. Can anyone help me in creating cummulative sum without getting this error? I am sharing full code from advance editor.
let
Source = Excel.Workbook(File.Contents("C:\Users\rabi.jaiswal\Desktop\hr_analytics\EMPLOYEE_ATTRITION_DATA.xlsx"), null, true),
Sheet1_Sheet = Source{[Item="Sheet1",Kind="Sheet"]}[Data],
#"Promoted Headers" = Table.PromoteHeaders(Sheet1_Sheet, [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"Emp_Id", Int64.Type}, {"Emp_Name", type text}, {"email", type text}, {"Join_Date", type date}, {"Joined_As", type text}, {"Department", type text}, {"Manger_Name", type text}, {"Current_Designation", type text}, {"Current_Designation_Start_Date", type date}, {"Work Hours", Int64.Type}, {"Standard_Work_Hours", Int64.Type}, {"Term_Date", type date}}),
#"Removed Columns" = Table.RemoveColumns(#"Changed Type",{"Emp_Id", "Emp_Name", "email", "Joined_As", "Department", "Manger_Name", "Current_Designation", "Current_Designation_Start_Date", "Work Hours", "Standard_Work_Hours"}),
#"Sorted Rows" = Table.Sort(#"Removed Columns",{{"Join_Date", Order.Ascending}}),
#"Added Index" = Table.AddIndexColumn(#"Sorted Rows", "Index", 1, 1, Int64.Type),
#"Renamed Columns" = Table.RenameColumns(#"Added Index",{{"Index", "Hire_Count"}}),
#"Added Conditional Column" = Table.AddColumn(#"Renamed Columns", "t_count", each if [Term_Date] = null then 0 else 1),
#"Added Custom" = Table.AddColumn(#"Added Conditional Column", "Term_total", each List.Sum(List.Range(#"Added Index"[t_count],0,[Hire_Count])))
in
#"Added Custom"
your index starts with one, so try
#"Added Custom" = Table.AddColumn(#"Added Conditional Column", "Term_total", each List.Sum(List.FirstN(#"Added Conditional Column"[t_count],[Index]))),
if your index started from zero, it would be
#"Added Custom" = Table.AddColumn(#"Added Conditional Column", "Term_total", each List.Sum(List.FirstN(#"Added Conditional Column"[t_count],[Index]+1))),

Concatenate strings in Power BI with Query editor

I trying to make function with Advanced Query editor. I make simple function and that function work well.Below you can see how is look like that code:
(page as number) as table =>
let
Source = Web.BrowserContents("https://www.neptun.mk/pocetna/categories/TV_AUDIO_VIDEO/televizori.nspx?items=100&page=" & Number.ToText(page)),
#"Extracted Table From Html" = Html.Table(Source, {{"Column1", ".product-list-item__content\-\-title"}, {"Column2", ".HappyCard"}, {"Column3", ".newPriceModel"}, {"Column4", "[width=""192""]", each [Attributes][src]?}}, [RowSelector=".product-list-item-grid"]),
#"Changed Type" = Table.TransformColumnTypes(#"Extracted Table From Html",{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}, {"Column4", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Custom", each "https://www.neptun.mk"),
#"Merged Columns" = Table.CombineColumns(#"Added Custom",{"Custom", "Column4"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged"),
#"Split Column by Delimiter" = Table.SplitColumn(#"Merged Columns", "Column2", Splitter.SplitTextByDelimiter(" ", QuoteStyle.Csv), {"Column2.1", "Column2.2", "Column2.3", "Column2.4"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"Column2.1", type text}, {"Column2.2", type text}, {"Column2.3", Int64.Type}, {"Column2.4", type text}}),
#"Split Column by Delimiter1" = Table.SplitColumn(#"Changed Type1", "Column3", Splitter.SplitTextByDelimiter(" ", QuoteStyle.Csv), {"Column3.1", "Column3.2", "Column3.3", "Column3.4"}),
#"Changed Type2" = Table.TransformColumnTypes(#"Split Column by Delimiter1",{{"Column3.1", type text}, {"Column3.2", type text}, {"Column3.3", Int64.Type}, {"Column3.4", type text}}),
#"Removed Other Columns" = Table.SelectColumns(#"Changed Type2",{"Column1", "Column2.3", "Column3.3", "Merged"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Other Columns",{{"Column1", "ImageURL"}, {"Column2.3", "RegularPrice"}, {"Column3.3", "DiscountedPrice"}, {"Merged", "Title"}})
in
#"Renamed Columns"
Most important part of this piece of code is third line which begin with `Source =
Web.BrowserContents("https://www.neptun.mk/pocetna/categories/TV_AUDIO_VIDEO/televizori.nspx?items=100&page=" & Number.ToText(page)),` Actually here I want to add some additional string and concatenate with this piece of code.
In purpose to do that first I want to declare one more variable which is call items
(page as number) as table =>
(items as number) as table =>
let
Source = Web.BrowserContents("https://www.neptun.mk/pocetna/categories/TV_AUDIO_VIDEO/televizori.nspx?items=100&page=" & Number.ToText(page)),
But now I have problem how to concatenate variable items into below part of code before part " & Number.ToText(page).I try with this line but is not work well:
Source = Web.BrowserContents("https://www.neptun.mk/pocetna/categories/TV_AUDIO_VIDEO/televizori.nspx?items="& Number.ToText(items),page=" & Number.ToText(page)),
So can anybody help me how to fix this code and concatenate into this line of code ?

Power BI: Total no. of records (S.No) get aggregated on performing union of CSV

I have 40-50 csv files for different cities. I would want to show the number of records across time filtered by city and hashtag.
The 'S.No' got aggregated and is showing the sum of records (I dont understand why did it shows the sum, shouldnt it be individual record numbers when performing Union?) i.e. Even if individual files have S.No from 1-50, the final table should show S.No from 1-2500. but that is not the case here.
Below image shows the data in query editor.
This image shows the table with aggregated sums in 'Data table view'
As we see, the S.No is shown aggregated.
UPDATE:
Pasting the query from Advance Editor:
let
Source = Folder.Files("C:\Users\Desktop\data collection\twitter_new_count"),
#"Filtered Hidden Files1" = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true),
#"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File from twitter_new_count", each #"Transform File from twitter_new_count"([Content])),
#"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}),
#"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "Transform File from twitter_new_count"}),
#"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transform File from twitter_new_count", Table.ColumnNames(#"Transform File from twitter_new_count"(#"Sample File"))),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Table Column1",{{"Source.Name", type text}, {"", Int64.Type}, {"ttext", type text}, {"date", type datetime}, {"isretweet", type logical}, {"retweetcount", Int64.Type}, {"favoritecount", Int64.Type}, {"score", Int64.Type}, {"term", type text}, {"city", type text}, {"country", type text}, {"duplicate", type logical}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each true),
#"Renamed Columns" = Table.RenameColumns(#"Filtered Rows",{{"", "S.No"}}),
#"Inserted Date" = Table.AddColumn(#"Renamed Columns", "Date.1", each DateTime.Date([date]), type date),
#"Inserted Time" = Table.AddColumn(#"Inserted Date", "Time", each DateTime.Time([date]), type time),
#"Renamed Columns2" = Table.RenameColumns(#"Inserted Time",{{"date", "Datentime"}, {"Date.1", "Date"}}),
#"Added Index" = Table.AddIndexColumn(#"Renamed Columns2", "Index", 0, 1),
#"Removed Columns" = Table.RemoveColumns(#"Added Index",{"Index"}),
#"Added Index1" = Table.AddIndexColumn(#"Removed Columns", "Index", 1, 1),
#"Renamed Columns3" = Table.RenameColumns(#"Added Index1",{{"Index", "Row Number"}}),
#"Inserted Hour" = Table.AddColumn(#"Renamed Columns3", "Hour", each Time.Hour([Datentime]), Int64.Type),
#"Filtered Rows1" = Table.SelectRows(#"Inserted Hour", each true),
#"Renamed Columns4" = Table.RenameColumns(#"Filtered Rows1",{{"Hour", "Hour of Day"}})
in
#"Renamed Columns4
I'm pretty sure those S.No values in the data table view are not aggregated, but rather they are just sorted differently than in the query editor view. Check the total number of rows to be sure.