I have the code below that
Searches the concatenation of the numeric parts in column STR of table Source within column VAL2 of table Codes. For example, for string abc.sec712.fir001.wpqdata, looks for 712001 in column VAL2. (CustomOUT1)
Searches the best match for values in column NUM of table Source within column VAL1 of table Codes. (Step `CustomOUT2)
These two steps are done separated using Table.AddColumn() in step CustomOUT1 and CustomOUT2.
Since both steps are using Table.AddColumn(), I'd like to join the logic of both steps in a single Table.AddColumn() step.
How can I do this?
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("ZY3LCsIwEEX/JesSMsl0kiwFobZqK4WsQhexVXDnC/x9UyVqEAbu5XCY6z1bbGtWMCBCjMmGwrO+72JVwmqjbIKNa2cokQx8YLVysSJaW2qT4LrbxErKAonEdq+X8cJ+5LfDqEHy4+kqBPDH+TKFe/hZ/mqqpFkDEJlWt1Wu4VvDXFs2LtOk+B8dng==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [TRS = _t, NUM = _t, STR = _t]),
Codes = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("XZDNjoMwDITfhXMPdn4cfOymqEUNzQqCWhX1/V9jbbPsYU8gT+absbetq6W0oXanLjKmJN/kCcB3n9PWLfM8LYvMOLmY0MlfcBEgmTrnXK+zzDwQq9OBA0DTxvM5P0eZoaPgfNRXFPsABzc/lYtEIbByfQw97eo85jkrNwJ5APVK9kFelGzemCAhqRelABydct07/a80lHKvg0ouUI8sDtVJXphepykXPQT2jplRXyYAcKbmda0t25k8s3UKIu7WYV0vqOjgKREIQNH0F/0ehq/81jtyiFFzzbuTH7LP/WG5xERgxyDE34Vaa5MtFJHJsD0Amba+Xt9tNU321MJOC++Zt1Ku7SYz8oxkVaU8Rov9/AA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [CODE = _t, VAL1 = _t, VAL2 = _t]),
CustomOUT1
= Table.AddColumn(
Source,
"OUT1",
(i) =>
try
Table.SelectRows(
Codes,
each Text.Middle(i[STR],7,3) & Text.Middle(i[STR],14,3) = [VAL2]
)[CODE]{0}
otherwise
"NOT FOUND"
,type text ),
CustomOUT2 = Table.AddColumn(
CustomOUT1,
"OUT2",
each
if [NUM] = "" then
null
else
let
t = Table.AddIndexColumn(
Table.SelectRows(
Codes,
(x) =>
let
s = List.Sort({x[VAL1], [NUM]}, each Text.Length(_))
in
Text.StartsWith(s{1}, s{0})
),
"Index"
)
in
if Table.RowCount(t) > 0 then
Table.First(
Table.Sort(
t,
(y) =>
Number.BitwiseShiftLeft(
Number.Abs(Text.Length([NUM]) - Text.Length(y[VAL1])),
32
)
+ y[Index]
)
)[CODE]
else
"NOT FOUND"
,type text)
in
CustomOUT2
Tables Source and Codes
Table with both outputs from two AddColumn Steps
Desired output
Easiest?
....
CustomOUT2 = Table.AddColumn(
CustomOUT1,
"OUT",
each
if [NUM] = "" then
[OUT1]
...
then right click and remove column OUT1
let Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("ZY3LCsIwEEX/JesSMsl0kiwFobZqK4WsQhexVXDnC/x9UyVqEAbu5XCY6z1bbGtWMCBCjMmGwrO+72JVwmqjbIKNa2cokQx8YLVysSJaW2qT4LrbxErKAonEdq+X8cJ+5LfDqEHy4+kqBPDH+TKFe/hZ/mqqpFkDEJlWt1Wu4VvDXFs2LtOk+B8dng==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [TRS = _t, NUM = _t, STR = _t]),
Codes = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("XZDNjoMwDITfhXMPdn4cfOymqEUNzQqCWhX1/V9jbbPsYU8gT+absbetq6W0oXanLjKmJN/kCcB3n9PWLfM8LYvMOLmY0MlfcBEgmTrnXK+zzDwQq9OBA0DTxvM5P0eZoaPgfNRXFPsABzc/lYtEIbByfQw97eo85jkrNwJ5APVK9kFelGzemCAhqRelABydct07/a80lHKvg0ouUI8sDtVJXphepykXPQT2jplRXyYAcKbmda0t25k8s3UKIu7WYV0vqOjgKREIQNH0F/0ehq/81jtyiFFzzbuTH7LP/WG5xERgxyDE34Vaa5MtFJHJsD0Amba+Xt9tNU321MJOC++Zt1Ku7SYz8oxkVaU8Rov9/AA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [CODE = _t, VAL1 = _t, VAL2 = _t]),
CustomOUT1
= Table.AddColumn(
Source,
"OUT1",
(i) =>
try
Table.SelectRows(
Codes,
each Text.Middle(i[STR],7,3) & Text.Middle(i[STR],14,3) = [VAL2]
)[CODE]{0}
otherwise
"NOT FOUND"
,type text ),
CustomOUT2 = Table.AddColumn(
CustomOUT1,
"OUT",
each
if [NUM] = "" then
[OUT1]
else
let
t = Table.AddIndexColumn(
Table.SelectRows(
Codes,
(x) =>
let
s = List.Sort({x[VAL1], [NUM]}, each Text.Length(_))
in
Text.StartsWith(s{1}, s{0})
),
"Index"
)
in
if Table.RowCount(t) > 0 then
Table.First(
Table.Sort(
t,
(y) =>
Number.BitwiseShiftLeft(
Number.Abs(Text.Length([NUM]) - Text.Length(y[VAL1])),
32
)
+ y[Index]
)
)[CODE]
else
"NOT FOUND"
,type text),
#"Removed Columns" = Table.RemoveColumns(CustomOUT2,{"OUT1"})
in #"Removed Columns"
I am new to data analysis and I'm wondering if I can get pointers for what I am facing at the moment.
I have an ICS calendar that I am trying to export into a spreadsheet. However, the data I recieve is organised as follows:
Data
Event: NAME XXX
Date: xx xx xx
Location: NOWHERE
URL: www.hi.com
Event: NAME YYY
Date: yy yy yy
Location: SOMEHWERE
URL: www.hello.com
... and so on
I need to be able promote the text before the : delimiter on every four rows as headers. so that my data looks like this:
Event
Date
Location
URL
NAME X
xx xx xx
SOMEHWERE
hello.com
NAME Y
xx xx xx
NOWHERE
bye.com
I can use SQL or Python or data visualisation software such as PowerBI, alternatively, good ol' Excel works fine.
I tried other tools and workarounds such as uploading the ICS calendar into my Outlook calendar and then exporting the calendar. This worked fine but it is a work around.
I would like to be able to load the information via the ICS link directly into a CSV/Excel because I am using the information to populate a PowerBI Dashboard.
This
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45Wci1LzSuxUvBz9HVViIiIUIrViVZySSxJtVKoqIAgsJBPfnJiSWZ+HlClf7iHa5ArWDQ0yMdKoby8XC8jUy85PxcshmxgZGQkkoGVlRCEZmCwv6+rRzimkak5OfkQU2MB", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Data = _t]),
#"Split Column by Delimiter" = Table.SplitColumn(
Source, "Data", Splitter.SplitTextByDelimiter(":", QuoteStyle.Csv), {"Data.1", "Data.2"}),
#"Added Index" = Table.AddIndexColumn(
#"Split Column by Delimiter", "Index", 1, 1, Int64.Type),
#"Added Custom" = Table.AddColumn(
#"Added Index", "Custom", each if Text.Contains([Data.1],"Event") then [Index] else null),
#"Filled Down" = Table.FillDown(
#"Added Custom",{"Custom"}),
#"Removed Columns" = Table.RemoveColumns(
#"Filled Down",{"Index"}),
#"Pivoted Column" = Table.Pivot(
#"Removed Columns", List.Distinct(#"Removed Columns"[Data.1]), "Data.1", "Data.2"),
#"Removed Columns1" = Table.RemoveColumns(
#"Pivoted Column",{"Custom"})
in
#"Removed Columns1"
is how to get from here:
to there:
In powerquery, try this on your sample data set provided above:
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Filtered Rows1" = Table.SelectRows(Source, each [Column1] <> null),
#"Added Index" = Table.AddIndexColumn(#"Filtered Rows1", "Index", 0, 1, Int64.Type),
#"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each if Text.Contains([Column1],"BEGIN:VEVENT") then [Index] else null),
#"Filled Down" = Table.FillDown(#"Added Custom",{"Custom"}),
#"Filtered Rows" = Table.SelectRows(#"Filled Down", each ([Custom] <> null)),
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Filtered Rows", {"Custom"}),
#"Replaced Value" = Table.ReplaceValue(#"Removed Errors","ORGANIZER;CN=","ORGANIZER/CN:",Replacer.ReplaceText,{"Column1"}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Replaced Value", "Column1", Splitter.SplitTextByEachDelimiter({":"}, QuoteStyle.Csv, false), {"Column1.1", "Column1.2"}),
#"Removed Columns" = Table.RemoveColumns(#"Split Column by Delimiter",{"Index"}),
removeHTML1 = Table.TransformColumns(#"Removed Columns",{{"Column1.2",each try Text.Combine(List.Select(List.Alternate(Text.SplitAny(_,"<>"),1,1,1), each _<>""), "") otherwise null, type text}}),
#"Pivoted Column" = Table.Pivot(removeHTML1, List.Distinct(removeHTML1[Column1.1]), "Column1.1", "Column1.2"),
extractEmail = Table.AddColumn(#"Pivoted Column", "email", each List.Last(Text.Split([#"ORGANIZER/CN"],":")))
in extractEmail
I want to add a custom column that extracts the text between the delimiters "~"
This is the input :
This is the output which I am expecting:
I have tried below query, but its not working
=Text.Select([#"Comments"]
{"A".."Z"} & {"1".."10"}&{"~"})
Could you please suggest me
This works in powerquery (M)
It assumes data is coming from Table1, into Column 1
Adjust code if that is not true
It also assumes there is a space between the ending ~ and the next word
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Replaced Value" = Table.ReplaceValue(Source,"~ ","::",Replacer.ReplaceText,{"Column1"}),
#"Adjust edge case" = Table.TransformColumns(#"Replaced Value",{{"Column1", each if Text.End(_,1)="~" then Text.Start(_,Text.Length(_)-1) &"::" else _, type text}}),
#"Added Custom" = Table.AddColumn(#"Adjust edge case", "Custom", each List.Difference(List.Transform(Text.Split([Column1],"~"), each Text.BeforeDelimiter(_,"::")),{""})),
ColumnNames=List.Transform({1..List.Max(List.Transform(#"Added Custom"[Custom], each List.Count(_)))}, each "Data "&Text.From(_)),
#"Added Custom2" = Table.AddColumn(#"Adjust edge case", "Custom", each Text.Combine(List.Skip(List.Transform(Text.Split([Column1],"~"), each Text.BeforeDelimiter(_,"::")),1),"~")),
#"Split Column by Delimiter" = Table.SplitColumn(#"Added Custom2", "Custom", Splitter.SplitTextByDelimiter("~", QuoteStyle.Csv), ColumnNames)
in #"Split Column by Delimiter"
could you please assist solving the following tasks:
F.e. I have data set:
What i need - to create a task with description, which discounts need to be check. It should be in following format though:
SKU within same brand with same discount depth should be merged into 1 row - Check Discount 10% for brand 1 for SKU's: cream & oil.
While others should remain as same rows as they have different discounts within brand:
Check Discount 20% for brand 2 for SKU detergent
Check Discount 15% for brand 2 for SKU tabs.
There is more levels of data, f.e. the task should be within same outlet (if there is x > 1 outlets, task will be multiplied by x according to amount of outlets). But I guess it should be easy further on if I get the method how to do the mentioned above task.
Should be pretty similar to the previous one, but I might be wrong
Monitor & Catalogue columns basically describe which rows can be merged. So the output out of this table should be 2 rows:
Check positioning of 1-oil and 2-tabs on the monitor
Check positioning of 1-cream and 2-detergent on the catalogue
There can be multiple levels of aggregation, i.e. on top of rows with 1's, there can be rows with 2's - meaning they should be merged in separate task as well. 0 in all cases means - don't take.
I understand it might be a little bit overcomplicated, but i'm looking to speed up this process in Power Query as it's currently being done with VBA analyzing each row and finding match positions.
Here's the desired result with input data:
Everything further is simple. I just eliminate brand-sku and group by task.
Thank you!
You guys are making it more complicated than it needs to be. The key here is that you can aggregate text columns when using Group By.
Here's how I'd do the first one:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUUouSk3MBdKGBqpKsToQsfzMHCQRIyA7JbUktSg9Na8EyDZCEi9JTCoGKTUFCsUCAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Brand = _t, SKU = _t, Discount = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Brand", Int64.Type}, {"SKU", type text}, {"Discount", Percentage.Type}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Brand", "Discount"}, {{"SKU", each Text.Combine([SKU],", "), type nullable text}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Result task", each "Check discount " & Number.ToText([Discount], "P0") & " for brand " & Number.ToText([Brand]) & " for SKU: " & [SKU], type text)
in
#"Added Custom"
Result:
Note that I've grouped on Brand and Discount and aggregated the SKU column but combining each row into a list separated by ", " using Text.Combine([SKU],", ") as the aggregating function instead of any of the default options you can choose. I usually pick Max as the aggregation and then replace that function, i.e. List.Max([SKU]), in the formula for that step.
Once you've done that grouping, you just need to string the pieces together in a custom column.
The second one can be done similarly with the added step of concatenating Brand and SKU into one column before grouping.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUUouSk3MBdIGQGyoFKsDEc3PzAHzQeIgMSMgKyW1JLUoPTWvBEU1SKYkMakYoTwWAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Brand = _t, SKU = _t, Monitor = _t, Catalogue = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Brand", Int64.Type}, {"SKU", type text}, {"Monitor", Int64.Type}, {"Catalogue", Int64.Type}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "BrandSKU", each Number.ToText([Brand]) & "-" & [SKU], type text),
#"Grouped Rows" = Table.Group(#"Added Custom", {"Monitor", "Catalogue"}, {{"BrandSKU", each Text.Combine([BrandSKU], ", "), type text}}),
#"Added Custom1" = Table.AddColumn(#"Grouped Rows", "Result task", each "Check placement of " & [BrandSKU] & " on " & (if [Catalogue] = 1 then "Catalogue" else "Monitor"), type text)
in
#"Added Custom1"
Here's the first one - note that the format you requested in the first one (data entered into separate rows within the same cell using alt+enter) isn't supported in powerquery, so I separated the data with commas instead.
Instructions
Add column>add index column
Highlight index columns>transform>pivot>sku as values>advanced options>don't aggregate
Highlight all of the columns to the right>transform>merge columns (choose a separator if you want one, I chose commas)
Transform>Replace ,, with , (may have to do a few times)
Change Brand to text, Discount to %
Add column>custom column formula = "Check discount " & Number.ToText([Discount]*100) & "% for brand " & [Brand] & " for SKU " & Text.Trim([Merged],",")
Before/After
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Brand", Int64.Type}, {"SKU", type text}, {"Discount", type number}}),
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 1, 1),
#"Pivoted Column" = Table.Pivot(Table.TransformColumnTypes(#"Added Index", {{"Index", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Added Index", {{"Index", type text}}, "en-US")[Index]), "Index", "SKU"),
#"Merged Columns" = Table.CombineColumns(#"Pivoted Column",{"1", "2", "3", "4", "5"},Combiner.CombineTextByDelimiter(",", QuoteStyle.None),"Merged"),
#"Changed Type1" = Table.TransformColumnTypes(#"Merged Columns",{{"Brand", type text}, {"Discount", Percentage.Type}}),
#"Added Custom" = Table.AddColumn(#"Changed Type1", "Custom", each "Check discount " & Number.ToText([Discount]*100) & "% for brand " & [Brand] & " for SKU " & Text.Trim([Merged],","))
in
#"Added Custom"
2nd Example Instructions
Note: For this one it is easies to do Monitor and Separator separately. Just filter for a different one each time.
Add column>add index column
Highlight index columns>transform>pivot>sku as values>advanced options>don't aggregate
Filter for Monitor =1
Delete Monitor & Catalogue columns
Merge remaining columns, use - as separator
Transpose
Merge columns using , as separator
Find and replace -- with - (may have to do a couple times)
Custom column> Use the formula ="Check place ment of " & Text.Trim([Merged]) & " on Monitor"
2nd Example Before/After
2nd Example M-Code
let
Source = Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Brand", Int64.Type}, {"SKU", type text}, {"Monitor", Int64.Type}, {"Catalogue", Int64.Type}}),
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1),
#"Pivoted Column" = Table.Pivot(Table.TransformColumnTypes(#"Added Index", {{"Index", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Added Index", {{"Index", type text}}, "en-US")[Index]), "Index", "SKU"),
#"Filtered Rows" = Table.SelectRows(#"Pivoted Column", each ([Monitor] = 1)),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Monitor", "Catalogue"}),
#"Merged Columns" = Table.CombineColumns(Table.TransformColumnTypes(#"Removed Columns", {{"Brand", type text}}, "en-US"),{"Brand", "0", "1", "2", "3"},Combiner.CombineTextByDelimiter("-", QuoteStyle.None),"Merged"),
#"Transposed Table" = Table.Transpose(#"Merged Columns"),
#"Merged Columns1" = Table.CombineColumns(#"Transposed Table",{"Column1", "Column2"},Combiner.CombineTextByDelimiter(", ", QuoteStyle.None),"Merged"),
#"Replaced Value" = Table.ReplaceValue(#"Merged Columns1","--","-",Replacer.ReplaceText,{"Merged"}),
#"Replaced Value1" = Table.ReplaceValue(#"Replaced Value","--","-",Replacer.ReplaceText,{"Merged"}),
#"Added Custom" = Table.AddColumn(#"Replaced Value1", "Custom", each "Check place ment of " & Text.Trim([Merged]) & " on Monitor")
in
#"Added Custom"
Hopefully that gets you started on how to apply PQ to your data! You may have to adjust slightly if your data sets vary.
Thanks for your advices, Hooded 0ne. Gave me the right direction.
I've done only 1st part though, here are some adjustments I made:
Added $ to SKU to find position later to be replaced with "," - now I can clear the delimiters from merge in 1st step via replace ";" with blank, replace "$" with "," and Text.End or Trim the first "," in the row
Added Select Columns to the step after "Pivot columns". There's dynamic list of columns, so I can't hardcode "1,2,3,4,5" like you did
Here's my final code for p1:
#"Add $" = Table.AddColumn(#"Filtered Rows", "SKU_SYMBOL", each "$"&[ROI_LKA_BASE.SKU]),
#"Add Index" = Table.AddIndexColumn(#"Add $", "Index", 0, 1, Int64.Type),
#"Added Custom" = Table.AddColumn(#"Add Index", "TextIndex", each "TASK_"&Number.ToText([Index])),
#"Removed Columns2" = Table.RemoveColumns(#"Added Custom",{"Index", "Shelf start", "Shelf End", "KAM", "Вид Инф. АУ", "Место размещ. АУ", "Адрес", "Attribute", "Value", "ROI_LKA_BASE.Мониторы", "ROI_LKA_BASE.Каталог", "ROI_LKA_BASE.Confirmed with customer", "ROI_LKA_BASE.Confirmed plan", "ROI_LKA_BASE.SKU"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns2", List.Distinct(#"Removed Columns2"[TextIndex]), "TextIndex", "SKU_SYMBOL"),
ColumnsToSelect = List.Select(Table.ColumnNames(#"Pivoted Column"),each Text.Contains(_,"TASK")),
#"Select Pivoted Columns" = Table.SelectColumns(#"Pivoted Column",ColumnsToSelect),
#"Merged Columns" = Table.CombineColumns(#"Pivoted Column",Table.ColumnNames(#"Select Pivoted Columns"),Combiner.CombineTextByDelimiter(";", QuoteStyle.None),"PIVOT_MERGED"),
#"Replaced Value" = Table.ReplaceValue(#"Merged Columns",";","",Replacer.ReplaceText,{"PIVOT_MERGED"}),
#"Replaced Value1" = Table.ReplaceValue(#"Replaced Value","$",",",Replacer.ReplaceText,{"PIVOT_MERGED"}),
#"Added Custom1" = Table.AddColumn(#"Replaced Value1", "TASK", each "Проверить скидку на " & [ROI_LKA_BASE.Бренд] & ": "
& Text.End([PIVOT_MERGED],Text.Length([PIVOT_MERGED])-1)),
#"Grouped Rows" = Table.Group(#"Added Custom1", {"Promo ID", "promotool_code", "ROI_LKA_BASE.Начало акции", "ROI_LKA_BASE.Конец акции", "ROI_LKA_BASE.Chain code", "ROI_LKA_BASE.Описание промо", "ROI_LKA_BASE.Сеть", "TASK"}, {{"Count", each Table.RowCount(_), Int64.Type}}
Will return later with my solution on 2nd part.
Thanks again Hooded 0ne, big help.