I am using a relatively new version of Power BI for Report Server.
I have this simplified data:
ImaginaryData =
DATATABLE (
"Fruit", STRING,
"Colour", STRING,
"Amount", INTEGER,
{
{ "Apple", "Red", 10 },
{ "Apple", "Green", 5 },
{ "Apple", "Blue", 17 },
{ "Pear", "Red", 100 },
{ "Pear", "Green", 65 },
{ "Pear", "Blue", 5 },
{ "Orange", "Red", 12 },
{ "Orange", "Green", 8 },
{ "Orange", "Blue", 38 }
} )
I then create a Matrix of the data:
I want to order this Matrix by amounts in the Blue column i.e. the Fruit Orange should be at the top of the list.
With a Table visual I hover over a column header and can then order by that column but that functionality does not appear for a Matrix.
How do I workaround this issue? [using a Table is not a solution]
You need to add a sortcolumn to your table, that holds the sum of the blue amounts for that particular fruit. Like this.
You cannot do this in DAX (calculated column), because then you can't sort [Fruit] by [Sort], because [Sort] is allready indirectly sorted by [Fruit].
So you need to use the Query-editor. I recreated your table in an excel-workbook to import it. Then I used the following M script
let
SortColour = "Blue",
Source = Excel.Workbook(File.Contents("C:\Users\XXXXX\Documents\ImaginaryData.xlsx"), null, true),
ImaginaryData_Table = Source{[Item="ImaginaryData",Kind="Table"]}[Data],
#"Changed Type" = Table.TransformColumnTypes(ImaginaryData_Table,{{"Fruit", type text}, {"Colour", type text}, {"Amount", Int64.Type}}),
JoinTable = Table.SelectRows(Table.Group(#"Changed Type", {"Fruit", "Colour"}, {{"Count", each List.Sum([Amount]), Int64.Type}}), each ([Colour] = SortColour)),
#"Merged Queries" = Table.NestedJoin(#"Changed Type",{"Fruit"},JoinTable,{"Fruit"},"SortTable",JoinKind.LeftOuter),
#"Expanded SortTable" = Table.ExpandTableColumn(#"Merged Queries", "SortTable", {"Count"}, {"Sort"})
in
#"Expanded SortTable"
After loading this query, you can sort [Fruit] by [Sort] in the Data view (Sort by Column on the Modeling tab). Then recreate the matrix visual and sort the Fruit Column descending by clicking the triangle in the visual.
When you add row { Pear / Blue / 50 } to the table in excel and refresh in PowerBI, the matrix changes to this:
Related
How to substitute code below with compact loop?
let
ParametersList = {"CustomerID","FirstName","LastName"},
Source1 = fnCheckId(srcTbl , ParametersList{0}),
Source2 = fnCheckId(Source1, ParametersList{1}),
Source3 = fnCheckId(Source2, ParametersList{2}),
Result = Source3
in
Result
Looping Problems:
It should loop ParametersList
current loop output table should work as input Table for next loop
SIMPLIFIED EXAMPLE DETAILS (Source File):
fnCheckId function example (in real business case much more complex):
(tbl as table, clm as text)=>
let
//tbl = srcTbl, clm = "FirstName",
#"Added Custom" = Table.AddColumn(tbl,"QA "&clm, each if Text.Length(Record.Field(_, clm))>3 then "Ok" else "Nok")
in
#"Added Custom"
Source Table:
Table.FromRows(
{
{1, "Bob", "Smith", "123-4567"},
{2, "Jim", "Brown", "987-6543"},
{3, "Paul", "Wick", "543-7890"}
},
{"CustomerID", "FirstName", "LastName", "Phone"}
)
Estimated Result Table:
With original function, how about
let srcTbl = Table.FromRows(
{
{1, "Bob", "Smith", "123-4567"},
{2, "Jim", "Brown", "987-6543"},
{3, "Paul", "Wick", "543-7890"}
},
{"CustomerID", "FirstName", "LastName", "Phone"}
),
List = {"CustomerID", "FirstName", "LastName"},
#"Unpivoted Only Selected Columns" = Table.Unpivot(srcTbl, List, "Attribute", "Value"),
Source1 = fnCheckId(#"Unpivoted Only Selected Columns","Value"),
#"Removed Columns" = Table.RemoveColumns(Source1,{"Value"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"QA Value", "Value"}}),
#"Change Title" = Table.TransformColumns(#"Renamed Columns",{{"Attribute",each "QA" & _, type text}}),
combined = #"Unpivoted Only Selected Columns" & #"Change Title",
#"Changed Type" = Table.TransformColumnTypes(combined,{{"Attribute", type text}, {"Value", type text}}) ,
#"Pivoted Column" = Table.Pivot(#"Changed Type", List.Distinct(#"Changed Type"[Attribute]), "Attribute", "Value")
in #"Pivoted Column"
or with changed function as below,
let srcTbl = Table.FromRows(
{
{1, "Bob", "Smith", "123-4567"},
{2, "Jim", "Brown", "987-6543"},
{3, "Paul", "Wick", "543-7890"}
},
{"CustomerID", "FirstName", "LastName", "Phone"}
),
List = {"CustomerID", "FirstName", "LastName"},
#"Unpivoted Only Selected Columns" = Table.Unpivot(srcTbl, List, "Attribute", "Value"),
#"Changed Type1" = Table.TransformColumnTypes(#"Unpivoted Only Selected Columns",{{"Value", type text}}),
#"Processed" = Table.TransformColumns(#"Changed Type1",{{"Value",each fnCheckId2(_), type text}}),
Namechange = Table.TransformColumns(Processed,{{"Attribute",each "QA "&_, type text}}),
combined = #"Changed Type1" & Namechange,
#"Pivoted Column" = Table.Pivot(combined, List.Distinct(combined[Attribute]), "Attribute", "Value")
in #"Pivoted Column"
with fnCheckId2
( clm as text)=>
let
z = if Text.Length(clm )>3 then "Ok" else "Nok"
in z
Please try this loop example:
loops ParametersList list
Use fnCheckId function output table as input for next loop
Do NOT require knowledge about looping function fnCheckId
Recursive function Loop_fnCheckId:
(Loop as number, inTbl as table, inPrm as list )=>
try Loop_fnCheckId(Loop-1, fnCheckId(inTbl, inPrm{Loop-1}),inPrm)
otherwise inTbl
code with loop:
let
ParametersList = {"CustomerID","FirstName","LastName"},
cntLoop = List.Count(ParametersList),
Result = Loop_fnCheckId(cntLoop,srcTbl, ParametersList)
in Result
P.S. The question: is it possible to rewrite it using each _ syntax and avoid additional looping function Loop_fnCheckId
I've been attempting to extract the geometry from an ESRI Rest endpoint. I return JSON and can drill down to the point where each row is a list of lists. I would like to concatenate all the points into a new row that Power Bi can display which I believe is in format
POLYGON((lon, lat lon2, lat2 lon3, lat3 lon4, lat4))
There is not a set number of points per polygon. In the example below there are 4 points that make up the polygon. If you check the endpoint url there are many polygons.
"geometry": {
"rings": [
[
[
-91.477749413304764,
31.470175721032774
],
[
-91.477709210911314,
31.470214015064812
],
[
-91.477676009740037,
31.470105771763997
],
[
-91.477749413304764,
31.470175721032774
]
]
]
}
Here is my current code
let
Source = Json.Document(Web.Contents("https://gispublic.ducks.org/arcgis/rest/services/WMU/PastUnits/MapServer/0/query?where=1%3D1&text=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&distance=&units=esriSRUnit_Foot&relationParam=&outFields=*&returnGeometry=true&returnTrueCurves=false&maxAllowableOffset=&geometryPrecision=&outSR=&havingClause=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&gdbVersion=&historicMoment=&returnDistinctValues=false&resultOffset=&resultRecordCount=&returnExtentOnly=false&datumTransformation=¶meterValues=&rangeValues=&quantizationParameters=&featureEncoding=esriDefault&f=pjson"), 65001),
features = Source[features],
#"convertedtoTable" = Table.FromList(features, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"expandedColumn" = Table.ExpandRecordColumn(#"convertedtoTable", "Column1", {"geometry"}, {"geometry"}),
geo = Table.ExpandRecordColumn(expandedColumn, "geometry", {"rings"}, {"geometry.rings"}),
#"geometry rings" = geo[geometry.rings],
#"Converted to Table" = Table.FromList(#"geometry rings", Splitter.SplitByNothing(), null, null, ExtraValues.Error),
Column1 = #"Converted to Table"[Column1],
#"Converted to Table1" = Table.FromList(Column1, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandListColumn(#"Converted to Table1", "Column1")
in
#"Expanded Column1"
You can
"drill down" a little further so that each list just contains a single polygon
Extract the list into a delimited array
split the array into columns where each row represents a polygon
With regard to the "splitting", if you will be doing this once, you can just accept the code generated by the UI.
If you will be doing this multiple times, and there might be different numbers of polygons in each run, it would be more efficient to calculate the number of columns needed.
But here is specimen code, replacing all after your #"geometry rings", but shortened as there are over 4000 columns generated.
#"geometry rings" = geo[geometry.rings],
//drill down to combine each ring into a single list
comb1 = List.Transform(#"geometry rings", each List.Combine(_)),
comb2 = List.Transform(comb1, each List.Combine(_)),
//convert to table
rings = Table.FromList(comb2,Splitter.SplitByNothing(),{"Rings"}),
#"Extracted Values" = Table.TransformColumns(rings, {"Rings", each Text.Combine(List.Transform(_, Text.From), ";"), type text}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Extracted Values", "Rings", Splitter.SplitTextByDelimiter(";", QuoteStyle.Csv), {"Rings.1", "Rings.2", ...,
#"Changed Type" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"Rings.1", type number}, {"Rings.2", type number}, ...
in
#"Changed Type
Edit: shortened code for dynamic numbers of columns
#"geometry rings" = geo[geometry.rings],
//drill down to combine each ring into a single list
comb1 = List.Transform(#"geometry rings", each List.Combine(_)),
comb2 = List.Transform(comb1, each List.Combine(_)),
numCols = List.Accumulate(comb2,
0,
(state,current)=> if state > List.Count(current) then state else List.Count(current)),
//convert to table
rings = Table.FromList(comb2,Splitter.SplitByNothing(),{"Rings"}),
#"Extracted Values" = Table.TransformColumns(rings, {"Rings", each Text.Combine(List.Transform(_, Text.From), ";"), type text}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Extracted Values", "Rings",
Splitter.SplitTextByDelimiter(";", QuoteStyle.Csv), numCols),
colTypes = List.Transform(Table.ColumnNames(#"Split Column by Delimiter"), each {_, Number.Type}),
#"Changed Type" = Table.TransformColumnTypes(#"Split Column by Delimiter",colTypes)
in
#"Changed Type"
I am trying to stack three columns like below on Power BI.
My example dataset is following:
Processo
Finalidade 1
Finalidade 2
Finalidade 3
1
Maçã
Banana
Mamão
2
Banana
3
Mamão
Banana
However, I would like that it stay like this:
Processo
Finalidade
1
Maçã
2
Banana
3
Mamão
1
Banana
2
3
Banana
1
Mamão
2
3
I tried to transpose columns but I can't this result.
Based on your input data and the hint in the comments I did it like that
let
Source = Excel.CurrentWorkbook(){[Name="tblData"]}[Content],
Step1 = Table.TransformColumnTypes(Source,{{"Processo", Int64.Type}, {"Finalidade 1", type text}, {"Finalidade 2", type text}, {"Finalidade 3", type text}}),
tblUnPivot = Table.ReplaceValue(Step1,null,"",Replacer.ReplaceValue,{"Finalidade 2"}),
replaceValues = Table.ReplaceValue(tblUnPivot,null,"",Replacer.ReplaceValue,{"Finalidade 3"}),
unpivot = Table.UnpivotOtherColumns(replaceValues, {"Processo"}, "Attribut", "Wert"),
sorted = Table.Sort(unpivot,{{"Attribut", Order.Ascending}, {"Processo", Order.Ascending}}),
removeCol = Table.RemoveColumns(sorted,{"Attribut"}),
renameCol = Table.RenameColumns(removeCol,{{"Wert", "Finalidade"}})
in
renameCol
Result looks like
But the solution strongly depends on the naming of the columns Finalidade 1 etc.
PS Another solution which is independet of the naming of the columns could look like that
let
Source = Excel.CurrentWorkbook(){[Name="tblData"]}[Content],
Step1 = Table.TransformColumnTypes(Source,{{"Processo", Int64.Type}, {"Finalidade 1", type text}, {"Finalidade 2", type text}, {"Finalidade 3", type text}}),
tblUnPivot = Table.ReplaceValue(Step1,null,"",Replacer.ReplaceValue,{"Finalidade 2"}),
replaceValues = Table.ReplaceValue(tblUnPivot,null,"",Replacer.ReplaceValue,{"Finalidade 3"}),
unpivot = Table.UnpivotOtherColumns(replaceValues, {"Processo"}, "Attribut", "Wert"),
addColumn = Table.AddColumn(unpivot, "colPos", each List.PositionOf(Table.ColumnNames(Step1),[Attribut])),
#"Neu angeordnete Spalten" = Table.ReorderColumns(addColumn,{"Processo", "colPos", "Attribut", "Wert"}),
removeCol1 = Table.RemoveColumns(#"Neu angeordnete Spalten",{"Attribut"}),
sortRows = Table.Sort(removeCol1,{{"colPos", Order.Ascending}, {"Processo", Order.Ascending}}),
renameCol = Table.RenameColumns(sortRows,{{"Wert", "Fianlidade"}}),
removeCol2 = Table.RemoveColumns(renameCol,{"colPos"})
in
removeCol2
Replace null with some characters. Unpivot other columns. Replace characters with null. Remove extra column
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Replaced Value" = Table.ReplaceValue(Source,null,"~~~",Replacer.ReplaceValue,{"Finalidade 1", "Finalidade 2", "Finalidade 3"}),
#"Unpivoted Other Columns1" = Table.UnpivotOtherColumns(#"Replaced Value", {"Processo"}, "Attribute", "Value"),
#"Replaced Value1" = Table.ReplaceValue(#"Unpivoted Other Columns1","~~~",null,Replacer.ReplaceValue,{"Value"}),
#"Removed Columns" = Table.RemoveColumns(#"Replaced Value1",{"Attribute"})
in #"Removed Columns"
I'm trying to scrape some data from a website, but I need the date/time included on the span tag, as showed below:
<span class="hourAgo ng-binding" title="07/07/2020 às 09:43:33">Há 3 horas</span>
The PowerQuery looks like that:
Source = Web.BrowserContents("https://www.reclameaqui.com.br/empresa/nestle/lista-reclamacoes/"),
#"Extracted Table From Html" =
Html.Table(Source, {{
"Column1", ".text-title"
}, {
"Column2", ".text-description"
}, {
"Column3", ".status-text"
}, {
"Column4", ".hourAgo" <<<<<<< Here's the class selector I got, but I need the title content
}, {
"Column5", ".mdi-map-marker + *"
}},
[RowSelector=".complain-list:nth-child(1) LI"]),
#"Changed Type" = Table.TransformColumnTypes(#"Extracted Table From Html",{{
"Column1", type text
}, {
"Column2", type text
}, {
"Column3", type text
}, {
"Column4", type text
}, {
"Column5", type text
}})
in
#"Changed Type"
All other columns are fine. That code returns me the "Há 3 horas" span content, so far.
You'll probably want to extract that title separately since it's within the span tag, which means you'll have to parse the site as text rather than HTML.
Split the HTML text by line feed (new line).
Convert to a table.
Filter to get just the rows containing "hourAgo".
Extract the date between the quotes.
let
Source = Web.BrowserContents("https://www.reclameaqui.com.br/empresa/nestle/lista-reclamacoes/"),
#"Split Text" = Text.Split(Source, "#(lf)"),
#"Converted to Table" = Table.FromList(#"Split Text", Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Filtered Rows" = Table.SelectRows(#"Converted to Table", each Text.Contains([Column1], "hourAgo")),
#"Extracted Text Between Delimiters" = Table.TransformColumns(#"Filtered Rows", {{"Column1", each Text.BetweenDelimiters(_, "<span class=""hourAgo ng-binding"" title=""", """>"), type text}})
in
#"Extracted Text Between Delimiters"
You can also change this slightly to include one of the other columns so you can merge back with your original table:
let
Source = Web.BrowserContents("https://www.reclameaqui.com.br/empresa/nestle/lista-reclamacoes/"),
#"Split Text" = Text.Split(Source, "#(lf)"),
#"Converted to Table" = Table.FromList(#"Split Text", Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Filtered Rows" = Table.SelectRows(#"Converted to Table", each Text.Contains([Column1], "hourAgo")),
#"Split Column by Delimiter" = Table.SplitColumn(#"Filtered Rows", "Column1", Splitter.SplitTextByDelimiter("</span>", QuoteStyle.Csv), {"Column1.1", "Column1.2", "Column1.3", "Column1.4", "Column1.5", "Column1.6", "Column1.7", "Column1.8", "Column1.9", "Column1.10", "Column1.11"}),
#"Removed Other Columns" = Table.SelectColumns(#"Split Column by Delimiter",{"Column1.4", "Column1.7"}),
#"Extracted Text After Delimiter" = Table.TransformColumns(#"Removed Other Columns", {{"Column1.4", each Text.BetweenDelimiters(_, "title=", ">"), type text}, {"Column1.7", each Text.Trim(Text.AfterDelimiter(_, ">")), type text}})
in
#"Extracted Text After Delimiter"
I load a huge excel table into power-query (using import .csv). This import automatically detects the column-types and change those respectively:
Table.TransformColumnTypes(
#"Höher gestufte Header",
{
{ "ID", Int64.Type },
{ "Country", type text },
{ "Customer", type text },
{ "Release Name", type text },
{ "Hardware Systems", type text },
{ "Service By", type text },
{ "Hwirelease Tags", type text },
{ "Country Tags", type text },
{ "Created", type datetime },
{ "Last Change", type datetime },
{ "Scope", type text },
{ "PPM PID", Int64.Type },
{ "Salesforce IDs", type text }
}
)
Problem:
The problem begins when I know change the source to a different .csv from an older date in which some of those columns mentioned before did nox exists (e.g. PPM PID).
I receive an error and have to manually delete the command which tries to change the respective column.
Question:
Is there some easy workaround, so that Power-Query only tried to change the columntype if the column exists? Or is my complete approach bad when column names tend to change it names over time?
One way to do this might be to:
Create a structure of pairs (i.e. list or record), where each pair represents a column name and its corresponding type. Include all possible columns (those which exist and those which don't).
Filter the collection to only keep the columns which are present in your table and then pass the filtered collection to Table.TransformColumnTypes.
To give an example:
let
#"Höher gestufte Header" = Table.FromColumns({
{1, 6, 45, 67},
{"US", "JA", "CA", "GB"}
}, {"ID", "Country"}),
typeTransformations = {{"ID", Int64.Type}, {"Country", type text}, {"Customer", type text}, {"Release Name", type text}, {"Hardware Systems", type text}, {"Service By", type text}, {"Hwirelease Tags", type text}, {"Country Tags", type text}, {"Created", type datetime}, {"Last Change", type datetime}, {"Scope", type text}, {"PPM PID", Int64.Type}, {"Salesforce IDs", type text}},
changeTypes = Table.TransformColumnTypes(#"Höher gestufte Header", List.Select(typeTransformations, each Table.HasColumns(#"Höher gestufte Header", _{0})))
in
changeTypes
You can use try otherwise:
#"Change Type ID" = try Table.TransformColumnTypes(#"Höher gestufte Header",{{"ID", Int64.Type}}) otherwise #"Höher gestufte Header",
#"Change Type Country" = try Table.TransformColumnTypes(#"Change Type ID",{{"Country", type text}}) otherwise #"Change Type ID",
...