In the Above Picture we have Items and NextPageLink. Data is present inside Items and NextPageLink Contains links for next page data.
I am trying to automate a rest-API call with a Json response having several pages.
The idea would be to automatically call the NextPageLink until there is no NextPageLink.
I want to get data into single table.
Here is the link i am using to pull data : https://prices.azure.com/api/retail/prices
Here is last page link : https://prices.azure.com/api/retail/prices?$skip=408500
Please help me
Create function ReadSingle
(offset) =>
let Source =Json.Document(Web.Contents(" https://prices.azure.com/api/retail/prices?$skip="& Number.ToText( offset )))[Items]
in Source
Use a loop to read in data. You could repeat until done, but frankly my computer locked up, so Im doing pages 0-25, by 100's
let Source
= List.Generate(
() => [ Offset = 0, Data = ReadSingle(0) ],
each [Offset] <= 2500,
each [ Data = ReadSingle( [Offset] ),
Offset = [Offset] + 100 ],
each [Data]
),
z=Table.FromColumns(Source),
#"Added Index" = Table.AddIndexColumn(z, "Index", 0, 1, Int64.Type),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Added Index", {"Index"}, "Attribute", "Value"),
#"Removed Columns" = Table.RemoveColumns(#"Unpivoted Other Columns",{"Index", "Attribute"}),
#"Expanded Value" = Table.ExpandRecordColumn(#"Removed Columns", "Value", {"currencyCode", "tierMinimumUnits", "retailPrice", "unitPrice", "armRegionName", "location", "effectiveStartDate", "meterId", "meterName", "productId", "skuId", "productName", "skuName", "serviceName", "serviceId", "serviceFamily", "unitOfMeasure", "type", "isPrimaryMeterRegion", "armSkuName"}, {"currencyCode", "tierMinimumUnits", "retailPrice", "unitPrice", "armRegionName", "location", "effectiveStartDate", "meterId", "meterName", "productId", "skuId", "productName", "skuName", "serviceName", "serviceId", "serviceFamily", "unitOfMeasure", "type", "isPrimaryMeterRegion", "armSkuName"})
in #"Expanded Value"
If you wanted to you could swap to repeat the loop until NextPageLink is a null. See my favorite reference on this use List.Generate to make API Calls in Power Query M
Related
Assuming Power BI dataset has columns with description added in the modelling view, then is there a way to generate data dictionary from Power BI report/dataset?
You can document using Power Query and Excel. Change your .pbix to a .pbit file and then run the following query to extract a full data dictionary. This is not my code and comes courtesy of RacketLuncher on Reddit.
let
Source = fUnzip(File.Contents("C:\Users\Your file.pbit")),
Filter_DataModelSchema = Table.SelectRows(Source, each ([FileName] = "DataModelSchema" and [Attributes]?[Hidden]? <> true)),
JSONFile = Json.Document(Filter_DataModelSchema{0}[Content],1200),
model = JSONFile[model], // Start from here to change whether you want to pick Tables, or Relationships, or other parts of the model.
tables = model[tables], // Here we are picking the tables metadata
#"Converted to Table" = Table.FromList(tables, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"name", "lineageTag", "modifiedTime", "structureModifiedTime", "columns", "partitions", "measures", "annotations"}, {"name", "lineageTag", "modifiedTime", "structureModifiedTime", "columns", "partitions", "measures", "annotations"}),
#"Expanded columns" = Table.ExpandListColumn(#"Expanded Column1", "columns"),
#"Expanded columns1" = Table.ExpandRecordColumn(#"Expanded columns", "columns", {"name", "dataType", "isKey", "sourceColumn", "formatString", "lineageTag", "summarizeBy", "annotations", "sortByColumn"}, {"columns.name", "columns.dataType", "columns.isKey", "columns.sourceColumn", "columns.formatString", "columns.lineageTag", "columns.summarizeBy", "columns.annotations", "columns.sortByColumn"}),
#"Expanded measures" = Table.ExpandListColumn(#"Expanded columns1", "measures"),
#"Expanded measures1" = Table.ExpandRecordColumn(#"Expanded measures", "measures", {"name", "expression", "formatString", "displayFolder", "lineageTag", "annotations"}, {"measures.name", "measures.expression", "measures.formatString", "measures.displayFolder", "measures.lineageTag", "measures.annotations"})
in
#"Expanded measures1"
I'm trying to model a data source in power bi, but I'm not getting it.
Could you help me with alternatives so I can create a new column? The data source is in excel and brings the data with subtotal by types (XPTO, XPT, etc). I want to put these types as corresponding values in the new column for your items. I tried via power query and dax, but I could not.
Original Source:
Modifications Needed
Source File
This assumes that you can identify the Subtotal Rows by checking the position of the first space in the first column:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added RowType" = Table.AddColumn(Source, "RowType", each Text.PositionOf([#"Centro financ./item orçamento"]," "), Int64.Type),
#"Added Type" = Table.AddColumn(#"Added RowType", "Type", each if [RowType] = 4 then Text.BetweenDelimiters([#"Centro financ./item orçamento"], " ", " ") else null, type text),
#"Filled Down" = Table.FillDown(#"Added Type",{"Type"}),
#"Filtered Rows" = Table.SelectRows(#"Filled Down", each ([RowType] = 8)),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"RowType"}),
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns",List.Combine({{"Type"}, Table.ColumnNames(Source)}))
in
#"Reordered Columns"
Another solution is to - Include another data source in Power BI, which will look something like this.
Item Type
615x92120 Mat1 XPTA
615x92121 Mat2 XPTA
615x92122 Mat3 XPTU
615x92123 Mat4 XPTU
And then do a Join between your existing table and this table to bring out the Type in your existing table. Once you have done that, you should be able to filter out to blanks or null which will be your delete lines.
Note :- This only works, if you know all the items and corresponding types in advance.
let
Source = Sql.Databases("xx.x.x.x"),
zzz_LIVE = Source{[Name="NAVzzz_LIVE"]}[Data],
#"COMPANY1$G_L Entry" = NAVzzzLIVE{[Schema="dbo",Item="COMPANY1$G_L Entry"]}[Data],
Pulling data from several entities (companies) in Microsoft Dynamics Navision with above code (Company1 as an example).
Trying to add a custom column with an unique identifier based on the name of the company.
G_L Account number G_L Account name Amount Company
10010 Revenue 100 Company1
22000 Rent 50 Company1
Is it possible to achieve this by M code?
Many thanks.
How about something like this?
let
Source = Sql.Databases("xx.x.x.x"),
zzz_LIVE = Source{[Name="NAVzzz_LIVE"]}[Data],
Name1 = "Company1",
Name2 = "Company2",
[...]
#"COMPANY1" = NAVzzzLIVE{[Schema="dbo",Item=Name1&"$G_L Entry"]}[Data],
#"Added Custom1" = Table.AddColumn(#"COMPANY1", "Company", each Name1),
#"COMPANY2" = NAVzzzLIVE{[Schema="dbo",Item=Name2&"$G_L Entry"]}[Data],
#"Added Custom2" = Table.AddColumn(#"COMPANY2", "Company", each Name2),
[...]
Table.Combine(#"Added Custom1",#"Added Custom2",[...])
That way you
I am not able to test this, but I think it might work for you:
let
Source = Sql.Databases("xx.x.x.x"),
zzz_LIVE = Source{[Name="NAVzzz_LIVE"]}[Data],
#"COMPANY1$G_L EntrySrc" = NAVzzzLIVE{[Schema="dbo",Item="COMPANY1$G_L Entry"]},
#"COMPANY1$G_L Entry" = #"COMPANY1$G_L EntrySrc"[Data],
#"Added Custom" = Table.AddColumn(#"COMPANY1$G_L Entry", "Company",
each Record.Field(#"COMPANY1$G_L EntrySrc", "Item"))
in
#"Added Custom"
I split the Navigation step (#"COMPANY1$G_L Entry") in two so that I could reference its record. Otherwise, the [Data] gets in the way.
If I'm correct, it will return COMPANY1$G_L Entry into your new column. You could change Record.Field(#"COMPANY1$G_L EntrySrc", "Item") to Text.Before(Record.Field(#"COMPANY1$G_L EntrySrc", "Item"), "$G_L") to only return COMPANY1.
I am running a query to update a historical data table within PowerBI. This update query calls the Adobe Analytics API, and pulls data based on a date range I specify.
As I want this to run as fast as possible, I would only like to query the data that I don't already have in my historical data table. I am okay at putting an end date for this query based off Date.Time.LocalNow().
#date(Date.Year(DateTime.LocalNow()), Date.Month(DateTime.LocalNow()), Date.Day(DateTime.LocalNow())-1)
However, where I am stuck is putting a start date. Ideally I would like to take the MaxDate from a table within my data model, and use this value in the query editor as my start date.
How would I do this?
UPDATE 1 - Full code Below
let
Source = AdobeAnalytics.Cubes()
myreportsuiteid = Source{[Id="myreportsuiteid"]}[Data],
#"Added Items" = Cube.Transform(myreportsuiteid,
{
{Cube.AddAndExpandDimensionColumn, "DateGranularity", {"year", "month", "day"}, {"Date Granularity.Level 1: Year", "Date Granularity.Level 2: Month", "Date Granularity.Level 3: Day"}},
{Cube.AddAndExpandDimensionColumn, "lasttouchchannel", {"lasttouchchannel"}, {"Last Touch Marketing Channel"}},
{Cube.AddMeasureColumn, "Unique Visitors", "uniquevisitors"},
{Cube.AddMeasureColumn, "Visits", "visits"},
{Cube.ApplyParameter, "DateRange", {#date(Date.Year(List.Max(Union[Date])), Date.Month(List.Max(Union[Date])), Date.Day(List.Max(Union[Date]))+1), #date(Date.Year(DateTime.LocalNow()), Date.Month(DateTime.LocalNow()), Date.Day(DateTime.LocalNow())-1)}},
{Cube.ApplyParameter, "Segment", {{"s300000554_5ae201be22fa9950dcdbcd92"}}}
})
in
#"Added Items"
UPDATE 2
WORKING CODE
Daily US is a table created thorough the Query Editor
let
Source = AdobeAnalytics.Cubes(),
todaysDate = Date.AddDays(DateTime.Date(DateTime.LocalNow()),-1),
maxDate = Date.AddDays(List.Max(#"Daily US"[Date]),1),
myreportsuiteid = Source{[Id="myreportsuiteid "]}[Data],
ERROR CODE
Union US is a table created outside of the Query Editor via Union US = DISTINCT(UNION('Seeker US','Daily US'))
let
Source = AdobeAnalytics.Cubes(),
todaysDate = Date.AddDays(DateTime.Date(DateTime.LocalNow()),-1),
maxDate = Date.AddDays(List.Max(#"Union US"[Date]),1),
myreportsuiteid = Source{[Id="myreportsuiteid "]}[Data],
If the column you want to reference for the MaxDate is named Union[Date], then you should be able to use List.Max to get the maximum value from that column.
Try this:
let
Source = AdobeAnalytics.Cubes(),
MaxDate = List.Max(Union[Date]),
TodaysDate = DateTime.Date(DateTime.LocalNow()),
myreportsuiteid = Source{[Id="myreportsuiteid"]}[Data],
#"Added Items" = Cube.Transform(myreportsuiteid,
{
{Cube.AddAndExpandDimensionColumn, "DateGranularity", {"year", "month", "day"}, {"Date Granularity.Level 1: Year", "Date Granularity.Level 2: Month", "Date Granularity.Level 3: Day"}},
{Cube.AddAndExpandDimensionColumn, "lasttouchchannel", {"lasttouchchannel"}, {"Last Touch Marketing Channel"}},
{Cube.AddMeasureColumn, "Unique Visitors", "uniquevisitors"},
{Cube.AddMeasureColumn, "Visits", "visits"},
{Cube.ApplyParameter, "DateRange", {MaxDate, TodaysDate}},
{Cube.ApplyParameter, "Segment", {{"s300000554_5ae201be22fa9950dcdbcd92"}}}
})
in
#"Added Items"
I want to profile every single data table I have in my Power BI report. By data profile I mean something like this:
Are there ways to make a data profile view in Power BI? DAX measure or calculated columns?
Alternatively, you can also recommend other data quality tools that can handle such tasks since I find it a bit difficult to achieve this result in Power BI.
Now I feel dumb after writing a manual query that did what it turns out Table.Profile does in one shot. However I will mention you can automatically get a profile for every table in your data set by using the #shared reference and filtering down to the tables:
let
Source = #shared,
#"Converted to Table" = Record.ToTable(Source),
#"Added Custom" = Table.AddColumn(#"Converted to Table", "TableCheck", each Type.Is(Value.Type([Value]), type table)),
#"Filtered Rows" = Table.SelectRows(#"Added Custom", each ([TableCheck] = true)),
#"Filtered Rows1" = Table.SelectRows(#"Filtered Rows", each [Name] <> "NAME_OF_THIS_QUERY"),
#"Added Custom1" = Table.AddColumn(#"Filtered Rows1", "Profile", each Table.Profile([Value])),
#"Expanded Profile" = Table.ExpandTableColumn(#"Added Custom1", "Profile", {"Column", "Min", "Max", "Average", "StandardDeviation", "Count", "NullCount", "DistinctCount"}, {"Profile.Column", "Profile.Min", "Profile.Max", "Profile.Average", "Profile.StandardDeviation", "Profile.Count", "Profile.NullCount", "Profile.DistinctCount"})
in
#"Expanded Profile"
And replace "NAME_OF_THIS_QUERY" with whatever you name the query so it doesn't try to profile itself.
Power BI has a built-in data profiler
Open Power BI and refer to the menu ribbon
Click Home
Click Edit Queries
Click View
Select Column Profile to view stats about your data
In the query editor, you can use the Table.Profile function on any table.
You can do multiple ones simultaneously like this:
= Table.Combine({Table.Profile(Table1),Table.Profile(Table2)})
Edit:
To see the profile, create a new Blank Query and define it as = Table.Profile(Table1). If you open the Advanced Editor, the M code looks like this:
let
Source = Table.Profile(Table1)
in
Source