Not able to schedule the dataset refresh in Power BI for Jira Reports - powerbi

While trying to schedule the dataset refresh for my Power BI Jira report (I am using Power BI Desktop for building the report and getting published to my Power BI account).
I am getting below error:
You can't schedule refresh for this dataset because the following data sources currently don't support refresh:
Data source for Query1
When I checked my "Data Source Settings" I can see the warning saying Some data sources may not be listed because of hand-authored queries
Here is my Power BI Query:
let
Source = Json.Document(Web.Contents(JIRA_URL,[RelativePath="/rest/api/2/search",Query=[jql=& TESTCASE_QUERY]])),
#"Converted to Table" = Record.ToTable(Source),
#"Transposed Table" = Table.Transpose(#"Converted to Table"),
#"Promoted Headers" = Table.PromoteHeaders(#"Transposed Table", [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"expand", type text}, {"startAt", Int64.Type}, {"maxResults", Int64.Type}, {"total", Int64.Type}, {"issues", type any}}),
#"Removed Other Columns" = Table.SelectColumns(#"Changed Type",{"total"}),
#"total" = #"Removed Other Columns"{0}[total],
#"startAt List" = List.Generate(()=>0, each _ < #"total", each _ +100), #"Converted to Table1" = Table.FromList(#"startAt List", Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table1",{{"Column1", "startAt"}}),
#"Added Custom" = Table.AddColumn(#"Renamed Columns", "URL", each JIRA_URL & "/rest/api/2/search?maxResults=100&jql=" & TESTCASE_QUERY & "&startAt=" & Text.From([startAt])), data = List.Transform(#"Added Custom"[URL], each Json.Document(Web.Contents(_))),
#"Converted to TableQuery" = Table.FromList(data, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded ColumnIssues" = Table.ExpandRecordColumn(#"Converted to TableQuery", "Column1", {"issues"}, {"issues"}),
#"Expanded issues" = Table.ExpandListColumn(#"Expanded ColumnIssues", "issues"),
#"Expanded issues1" = Table.ExpandRecordColumn(#"Expanded issues", "issues", {"id", "key", "fields"}, {"id", "key", "fields"}) in
#"Expanded issues1"
I tried using the relative path as well, however, not able to solve this issue. Is it because I have used one more query inside as well #"Added Custom" = Table.AddColumn(#"Renamed Columns", "URL", each JIRA_URL & "/rest/api/2/search?maxResults=100&jql=" & QUERY & "&startAt=" & Text.From([startAt])), data = List.Transform(#"Added Custom"[URL], each Json.Document(Web.Contents(_))),
What could be best solution.
I was able to fix the above issue in my PowerBI desktop with below sample code
let
Source = Json.Document(
Web.Contents(JIRA_URL,
[
RelativePath="rest/api/2/search",
Query=
[
maxResults="100",
jql= EPICS_QUERY,
startAt="0",
apikey="MjY0MzgyODgyNDg4OnRnBcfBqhio"
]
]
)),
numIssues = Source[total],
startAtList = List.Generate(()=>0, each _ < numIssues, each _ +100),
data = List.Transform(startAtList, each Json.Document(Web.Contents(JIRA_URL,
[
RelativePath="rest/api/2/search",
Query=
[
maxResults="100",
jql=EPICS_QUERY,
startAt=Text.From(_),
apikey="MjY0MzgyODgyNDg4OnRnBcfBqhio"
]
]))),
iLL = List.Generate(
() => [i=-1, iL={} ],
each [i] < List.Count(data),
each [
i = [i]+1,
iL = data{i}[issues]
],
each [iL]
),
// and finally, collapse that list of lists into just a single list (of issues)
issues = List.Combine(iLL),
#"Converted to Table" = Table.FromList(issues, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"key", "fields"}, {"issue", "fields"}),
#"Expanded fields" = Table.ExpandRecordColumn(#"Expanded Column1", "fields", {"assignee", "created", "creator", "description", "issuetype", "parent", "priority", "project", "reporter", "resolution", "resolutiondate", "status", "summary", "updated"}, {"assigneeF", "created", "creatorF", "description", "issuetypeF", "parentF", "priorityF", "projectF", "reporterF", "resolutionF", "resolutiondate", "statusF", "summary", "updated"}),
#"Expanded assignee" = Table.ExpandRecordColumn(#"Expanded fields", "assigneeF", {"key"}, {"assignee"}),
#"Expanded creator" = Table.ExpandRecordColumn(#"Expanded assignee", "creatorF", {"key"}, {"creator"}),
#"Expanded issuetype" = Table.ExpandRecordColumn(#"Expanded creator", "issuetypeF", {"name"}, {"issuetype"}),
#"Expanded priority" = Table.ExpandRecordColumn(#"Expanded issuetype", "priorityF", {"name"}, {"priority"}),
#"Expanded project" = Table.ExpandRecordColumn(#"Expanded priority", "projectF", {"key"}, {"project"}),
#"Expanded reporter" = Table.ExpandRecordColumn(#"Expanded project", "reporterF", {"key"}, {"reporter"}),
#"Expanded resolution" = Table.ExpandRecordColumn(#"Expanded reporter", "resolutionF", {"name"}, {"resolution"}),
#"Expanded status" = Table.ExpandRecordColumn(#"Expanded resolution", "statusF", {"name"}, {"status"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded status",{{"created", type datetimezone}, {"resolutiondate", type datetimezone}, {"updated", type datetimezone}}),
#"Expanded parentF" = Table.ExpandRecordColumn(#"Changed Type", "parentF", {"key"}, {"parent"})
in
#"Expanded parentF"

Related

Power Query. How can I dynamically transform a list of records to columns?

I'm returning some JSON data from an API. There's an ID, a bunch of other fields (not included in this), and most importantly a List. The List contains a number of records (the same number and structure for each row)
I'm trying to map the records to columns rather than having to "Expand to New Rows". Each record in the list contains 3 fields (ID, Value & Text).
This is the current structure:
I would like to transform the list of records to look something like this:
The number of records within the List can change. So today I have 2 records in a List for each ID, but tomorrow there could be 4 records. So I need something dynamic that will add a new column into the table based on each record available in the list.
Any help would be much appreciated
See if this works. Built in sample.
let Source = Table.AddColumn(#table({"ID"}, {{"111"}, {"222"},{"333"}}), "Custom", each List.Repeat({[ID="Field", Value="BOB",Text="ASDSD"]},3)),
#"Added Custom2" = Table.AddColumn(Source, "Custom.2", each Table.FromRecords([Custom])),
#"Added Index" = Table.AddIndexColumn(#"Added Custom2", "Index", 0, 1, Int64.Type),
Names= Table.ColumnNames ( Table.Combine ( #"Added Index"[Custom.2] ) ),
NewNames=List.Transform(Names, each "extracted"&_),
Expand= Table.ExpandTableColumn ( #"Added Index", "Custom.2", Names,NewNames ),
#"Removed Columns" = Table.RemoveColumns(Expand,{"Custom"}),
Unpivot=Table.Unpivot(#"Removed Columns",NewNames,"attribute","value"),
#"Grouped Rows" = Table.Group(Unpivot, {"Index"}, {{"data", each
Table.TransformColumns(
Table.AddIndexColumn(_, "Index2", 1, 1, Int64.Type)
,{{"Index2", each Number.RoundUp(_/ List.Count(NewNames),0), type number}})
, type table }}),
#"Removed Other Columns" = Table.SelectColumns(#"Grouped Rows",{"data"}),
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(Table.Column(#"Removed Other Columns", "data"), each if _ is table then Table.ColumnNames(_) else {}))),
#"Expanded Part2" = Table.ExpandTableColumn(#"Removed Other Columns", "data",ColumnsToExpand ,ColumnsToExpand ),
#"Merged Columns1" = Table.CombineColumns(Table.TransformColumnTypes(#"Expanded Part2", {{"Index2", type text}}, "en-US"),{"Index2", "attribute"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged"),
#"Pivoted Column" = Table.Pivot(#"Merged Columns1", List.Distinct(#"Merged Columns1"[Merged]), "Merged", "value")
in #"Pivoted Column"

Power BI - The column 'Column1' of the table wasn't found error

I am getting below error when I am trying to apply the code and load the query.
The column 'Column1' of the table wasn't found
Here is the sample code of query:
let
BaseUrl = "https://jira.tools.com/rest/api/2/search?jql=project in ('ABC') AND issuetype = 'Test Case'",
JiraIDPerPage = 1000,
GetJson = (Url) =>
let
RawData = Web.Contents(Url),
Json = Json.Document(RawData)
in Json,
GetJiraIDCount = () =>
let Url = BaseUrl & "&maxResults=0",
Json = GetJson(Url),
Count = Json[#"total"]
in Count,
GetPage = (Index) =>
let Skip = "&startAt=" & Text.From(Index * JiraIDPerPage),
Top = "&maxResults=" & Text.From(JiraIDPerPage),
Url = BaseUrl & Skip & Top,
Json = GetJson(Url),
Value = Json[#"issues"]
in Value,
JiraIDCount = List.Max({ JiraIDPerPage, GetJiraIDCount() }),
PageCount = Number.RoundUp(JiraIDCount / JiraIDPerPage),
PageIndices = { 0 .. PageCount - 1 },
Pages = List.Transform(PageIndices, each GetPage(_)),
JiraID = List.Union(Pages),
Table = Table.FromList(JiraID, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(Table,"Column1", {"id", "key", "fields"}, {"Column1.id", "Column1.key", "Column1.fields"})
in
#"Expanded Column1"
I can't see any exceptions getting generated, However I try to apply the changes and when power BI loads the data it gives above error.
I was able to fix the above issue with below code.
let
Source = Json.Document(
Web.Contents(JIRA_URL,
[
RelativePath="rest/api/2/search",
Query=
[
maxResults="100",
jql= EPICS_QUERY,
startAt="0",
apikey="MjY0MzgyODgyNDg4On5RnBcfBqhio"
]
]
)),
numIssues = Source[total],
startAtList = List.Generate(()=>0, each _ < numIssues, each _ +100),
data = List.Transform(startAtList, each Json.Document(Web.Contents(JIRA_URL,
[
RelativePath="rest/api/2/search",
Query=
[
maxResults="100",
jql=EPICS_QUERY,
startAt=Text.From(_),
apikey="MjY0MzgyODgyNDg4On5RnBcfBqhio"
]
]))),
iLL = List.Generate(
() => [i=-1, iL={} ],
each [i] < List.Count(data),
each [
i = [i]+1,
iL = data{i}[issues]
],
each [iL]
),
// and finally, collapse that list of lists into just a single list (of issues)
issues = List.Combine(iLL),
#"Converted to Table" = Table.FromList(issues, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"key", "fields"}, {"issue", "fields"}),
#"Expanded fields" = Table.ExpandRecordColumn(#"Expanded Column1", "fields", {"assignee", "created", "creator", "description", "issuetype", "parent", "priority", "project", "reporter", "resolution", "resolutiondate", "status", "summary", "updated"}, {"assigneeF", "created", "creatorF", "description", "issuetypeF", "parentF", "priorityF", "projectF", "reporterF", "resolutionF", "resolutiondate", "statusF", "summary", "updated"}),
#"Expanded assignee" = Table.ExpandRecordColumn(#"Expanded fields", "assigneeF", {"key"}, {"assignee"}),
#"Expanded creator" = Table.ExpandRecordColumn(#"Expanded assignee", "creatorF", {"key"}, {"creator"}),
#"Expanded issuetype" = Table.ExpandRecordColumn(#"Expanded creator", "issuetypeF", {"name"}, {"issuetype"}),
#"Expanded priority" = Table.ExpandRecordColumn(#"Expanded issuetype", "priorityF", {"name"}, {"priority"}),
#"Expanded project" = Table.ExpandRecordColumn(#"Expanded priority", "projectF", {"key"}, {"project"}),
#"Expanded reporter" = Table.ExpandRecordColumn(#"Expanded project", "reporterF", {"key"}, {"reporter"}),
#"Expanded resolution" = Table.ExpandRecordColumn(#"Expanded reporter", "resolutionF", {"name"}, {"resolution"}),
#"Expanded status" = Table.ExpandRecordColumn(#"Expanded resolution", "statusF", {"name"}, {"status"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded status",{{"created", type datetimezone}, {"resolutiondate", type datetimezone}, {"updated", type datetimezone}}),
#"Expanded parentF" = Table.ExpandRecordColumn(#"Changed Type", "parentF", {"key"}, {"parent"})
in
#"Expanded parentF"

Power Query data load: large dataset causes 'expression.error column not found'

When I try to load the query below with the entire data set (ie, around 100,000 rows in total) it errors with:
"Expression.Error column "Column1" not found" (Column1 refers to a column which I expanded in a function and then call in the query)
When I limit the dataset to call only ~900 rows, I do NOT get the error and the dataset loads perfectly.
My conclusion is that there is NOT ACTUALLY an error in my query, but somehow the size of the dataset is causing this.
QUESTION: How can I prevent this error and get my whole dataset to
load?
*********QUERY THAT DOES NOT LOAD:
let
Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
#"Invoked Custom Function" = Table.AddColumn(Source, "FnGet", each FnGet([companyID])),
#"Expanded FnGet" = Table.ExpandTableColumn(#"Invoked Custom Function", "FnGet", {"date", "articleUrl", "title", "summary", "reads"}, {"date", "articleUrl", "title", "summary", "reads"}) in
#"Expanded FnGet"
***QUERY THAT HAS NO ERROR ON LOAD:
NOTE STEP 1 "Kept First Rows" which reduces the dataset to approx 1000 rows
let
Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
#"Kept First Rows" = Table.FirstN(Source,10),
#"Invoked Custom Function" = Table.AddColumn(#"Kept First Rows", "FnGet", each FnGet([companyID])),
#"Expanded FnGet" = Table.ExpandTableColumn(#"Invoked Custom Function", "FnGet", {"date", "articleUrl", "title", "summary", "reads"}, {"date", "articleUrl", "title", "summary", "reads"}) in
#"Expanded FnGet"
*************** ADDITIONAL INFORMATION ******************
The query references a function "FnGet" which references Column1 in step 3:
let
Source = (companyID as any) => let
Source = Json.Document(Web.Contents("https://www.lexology.com/api/v1/track/clients/articles?companyId=" & Number.ToText(companyID)&"&limit=100" , [Headers=[ApiKey="ABCD12345"]])),
#"Converted to Table" = Record.ToTable(Source),
Value = #"Converted to Table"{7}[Value],
#"Converted to Table1" = Table.FromList(Value, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table1", "Column1", {"date", "articleUrl", "title", "summary", "reads"}, {"date", "articleUrl", "title", "summary", "reads"})
in
#"Expanded Column1"
in
Source

Create list of date ranges from a list of dates using power query

Please help me create a function in Power Query.
At one of the steps of the query, as a result, I get a list of dates. Some go sequentially, some separately. The quantity is not fixed.
Example (MM.DD.YYYY):
{01/01/2019,
01/02/2019,
01/03/2019,
01/05/2019,
01/06/2019,
01/08/2019}
I need to determine all intervals of consecutive dates and reflect the list of such intervals. The interval is set by the start and end dates. If there is one continuous date, then it is the beginning and the end.
An example from the previous data:
{{01/01/2019, 01/03/2019},
{01/05/2019, 01/06/2019},
{01/08/2019, 01/08/2019}}
Please help me write a function to solve this problem.
In my data, there are about 10,000 lines, each of which has a list attached up to 365 days. It is desirable that the function works quickly.
It feels like list.generate can help, but I don't understand this function very well.
This function, which I called Parse Dates, should do it:
(dateList) =>
let
#"Converted to Table" = Table.FromList(dateList, Splitter.SplitByNothing(), {"Dates"}, null, ExtraValues.Error),
#"Changed Type" = Table.TransformColumnTypes(#"Converted to Table",{{"Dates", type date}}),
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1),
#"Added Start" = Table.AddColumn(#"Added Index", "Start", each try if #"Added Index"{[Index]-1}[Dates] = Date.AddDays([Dates],-1) then null else [Dates] otherwise [Dates]),
#"Added End" = Table.AddColumn(#"Added Start", "End", each try if #"Added Start"{[Index]+1}[Dates] = Date.AddDays([Dates],1) then null else [Dates] otherwise [Dates]),
#"Added Custom1" = Table.AddColumn(#"Added End", "Group", each "Group"),
#"Grouped Rows" = Table.Group(#"Added Custom1", {"Group"}, {{"Start", each List.RemoveNulls([Start]), type anynonnull}, {"End", each List.RemoveNulls([End]), type anynonnull}}),
#"Added Custom2" = Table.AddColumn(#"Grouped Rows", "Tabled", each Table.FromColumns({[Start],[End]},{"Start","End"})),
#"Removed Other Columns" = Table.SelectColumns(#"Added Custom2",{"Tabled"}),
#"Expanded Tabled" = Table.ExpandTableColumn(#"Removed Other Columns", "Tabled", {"Start", "End"}, {"Start", "End"}),
#"Added Custom3" = Table.AddColumn(#"Expanded Tabled", "Custom", each List.Dates([Start],Number.From([End]-[Start])+1,#duration(1,0,0,0))),
#"Removed Other Columns1" = Table.SelectColumns(#"Added Custom3",{"Custom"})
in
#"Removed Other Columns1"
I invoked it with this:
let
Source = #"Parse Dates"(#"Dates List")
in
Source
...against this list, which I called Dates List:
...to get this result:
I managed to figure out how to use the List.Generate function to solve this problem.
This function works a little faster for me.
I called it fn_ListOfDatesToDateRanges.
To invoke it, you must pass a column in each row of which there is a list of dates.
Information from the KenR blog helped me with development.
To compare performance, I used an array with about 250 thousand lines. The increase in speed was 45 seconds versus 1 minute ~ (-33%)
Test file with used functions is here
(Dates)=>
let
InputData = List.Transform(List.Sort(Dates,Order.Ascending), each DateTime.Date(DateTime.From(_, "en-US"))),
DateRangesGen = List.Generate(
()=> [Date=null, Counter=0],
each [Counter]<=List.Count(InputData),
each [
Date =
let
CurrentRowDate = InputData{[Counter]},
PreviousRowDate = try InputData{[Counter]-1} otherwise null,
NextRowDate = try InputData{[Counter]+1} otherwise null,
MyDate = [Start_Date=
(if PreviousRowDate = null then CurrentRowDate else
if CurrentRowDate = Date.AddDays(Replacer.ReplaceValue(PreviousRowDate,null,0),1) then null else CurrentRowDate),
End_Date=(
if NextRowDate = null then CurrentRowDate else
if CurrentRowDate=Date.AddDays(Replacer.ReplaceValue(NextRowDate,null,0),-1) then null else CurrentRowDate)
]
in
MyDate,
Counter=[Counter]+1],
each [Date]),
DateRanges = Table.ExpandTableColumn(Table.SelectColumns(Table.AddColumn(Table.Group(Table.AddColumn(Table.ExpandRecordColumn(Table.FromList(DateRangesGen, Splitter.SplitByNothing(), null, null, ExtraValues.Error), "Column1", {"Start_Date", "End_Date"}, {"Start_Date", "End_Date"}), "Group", each "Group"), "Group", {{"Start_Date", each List.RemoveNulls([Start_Date]), type anynonnull}, {"End_Date", each List.RemoveNulls([End_Date]), type anynonnull}}), "Tabled", each Table.FromColumns({[Start_Date],[End_Date]},{"Start_Date","End_Date"})),{"Tabled"}), "Tabled", {"Start_Date", "End_Date"}, {"Start_Date", "End_Date"})
in
DateRanges

Power Query List To Table

I have trouble transforming a list to table.
So this:
{"123","1.1","321","12","345","345"}
should be transformed to this
{{"123","1.1"},{"321","12"},{"345","345"}}
even better if there where a function that you could easily transform also to this
{{"123","1.1","321"},{"12","345","345"}}
Might not be the best way:
let
ListToTable = (sourceList as list, columnCount as number) as table =>
let
#"Converted to Table" = Table.FromList(sourceList , Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Added Index" = Table.AddIndexColumn(#"Converted to Table", "Index", 0, 1),
#"Integer-Divided Column" = Table.TransformColumns(#"Added Index", {{"Index", each Number.IntegerDivide(_, columnCount), Int64.Type}}),
#"Grouped Rows" = Table.FromRecords(List.Transform(Table.Group(#"Integer-Divided Column", {"Index"}, {{"Rows", each Table.SelectColumns(_, "Column1"), type table}})[Rows], (rowTable) => Table.First(Table.Transpose(rowTable))))
in
#"Grouped Rows"
in
ListToTable({1,2,3,4,5,6}, 3)