How do I schedule refresh of a Web.Contents data source? - powerbi

Question

I am trying to set a 'Scheduled Refresh' on a dataset in the Power BI web app (https://app.powerbi.com).
Normally I should see these options in the dataset settings:
But when I go to settings I am greeted by this warning:
and no way to select the 'Gateway Connection' or data source settings.
I found a useful article which explains a problem with Web.Contents and how to get around it:
https://blog.crossjoin.co.uk/2016/08/23/web-contents-m-functions-and-dataset-refresh-errors-in-power-bi/
I applied this and it still doesn't work.
In Power BI Desktop no data sources are listed as I am using a hand-authored query.
The way it works is there is a main query (Log Scroll) which calls a recursive function query (RecursiveFetch). The function then calls a Web API which works by sending a new page of JSON data everytime it is called, in a sort of 'scrolling' manner.
The Log Scroll query looks like this:
let
url = "http://exampleURL:1000"
Source = RecursiveFetch(url, 5, null, null)
in
Source
The RecursiveFetch looks like this:
let
RecursiveFetch= (url, scrollCount, scrollID, counter) =>
let
Counter = if (counter = null) then 0 else counter,
Results = if (scrollID = null) then
Json.Document(Web.Contents(url,
[
Headers=[
#"Authorization"="Basic <key here>",
#"Content-Type"="application/json"
]
]
))
else
Json.Document(Web.Contents(url,
[
Content = Text.ToBinary(scrollID),
Headers=[
#"Authorization"="Basic <key here>",
#"Content-Type"="application/json"
]
]
)),
ParsedResults = Table.FromList(Results[hits][hits], Splitter.SplitByNothing(), null, null, ExtraValues.Error),
Return = if (Counter < scrollCount) then
ParsedResults & RecursiveFetch(url, scrollCount scrollID, Counter)
else
ParsedResults
in
Return
in
RecursiveFetch
It all works perfectly in Power BI Desktop but when I publish it to the web app I get the errors shown above.
I have manually set up a data source in my Gateway Cluster which connects fine to the URL with the same credentials that the hand-authored query uses.
How do I get this all to work? Is there something I have missed?

Answer 1

It all works perfectly in Power BI Desktop but when I publish it to the web app I get the errors shown above.
To fix that, Web.Contents needs to use options[RelativePath] and options[Query] (or Content for HTTP POST)
The original:
"https://data.gov.uk/api/3/action/package_search?q=" & Term
Will use:
let
BaseUrl = "https://data.gov.uk",
Options = [
RelativePath = "/api/3/action/package_search",
Headers = [
Accept="application/json"
],
Query = [
q = Term
]
],
// wrap 'Response' in 'Binary.Buffer' if you are using it multiple times
response = Web.Contents(BaseUrl, Options),
buffered = Binary.Buffer(response),
response_metadata = Value.Metadata(response),
status_code = response_metadata[Response.Status],
from_json = Json.Document(final_result)
in
from_json
the parameter url is going to be the minimum url possible, otherwise the service will think your dynamic request is actually unchanged -- causing the original refresh error.