PowerQuery List.Generate: Order of Argument Evaluation? - powerbi

Regarding PowerQuery's List.Generate function.
Could someone please help me understand why the first version of this List.Generate call returns 2 elements (desired result), while the second returns only 1 element (only the initial value - an empty list)? The only difference in the two calls is found on line 3.
Working query
Source =
List.Generate(
()=>[Page = 1, NextPage = "next", Response = [next = "next", results = {}]],
each [Page] <= MaxPages and NextPage <> null,
each[
Page = [Page] + 1,
Response = Json.Document(Web.Contents("https://platform.myapi.com/public_api?",
[
Query=
[
page=Text.From([Page]),
page_size=Text.From(1000)
],
Headers=[#"x-api-key"=#"ApiKey"],
RelativePath=relativePath
]
)
),
NextPage = [Response][next]
],
each [Response][results]
), ...
Correct. 2 lists - the initial, empty list, and the results of the API call
Query I think should work, but doesn't
List.Generate(
()=>[Page = 1, NextPage = "next", Response = [next = "next", results = {}]],
each [Page] <= MaxPages and [Response][next] <> null, // only change
each[
Page = [Page] + 1,
Response = Json.Document(Web.Contents("https://platform.myapi.com/public_api?",
[
Query=
[
page=Text.From([Page]),
page_size=Text.From(1000)
],
Headers=[#"x-api-key"=#"ApiKey"],
RelativePath=relativePath
]
)
),
NextPage = [Response][next]
],
each [Response][results]
),...
Incorrect. Only the initial empty list
For clarity, here is sample output from the API. It’s a paginated API with 1,000 records per page. So, since there are only 697 records returned, there is no further page to pull, and the next element is returned as null. There are 697 elements in the results array.

In you data, as you state, Response[next] is null (since you only have 687 items).
So the condition:
each [Page] <= MaxPages and [Response][next] <> null,
Is never met on line 3 of query 2 and no further processing happens.
In the first query, NextPage = "next", so this works.

Related

Power BI Iterative API Loop

I am attempting (and can successfully do so) to connect to an API and loop through several iterations of the API call in order to grab the next_page value, put it in a list and then call the list.
Unfortunately, when this is published to the PBI service I am unable to refresh there and indeed 'Data Source Settings' tells me I have a 'hand-authored query'.
I have attempted to follow Chris Webbs' blog post around the usage of query parameters and relative path, but if I use this I just get a constant loop of the first page that's hit.
The Start Epoch Time is a helper to ensure I only grab data less than 3 months old.
let
iterations = 10000, // Number of MAXIMUM iterations
url = "https://www.zopim.com/api/v2/" & "incremental/" & "chats?fields=chats(*)" & "&start_time=" & Number.ToText( StartEpochTime ),
FnGetOnePage =
(url) as record =>
let
Source1 = Json.Document(Web.Contents(url, [Headers=[Authorization="Bearer MY AUTHORIZATION KEY"]])),
data = try Source1[chats] otherwise null, //get the data of the first page
next = try Source1[next_page] otherwise null, // the script ask if there is another page*//*
res = [Data=data, Next=next]
in
res,
GeneratedList =
List.Generate(
()=>[i=0, res = FnGetOnePage(url)],
each [i]<iterations and [res][Data]<>null,
each [i=[i]+1, res = FnGetOnePage([res][Next])],
each [res][Data])
Lookups
If Source1 exists, but [chats] may not, you can simplify
= try Source1[chats] otherwise null
to
= Source1[chats]?
Plus it you don't lose non-lookup errors.
m-spec-operators
Chris Web Method
should be something closer to this.
let
Headers = [
Accept="application/json"
],
BaseUrl = "https://www.zopim.com", // very important
Options = [
RelativePath = "api/v2/incremental/chats",
Headers = [
Accept="application/json"
],
Query = [
fields = "chats(*)",
start_time = Number.ToText( StartEpocTime )
],
Response = Web.Contents(BaseUrl, Options),
Result = Json.Document(Response) // skip if it's not JSON
in
Result
Here's an example of a reusable Web.Contents function
helper function
let
/*
from: <https://github.com/ninmonkey/Ninmonkey.PowerQueryLib/blob/master/source/WebRequest_Simple.pq>
Wrapper for Web.Contents returns response metadata
for options, see: <https://learn.microsoft.com/en-us/powerquery-m/web-contents#__toc360793395>
Details on preventing "Refresh Errors", using 'Query' and 'RelativePath':
- Not using Query and Relative path cause refresh errors:
<https://blog.crossjoin.co.uk/2016/08/23/web-contents-m-functions-and-dataset-refresh-errors-in-power-bi/>
- You can opt-in to Skip-Test:
<https://blog.crossjoin.co.uk/2019/04/25/skip-test-connection-power-bi-refresh-failures/>
- Debugging and tracing the HTTP requests
<https://blog.crossjoin.co.uk/2019/11/17/troubleshooting-web-service-refresh-problems-in-power-bi-with-the-power-query-diagnostics-feature/>
update:
- MaybeErrResponse: Quick example of parsing an error result.
- Raw text is returned, this is useful when there's an error
- now response[json] does not throw, when the data isn't json to begin with (false errors)
*/
WebRequest_Simple
= (
base_url as text,
optional relative_path as nullable text,
optional options as nullable record
)
as record =>
let
headers = options[Headers]?, //or: ?? [ Accept = "application/json" ],
merged_options = [
Query = options[Query]?,
RelativePath = relative_path,
ManualStatusHandling = options[ManualStatusHandling]? ?? { 400, 404, 406 },
Headers = headers
],
bytes = Web.Contents(base_url, merged_options),
response = Binary.Buffer(bytes),
response_metadata = Value.Metadata( bytes ),
status_code = response_metadata[Response.Status]?,
response_text = Text.Combine( Lines.FromBinary(response,null,null, TextEncoding.Utf8), "" ),
json = Json.Document(response),
IsJsonX = not (try json)[HasError],
Final = [
request_url = metadata[Content.Uri](),
response_text = response_text,
status_code = status_code,
metadata = response_metadata,
IsJson = IsJsonX,
response = response,
json = if IsJsonX then json else null
]
in
Final,
tests = {
WebRequest_Simple("https://httpbin.org", "json"), // expect: json
WebRequest_Simple("https://www.google.com"), // expect: html
WebRequest_Simple("https://httpbin.org", "/headers"),
WebRequest_Simple("https://httpbin.org", "/status/codes/406"), // exect 404
WebRequest_Simple("https://httpbin.org", "/status/406"), // exect 406
WebRequest_Simple("https://httpbin.org", "/get", [ Text = "Hello World"])
},
FinalResults = Table.FromRecords(tests,
type table[
status_code = Int64.Type, request_url = text,
metadata = record,
response_text = text,
IsJson = logical, json = any,
response = binary
],
MissingField.Error
)
in
FinalResults

How to save a table from Datatables to a database in Django?

I'm using Datatables to display tables in Django on the server side. After searching for a phrase, I have a button ready to save the current table after filtering to Excel. I would like the second button to create a new table in the database and save the same table to it as for Excel.
I don't know how I can send AJAX data to Django and read it in views in such a way that I can query the database.
javascript:
$(document).ready(function () {
function newexportaction(e, dt, button, config) {
var self = this;
var oldStart = dt.settings()[0]._iDisplayStart;
dt.one('preXhr', function (e, s, data) {
data.start = 0;
data.length = 2147483647;
dt.one('preDraw', function (e, settings) {
// Call the original action function
if (button[0].className.indexOf('buttons-excel') >= 0) {
$.fn.dataTable.ext.buttons.excelHtml5.available(dt, config) ?
$.fn.dataTable.ext.buttons.excelHtml5.action.call(self, e, dt, button, config) :
$.fn.dataTable.ext.buttons.excelFlash.action.call(self, e, dt, button, config);
}
dt.one('preXhr', function (e, s, data) {
settings._iDisplayStart = oldStart;
data.start = oldStart;
});
});
});
dt.ajax.reload();
}
$("#tb").DataTable({
"lengthChange": true,
"paging": true,
"searching": true,
"oLanguage": {
"sSearch": "Szukaj:",
},
"language": {
"processing": "Przetwarzanie",
"lengthMenu": "Pokaż _MENU_ elementów",
"info": "Wyświetlono od _START_ do _END_ z _TOTAL_ elementów",
"zeroRecords": "Nie znaleziono pasujących elementów",
"paginate": {
"first": "Pierwsza",
"last": "Ostatnia",
"next": "Kolejna",
"previous": "Poprzednia"
},
"emptyTable": "Brak danych do wyświetlenia",
},
"processing": true,
"serverSide": true,
buttons: [
{
extend: 'excel',
titleAttr: 'Excel',
title: '',
action: newexportaction
},
],
dom: 'B<"clear">lfrtip',
// "destroy": true,
"pageLength": 15,
"ordering": false,
"ajax": {
"url": "paging2/",
"type": "get"
},
});
});
views.py:
def paging2(request):
draw = int (request.GET.get ('draw')) # record the number of operations
start = int (request.GET.get ('start')) # start position
length = int (request.GET.get ('length')) # length of each page
search_key = request.GET.get ('search[value]') # search keyword
order_column = request.GET.get ('order [0] [column]') # sort field index
order_column = request.GET.get ('order [0] [dir]') #Ordering rule: ase / desc
# sql query all data, then do paging, the speed is slow
# if search_key:
# result = query(search_key)
# else:
# result = select_all()
# data = result[start:start+length]
# count = len(result)
# sql for paging, fast
if search_key:
sql_search = "SELECT NAZWA,TEL,NIP,Adres,WWW, EMAIL, Branza, NAZWA_SCRAPERA FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL AND Branza like '%%%s%%'" % search_key
sql_search_len = "SELECT COUNT(*) FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL AND Branza like '%%%s%%'"% search_key
result, count = query(sql_search, sql_search_len)
data = result[start:start + length]
else:
sql = """
SELECT NAZWA,TEL,NIP,Adres,WWW, EMAIL, Branza, NAZWA_SCRAPERA FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL
LIMIT %s OFFSET %s
"""
data = select_by_page(length, start, sql)
# data = select_by_page(start, start + length)
sql_len = "SELECT COUNT(*) FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL"
count = all_count(sql_len)
dic = {
'draw': draw,
'recordsTotal': count,
'recordsFiltered': count,
'data': data,
}
return HttpResponse(json.dumps(dic), content_type='application/json')

Replacing blank with zero for a mesure gives som unexpected extra rows

I have a little puzzle that annoys me in PowerBI/DAX. I'm not looking for workarounds - I am looking for an explanation of what is going on.
I've created some sample data to reconstruct the problem.
Here are my two sample tables written in DAX:
Events =
DATATABLE (
"Course", STRING,
"WeekNo", INTEGER,
"Name", STRING,
"Status", STRING,
{
{ "Python", 1, "Joe", "OnSite" },
{ "Python", 1, "Donald", "Video" },
{ "DAX", 2, "Joe", "OnSite" },
{ "DAX", 2, "Hillary", "Video" },
{ "DAX", 3, "Joe", "OnSite" },
{ "DAX", 3, "Hillary", "OnSite" },
{ "DAX", 3, "Donald", "OnSite" }
}
)
WeekNumbers =
DATATABLE ( "WeekNumber", INTEGER, { { 1 }, { 2 }, { 3 }, { 4 } } )
I have a table with events and another table with all weeknumbers and there is a (m:1) relation between them on the weekNo/weeknumber (I've given them different names to easily distinguish them in this example). I have a slicer in PowerBI on the weeknumber. And I have a table which shows aggregation and counts the participants based on the status with the following measures:
#OnSite = COUNTROWS(FILTER(events,Events[Status]="OnSite"))
#Video = COUNTROWS(FILTER(events,Events[Status]="Video"))
I visualize the two measures in a table together with the Course and the weekNo. With the slicer on weekNumber 3 there are nobody with status video so #video is blank. See screenshot.
Then I decided to create a new measure which should show a 0 instead of blank for the #video:
#VideoWithZero = VAR counter=COUNTROWS(FILTER(events,Events[Status]="Video"))
RETURN IF(ISBLANK(counter),0,counter)
I add the #VideoWithZero to the table and get a lot of extra rows in the table for the other weekNo's:
So my question is - Why do I get the extra rows for week 1 and 2 in the table? I would expect my filter on WeekNumber to filter them out.
The filter is being applied to the context of the query executed, and then the measures are calculated. Now the issue is that one of your measures is always returning a value (0), so regardless of your context it will always show a result, thus making it seem that it is ignoring the filter.
One way I tend to get implement this is by providing some additional context to when I might want to show 0 instead of blank. In your case it would be when one of the counts is not blank:
#OnSite =
VAR video = COUNTROWS(FILTER(events,Events[Status]="Video"))
VAR onsite = COUNTROWS(FILTER(events,Events[Status]="OnSite"))
RETURN IF(ISBLANK(video), onsite, onsite + 0) //+0 will force it not to be blank
#Video =
VAR video = COUNTROWS(FILTER(events,Events[Status]="Video"))
VAR onsite = COUNTROWS(FILTER(events,Events[Status]="OnSite"))
RETURN IF(ISBLANK(onsite), video, video + 0)
So on the OnSite measure it will check if there are Videos and if so, it adds +0 to the result of the OnSite count to force it not to be blank (and vice versa)
One other way could be to count total rows and subtract the ones different to the status you need:
#OnSite =
VAR total= COUNTROWS(Events[Status])
VAR notOnsite = COUNTROWS(FILTER(events,Events[Status]<>"OnSite"))
RETURN total - notOnsite
#Video =
VAR total= COUNTROWS(Events[Status])
VAR notVideo= COUNTROWS(FILTER(events,Events[Status]<>"Video"))
RETURN total - notVideo

DRF formatting XLSX content

I am trying to set a different color on every second row in XLSX file. From the documentation I see that I can pass some conditions using body property or get_body() method, but this only allows me to set somewhat "static" conditions. Here is the ViewSet config responsible for rendering the XLSX file:
class MyViewSet(XLSXFileMixin, ModelViewSet):
def get_renderers(self) -> List[BaseRenderer]:
if self.action == "export":
return [XLSXRenderer()]
else:
return super().get_renderers()
#action(methods=["GET"], detail=False)
def export(self, request: Request) -> Response:
serializer = self.get_serializer(self.get_queryset(), many=True)
return Response(serializer.data)
# Properties for XLSX
column_header = {
"titles": [
"Hostname", "Operating System", "OS name", "OS family", "OS version", "Domain", "Serial number",
"Available patches",
],
"tab_title": "Endpoints",
"style": {
"font": {
"size": 14,
"color": "FFFFFF",
},
"fill": {
"start_color": "3F803F",
"fill_type": "solid",
}
}
}
body = {
"style": {
"font": {
"size": 12,
"color": "FFFFFF"
},
"fill": {
"fill_type": "solid",
"start_color": "2B2B2B"
},
}
}
OK. I got the answer after some digging through the source code. The render method of XLSXRenderer has this piece of code:
for row in results:
column_count = 0
row_count += 1
flatten_row = self._flatten(row)
for column_name, value in flatten_row.items():
if column_name == "row_color":
continue
column_count += 1
cell = ws.cell(
row=row_count, column=column_count, value=value,
)
cell.style = body_style
ws.row_dimensions[row_count].height = body.get("height", 40)
if "row_color" in row:
last_letter = get_column_letter(column_count)
cell_range = ws[
"A{}".format(row_count): "{}{}".format(last_letter, row_count)
]
fill = PatternFill(fill_type="solid", start_color=row["row_color"])
for r in cell_range:
for c in r:
c.fill = fill
So when I added a field row_color in my serializer as SerializerMethodField I was able to define a function that colors rows:
def get_row_color(self, obj: Endpoint) -> str:
"""
This method returns color value for row in XLSX sheet.
(*self.instance,) extends queryset to a list (it must be a queryset, not a single Endpoint).
.index(obj) gets index of currently serialized object in that list.
As the last step one out of two values from the list is chosen using modulo 2 operation on the index.
"""
return ["353535", "2B2B2B"][(*self.instance,).index(obj) % 2]

update list element in dynamodb

I have a list of maps as one field of a DynamoDB table. How can I update a specific element (or, rather element field ?)
Trying something like
rc = table.update_item(Key={ 'username' : user },
UpdateExpression="set list[:i].field = :nd",
ExpressionAttributeValues={
':i' : itemnum,
':nd': data,
},
ReturnValues="UPDATED_NEW"
);
But I am getting an error:
Invalid UpdateExpression: Syntax error; token: ":i", near: "[:i]"
Any ideas how can I reference list element with variable number. Thanks.
Use a literal instead:
rc = table.update_item(Key={ 'username' : user },
UpdateExpression="set list[" + itemnum + "].field = :nd",
ExpressionAttributeValues={
':nd': data,
},
ReturnValues="UPDATED_NEW"
);
I'm adding an answer to #Snedden27 "Does this have any security risk if itemnum is send by the end user?"
Yes, in theory a user could inject something here in certain circumstances (e.g. if you are updating a list and the list items are also lists then the user could input "3][2" which would make a valid expression that updates a nested list and screws up your data structure.
Something like parseInt on the input should prevent this injection:
rc = table.update_item(Key={ 'username' : user },
UpdateExpression="set parent_list[" + parseInt(itemnum) + "] = :nd",
ExpressionAttributeValues={
':nd': child_list,
},
ReturnValues="UPDATED_NEW"
);