I am working on a custom connector for Power Bi to get Data from multiple APIs.
The user could select the API instance then the list of all available tables in this API should be shown in a list field. Then write the desired query in the Query field.
Actually I could show the of the list of tables when I hardcoded them in my code.
I want to dynalically show the tables depending on the selected API.
for example:
API 1 => table1, table2
API 2 => table3, table4, table5...
I use a function to get this list.
My question is how to render the list of table so it shows the table dynamically without hardcode it!
The hardcoded list of tables:
MyQueryType = type function (
APIType as (type text meta [
Documentation.FieldCaption = "API name",
Documentation.FieldDescription = "Enter the API name",
Documentation.AllowedValues = { "BS_PARAMETER", "Elastic API DSL", "Elastic API SQL" }
//Documentation.AllowedValues = Indices2
]),
PartnerCodeType as (type text meta [
Documentation.FieldCaption = "Partner code",
Documentation.FieldDescription = "Enter the partner code",
Documentation.AllowedValues = { "jp_demo", "bilan_sanguin" }
]),
TableType as (type text meta [
Documentation.FieldCaption = "Table",
Documentation.FieldDescription = "Enter you query",
Documentation.AllowedValues = { "table1", "table2", "table3", "table4", "table5" }
]),
QueryType as (type text meta [
Documentation.FieldCaption = "Query",
Documentation.FieldDescription = "Enter you query",
Documentation.SampleValues = { "{ ""query"": { ""match_all"":{} }}" },
Formatting.IsMultiLine = true
])
)
as table meta [
Documentation.Name = "My Custom Query",
Documentation.LongDescription = "Long Desc"
];
I created a function Indices() that gets the list of table for each API then I call it in the MyQueryType function:
MyQueryType = type function (
APIType as (type text meta [
Documentation.FieldCaption = "API name",
Documentation.FieldDescription = "Enter the API name",
Documentation.AllowedValues = { "BS_PARAMETER", "Elastic API DSL", "Elastic API SQL" }
//Documentation.AllowedValues = Indices2
]),
PartnerCodeType as (type text meta [
Documentation.FieldCaption = "Partner code",
Documentation.FieldDescription = "Enter the partner code",
Documentation.AllowedValues = { "jp_demo", "bilan_sanguin" }
]),
TableType as (type text meta [
Documentation.FieldCaption = "Table",
Documentation.FieldDescription = "dec",
Documentation.AllowedValues = Indices()
]),
QueryType as (type text meta [
Documentation.FieldCaption = "Query",
Documentation.FieldDescription = "Enter you query",
Documentation.SampleValues = { "{ ""query"": { ""match_all"":{} }}" },
Formatting.IsMultiLine = true
])
)
as table meta [
Documentation.Name = "My Custom Query",
Documentation.LongDescription = "Long Desc"
];
//Other code snippet
//Get ES Metadata: indexes
Indices =
let
api_url = "http://MyURL:9200/_cat/indices?format=json",
response = Web.Contents(api_url),
source = Json.Document(response),
#"Converted to Table" = Table.FromList(source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"index"}, {"Column1.index"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Column1",{{"Column1.index", type text}}),
mylist = Table.ToList(#"Expanded Column1")
//mylistb = Table.TransformColumnTypes(mylist,{{"Column1.index", type text}})
//totext = Text.Combine(mylist, ",")
in
//#"Changed Type";
mylist;
Related
I am getting below error when I am trying to apply the code and load the query.
The column 'Column1' of the table wasn't found
Here is the sample code of query:
let
BaseUrl = "https://jira.tools.com/rest/api/2/search?jql=project in ('ABC') AND issuetype = 'Test Case'",
JiraIDPerPage = 1000,
GetJson = (Url) =>
let
RawData = Web.Contents(Url),
Json = Json.Document(RawData)
in Json,
GetJiraIDCount = () =>
let Url = BaseUrl & "&maxResults=0",
Json = GetJson(Url),
Count = Json[#"total"]
in Count,
GetPage = (Index) =>
let Skip = "&startAt=" & Text.From(Index * JiraIDPerPage),
Top = "&maxResults=" & Text.From(JiraIDPerPage),
Url = BaseUrl & Skip & Top,
Json = GetJson(Url),
Value = Json[#"issues"]
in Value,
JiraIDCount = List.Max({ JiraIDPerPage, GetJiraIDCount() }),
PageCount = Number.RoundUp(JiraIDCount / JiraIDPerPage),
PageIndices = { 0 .. PageCount - 1 },
Pages = List.Transform(PageIndices, each GetPage(_)),
JiraID = List.Union(Pages),
Table = Table.FromList(JiraID, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(Table,"Column1", {"id", "key", "fields"}, {"Column1.id", "Column1.key", "Column1.fields"})
in
#"Expanded Column1"
I can't see any exceptions getting generated, However I try to apply the changes and when power BI loads the data it gives above error.
I was able to fix the above issue with below code.
let
Source = Json.Document(
Web.Contents(JIRA_URL,
[
RelativePath="rest/api/2/search",
Query=
[
maxResults="100",
jql= EPICS_QUERY,
startAt="0",
apikey="MjY0MzgyODgyNDg4On5RnBcfBqhio"
]
]
)),
numIssues = Source[total],
startAtList = List.Generate(()=>0, each _ < numIssues, each _ +100),
data = List.Transform(startAtList, each Json.Document(Web.Contents(JIRA_URL,
[
RelativePath="rest/api/2/search",
Query=
[
maxResults="100",
jql=EPICS_QUERY,
startAt=Text.From(_),
apikey="MjY0MzgyODgyNDg4On5RnBcfBqhio"
]
]))),
iLL = List.Generate(
() => [i=-1, iL={} ],
each [i] < List.Count(data),
each [
i = [i]+1,
iL = data{i}[issues]
],
each [iL]
),
// and finally, collapse that list of lists into just a single list (of issues)
issues = List.Combine(iLL),
#"Converted to Table" = Table.FromList(issues, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"key", "fields"}, {"issue", "fields"}),
#"Expanded fields" = Table.ExpandRecordColumn(#"Expanded Column1", "fields", {"assignee", "created", "creator", "description", "issuetype", "parent", "priority", "project", "reporter", "resolution", "resolutiondate", "status", "summary", "updated"}, {"assigneeF", "created", "creatorF", "description", "issuetypeF", "parentF", "priorityF", "projectF", "reporterF", "resolutionF", "resolutiondate", "statusF", "summary", "updated"}),
#"Expanded assignee" = Table.ExpandRecordColumn(#"Expanded fields", "assigneeF", {"key"}, {"assignee"}),
#"Expanded creator" = Table.ExpandRecordColumn(#"Expanded assignee", "creatorF", {"key"}, {"creator"}),
#"Expanded issuetype" = Table.ExpandRecordColumn(#"Expanded creator", "issuetypeF", {"name"}, {"issuetype"}),
#"Expanded priority" = Table.ExpandRecordColumn(#"Expanded issuetype", "priorityF", {"name"}, {"priority"}),
#"Expanded project" = Table.ExpandRecordColumn(#"Expanded priority", "projectF", {"key"}, {"project"}),
#"Expanded reporter" = Table.ExpandRecordColumn(#"Expanded project", "reporterF", {"key"}, {"reporter"}),
#"Expanded resolution" = Table.ExpandRecordColumn(#"Expanded reporter", "resolutionF", {"name"}, {"resolution"}),
#"Expanded status" = Table.ExpandRecordColumn(#"Expanded resolution", "statusF", {"name"}, {"status"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded status",{{"created", type datetimezone}, {"resolutiondate", type datetimezone}, {"updated", type datetimezone}}),
#"Expanded parentF" = Table.ExpandRecordColumn(#"Changed Type", "parentF", {"key"}, {"parent"})
in
#"Expanded parentF"
I'm using Datatables to display tables in Django on the server side. After searching for a phrase, I have a button ready to save the current table after filtering to Excel. I would like the second button to create a new table in the database and save the same table to it as for Excel.
I don't know how I can send AJAX data to Django and read it in views in such a way that I can query the database.
javascript:
$(document).ready(function () {
function newexportaction(e, dt, button, config) {
var self = this;
var oldStart = dt.settings()[0]._iDisplayStart;
dt.one('preXhr', function (e, s, data) {
data.start = 0;
data.length = 2147483647;
dt.one('preDraw', function (e, settings) {
// Call the original action function
if (button[0].className.indexOf('buttons-excel') >= 0) {
$.fn.dataTable.ext.buttons.excelHtml5.available(dt, config) ?
$.fn.dataTable.ext.buttons.excelHtml5.action.call(self, e, dt, button, config) :
$.fn.dataTable.ext.buttons.excelFlash.action.call(self, e, dt, button, config);
}
dt.one('preXhr', function (e, s, data) {
settings._iDisplayStart = oldStart;
data.start = oldStart;
});
});
});
dt.ajax.reload();
}
$("#tb").DataTable({
"lengthChange": true,
"paging": true,
"searching": true,
"oLanguage": {
"sSearch": "Szukaj:",
},
"language": {
"processing": "Przetwarzanie",
"lengthMenu": "Pokaż _MENU_ elementów",
"info": "Wyświetlono od _START_ do _END_ z _TOTAL_ elementów",
"zeroRecords": "Nie znaleziono pasujących elementów",
"paginate": {
"first": "Pierwsza",
"last": "Ostatnia",
"next": "Kolejna",
"previous": "Poprzednia"
},
"emptyTable": "Brak danych do wyświetlenia",
},
"processing": true,
"serverSide": true,
buttons: [
{
extend: 'excel',
titleAttr: 'Excel',
title: '',
action: newexportaction
},
],
dom: 'B<"clear">lfrtip',
// "destroy": true,
"pageLength": 15,
"ordering": false,
"ajax": {
"url": "paging2/",
"type": "get"
},
});
});
views.py:
def paging2(request):
draw = int (request.GET.get ('draw')) # record the number of operations
start = int (request.GET.get ('start')) # start position
length = int (request.GET.get ('length')) # length of each page
search_key = request.GET.get ('search[value]') # search keyword
order_column = request.GET.get ('order [0] [column]') # sort field index
order_column = request.GET.get ('order [0] [dir]') #Ordering rule: ase / desc
# sql query all data, then do paging, the speed is slow
# if search_key:
# result = query(search_key)
# else:
# result = select_all()
# data = result[start:start+length]
# count = len(result)
# sql for paging, fast
if search_key:
sql_search = "SELECT NAZWA,TEL,NIP,Adres,WWW, EMAIL, Branza, NAZWA_SCRAPERA FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL AND Branza like '%%%s%%'" % search_key
sql_search_len = "SELECT COUNT(*) FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL AND Branza like '%%%s%%'"% search_key
result, count = query(sql_search, sql_search_len)
data = result[start:start + length]
else:
sql = """
SELECT NAZWA,TEL,NIP,Adres,WWW, EMAIL, Branza, NAZWA_SCRAPERA FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL
LIMIT %s OFFSET %s
"""
data = select_by_page(length, start, sql)
# data = select_by_page(start, start + length)
sql_len = "SELECT COUNT(*) FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL"
count = all_count(sql_len)
dic = {
'draw': draw,
'recordsTotal': count,
'recordsFiltered': count,
'data': data,
}
return HttpResponse(json.dumps(dic), content_type='application/json')
I'm attempting to import data from Azure Application Insights into PowerBI. The issue is that, regardless of the timespan I set, I seem to only be pulling about a week's worth of data. Here's what the M query looks like:
let AnalyticsQuery =
let Source = Json.Document(Web.Contents("https://api.applicationinsights.io/v1/apps/<uuid>/query",
[Query=[#"query"="customEvents
| project customDimensions
",#"x-ms-app"="AAPBI",#"timespan"="P30D"],Timeout=#duration(0,0,60,0)])),
TypeMap = #table(
{ "AnalyticsTypes", "Type" },
{
{ "string", Text.Type },
{ "int", Int32.Type },
{ "long", Int64.Type },
{ "real", Double.Type },
{ "timespan", Duration.Type },
{ "datetime", DateTimeZone.Type },
{ "bool", Logical.Type },
{ "guid", Text.Type },
{ "dynamic", Text.Type }
}),
DataTable = Source[tables]{0},
Columns = Table.FromRecords(DataTable[columns]),
ColumnsWithType = Table.Join(Columns, {"type"}, TypeMap , {"AnalyticsTypes"}),
Rows = Table.FromRows(DataTable[rows], Columns[name]),
Table = Table.TransformColumnTypes(Rows, Table.ToList(ColumnsWithType, (c) => { c{0}, c{3}}))
in
Table
in
AnalyticsQuery
I was thinking this was a size issue, but I've already narrowed it down to a single column (albeit a wide one) and it's still not returning any more data.
Narrowing the returned dataset to two columns has increased the dataset to include a few weeks instead of less than a week, but I'm still looking for a bigger dataset. Here's the latest query:
let AnalyticsQuery =
let Source = Json.Document(Web.Contents("https://api.applicationinsights.io/v1/apps/<uuid>/query",
[Query=[#"query"="customEvents
| extend d=parse_json(customDimensions)
| project timestamp, d[""Properties""]
| order by timestamp desc
| where timestamp <= now() and d_Properties <> """"
",#"x-ms-app"="AAPBI"],Timeout=#duration(0,0,4,0)])),
TypeMap = #table(
{ "AnalyticsTypes", "Type" },
{
{ "string", Text.Type },
{ "int", Int32.Type },
{ "long", Int64.Type },
{ "real", Double.Type },
{ "timespan", Duration.Type },
{ "datetime", DateTimeZone.Type },
{ "bool", Logical.Type },
{ "guid", Text.Type },
{ "dynamic", Text.Type }
}),
DataTable = Source[tables]{0},
Columns = Table.FromRecords(DataTable[columns]),
ColumnsWithType = Table.Join(Columns, {"type"}, TypeMap , {"AnalyticsTypes"}),
Rows = Table.FromRows(DataTable[rows], Columns[name]),
Table = Table.TransformColumnTypes(Rows, Table.ToList(ColumnsWithType, (c) => { c{0}, c{3}}))
in
Table,
#"Sorted Rows" = Table.Sort(AnalyticsQuery,{{"timestamp", Order.Ascending}})
in
#"Sorted Rows"
You should look into either table buffering or directquery: see this discussion
When we using multiple IN conditions clause with QueryRequest, it only data which matches with the first item provided in IN condition.
I have Json format has
{
[
{
CID:1,
PId:1,
},
{
CID:2,
PID:2,
}
],
Age:30,
STATE:[
"AP", "TS"
],
Brands:[
"SAMSUNG","NOKIA","LG"
]
}
var filterExpression = " ((#CID= :CID1 and #PID = :PID1) or (#CID = :CID2 and #PID = :PID2) and #State in ( :state1, :state2) and #Brands in ( :brands1, :brands2)";
// Create Query request
QueryRequest request1 = new QueryRequest
{
TableName = "Table1",
IndexName = "Age-index",
KeyConditionExpression = "Age = :Age",
FilterExpression = filterExpression,
ExpressionAttributeNames = expressionAttributeNames,
ExpressionAttributeValues = expressionAttributeValues
};
var result = await dynamoClient.QueryAsync(request1);
I have 3 POCO classes as below;
Continent has many ContinentPart
Continent has many Country
Country has many City
I want to get Continens with ContinentParts, Countries and Cities
using (IDbConnection db = new SqlConnection(_conf["ConnectionStrings:WorkConStr"]))
{
string query = #"SELECT * FROM Continent as c
LEFT JOIN ContinentPart as cp ON c.ContinentId=cp.ContinentId
LEFT JOIN Country as co ON c.ContinentId=co.ContinentId
LEFT JOIN City ci ON co.CountryId=ci.CountryId
ORDER BY c.RecDate DESC";
var continentDictionary = new Dictionary<int, Continent>();
var list = db.Query<Continent, ContinentPart, Country, City, Continent>(
query,
map: (continent, continentPart, country, city) =>
{
Continent m;
if (!continentDictionary.TryGetValue(continent.ContinentId, out m))
{
continentDictionary.Add(continent.ContinentId, m = continent);
}
if (m.ContinentParts == null)
m.ContinentParts = new List<ContinentPart>();
m.ContinentParts.Add(continentPart);
if (m.Countries == null)
m.Countries = new List<Country>();
m.Countries.Add(country);
foreach (var h in m.Countries)
{
if (h.Cities == null)
h.Cities = new List<City>();
h.Cities.Add(city);
}
return m;
},
param: new { },
splitOn: "")
.Distinct()
.ToList();
return list;
}
You have two options here. You can use Multiple Resultsets feature:
https://medium.com/dapper-net/handling-multiple-resultsets-4b108a8c5172
Or, another option, better in my opinion, is to return the joined result as a JSON, for example with a query like the following:
select
continents.continent_id,
continents.continent,
countries.country_id,
countries.country,
cities.city_id,
cities.city
from
dbo.Continent continents
inner join
dbo.Country countries on continents.continent_id = countries.continent_id
inner join
dbo.City cities on countries.country_id = cities.country_id
for
json auto
that returns a JSON like this:
{
"continent_id": 1,
"continent": "Africa",
"countries":
[
{
"country_id": 1,
"country": "Nigeria",
"cities":
[
{
"city_id": 1,
"city": "Lagos"
}, {
"city_id": 2,
"city": "Abuja"
}
]
}
]
}
and that can then be turned into a complex object using Custom Type Handling:
https://medium.com/dapper-net/custom-type-handling-4b447b97c620
Links are to articles I've wrote on the subject.