Dynamic dataset url in Apache Superset with Apache Drill - apache-superset

I connect Apache Superset with apache Drill via sqlalchemy-drill.
Here is connection:
"type": "http",
"cacheResults": true,
"connections": {
"get": {
"url": "http://localhost:5500/",
"method": "GET",
"headers": null,
"authType": "none",
"userName": null,
"password": null,
"postBody": null,
"params": null,
"dataPath": null,
"requireTail": true,
"inputType": "json",
"xmlDataLevel": 1
}
},
"timeout": 5,
"proxyType": "direct",
"enabled": true
In Superset I getting the data:
SELECT * FROM api.get.`link`
But in api I need to get the data via dynamic urls like "link/1", "link/2" etc.
I'm trying to use Jinja template for inserting urla parameter into query:
SELECT * FROM api.get.{{ url_param('url') }}
And then in the dashbord use url like "http://localhost:8088/superset/dashboard/1/?url=link/1" for accessing to parameter? but it doesn't work.
Is there any way to use dynamic url for access to api from Superset?

Need to set ENABLE_TEMPLATE_PROCESSING in superset_config.py to TRUE and all works

Related

How to add a label to the combination of the url pattern and http method?

In our project, we must register all our endpoints with their related HTTP method to the third-party service. To organize the list of endpoints, we want to add labels to the combination of the URL and HTTP method. But I can't find a way to make this happen:/
Can anyone help me?
for more detail:
I have this path:
path("brands/", views.GetBrand.as_view({"get": "list", "post": "create"}))
I want to extract this information from the above line:
[
{
"label": ["get-brand", "brand-management"],
"method": "GET",
"url": "api/brands/"
},
{
"label": ["create-brand", "brand-management"],
"method": "post",
"url": "api/brands/"
}
I assumed that we can read the labels from the attribute that we set on the class view, can't I?!

Account Locked attribute not getting added in response for scim2 GET Users API in wso2

I am trying retrieve the user list which have locked accounts in WSO2 IS 5.9 version.
I tried after adding account lock attribute to below claims:
http://schemas.xmlsoap.org/ws/2005/05/identity
urn:ietf:params:scim:schemas:core:2.0
urn:ietf:params:scim:schemas:core:2.0:User
urn:ietf:params:scim:schemas:extension:enterprise:2.0:User
I have followed below URL as well to add the custom claim:
https://is.docs.wso2.com/en/5.9.0/develop/extending-scim2-user-schemas/
{ "attributeURI":"urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:accountLock",
"attributeName":"accountLock",
"dataType":"boolean",
"multiValued":"false",
"description":"Account lock",
"required":"false",
"caseExact":"false",
"mutability":"readwrite",
"returned":"default",
"uniqueness":"none",
"subAttributes":"null",
"multiValuedAttributeChildName":"null",
"canonicalValues":[],
"referenceTypes":[]
}
But still i am not able to get the accountLock attribute in response to GET Users API of scim2.
Response
"totalResults": 10,
"startIndex": 1,
"itemsPerPage": 10,
"schemas": [
"urn:ietf:params:scim:api:messages:2.0:ListResponse"
],
"Resources": [
{
"emails": [
"divya#abc.com"
],
"meta": {
"created": "2020-06-25T07:49:35.465Z",
"lastModified": "2020-06-25T11:20:13.482Z",
"resourceType": "User"
},
"name": {
"givenName": "guest",
"familyName": "guest"
},
"groups": [
{
"display": "Application/sp1"
},
{
"display": "Application/sp2"
},
{
"display": "Application/Read"
}
],
"id": "9ffbed2e-3703-470c-a2c8-e738f4c09709",
"userName": "guest12"
}
]}```
The following reasons may cause to accoutLock attribute does not appear in SCIM2 GET user response.
You might missed to add the new attribute ( "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:accountLock") as a sub attribute of urn:ietf:params:scim:schemas:extension:enterprise:2.0:User object. (Point 3 in https://is.docs.wso2.com/en/5.9.0/develop/extending-scim2-user-schemas/#extending-the-scim-20-api.
"subAttributes":"accoutLock verifyEmail askPassword employeeNumber costCenter organization division department manager")
"attributeURI":"urn:ietf:params:scim:schemas:extension:enterprise:2.0:User",
"attributeName":"urn:ietf:params:scim:schemas:extension:enterprise:2.0:User",
"dataType":"complex",
"multiValued":"false",
"description":"Enterprise User",
"required":"false",
"caseExact":"false",
"mutability":"readWrite",
"returned":"default",
"uniqueness":"none",
"subAttributes":"accoutLock verifyEmail askPassword employeeNumber costCenter organization division department manager",
"canonicalValues":[],
"referenceTypes":["external"]
}```
The mapped attribute of the added custom claim (https://is.docs.wso2.com/en/5.9.0/develop/extending-scim2-user-schemas/#add-the-custom-claim) should be an existing attribute in LDAP schema if you are using the default LDAP userstore. (However, if you have done this mistake you won't be able to update/add claim value. It gives One or more attributes you are trying to add/update are not supported by underlying LDAP for user: error)
The response of SCIM2 GET users doesn't contain the attributes which don't have a value. Thus, set true/false to the claim value.
Moreover, it is enough to add the new attribute to urn:ietf:params:scim:schemas:extension:enterprise:2.0:User claim dialect. Follow steps in https://is.docs.wso2.com/en/5.9.0/develop/extending-scim2-user-schemas/

'TypeError: Cannot initialize connector undefined: Cannot read property 'root' of undefined'

I'm using loopback3.x. I want to integrate 3rd party APIs with loopback. For that while using loopback-connector-rest shows the error 'TypeError: Cannot initialize connector undefined: Cannot read property 'root' of undefined'. How to fix it?
Datasource configuration
"restDataSource": {
"name": "restDataSource",
"baseURL": "",
"crud": true,
"connector": "rest",
"debug": false,
"options": {
"headers": {
"accept": "application/json",
"content-type": "application/json",
"authorization": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
"strictSSL": false
},
"operations": [{
"template": {
"method": "POST",
"url": "https://fcm.googleapis.com/fcm/send",
"options": {
"strictSSL": true,
"useQuerystring": true
}
},
"functions": {
"notify": ["title", "text", "click_action", "keyname", "to"]
}
}]
}
Error
TypeError: Cannot create data source "restDataSource": Cannot initialize connector "rest": Cannot read property 'root' of undefined
at /home/veena-msl/Documents/care-doc-api/node_modules/loopback-connector-rest/lib/rest-connector.js:93:28
at Array.forEach (<anonymous>)
at /home/veena-msl/Documents/care-doc-api/node_modules/loopback-connector-rest/lib/rest-connector.js:87:22
at Array.forEach (<anonymous>)
at Function.initializeDataSource [as initialize] (/home/veena-msl/Documents/care-doc-api/node_modules/loopback-connector-rest/lib/rest-connector.js:52:25)
at DataSource.setup (/home/veena-msl/Documents/care-doc-api/node_modules/loopback-datasource-juggler/lib/datasource.js:493:19)
at new DataSource (/home/veena-msl/Documents/care-doc-api/node_modules/loopback-datasource-juggler/lib/datasource.js:138:8)
at Registry.createDataSource (/home/veena-msl/Documents/care-doc-api/node_modules/loopback/lib/registry.js:364:12)
at dataSourcesFromConfig (/home/veena-msl/Documents/care-doc-api/node_modules/loopback/lib/application.js:570:19)
at Function.app.dataSource (/home/veena-msl/Documents/care-doc-api/node_modules/loopback/lib/application.js:269:14)
at /home/veena-msl/Documents/care-doc-api/node_modules/loopback-boot/lib/executor.js:191:9
at /home/veena-msl/Documents/care-doc-api/node_modules/loopback-boot/lib/executor.js:282:5
at Array.forEach (<anonymous>)
at forEachKeyedObject (/home/veena-msl/Documents/care-doc-api/node_modules/loopback-boot/lib/executor.js:281:20)
at setupDataSources (/home/veena-msl/Documents/care-doc-api/node_modules/loopback-boot/lib/executor.js:181:3)
at execute (/home/veena-msl/Documents/care-doc-api/node_modules/loopback-boot/lib/executor.js:39:3)
In the functions section, you have defined a function notify accepting several input arguments: title, text, click_action, keyname, to. However, the template section does not provide any information on how to map those arguments to an HTTP request. For example, is the title supposed to be sent via URL query or in the request body?
IIUC, you are trying to use the legacy Firbase Cloud Messaging HTTP API as described here: https://firebase.google.com/docs/cloud-messaging/http-server-ref Based on that documentation, I think all arguments of your function should be sent in the request body.
It looks like LoopBack's REST connector is not correctly detecting and handling the situation when an input argument is not mapped to any HTTP source. It should not be crashing, feel free to open a bug report in https://github.com/strongloop/loopback-connector-rest/issues
Here is a configuration that does not crash the server. I don't have a Firebase account to verify that it's working as expected.
"template": {
"method": "POST",
"url": "https://fcm.googleapis.com/fcm/send",
"options": {
"strictSSL": true,
"useQuerystring": true
},
"body": {
"title": "{title:string}",
"text": "{text:string}",
"click_action": "{click_action:string}",
"keyname": "{keyname:string}",
"to": "{to:string}"
}
},
You can learn more about different ways how to configure input arguments in the connector documentation: https://loopback.io/doc/en/lb3/REST-connector.html#defining-a-custom-method-using-a-template

Loopback 4 The swagger generated filters don't seem to work

I set up a basic model and controller connected to mysql database. Everything looks good and is working.
My Definition Model looks like this.
import { Entity, model, property } from '#loopback/repository';
#model({
settings: {
mysql: {
table: 'definition'
}
}
})
export class Definition extends Entity {
#property({
type: 'number',
id: true,
generated: true,
forceId: true,
required: false,
description: 'The unique identifier for a definition',
})
id: number;
#property({
type: 'string',
required: true,
})
name: string;
#property({
type: 'string',
required: true,
})
process_id: string;
constructor(data?: Partial<Definition>) {
super(data);
}
}
export interface DefinitionRelations {
// describe navigational properties here
}
export type DefinitionWithRelations = Definition & DefinitionRelations;
When I have it up and running and click on the explorer to test it out.
I click on "Try it out" for get /definitions and the only editable field that gets enabled is the filter field.
It is repopulated with a value like this...
{
"where": {},
"fields": {
"id": true,
"name": true,
"process_id": true
},
"offset": 0,
"limit": 0,
"skip": 0,
"order": [
"string"
]
}
I have to clear that to get it to work which is fine. When I run it with no filter it returns these results.
[
{
"id": 10,
"name": "Place Manual Payoff Order Process Config",
"process_id": "Process_PlaceManualPayoffOrderProcessConfig"
},
{
"id": 11,
"name": "test",
"process_id": "Test"
},
{
"id": 12,
"name": "test2",
"process_id": "test2"
}
]
I am trying to use the filter expression to only return ones with a specific process_id field. So I change the filter to look like this.
{
"where": {"process_id": "test2"}
}
And it still returns the same results.
[
{
"id": 10,
"name": "Place Manual Payoff Order Process Config",
"process_id": "Process_PlaceManualPayoffOrderProcessConfig"
},
{
"id": 11,
"name": "test",
"process_id": "Test"
},
{
"id": 12,
"name": "test2",
"process_id": "test2"
}
]
Do filters currently work in Loopback 4 or am I using them incorrectly?
EDIT: if I post the filters in the URL string they work. It seems as though the openapi ui isn't generating that part of the filter Into the URL string.
It seems as though the openapi ui isn't generating that part of the filter Into the URL string.
Yes, this a precise description.
OpenAPI Spec version 3.x does not specify how to serialize deeply-nested values to URL queries and swagger-js, the library powering swagger-ui (which powers LoopBack's REST API Explorer), silently ignores such values.
The issue has been reported and is being discussed here:
swagger-api/swagger-js#1385
OAI/OpenAPI-Specification#1706

HIVE_INVALID_METADATA in Amazon Athena

How can I work around the following error in Amazon Athena?
HIVE_INVALID_METADATA: com.facebook.presto.hive.DataCatalogException: Error: : expected at the position 8 of 'struct<x-amz-request-id:string,action:string,label:string,category:string,when:string>' but '-' is found. (Service: null; Status Code: 0; Error Code: null; Request ID: null)
When looking at position 8 in the database table connected to Athena generated by AWS Glue, I can see that it has a column named attributes with a corresponding struct data type:
struct <
x-amz-request-id:string,
action:string,
label:string,
category:string,
when:string
>
My guess is that the error occurs because the attributes field is not always populated (c.f. the _session.start event below) and does not always contain all fields (e.g. the DocumentHandling event below does not contain the attributes.x-amz-request-id field). What is the appropriate way to address this problem? Can I make a column optional in Glue? Can (should?) Glue fill the struct with empty strings? Other options?
Background: I have the following backend structure:
Amazon PinPoint Analytics collects metrics from my application.
The PinPoint event stream has been configured to forward the events to an Amazon Kinesis Firehose delivery stream.
Kinesis Firehose writes data to S3
Use AWS Glue to crawl S3
Use Athena to write queries based on the databases and tables generated by AWS Glue
I can see PinPoint events successfully being added to json files in S3, e.g.
First event in a file:
{
"event_type": "_session.start",
"event_timestamp": 1524835188519,
"arrival_timestamp": 1524835192884,
"event_version": "3.1",
"application": {
"app_id": "[an app id]",
"cognito_identity_pool_id": "[a pool id]",
"sdk": {
"name": "Mozilla",
"version": "5.0"
}
},
"client": {
"client_id": "[a client id]",
"cognito_id": "[a cognito id]"
},
"device": {
"locale": {
"code": "en_GB",
"country": "GB",
"language": "en"
},
"make": "generic web browser",
"model": "Unknown",
"platform": {
"name": "macos",
"version": "10.12.6"
}
},
"session": {
"session_id": "[a session id]",
"start_timestamp": 1524835188519
},
"attributes": {},
"client_context": {
"custom": {
"legacy_identifier": "50ebf77917c74f9590c0c0abbe5522d2"
}
},
"awsAccountId": "672057540201"
}
Second event in the same file:
{
"event_type": "DocumentHandling",
"event_timestamp": 1524835194932,
"arrival_timestamp": 1524835200692,
"event_version": "3.1",
"application": {
"app_id": "[an app id]",
"cognito_identity_pool_id": "[a pool id]",
"sdk": {
"name": "Mozilla",
"version": "5.0"
}
},
"client": {
"client_id": "[a client id]",
"cognito_id": "[a cognito id]"
},
"device": {
"locale": {
"code": "en_GB",
"country": "GB",
"language": "en"
},
"make": "generic web browser",
"model": "Unknown",
"platform": {
"name": "macos",
"version": "10.12.6"
}
},
"session": {},
"attributes": {
"action": "Button-click",
"label": "FavoriteStar",
"category": "Navigation"
},
"metrics": {
"details": 40.0
},
"client_context": {
"custom": {
"legacy_identifier": "50ebf77917c74f9590c0c0abbe5522d2"
}
},
"awsAccountId": "[aws account id]"
}
Next, AWS Glue has generated a database and a table. Specifically, I see that there is a column named attributes that has the value of
struct <
x-amz-request-id:string,
action:string,
label:string,
category:string,
when:string
>
However, when I attempt to Preview table from Athena, i.e. execute the query
SELECT * FROM "pinpoint-test"."pinpoint_testfirehose" limit 10;
I get the error message described earlier.
Side note, I have tried to remove the attributes field (by editing the database table from Glue), but that results in Internal error when executing the SQL query from Athena.
This is a known limitation. Athena table and database names allow only underscore special characters#
Athena table and database names cannot contain special characters, other than underscore (_).
Source: http://docs.aws.amazon.com/athena/latest/ug/known-limitations.html
Use tick (`) when table name has - in the name
Example:
SELECT * FROM `pinpoint-test`.`pinpoint_testfirehose` limit 10;
Make sure you select "default" database on the left pane.
I believe the problem is your struct element name: x-amz-request-id
The "-" in the name.
I'm currently dealing with a similar issue since my elements in my struct have "::" in the name.
Sample data:
some_key: {
"system::date": date,
"system::nps_rating": 0
}
Glue derived struct Schema (it tried to escape them with ):
struct <
system\:\:date:String
system\:\:nps_rating:Int
>
But that still gives me an error in Athena.
I don't have a good solution for this other than changing Struct to STRING and trying to process the data that way.