Cubejs Access Other Cubes By Variable - cube.js

Given the following Cube, I can use a function to create dynamic measures.
cube(`Creatives`, {
sql: `SELECT * FROM public.creatives`,
joins: {
Events: {
relationship: `hasMany`,
sql: `${Events}.creative_id = ${CUBE}.id`,
},
},
measures: {
...['impression'].reduce((all, event) => {
return {
...all,
[`Total_${event}_events`]: {
type: `count`,
title: `Total ${event} events`,
filters: [
{
sql: `${Events.type} = '${event}'`,
},
],
},
};
}, {}),
},
dimensions: {
...
},
});
But when I try to move the reducer to a function, similar to the examples I get
ReferenceEvents is not defined
Which obviously is because there's no variable within the scope of my function, where previously Cubes interpolation of the string was replacing it for me.
How can I get access to other Cubes in functions similar to the below? I.e. get (or pass in Events to createTotalEventsMeasure()
const createTotalEventsMeasure = (event) => ({
[`Total_${event}_events`]: {
type: `count`,
title: `Total ${event} events`,
filters: [
{
sql: (CUBE) => `${Events.type} = '${event}'`,
},
],
},
});
cube(`Creatives`, {
sql: `SELECT * FROM public.creatives`,
joins: {
Events: {
relationship: `hasMany`,
sql: `${Events}.creative_id = ${CUBE}.id`,
},
},
measures: {
...['impression'].reduce(
(all, event) => ({
...all,
...createTotalEventsMeasure(event),
}),
{}
),
},
dimensions: {
...
},
});

Related

AWS Glue - finding schema reference for table

Using AWS CDK 2 for creating schemas and tables, I seem to have problems linking the schemaReference
const schema = new glue.CfnSchema(this, "User", {
compatibility: "NONE",
dataFormat: "JSON",
name: "user",
schemaDefinition: JSON.stringify(userSchema),
});
new glue.CfnTable(
this,
"UserTable",
{
catalogId: this.account,
databaseName: "my_db",
tableInput: {
name: "users",
tableType: "EXTERNAL_TABLE",
storageDescriptor: {
location: "my_db.public.users",
schemaReference: schema,
},
parameters: {
classification: "postgresql",
typeOfData: "table",
connectionName: "rds_conn",
},
},
}
);
It seems like I'd expect schemaReference to be able to use the Cfn output in some way? I can only get this working by hard coding a schemaReference object with a schemaVersionId that I find in the console.
My solution was to lock the schema version in its definition, then to reference the schema by name. Example
new glue.CfnSchema(this, "User", {
name: "user",
// ...
checkpointVersion: {
versionNumber: 1,
},
});
new glue.CfnTable(
this,
"UserTable",
{
// ...
tableInput: {
// ...
storageDescriptor: {
// ...
schemaReference: {
schemaId: {
registryName: "default-registry",
schemaName: "user",
},
schemaVersionNumber: 1,
},
},
},
}
);
Though verbose, it has the advantage of being portable across stacks.

BatchWriteItemCommand with AWS.DynamoDB class using AWS SDK V3 in Nodejs

I have been trying for hours to perform a DynamoDB DeleteRequest using BatchWriteItemCommand but I keep getting the following error:
Error ValidationException: 1 validation error detected: Value null at 'requestItems.td_notes_sdk.member.1.member.deleteRequest.key' failed to satisfy constraint: Member must not be null
This is what my table looks like:
Partition key: user_id (string)
Sort key: timestamp (number)
DynamoDB Screenshot
This is what my code looks like:
// Import required AWS SDK clients and commands for Node.js
import {
DynamoDBClient,
BatchWriteItemCommand,
} from "#aws-sdk/client-dynamodb";
// Set the parameters
export const params = {
RequestItems: {
"td_notes_sdk": [
{
DeleteRequest: {
Item: {
Key: {
user_id: { S : "bb" },
timestamp: { N : 2 },
},
},
},
},
],
},
};
export const run = async () => {
const ddbClient = new DynamoDBClient({ region: "us-east-2" });
try {
const data = await ddbClient.send(new BatchWriteItemCommand(params));
console.log("Success, items inserted", data);
return data;
} catch (err) {
console.log("Error", err);
}
};
run();
Here are some resources that I've been trying to follow along with:
Resource 1: Writing items in Batch Example
Resource 2: AWS Javascript SDK v3 Documentation
Update: BatchWrite PutRequest work with the code below, so I know that the structure of my keys/attributes is closer to being correct. Still does not work for DeleteRequest.
export const params = {
RequestItems: {
"td_notes_sdk": [
{
PutRequest: {
Item: {
user_id: { "S": "bb" },
timestamp: { "N": "5" },
},
},
},
],
},
};
You don't supply an Item when deleting an item. You supply a Key.
Here is a working example:
const params_delete = {
RequestItems: {
"td_notes_sdk": [
{
DeleteRequest: {
Key: {
user_id: { S: "bb" },
timestamp: { N: "2" },
},
},
},
],
},
};
const delete_batch = async () => {
const ddbClient = new DynamoDBClient({ region: "us-east-2" });
try {
const data = await ddbClient.send(new BatchWriteItemCommand(params_delete));
console.log("Success, item deleted");
return data;
} catch (err) {
console.log("Error", err);
}
};
delete_batch();

chart js not dispalying data array that comes from an axios request

I have an API end point that returns an array of 24 values that I want to use in my chartjs within a vue component.
when the page loads I get no errors but the bars on the charts just don't show and I don't know why.
EDIT: I noticed that the async function returns a promise instead of the actual data:
async filterData() {
await this.$axios.get('/api/data_app/job_count_by_hour/')
.then(response => {
return this.chart_data = response.data;
})
}
here is the data return code, I have a function that populates the chart_data array :
data(){
return {
form:{
day: 'select day',
workspace:'',
machine_family: [],
duration: []
},
res: [],
total:[],
chart_data: [],
url: '/api/jobs/job_count_by_hour/',
days: ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "sunday"],
barChart2: {
labels: ["6h", "7h", "8h", "9h","10h","11h", "12h", "13h", "14h", "15h", "16h", "17h", "18h", "19h", "20h", "21h","22h", "23h", "00h"],
datasets: [{
label: ["popularity"],
backgroundColor:"#f93232" ,
data: this.chart_data
},],
},
}
},
methods: {
async filterData() {
let _url = `${this.url}`
await this.$axios.get(_url)
.then(response => {
this.chart_data = response.data;
})
return this.chart_data
},
},
mounted() {
this.filterData()
}
}
this is the chart component:
<script>
import { Line } from 'vue-chartjs'
export default {
extends: Line,
props: {
chartdata: {
type: Object,
default: null
},
options: {
type: Object,
default: null
}
},
mounted () {
this.renderChart(this.chartdata, this.options)
}
}
in the parent component It looks like this:
en <BarChart :labels="barChart2.labels"
:datasets="barChart2.datasets"
:height="100"
>
</BarChart>ter code here
Turns out that when you try to update nested data, the component doesn't re-render.
This is how I solved it, I put the entire object in an update function and call that function when i get my data from the back end, I hope this helps!:
methods: {
onInput(value) {
this.filterData()
},
updateChart(data) {
this.datasets = [{
label: ["popularity"],
backgroundColor:"#f93232",
data: data
}]
},
async loadData() {
await this.$axios.get(this.url)
.then(response => {
this.updateChart(response.data)
})
},
},
mounted() {
this.loadData()
},

In cube.js, how would I query over contents of an array?

Let's say I have an array of info (ex: tags: ['red', 'blue', 'green'] ) in my data. How would I add that to my cube(s) to do something like a filter on tags array contains a particular value?
I'm specifically using the Athena driver with pre-aggregation into Aurora Postgres.
This is where I have gotten so far, but it's not quite there yet.
cube(`Events`, {
sql: `select * from events`,
joins: {
Tags: {
relationship: `hasMany`,
sql: `${tags}.id = ${tags}`
}
},
});
cube(`Tags`, {
sql: `UNNEST(tags) t (id, idx)`,
dimensions: {
tag: {
sql: `id`,
type: `string`
}
}
});
It's a right direction. Primary keys and select for events table in tags cube should be added:
cube(`Events`, {
sql: `select * from events`,
joins: {
Tags: {
relationship: `hasMany`,
sql: `${Events}.id = ${Tags}.id`
}
},
dimensions: {
id: {
sql: `id`,
type: `string`,
primaryKey: true
}
}
});
cube(`Tags`, {
sql: `select e.id, t.id as tag from events e CROSS JOIN UNNEST(tags) t (id, idx)`,
dimensions: {
id: {
sql: `id || tag`,
type: `string`,
primaryKey: true
},
tag: {
sql: `tag`,
type: `string`
}
}
});
Then equals filter on a Tags cube can be used to query it:
{
"measures": ["Events.count"],
"filters": [{
"dimension": "Tags.tag",
"operator": "equals",
"values": ["red"]
}]
}

How to aggregate data from another cube in cubejs?

I have the following cubes (I'm only showing the data necessary to reproduce the problem):
SentMessages:
cube(`SentMessages`, {
sql: `Select * from messages_sent`,
dimensions: {
campaignId: {
sql: `campaign_id`,
type: `number`
},
phone: {
sql: `phone_number`,
type: `number`
}
}
});
Campaigns:
cube(`Campaign`, {
sql: `SELECT * FROM campaign`,
joins: {
SentMessages: {
sql: `${Campaign}.id = ${SentMessages}.campaign_id`,
relationship: `hasMany`
}
},
measures: {
messageSentCount: {
sql: `${SentMessages}.phone`,
type: `count`
}
},
dimensions: {
name: {
sql: `name`,
type: `string`
},
}
});
The query being sent looks like this:
"query": {
"dimensions": ["Campaign.name"],
"timeDimensions": [
{
"dimension": "Campaign.createdOn",
"granularity": "day"
}
],
"measures": [
"Campaign.messageSentCount"
],
"filters": []
},
"authInfo": {
"iat": 1578961890,
"exp": 1579048290
},
"requestId": "da7bf907-90de-4ba0-80f8-1a802dd442f6"
For some reason this is resulting in the following error:
Error: 'Campaign.messageSentCount' references cubes that lead to row multiplication. Please rewrite it using sub query.
I've searched quite a bit on this error and cant find anything. Can someone please help or provide some insight into the problem? It would be really nice if the framework could show the erroneous sql generated just for troubleshooting purposes.
Campaign has many SentMessages and if joined to calculate Campaign.messageSentCount this calculation results might be affected. There's a simple check that ensures there're no hasMany cubes referenced inside aggregation function. This simple sanity check is required to avoid situation which leads to incorrect calculation results. For example if ReceivedMessages is also added as a join to the Campaign then Campaign.messageSentCount will generate incorrect results if ReceivedMessages and SentMessages are selected simultaneously.
To avoid this sanity check error, substitution with sub query is expected here as follows:
SentMessages:
cube(`SentMessages`, {
sql: `Select * from messages_sent`,
measures: {
count: {
type: `count`
}
},
dimensions: {
campaignId: {
sql: `campaign_id`,
type: `number`
},
phone: {
sql: `phone_number`,
type: `number`
}
}
});
Campaigns:
cube(`Campaign`, {
sql: `SELECT * FROM campaign`,
joins: {
SentMessages: {
sql: `${Campaign}.id = ${SentMessages}.campaign_id`,
relationship: `hasMany`
}
},
measures: {
totalMessageSendCount: {
sql: `${messageSentCount}`,
type: `sum`
}
},
dimensions: {
messageSentCount: {
sql: `${SentMessages.count}`,
type: `number`,
subQuery: true
},
name: {
sql: `name`,
type: `string`
},
}
});
For cases where Campaign.messageSentCount doesn't make any sense as a dimension, schema can be simplified and SentMessages.count can be used directly.
I figured part of this out on my own (at least the solution part), figured I'd post in case anyone else was having difficulty:
It appears that this definition is problematic (and uncessary):
messageSentCount: {
sql: `${SentMessages}.phone`,
type: `count`
}
I believe the correct way to do this is to add a measure to the table you want the COUNT to be applied to. In this query I want a count of SentMessages.phone (as shown above), so the following should be added to the SentMessages cube.
count: {
sql: `phone`
type: `count`,
},
Then the query works simply as follows:
"query": {
"dimensions": [
"Campaign.name"
],
"timeDimensions": [
{
"dimension": "SentMessages.createdOn",
"granularity": "day"
}
],
"measures": [
"SentMessages.count"
],
"filters": []
},
"authInfo": {
"iat": 1578964732,
"exp": 1579051132
},
"requestId": "c84b4596-2ee8-48e7-8e0a-974eb284dde3"
And it works as expected. I still don't understand the row multiplication error and why this measure doesn't work if placed on the Campaign cube. I will wait to accept this answer as i found this experimentally and still unclear of the problem.