How to aggregate data from another cube in cubejs? - cube.js

I have the following cubes (I'm only showing the data necessary to reproduce the problem):
SentMessages:
cube(`SentMessages`, {
sql: `Select * from messages_sent`,
dimensions: {
campaignId: {
sql: `campaign_id`,
type: `number`
},
phone: {
sql: `phone_number`,
type: `number`
}
}
});
Campaigns:
cube(`Campaign`, {
sql: `SELECT * FROM campaign`,
joins: {
SentMessages: {
sql: `${Campaign}.id = ${SentMessages}.campaign_id`,
relationship: `hasMany`
}
},
measures: {
messageSentCount: {
sql: `${SentMessages}.phone`,
type: `count`
}
},
dimensions: {
name: {
sql: `name`,
type: `string`
},
}
});
The query being sent looks like this:
"query": {
"dimensions": ["Campaign.name"],
"timeDimensions": [
{
"dimension": "Campaign.createdOn",
"granularity": "day"
}
],
"measures": [
"Campaign.messageSentCount"
],
"filters": []
},
"authInfo": {
"iat": 1578961890,
"exp": 1579048290
},
"requestId": "da7bf907-90de-4ba0-80f8-1a802dd442f6"
For some reason this is resulting in the following error:
Error: 'Campaign.messageSentCount' references cubes that lead to row multiplication. Please rewrite it using sub query.
I've searched quite a bit on this error and cant find anything. Can someone please help or provide some insight into the problem? It would be really nice if the framework could show the erroneous sql generated just for troubleshooting purposes.

Campaign has many SentMessages and if joined to calculate Campaign.messageSentCount this calculation results might be affected. There's a simple check that ensures there're no hasMany cubes referenced inside aggregation function. This simple sanity check is required to avoid situation which leads to incorrect calculation results. For example if ReceivedMessages is also added as a join to the Campaign then Campaign.messageSentCount will generate incorrect results if ReceivedMessages and SentMessages are selected simultaneously.
To avoid this sanity check error, substitution with sub query is expected here as follows:
SentMessages:
cube(`SentMessages`, {
sql: `Select * from messages_sent`,
measures: {
count: {
type: `count`
}
},
dimensions: {
campaignId: {
sql: `campaign_id`,
type: `number`
},
phone: {
sql: `phone_number`,
type: `number`
}
}
});
Campaigns:
cube(`Campaign`, {
sql: `SELECT * FROM campaign`,
joins: {
SentMessages: {
sql: `${Campaign}.id = ${SentMessages}.campaign_id`,
relationship: `hasMany`
}
},
measures: {
totalMessageSendCount: {
sql: `${messageSentCount}`,
type: `sum`
}
},
dimensions: {
messageSentCount: {
sql: `${SentMessages.count}`,
type: `number`,
subQuery: true
},
name: {
sql: `name`,
type: `string`
},
}
});
For cases where Campaign.messageSentCount doesn't make any sense as a dimension, schema can be simplified and SentMessages.count can be used directly.

I figured part of this out on my own (at least the solution part), figured I'd post in case anyone else was having difficulty:
It appears that this definition is problematic (and uncessary):
messageSentCount: {
sql: `${SentMessages}.phone`,
type: `count`
}
I believe the correct way to do this is to add a measure to the table you want the COUNT to be applied to. In this query I want a count of SentMessages.phone (as shown above), so the following should be added to the SentMessages cube.
count: {
sql: `phone`
type: `count`,
},
Then the query works simply as follows:
"query": {
"dimensions": [
"Campaign.name"
],
"timeDimensions": [
{
"dimension": "SentMessages.createdOn",
"granularity": "day"
}
],
"measures": [
"SentMessages.count"
],
"filters": []
},
"authInfo": {
"iat": 1578964732,
"exp": 1579051132
},
"requestId": "c84b4596-2ee8-48e7-8e0a-974eb284dde3"
And it works as expected. I still don't understand the row multiplication error and why this measure doesn't work if placed on the Campaign cube. I will wait to accept this answer as i found this experimentally and still unclear of the problem.

Related

Cubejs Access Other Cubes By Variable

Given the following Cube, I can use a function to create dynamic measures.
cube(`Creatives`, {
sql: `SELECT * FROM public.creatives`,
joins: {
Events: {
relationship: `hasMany`,
sql: `${Events}.creative_id = ${CUBE}.id`,
},
},
measures: {
...['impression'].reduce((all, event) => {
return {
...all,
[`Total_${event}_events`]: {
type: `count`,
title: `Total ${event} events`,
filters: [
{
sql: `${Events.type} = '${event}'`,
},
],
},
};
}, {}),
},
dimensions: {
...
},
});
But when I try to move the reducer to a function, similar to the examples I get
ReferenceEvents is not defined
Which obviously is because there's no variable within the scope of my function, where previously Cubes interpolation of the string was replacing it for me.
How can I get access to other Cubes in functions similar to the below? I.e. get (or pass in Events to createTotalEventsMeasure()
const createTotalEventsMeasure = (event) => ({
[`Total_${event}_events`]: {
type: `count`,
title: `Total ${event} events`,
filters: [
{
sql: (CUBE) => `${Events.type} = '${event}'`,
},
],
},
});
cube(`Creatives`, {
sql: `SELECT * FROM public.creatives`,
joins: {
Events: {
relationship: `hasMany`,
sql: `${Events}.creative_id = ${CUBE}.id`,
},
},
measures: {
...['impression'].reduce(
(all, event) => ({
...all,
...createTotalEventsMeasure(event),
}),
{}
),
},
dimensions: {
...
},
});

cube.js join compile "error does not match any of the allowed types"

I have 2 tables, one contains daily data and the other contains attributes that I would like to use for segmenting data.
I got the following Error when I try to compile my cube.js schema.
cube.js error
Error: Error: Compile errors: DailyVolumes cube: "dimensions.wellId"
does not match any of the allowed types Possible reasons (one of): *
(dimensions.wellId.case) is required * (dimensions.wellId.sql = () =>
well_id) is not allowed * (dimensions.wellId.primary_key = true) is
not allowed
The followings are my tables DDL and cube.js Schemas:
drop table if exists daily_volumes;
drop table if exists wells;
create table if not exists wells (
id integer not null,
well_name varchar(255),
api_10 varchar(13),
area varchar(255),
run varchar(255),
engineering_id varchar(50),
accounting_id varchar(50),
active_flag int,
primary key (id)
);
create table if not exists daily_volumes(
well_id integer not null,
record_date timestamp not null,
oil_prod_bbl float not null,
water_prod_bbl float not null,
gas_prod_mcf float not null,
primary key (well_id, record_date),
constraint fk_well_id foreign key (well_id) references wells(id)
);
schema/Wells.js
cube(`Wells`, {
sql: `SELECT * FROM public.wells`,
preAggregations: {
// Pre-Aggregations definitions go here
// Learn more here: https://cube.dev/docs/caching/pre-aggregations/getting-started
},
joins: {
},
measures: {
count: {
type: `count`,
drillMembers: [id, wellName, engineeringId, accountingId]
}
},
dimensions: {
id: {
sql: `id`,
type: `number`,
primaryKey: true
},
wellName: {
sql: `well_name`,
type: `string`
},
api10: {
sql: `api_10`,
type: `string`,
title: `Api 10`
},
area: {
sql: `area`,
type: `string`
},
run: {
sql: `run`,
type: `string`
},
engineeringId: {
sql: `engineering_id`,
type: `string`
},
accountingId: {
sql: `accounting_id`,
type: `string`
}
},
dataSource: `default`
});
schema/DailyVolumes.js
cube(`DailyVolumes`, {
sql: `SELECT * FROM public.daily_volumes`,
preAggregations: {
// Pre-Aggregations definitions go here
// Learn more here: https://cube.dev/docs/caching/pre-aggregations/getting-started
},
joins: {
Wells: {
sql: `${CUBE}.well_id = ${Wells}.id`,
relationship: `belongsTo`,
},
},
measures: {
count: {
type: `count`,
sql: `id`,
// drillMembers: [recordDate],
},
},
dimensions: {
recordDate: {
sql: `record_date`,
type: `time`,
},
wellId: {
sql: `well_id`,
type: `number`,
primary_key: true,
},
},
dataSource: `default`,
});
I think the issue was that you used primary_key (snake case) instead of primaryKey (camel case), as described in docs: https://cube.dev/docs/schema/reference/dimensions#primary-key
I also have to admit that the error message is not very helpful now.
Setting primaryKey to true will change the default value of the shown
parameter to false. If you still want shown to be true — set it
manually.
Extracted from Documentation Page.

Cube.js time interval best practice

I have table with working time interval. what is the best way to model this in cube.js to allow time dimension queries like total worker working time, between dates, total worker time in day and so on.
Thanks!
the table looks like:
CREATE working_times test_timestamp (
id INT AUTO_INCREMENT PRIMARY KEY,
workerId VARCHAR(255) NOT NULL,
from TIMESTAMP,
to TIMESTAMP
);
and the cube:
cube(`WorkingTimes`, {
sql: `SELECT * FROM db.working_times`,
measures: {
???
},
dimensions: {
from: {
sql: `from`,
type: `time`
},
to: {
sql: `to`,
type: `time`
},
},
});
It can be defined as a timestamp difference. Pretending it's MySQL:
cube(`WorkingTimes`, {
sql: `SELECT * FROM db.working_times`,
measures: {
workingMinutes: {
sql: `TIMESTAMPDIFF(MINUTE, ${to}, ${from})`,
type: `sum`
}
},
dimensions: {
from: {
sql: `from`,
type: `time`
},
to: {
sql: `to`,
type: `time`
},
},
});

In cube.js, how would I query over contents of an array?

Let's say I have an array of info (ex: tags: ['red', 'blue', 'green'] ) in my data. How would I add that to my cube(s) to do something like a filter on tags array contains a particular value?
I'm specifically using the Athena driver with pre-aggregation into Aurora Postgres.
This is where I have gotten so far, but it's not quite there yet.
cube(`Events`, {
sql: `select * from events`,
joins: {
Tags: {
relationship: `hasMany`,
sql: `${tags}.id = ${tags}`
}
},
});
cube(`Tags`, {
sql: `UNNEST(tags) t (id, idx)`,
dimensions: {
tag: {
sql: `id`,
type: `string`
}
}
});
It's a right direction. Primary keys and select for events table in tags cube should be added:
cube(`Events`, {
sql: `select * from events`,
joins: {
Tags: {
relationship: `hasMany`,
sql: `${Events}.id = ${Tags}.id`
}
},
dimensions: {
id: {
sql: `id`,
type: `string`,
primaryKey: true
}
}
});
cube(`Tags`, {
sql: `select e.id, t.id as tag from events e CROSS JOIN UNNEST(tags) t (id, idx)`,
dimensions: {
id: {
sql: `id || tag`,
type: `string`,
primaryKey: true
},
tag: {
sql: `tag`,
type: `string`
}
}
});
Then equals filter on a Tags cube can be used to query it:
{
"measures": ["Events.count"],
"filters": [{
"dimension": "Tags.tag",
"operator": "equals",
"values": ["red"]
}]
}

couchdb - startkey endkey doesn't filter records with key arrays as expected

I have Couch-DB map-reduce view which outputs this;
{
rows: [
{
key: [
"2014-08-20",
2,
"registration"
],
value: 2
},
{
key: [
"2014-08-20",
2,
"search"
],
value: 3
},
{
key: [
"2014-08-21",
2,
"registration"
],
value: 3
},
{
key: [
"2014-08-21",
2,
"search"
],
value: 4
}
]
}
I need to query all the records that has between 2014-08-20 and 2014-08-21 Also the same time I need the integer value in the middle to be 2 and the last value to be "registration".
My curl request URL looks like this
BASE_URL?group=true&startkey=["2014-08-20",1,"registration"]
&endkey=["2014-08-21",1,"registration"]
This filters out the date but none of the other elements (the integer value and the string). Anyone got an idea whats happening?
Any help would be appreciated.
UPDATE
My Document structure looks something like this
[
{
platform_version: 2,
UDID: "EWHSJeNp20sBFuzqcorkKVVi",
session: {
timestamp: "2014-08-20T00:00:00.000Z",
session_id: "kOnNIhCNQ31LlkpEPQ7XnN1D",
ip: "202.150.213.66",
location: "1.30324,103.5498"
},
events: [
{
event_type: "search",
timestamp: "2014-08-21T00:00:00.000Z",
location: "1.30354,103.5301",
attributes: {
}
}
]
},
{
platform_version: 2,
UDID: "EWHSJeNp20sBFuzqcorkKVVi",
session: {
timestamp: "2014-08-21T00:00:00.000Z",
session_id: "kOnNIhCNQ31LlkpEPQ7XnN1D",
ip: "202.150.213.66",
location: "1.30324,103.5498"
},
events: [
{
event_type: "search",
timestamp: "2014-08-21T00:00:00.000Z",
location: "1.30354,103.5301",
attributes: {
}
}
]
},
{
platform_version: 2,
UDID: "EWHSJeNp20sBFuzqcorkKVVi",
session: {
timestamp: "2014-08-20T00:00:00.000Z",
session_id: "kOnNIhCNQ31LlkpEPQ7XnN1D",
ip: "202.150.213.66",
location: "1.30324,103.5498"
},
events: [
{
event_type: "click",
timestamp: "2014-08-21T00:00:00.000Z",
location: "1.30354,103.5301",
attributes: {
}
}
]
}
]
and the map reduce function looks like this.
function(doc) {
date = doc.session.timestamp.split("T")[0];
eventArray = doc.events;
for (i = 0; i < eventArray.length; i++) {
emit([doc.app_version,eventArray[i].event_type,date],1);
}
}
It started working after I change the order of the keys. but still, I can't use a wildcard to query all the event types.
You are getting documents with different second array item different then 1 because in the CouchDB the limits (startkey and endkey) are compared to map keys using lexicographical order. In the lexicographical order [1,1] < [1,2] < [2,1] < [2,2].
What you need is either multidimensional queries (which are not supported by CouchDB), additional client-side filtering (which may increase data transferred between CouchDB and your app) or additional server-side filtering with list function (which increase processing time of queries).
If your app needs filtering using range only on the first element of key array (just like in your example query), you would solve your problem easily by placing the item at the last position of array, eg. emitting [1, "registration", "2014-08-21"] instead of ["2014-08-21", 1, "registration"].