why linked/coral project not support transform implicit join to relNode ?
how to support coral support implicit join transform to relNode
Related
I am using Apache Calcite to validate and rewrite SQL based on policies that put certain restrictions on these SQL queries. I am trying to modify a RelNode tree in order to rewrite the query to enforce these restrictions. I want to be able to remove certain parts from a query (after it has been validated). For example, I want to be able to remove projection fields (which I managed to do using RelBuilder.projectExcept) and to remove a table scan and its corresponding column references from a query.
Simple example:
SELECT a.foo, b.bar, c.baz
FROM a, b, c
WHERE a.index = b.index AND b.index = c.index
Let's say we want to remove table c from the query, to get to the following:
SELECT a.foo, b.bar
FROM a, b
WHERE a.index = b.index
I have tried using RelBuilder but this does not support removing nodes from the tree. I have also thought about an approach using RelVisitor but this seems quite complicated for this purpose. I think it would essentially require building a new RelNode tree. Lastly, implementing rules using RelRule seems like it would be a suitable option, but I cannot figure out from the Calcite documentation how to remove a particular RelNode and how to parameterize this (e.g. conditionally apply the rule if the table name is c).
Can anyone point me to a good approach? Alternatively, would it be easier to just modify the SqlNode parse tree?
A rule transforms (in this case TransformationRule) a RelNode to an equivalent RelNode i.e both should have the same row. Assumming you want to use HepPlanner with your custom rule registered and if the rule matches, it will eventually check whether the original rel and the transformed rel have the same row using RelOptUtil#verifyTypeEquivalence. I think mutating the relNode via RelVisitor or mutating the sqlNode via SqlVisitor is your best bet.
I am trying to convert sql query to Tinkerpop Gremlin. sql2Gremlin library does it but it looks on join as relation while I am relying on no join approach where you can refer relations with dot as delimiter between two entity.
I have parsed and validated query and I have RelRoot object.
Apache calcite returns RelRoot object which is root of algebraic expression.
Lets say I dont want to apply any query optimization, How do i use my RelNode Visitor to transform the RelRoot into TinkerPop Gremlin DSL.
Ideally I would first use From clause and then apply filters defined in where clause? How is select, filters, From clause represent in RelRoot tree?
What does apache calcite means by relational expression or RelNode?
Rephrasing the same question without TinkerPop Gremlin context:
How should I use RelRoot visitor to visit the RelRoot and transform the query to another DSL?
I don't know why you insist on RelRoot and not RelNode tree, but Apache Calcite is doing its optimizations of relational algebra in RelNode stack. There is a class called RelVisitor that you might find interesting, since it can do exactly what you need: visit all RelNodes. You can then extract information you need from them and build your DSL with it.
EDIT: In RelVisitor, you have access to the parent node and the child nodes of the currently visited node. You can extract all the information usually available to the RelNode object (see docs), and if you cast it to specific relational algebra operation, for example, Project, you can extract what fields are inside Project operation by doing node.getRowType().getFieldList().forEach(field -> names.add(field.getName())), where names is a previously defined Set<String>. You can find the full code here.
You should also take a look at the algebra docs to understand how SQL maps to relational algebra in Calcite before attempting this.
I am working on building a shiny app which would enable user to perform mathematical operations on the dataset fields.
Suppose my dataset has fields F1,F2,F3,F4, my intention is to provide end user with ability to generate any mathematical equation using operators(+-/*,log,sin) on these fields.
For Ex : - F1 + log(F2) / F3* (F4). For convenience the user should have the ability to generate this equation with option of drag and drop ( on both operators and fields). Please refer to this link for more clarity on my requirement
https://support.formstack.com/customer/portal/articles/1882321-new-and-improved-calculation-builder
I have tried making use of shiny packages such as shinyjqui and shinyDND with no luck so far.
suppose my query is: "select b.* from sourav_test.test1 a inner join sourav_test.test2 b on a.id=b.id". I need the best and efficient approach for apache beam to write this.
In Apache Beam SDK 2.5 a great approach is using the join library which performs SQL like joins. In the case of inner joins, the syntax would be as follows:
innerJoin(PCollection<KV<K,V1>> leftCollection,PCollection<KV<K,V2>> rightCollection)
Relating to your case, the left and side collections represents the collections to be inner joined. The K value would be the type of the key related to both collections. The Vs would represent the values of each collection respectively.
Why doesn't Informatica's joiner transformation support !=, >=, <= operators?
Why should they come up with a concept like lookup?
Joiner transformation is used for vertical consolidation.
e.g
order-tbl
order-id, item-id, item-qty
item-tbl
item-id, item-price, item-desc
Using a join condition on order-tbl.item-id = item-tbl.item-id you could print a report like this
order-id, item-id, item-price, item-desc
For vertical consolidation, I can't think of a scenerio needing other conditions like !=, >=, <=.
With lookup transformation, some core ETL tasks are made easy like
identifying if the incoming record is a new record (primary key doesn't exist) or an update to the existing record;
lookup a value e.g. lookup item-price from item-tbl to calculate order total.
Now you can join heterogeneous sources with a non-equi join condition using a Lookup Transformation's "Multi Match" feature
You can download sample from following Informatica Marketplace
https://community.informatica.com/solutions/mapping_multi_match_lookup_join