I need some help with processing queries using Apache Calcite, I am currently using Planner to process my queries.
My code looks like this:
// custom config
FrameworkConfig frameworkConfig = Frameworks.newConfigBuilder()
.parserConfig(parserConfig)
.defaultSchema(rootSchema)
.sqlToRelConverterConfig(convertConfig)
.operatorTable(operatorTable)
.build();
Planner planner = Frameworks.getPlanner(frameworkConfig);
String sql = "Select id, name from t where id in (?, ?)";
SqlNode parsedSqlNode = planner.parse(sql);
SqlNode validatedSqlNode = planner.validate(parsedSqlNode);
RelRoot relRoot = planner.rel(validatedSqlNode);
PreparedStatement preparedStatement = RelRunners.run(relRoot.rel);
preparedStatement.setLong(1, 11111);
preparedStatement.setLong(2, 22222);
ResultSet resultSet = preparedStatement.executeQuery();
// ....
It works fine without SQL parameters, but there are some queries with SQL parameters.
While using SQL parameter and setting the parameter(preparedStatement.setLong), I got the following error message:
Caused by: java.sql.SQLException: parameter ordinal 1 out of range
at org.apache.calcite.avatica.Helper.createException(Helper.java:60)
at org.apache.calcite.avatica.AvaticaPreparedStatement.getParameter(AvaticaPreparedStatement.java:427)
at org.apache.calcite.avatica.AvaticaPreparedStatement.getSite(AvaticaPreparedStatement.java:434)
at org.apache.calcite.avatica.AvaticaPreparedStatement.setLong(AvaticaPreparedStatement.java:178)
SQL parameters are obviously not being handled correctly. What is the proper way to define and set SQL parameters when using Planner in a case like this?
Related
i want to generate sql use calcite. like this
org.apache.calcite.rel.rel2sql.RelToSqlConverterTest#testAntiJoin
final FrameworkConfig frameworkConfig = Frameworks.newConfigBuilder()
.parserConfig(SqlParser.Config.DEFAULT)
// .defaultSchema(schema)
.build();
final RelBuilder builder = RelBuilder.create(frameworkConfig);
final RelBuilder builder = relBuilder();
final RelNode root = builder
.scan("DEPT")
.scan("EMP")
.join(
JoinRelType.ANTI, builder.equals(
builder.field(2, 1, "DEPTNO"),
builder.field(2, 0, "DEPTNO")))
.project(builder.field("DEPTNO"))
.build();
but if i don't set the schema, the exception table not found will be throw.
is there any way to generate sql without schema info.
the aim is generate sql. just generate sql.
reply for first answer. because comment character length limit.
My scenario is Business Intelligence. DataSource can be many, such as Hive, ClickHouse, and so on. And there are many tables. I also need to dynamically delete or add datasource. So I don't think it's appropriate for Calcite to be aware of all the data sources. I have two more questions:
How to create 'free-standing' table objects as you said
Check whether SqlNode can be used to do this. for example:
SqlIdentifier from = new SqlIdentifier("testTable", SqlParserPos.QUOTED_ZERO);
SqlNode[] nodes = new SqlNode[2];
nodes[0] = new SqlIdentifier("a", SqlParserPos.QUOTED_ZERO);
nodes[1] = SqlLiteral.createExactNumeric("1", SqlParserPos.QUOTED_ZERO);
SqlNode where = new SqlBasicCall(SqlStdOperatorTable.EQUALS, nodes, SqlParserPos.QUOTED_ZERO);
SqlIdentifier selectNode = new SqlIdentifier("a", SqlParserPos.QUOTED_ZERO);
SqlSelect select = new SqlSelect(SqlParserPos.QUOTED_ZERO, SqlNodeList.EMPTY,
new SqlNodeList(Arrays.asList(selectNode), SqlParserPos.QUOTED_ZERO),
from,
where,
null,
null,
null,
null,
null,
null,
null);
SqlString sqlString = select.toSqlString(CalciteSqlDialect.DEFAULT);
System.out.println(sqlString.getSql());
Only one method in RelBuilder uses a RelOptSchema: scan(String...) (and its variant Scan(Iterable<String>)). Which makes sense when you consider that the purpose of RelOptSchema is as a directory service, converting a table name (or table path, consisting of a table name qualified with catalog and/or schema names) into a RelOptTable object.
If you have 'free-standing' table objects that are not accessed via a namespace then you can create TableScan relational expressions directly and then call RelBuilder.push(RelNode) to add them to the stack. Since you never call RelBuilder.scan you can create RelBuilder with a null RelOptSchema.
But in your case, it looks as if you don't have free-standing table objects. That's a problem for Calcite, because it needs to know that your "EMP" table has a field called "DEPTNO" and it has type INTEGER.
So I suggest that you create a 'virtual' schema that contains type information but is not necessarily backed by real tables. The MockCatalogReader class, used in several of Calcite's tests, is a good example to follow.
I have a gcp based environment. I use standard SQL scripting in gcp BigQuery and federated query to cloudsql MySql. Federated query selects data from cloudsql mysql database. I need to select data from cloudsql mysql database based on condition that depends on data in BigQuery. I use variables in standard sql scriping in gcp bigquery to store the value that I select from bigquery. I want to value of this variable in the where clause of mysql query. See following example where I select a date from BigQuery and store it in a variable "BQ_LAST_DATETIME".
DECLARE BQ_LAST_DATETIME DATETIME
SET BQ_LAST_DATETIME = (select max(date_created) from bq_my_dataset.bq_my_table);
Since I am using bigquery federated query to read data out of cloudsql database (https://cloud.google.com/bigquery/docs/cloud-sql-federated-queries) as shown below and I want to use value that I stored in the variable "BQ_LAST_DATETIME" in the mysql query where clause
SELECT * FROM EXTERNAL_QUERY("my-gcp-project.my-region.my-connection2-cloudsql", "select * from mysqlschema.mysql_table where where date_created = #BQ_LAST_DATETIME;" );
Please note that in above query I have used "#BQ_LAST_DATETIME" as a placeholder to show what I want to achieve. I am not sure if I can directly use bigquery scripting variable as query parameter in the "external" query part of federated query.
Any suggestions on how to achieve parametrization of external queries in federated query, or if you know how I could achieve effect similar to what my intent is?
I actually tried following as depicted . I used bigquery scripting variable as query parameter in the "external" query part of federated query. only nuance here is that since the I was dealing with dates I performed a cast and also since the date variable actually is treated as a string I formatted it back to date using mysql STR_TO_DATE as follows
DECLARE BQ_LAST_DATETIME DATETIME
SET BQ_LAST_DATETIME = (select max(date_created) from bq_my_dataset.bq_my_table);
SET BQ_LAST_DATE= CAST(BQ_LAST_DATETIME AS DATE);
SELECT * FROM EXTERNAL_QUERY("my-gcp-project.my-region.my-connection2-cloudsql", "select * from mysqlschema.mysql_table where where date_created = STR_TO_DATE(#BQ_LAST_DATE,'%Y-%m-%d') ;" );
While this query is accepted by parser it is NOT giving expected result.
Basically the value of the variable #BQ_LAST_DATE does not seem to get to MySQL query as expected.
Does anyone know what am I missing ?
Thanks a lot for your help
You can try EXECUTE IMMEDIATE:
DECLARE BQ_LAST_DATETIME STRING;
DECLARE DSQL STRING;
SET BQ_LAST_DATETIME = 'SELECT max(date_created) from bq_my_dataset.bq_my_table';
SET DSQL = '"select * from mysqlschema.mysql_table where date_created = (' || BQ_LAST_DATETIME || ')"';
EXECUTE IMMEDIATE 'SELECT * FROM EXTERNAL_QUERY("my-gcp-project.my-region.my-connection2-cloudsql",' || DSQL || ');'
I'm using pro c/c++ in a unix environment. Inside a c function I create a simple select statement which is not inside a loop and autocommit is disabled.
This is the dynamic sql;
char sql_statement[200];
int num1, num2;
...
snprintf(sql_statement, sizeof(sql_sattement), "select column1, column2 from '%s' where num = '%d' and code_cfg = '%d'", "customer", 0, 0);
exec sql prepare instruction_to_execute from :sql_statement;
exec sql declare crs cursor for instruction_to_execute;
exec sql open crs;
exec sql fetch crs into :num1, :num2;
...
Executing this code gives me ORA-01002 error. As I said before, this code is not inside a loop and autocommit is off.
But if I write this code statically, works fine. Below the code:
EXEC SQL
SELECT column1, column2
INTO :num1, :num2
FROM CUSTOMER
where num = 0
AND code_cfg = 0;
SQL instruction is simple. For my understanding this code should be executed fine.
In SO I found some answers:
ORA-01002: fetch out of sequence C++
ORA-01002: fetch out of sequence
Hibernate error “ORA-01002” when persisting entity with custom Sequence Generator
ORA-01002: fetch out of sequence
java.sql.SQLException: ORA-01002: fetch out of sequence
How do I create an insert query in Doctrine that will perform the same function as the following SQL query:
INSERT INTO target (tgt_col1, tgt_col2)
SELECT 'flag' as marker, src_col2 FROM source
WHERE src_col1='mycriteria'
Doctrine documentation says:
If you want to execute DELETE, UPDATE or INSERT statements the Native
SQL API cannot be used and will probably throw errors. Use
EntityManager#getConnection() to access the native database connection
and call the executeUpdate() method for these queries.
Examples
// Get entity manager from your context.
$em = $this->getEntityManager();
/**
* 1. Raw query
*/
$query1 = "
INSERT INTO target (tgt_col1, tgt_col2)
SELECT 'flag' as marker, src_col2 FROM source
WHERE src_col1='mycriteria'
";
$affectedRows1 = $em->getConnection()->executeUpdate($query1);
/**
* 2. Query using class metadata.
*/
$metadata = $em->getClassMetadata(Your\NameSpace\Entity\Target::class);
$tableName = $metadata->getTableName();
$niceTitle = $metadata->getColumnName('niceTitle');
$bigDescription = $metadata->getColumnName('bigDescription');
$metadata2 = $em->getClassMetadata(Your\NameSpace\Entity\Source::class);
$table2Name = $metadata2->getTableName();
$smallDescription = $metadata2->getColumnName('smallDescription');
$query2 = "
INSERT INTO $tableName ($niceTitle, $bigDescription)
SELECT 'hardcoded title', $smallDescription FROM $table2Name
WHERE $niceTitle = 'mycriteria'
";
$affectedRows2 = $em->getConnection()->executeUpdate($query2);
I'm still not convinced it's the right approach you are taking but if you really need an SQL query to be run for whatever reason you can do that in Doctrine with $entityManager->createNativeQuery(); function:
http://doctrine-orm.readthedocs.org/en/latest/reference/native-sql.html
Doctrine isn't a tool for query manipulation. The whole idea is to work on Entity level, not the SQL level (tables, etc). Doctrine's 2 QueryBuilder doesn't even support INSERT operations via DQL.
A small snippet of pseudo code below to illustrate how it can be done in "Doctrine's way":
$qb = $entityManager->createQueryBuilder();
$qb->select('s')
->from('\Foo\Source\Entity', 's')
->where('s.col1 = :col1')
->setParameter('col1', 'mycriteria');
$sourceEntities = $qb->getQuery()->getResult();
foreach($sourceEntities as $sourceEntity) {
$targetEntity = new \Foo\Target\Entity();
$targetEntity->col1 = $sourceEntity->col1;
$targetEntity->col2 = $sourceEntity->col2;
$entityManager->persist($targetEntity);
}
$entityManager->flush();
Is it possible to retrieve a List<Date> instead of a List<SomeBean> using Ebean ?
For example, I have this model:
Days(id, name, day);
I'd like to do something like:
List<Date> dates = Ebean.createQuery(Date.class, "SELECT day FROM days").findList();
Of course, this doesn't work and returns this:
PersistenceException: java.util.Date is NOT an Entity Bean registered with this server?
How can I do that?
you can use sqlQuery see SqlQuery in Ebean :
The database retrieves time and date data with a type LocalDateTime
String sql = "select day from days";
List<SqlRow> row = DB.sqlQuery(sql).findList();
List<LocalDateTime> days = row.stream().map(r->(LocalDateTime) r.get("day")).collect(Collectors.toList());