I have a project that requires using Calcite's SQL Parser to parse large amounts of DDL statements which are in heavy MySQL dialect.
I have isolated an issue which can be illustrated with this specific example:
create table `t1` (x int(11))
The astute reader may recognize int(11) is MySQL dialect. I need to be able to parse this. Removing the (11) in this example is not an acceptable fix.
Here's a Java JUnit test that reproduces the problem:
public class SqlParserMysqlTest {
private static final Config CONFIG = SqlParser.Config.DEFAULT
.withLex(Lex.MYSQL).withConformance(SqlConformanceEnum.MYSQL_5)
.withParserFactory(SqlDdlParserImpl.FACTORY);
#Test
public void testMysqlCreate() throws SqlParseException {
SqlParser sqlParser = SqlParser.create(
new SourceStringReader("create table `t1` (x int(11))"), CONFIG);
SqlNode sqlNode = sqlParser.parseQuery();
Assert.assertEquals(SqlKind.CREATE_TABLE, sqlNode.getKind());
SqlCreateTable create = (SqlCreateTable) sqlNode;
Assert.assertEquals("T1", create.name.getSimple());
final SqlColumnDeclaration sqlColumnDeclaration = (SqlColumnDeclaration) create.columnList
.get(0);
Assert.assertEquals("X", sqlColumnDeclaration.name.getSimple());
Assert.assertEquals("INTEGER",
sqlColumnDeclaration.dataType.getTypeName().getSimple());
}
}
POM dependencies:
<dependency>
<groupId>org.apache.calcite</groupId>
<artifactId>calcite-core</artifactId>
<version>1.26.0</version>
</dependency>
<dependency>
<groupId>org.apache.calcite</groupId>
<artifactId>calcite-server</artifactId>
<version>1.26.0</version>
</dependency>
The issue is that the parser barfs on the parenthesis in int(11):
org.apache.calcite.sql.parser.SqlParseException: Encountered "(" at line 1, column 25.
Was expecting one of:
"ARRAY" ...
"AS" ...
"DEFAULT" ...
"GENERATED" ...
"MULTISET" ...
"NOT" ...
"NULL" ...
")" ...
"," ...
at org.apache.calcite.sql.parser.ddl.SqlDdlParserImpl.convertException(SqlDdlParserImpl.java:394)
at org.apache.calcite.sql.parser.ddl.SqlDdlParserImpl.normalizeException(SqlDdlParserImpl.java:157)
at org.apache.calcite.sql.parser.SqlParser.handleException(SqlParser.java:140)
at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:155)
I know that other people are successfully using Calcite to parse MySQL syntax, so what am I doing wrong?
From the documentation on table elements, it looks like the corrected syntax could be create table `t1` (x int 11) (losing the internal parentheses) but I have no way to test it.
Related
I'm using some custom DQL functions to filter rows by some JSONB fields in PostgreSQL. Here's my query function:
private function findTopLevelResources(): array {
return $this->createQueryBuilder('r')
->where("JSON_EXISTS(r.contents, '-1') = FALSE")
->getQuery()
->getResult();
}
Running this code results in DriverException from AbstractPostgreSQLDriver:
An exception occurred while executing 'SELECT r0_.id AS id_0, r0_.marking AS marking_1, r0_.contents AS contents_2, r0_.kind_id AS kind_id_3 FROM resource r0_ WHERE r0_.contents?'-1' = false':
SQLSTATE[3F000]: Invalid schema name: 7 ERROR: schema "r0_" does not exist
LINE 1: ... r0_.kind_id AS kind_id_3 FROM resource r0_ WHERE r0_.conten...
^
I tried to execute the raw SQL query manually from PHPStorm and it worked, no errors.
How do I get this to work in Doctrine?
Why doesn't this query work with Doctrine, but does when I test it manually?
Here's JSON_EXISTS: (based on syslogic/doctrine-json-functions)
class JsonExists extends FunctionNode
{
const FUNCTION_NAME = 'JSON_EXISTS';
const OPERATOR = '?';
public $jsonData;
public $jsonPath;
public function getSql(SqlWalker $sqlWalker)
{
$jsonData = $sqlWalker->walkStringPrimary($this->jsonData);
$jsonPath = $this->jsonPath->value;
return $jsonData . self::OPERATOR . "'$jsonPath'";
}
public function parse(Parser $parser)
{
$parser->match(Lexer::T_IDENTIFIER);
$parser->match(Lexer::T_OPEN_PARENTHESIS);
$this->jsonData = $parser->StringPrimary();
$parser->match(Lexer::T_COMMA);
$this->jsonPath = $parser->StringPrimary();
$parser->match(Lexer::T_CLOSE_PARENTHESIS);
}
}
Registered via Symfony's YAML config like this:
doctrine:
orm:
dql:
numeric_functions:
json_exists: Syslogic\DoctrineJsonFunctions\Query\AST\Functions\Postgresql\JsonExists
Versions of stuff:
PHP 7.1.1
doctrine/dbal v2.6.1
doctrine/orm dev-master e3ecec3 (== 2.6.x-dev)
symfony/symfony v3.3.4
The error message is a false clue.
Actual problem is caused by PDO (this is why it works from PHPStorm). When it sees a query like this:
SELECT * FROM foo WHERE contents?'bar'
It treats it like a parametrized query and the question mark ? as a parameter. For some reason it sometimes results in nonsensical error messages. This specific case could be solved by adding a space after the question mark, but it won't work for operators ?| and ?& which can't have a space in the middle.
The solution is to use functions corresponding to operators (this question saved the day). One can find out how they are called using queries like this one:
SELECT oprname, oprcode FROM pg_operator WHERE oprname IN ('?', '?|', '?&')
Here's the part of result related to JSON:
? → jsonb_exists
?| → jsonb_exists_any
?& → jsonb_exists_all
So instead of previous query which causes problems via PDO, one can use this equivalent one:
SELECT * FROM foo WHERE jsonb_exists(contents, 'bar')
Context
SDN 3.3.0.RELEASE / 3.4.0.M1
Neo4j 2.1.7 in distant server mode.
Use case
I have an existing Person in database which can have multiple PersonTranslatedContent that can be in several languages.
So basicaly, my modelisation is like :
(:Person)-[:TRANSLATION {lang:"fr}]->(:PersonTranslatedContent)
Problem
When I create a PersonTranslatedContent node and a TRANSLATION relation to link with my Person, the 'lang' property is not persisted on the relationship.
The nodes are corecctly created, but when I query the database from the Neo4j browser, my relationship has only one property : _type__ : PersonToPersonTranslatedContent
Analysis
When logging the HTTP request received by Neo4j, the requets performed are in this order :
1. MATCH (n) WHERE id(n) = {id_n} MATCH (m) WHERE id(m) = {id_m} CREATE (n)-[r:`TRANSLATION`]->(m) SET r={props} RETURN id(r) as id, type(r) as type, r as properties, id(startNode(r)) as start, id(endNode(r)) as end
2. START r=rel({id}) SET r.`_ _type_ _` = {value}
3. START r=rel({id}) RETURN id(r) as id, type(r) as type, r as properties, id(startNode(r)) as start, id(endNode(r)) as end
4. **START r=rel({id}) SET r.`lang` = {value}** <-- here the lang property seems to be correctly set !
5. START r=rel({id}) SET r = {props} <- here, props = {"_ _type_ _" : PersonToPersonTranslatedContent"}
All those REST calls are done within a simple call to personToPersonTranslatedContentRepository.save(). I followed the white rabit in debug mode and here is the shortened call stack :
Neo4jEntityConverterImpl.write() --> entityStateHandler.useOrCreateState() --> RestAPICypherImpl.createRelationship() (correspond to bullet 1)
Neo4jEntityConverterImpl.write() --> typeMapper.writeType() --> RestAPICypherImpl.setPropertyOnEntity() (correspond to bullet 2)
Neo4jEntityConverterImpl.write() --> sourceStateTransmitter.copyPropertiesTo() --> persistentEntity.doWithProperties() --> RestAPICypherImpl.setPropertyOnEntity() (correspond to bullet 4)
Neo4jEntityConverterImpl.write() --> sourceStateTransmitter.copyPropertiesTo() --> ((UpdateableState)target).flush() --> RestAPICypherImpl.setPropertiesOnEntity() (correspond to bullet 5)
So, in my opinion considering what I know and what I saw during debug, the problem seems to be around the "propertyData" attribute of class RestEntity which is used in the ((UpdateableState)target).flush() ! It always hold the value {"_ type _" : PersonToPersonTranslatedContent"} but never contains my "lang" property.
Note : my problem is the same as the one explain here 3.3.0.M1 : Properties on RelationShipEntity not saved when using CypherRestGraphDatabase?. His post has no satisfying answer.
Can you help me (and him I guess) ?
Thx :)
13/07/15 : Updated
I finally manage to resolve my problem by using :
Map<String, Object> properties = new HashMap<>();
properties.put("lang", lang);
properties.put("__type__", "UserToUserTranslatedContent");
neo4jOperations.createRelationshipBetween(neo4jOperations.getNode(user.getId()), neo4jOperations.getNode(translatedContent.getId()), RelationNames.TRANSLATION, properties);
But it is still strange that simple save() operation do not work as expected
For people looking for a fix, I made a report concerning this problem (https://jira.spring.io/browse/DATAGRAPH-699) and a quick way to use SDN like before (< 3.3.x) is doing like this:
remove "spring-data-neo4j" and "spring-data-neo4j-rest" from your build.gradle and add these lines:
repositories {
maven {
url "https://jitpack.io"
}
}
dependencies {
compile 'org.springframework:spring-orm:4.1.7.RELEASE'
compile 'org.springframework:spring-aspects:4.1.7.RELEASE'
compile 'com.github.nousmotards:spring-data-neo4j:3.3.1.NM1'
}
Hope this will help in the mean time of having a real fix ;)
for some reason I need to query 'case-sensitive' in MySql + doctrine 2. Is it possible?
neither
$em->find('UserEn', 'Bob')
nor
$q = $this->em->createQuery('select u from UserEn u where u.name = :name');
$q->setParameter('name', 'Bob');
$result = $q->getResult();
is working. Any idea?
Maybe you are using a MySQL collation ending with "_ci", like "utf8_general_ci". "ci" stands for "case insensitive".
If this is the case, it is not a Doctrine issue, but a MySQL issue.
See http://dev.mysql.com/doc/refman/5.0/en/case-sensitivity.html
"The default character set and collation are latin1 and latin1_swedish_ci, so nonbinary string comparisons are case insensitive by default."
For those that are unable to change their database collation, you can use the BINARY operator in order to force case sensitivity on the criteria.
The BINARY operator casts the string following it to a binary string.
This is an easy way to force a comparison to be done byte by byte
rather than character by character. BINARY also causes trailing spaces
to be significant.
See MySQL BINARY Operator for more details.
To enable the BINARY operator in Doctrine DQLs, you can install the Doctrine Extensions library.
Or create your own Binary String Function like so.
use Doctrine\ORM\Query\AST\Functions\FunctionNode,
Doctrine\ORM\Query\Lexer;
class Binary extends FunctionNode
{
private $stringPrimary;
public function parse(\Doctrine\ORM\Query\Parser $parser)
{
$parser->match(Lexer::T_IDENTIFIER);
$parser->match(Lexer::T_OPEN_PARENTHESIS);
$this->stringPrimary = $parser->StringPrimary();
$parser->match(Lexer::T_CLOSE_PARENTHESIS);
}
public function getSql(\Doctrine\ORM\Query\SqlWalker $sqlWalker)
{
return 'BINARY('.$sqlWalker->walkSimpleArithmeticExpression($this->stringPrimary).')';
}
}
Next you'll need to register the binary string function with your doctrine configuration. You can do so in your configuration settings or add it as needed like so.
$em->getConfiguration()->addCustomStringFunction('binary', 'DoctrineExtensions\\Query\\Mysql\\Binary');
Then you will be able to use the binary(...) function in your DQL like so.
$q = $em->createQuery('select u from UserEn u where binary(u.name) = :name');
echo $q->getSQL();
/* SELECT ... FROM ... WHERE BINARY(u0_.name) = ? */
To add the binary string function using the Symfony Framework, in your config.yml file change the doctrine.orm.entity_managers.%entity_manager%.dql setting like so.
doctrine:
orm:
#...
entity_managers:
#...
default:
#...
dql:
#...
string_functions:
#...
binary: 'DoctrineExtensions\Query\Mysql\Binary'
See Symfony Doctrine Configuration Documentation for more details
It's not Doctrine issue you have to change table collation to binary then case sensitive would work.
Do alter table and change this
CHARSET=utf8 COLLATE=utf8_general_ci
to this
CHARSET=utf8 COLLATE=utf8_bin
Is there a way to search all the mappings, sessions, etc. in Informatica for a text string contained within a SQL override?
For example, suppose I know a certain stored procedure (SP_FOO) is being called somewhere in an INFA process, but I don't know where exactly. Somewhere I think there is a Post SQL on a source or target calling it. Could I search all the sessions for Post SQL containing SP_FOO ? (Similar to what I could do with grep with source code.)
You can use Repository queries for querying REPO tables(if you have enough access) to get data related with all the mappings,transformations,sessions etc.
Please use the below link to get almost all kind of repo queries.Ur answers can be find in the below link.
https://uisapp2.iu.edu/confluence-prd/display/EDW/Querying+PowerCenter+data
select *--distinct sbj.SUBJECT_AREA,m.PARENT_MAPPING_NAME
from REP_SUBJECT sbj,REP_ALL_MAPPINGS m,REP_WIDGET_INST w,REP_WIDGET_ATTR wa
where sbj.SUBJECT_ID = m.SUBJECT_ID AND
m.MAPPING_ID = w.MAPPING_ID AND
w.WIDGET_ID = wa.WIDGET_ID
and sbj.SUBJECT_AREA in ('TLR','PPM_PNLST_WEB','PPM_CURRENCY','OLA','ODS','MMS','IT_METRIC','E_CONSENT','EDW','EDD','EDC','ABS')
and (UPPER(ATTR_VALUE) like '%PSA_CONTACT_EVENT%'
-- or UPPER(ATTR_VALUE) like '%PSA_MEMBER_CHARACTERISTIC%'
-- or UPPER(ATTR_VALUE) like '%PSA_REPORTING_HH_CHRSTC%'
-- or UPPER(ATTR_VALUE) like '%PSA_REPORTING_MEMBER_CHRSTC%'
)
--and m.PARENT_MAPPING_NAME like '%ARM%'
order by 1
Please let me know if you have any issues.
Another less scientific way to do this is to export the workflow(s) as XML and use a text editor to search through them for the stored procedure name.
If you have read access to the schema where the informatica repository resides, try this.
SELECT DISTINCT f.subj_name folder, e.mapping_name, object_type_name,
b.instance_name, a.attr_value
FROM opb_widget_attr a,
opb_widget_inst b,
opb_object_type c,
opb_attr d,
opb_mapping e,
opb_subject f
WHERE a.widget_id = b.widget_id
AND b.widget_type = c.object_type_id
AND ( object_type_name = 'Source Qualifier'
OR object_type_name LIKE '%Lookup%'
)
AND a.widget_id = b.widget_id
AND a.attr_id = d.attr_id
AND c.object_type_id = d.object_type_id
AND attr_name IN ('Sql Query')--, 'Lookup Sql Override')
AND b.mapping_id = e.mapping_id
AND e.subject_id = f.subj_id
AND a.attr_value is not null
--AND UPPER (a.attr_value) LIKE UPPER ('%currency%')
Yes. There is a small java based tool called Informatica Meta Query.
Using that tool, you can search for any information that is present in the Informatica meta data tables.
If you cannot find that tool, you can write queries directly in the Informatica Meta data tables to get the required information.
Adding few more lines to solution provided by Data Origin and Sandeep.
It is highly advised not to query repository tables directly. Rather, you can create synonyms or views and then query those objects to avoid any damage to rep tables.
In our dev/ prod environment application programmers are not granted any direct access to repo. tables.
As querying the Informatica database isn't the best idea, I would suggest you to export all the workflows in your folder into xml using Repository Manager. From Rep Mgr you can select all of them once and export them at once. Then write a java program to search the pattern from the xml's you have.
I have written a sample prog here, please modify it as per your requirement:
make a spec file with workflow names(specFileName).
main()
{
try {
File inFile = new File(specFileName);
BufferedReader reader = new BufferedReader(newFileReader(infile));
String tectToSearch = '<YourString>';
String currentLine;
while((currentLine = reader.readLine()) != null)
{
//trim newline when comparing with String
String trimmedLine = currentLine.trim();
if(currentline has the string pattern)
{
SOP(specFileName); //specfile name
}
}
reader.close();
}
catch(IOException ex)
{
System.out.println("Error reading to file '" + specFileName +"'");
}
}
Is there a way to auto quote reserved words with Doctrine 2 when using $entityManager->find('entity', id) ?
When using the query builder this can be done but there should be a global configuration setting that does this? I don't want to have to specify it in the annotations for the reserved words.
This was an issue I raised a while back with the Doctrine team.
https://github.com/doctrine/doctrine2/issues/2409
The ticket was closed with the comment:
You have to manually escape characters with #Column(name="`integer`")
So I guess you'd need to deal with any reserved keywords in your annotations
4.6. Quoting Reserved Words
Sometimes it is necessary to quote a column or table name because of reserved word conflicts. Doctrine does not quote identifiers automatically, because it leads to more problems than it would solve. Quoting tables and column names needs to be done explicitly using ticks in the definition.
<?php
/** #Column(name="`number`", type="integer") */
private $number;
Doctrine will then quote this column name in all SQL statements according to the used database platform.
Identifier Quoting does not work for join column names or discriminator column names unless you are using a custom QuoteStrategy.
For more control over column quoting the Doctrine\ORM\Mapping\QuoteStrategy interface was introduced in 2.3. It is invoked for every column, table, alias and other SQL names. You can implement the QuoteStrategy and set it by calling Doctrine\ORM\Configuration#setQuoteStrategy().
The ANSI Quote Strategy was added, which assumes quoting is not necessary for any SQL name. You can use it with the following code:
<?php
use Doctrine\ORM\Mapping\AnsiQuoteStrategy;
$configuration->setQuoteStrategy(new AnsiQuoteStrategy());
http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/basic-mapping.html#quoting-reserved-words
It's not implemented by Doctrine just because it's too platform-depending.
All you need, is implement own QuoteStrategy.
For example, for symfony project:
Copy-paste vendor AnsiQuoteStrategy class, rename it and make some quoting:
AppBundle/ORM/QuoteStrategy.php
namespace AppBundle\ORM;
use Doctrine\DBAL\Platforms\AbstractPlatform;
use Doctrine\ORM\Mapping as M;
class QuoteStrategy implements M\QuoteStrategy
{
private function quote($token, AbstractPlatform $platform)
{
// implement your quote strategy
switch ($platform->getName()) {
case 'mysql':
default:
return '`' . $token . '`';
}
}
// add quoting to appropriate methods
public function getColumnName($fieldName, M\ClassMetadata $class, AbstractPlatform $platform)
{
return $this->quote($class->fieldMappings[$fieldName]['columnName'], $platform);
}
// ... Rest methods
}
Then, register your quote strategy as a service:
src/AppBundle/Resources/config/services.yml
app.orm.quote_strategy:
class: AppBundle\ORM\QuoteStrategy
public: false
Then, use it for your entitymanager configuration:
app/config/config.yml
orm:
entity_managers:
default:
quote_strategy: app.orm.quote_strategy
That is all :)
Per the statement made by #tim-lytle, I have re-raised the issue. This really should be included with Doctrine ORM's scope of safety.
https://github.com/doctrine/doctrine2/issues/5874