To create a table in Rails 4 a database adapter should implement visit_TableDefinition and return an SQL string from it. Problem is, I also need to create a SEQUENCE and the DB driver in use (ruby-oci8) can't execute multiple statements for some reason. Is there a way to solve this?
Related
I am fairly new to Great Expectations - and have a question. Essentially I have a PostgreSQL database, and every time I run my data pipeline, i want to validate a specific subset of the PostgreSQL table based off some key. Eg: If the data pipeline is run every day, the would be a field called current_batch. And the validation would occur for the below query:
SELECT * FROM jobs WHERE current_batch = <input_batch>.
I am unsure the best way to complete this. I am a using v3-api of great expectations and am a bit confused as to whether to use a checkpoint, or a validator. I assume I want to use a checkpoint but I can't seem to figure out how to create a checkpoint, but then only validate a specific subset of the PostgreSQL datasource.
Any help or guidance would be much appreciated.
Thanks,
I completely understand your confusion because I am working with GE too and the documentation is not really clear.
First of all "Validators" are now called "Checkpoints", so they are not a different entity, as you can read here.
I am working on an Oracle database and the only way I found to apply a query before testing my data with expectations is to put the query inside the checkpoint.
To create a checkpoint you should run the great_expectations checkpoint new command from your terminal. After creating it, you should add the "query" field inside the .yml file that is your checkpoint.
Below you can see a snippet of a checkpoint I am working with. When I want to validate my data, I run the command great_expectations checkpoint run check1
name: check1
module_name: great_expectations.checkpoint
class_name: LegacyCheckpoint
batches:
- batch_kwargs:
table: pso
schema: test
query: SELECT p AS c,
[ ... ]
AND lsr = c)
datasource: my_database
data_asset_name: test.pso
expectation_suite_names:
- exp_suite1
Hope this helps! Feel free to ask if you have any doubts :)
I managed this using Views (in Postgres). Before running GE, I create (or replace the existing) view as a query with all necessary joins, filtering, aggregations, etc. And then specify the name of this view in GE checkpoints.
Yes, it is not the ideal solution. I would rather use a query in checkpoints too. But as a workaround, it covers all my cases.
Let's have view like this:
CREATE OR REPLACE VIEW table_to_check_1_today AS
SELECT * FROM initial_table
WHERE dt = current_date;
And checkpoint be configured something like this:
name: my_task.my_check
config_version: 1.0
validations:
- expectation_suite_name: my_task.my_suite
batch_request:
datasource_name: my_datasource
data_connector_name: default_inferred_data_connector_name
data_asset_name: table_to_check_1_today
Yes, a view can be created using the "current_date" - and the checkpoint can simply run the view. However, this would mean that the variable (current_date) is stored in the database - which may not be desirable; you might want to run the query in the checkpoint for a different date - which could be coming from a environment variable or elsewhere - to the CLI or python/notebook
Yet to find a solution where we can substitute a string in the checkpoint query; using a config variable from the file is a very static way - there may be different checkpoints running for different dates.
We have a situation where we are dealing with a relational source(Oracle). The system is developed in a way where we have to first execute a package which will enable data read from Oracle and user will be able to get results out of select statement. I am trying to find a way on how to implement this in informatica mapping.
What we tried
1. In PreSQL we tried to execute the package and in SQL query we wrote select statement - data not getting loaded in target.
2. In PreSQL we wrote a block in which we are executing the package and just after that(within same beging...end block) we wrote insert statement on top of select statement - This is inserting data through insert statement however I am not in favor of this solution as both source and target are dummy which will confuse people in future.
Is there any possibility to implement this solution somehow by using 1st option.
Please help and suggest.
Thanks
The stored procedure transformation is there for this purpose configure it to execute source pre load
Pre-Sql and data read are not a part of same session. From what I understand, this needs to be done within the same session as otherwise the read is granted only for the session.
What you can do, is create a stored procedure/package that will grant read access and then return the data. Use it as a SQL Override on your SQ. This way SQ will read the data as usual. The concept:
CREATE PROCEDURE ReadMyData AS
BEGIN
execute immediate 'GiveMeTheReadAccess';
select * from MyTable;
END;
And use the ReadMyData on the Source Qualifier.
I am trying to create a rather complex trigger in a doctrine migrations class:
https://github.com/2ndQuadrant/audit-trigger/blob/master/audit.sql
The first impulse was just to put the whole trigger code in one big blob and add it
with:
public function up(Schema $schema)
{
$this->addSql($triggerSqlInABigBlob);
}
However the migration fails
SQLSTATE[42601]: Syntax error: 7 ERROR: cannot insert multiple commands into a prepared statement
Is this even possible to manage in a doctrine migration? Is there a workaround / best practice to do this?
addSql in Doctrine's AbstractMigration expects one SQL command or array of multiple SQL commands. What you send is a string containig multiple SQL commands, which is not allowed. You can try this:
public function up(Schema $schema)
{
$this->addSql(explode(';',$triggerSqlInABigBlob));
}
That should convert the string into an array where each element is one SQL command. The comments might a problem though.
I was wondering if you can help me out with my current problem which is to insert data into multiple tables in my relational database using a single form. I am fairly new to APEX but do have a little bit of background on mysql and php programming. In the past, I normally achieve such task by creating a view of all the columns from different table that I want to populate and using a simple insert commands but doing the same thing in apex gives me and error stating that "ORA-01779: cannot modify a column which maps to a non key-preserved table".
In Oracle you can not just update a view which has eg a JOIN clause. Oracle will not map all columns back to the source tables: one table might while the others won't. This isn't an apex problem: if you were to run an update against your view in the db you would get this error just as well.
If you want to have your apex screen remain as transparent as possible, then you may want to consider user an instead-of trigger on the view. You will have to write the correct dml statements in this trigger though in order to ensure your data is pushed through correctly to all tables.
Another option is to use the view only to fetch, and use different processes to push the data to the correct tables. Using data-layer packages might alleviate the use of code stored in apex (eg having a lot of plsql code in apex itself is usually not favored and is rather stored in packages).
Create items and get all the items values and use PL/SQL on submit button.
Eg: p1_party_Name, p2_Service_Name
BEGIN;
INSERT INTO par VALUES(par_party_uid_seq.nextval,:p1_Party_name);
INSERT INTO par VALUES(ser_service_uid_seq.nextval,:p2_Service_name);
END;
i want to know how to drop a constraint only if it exists. is there any single line statement present in mysql server which will allow me to do this.
i have tried the following command but unable to get the desire output
alter table airlines
drop foreign key if exits FK_airlines;
any help to this really help me to go forward in mysql
I do not believe this is possible in a single line, unless you are willing to detect the error and move on (not a bad thing).
The INFORMATION_SCHEMA database contains the info you need to tell if the foreign key exists, so you could implement it in a 2 step process.
http://dev.mysql.com/doc/refman/5.1/en/table-constraints-table.html
yep not possible the if exist is available only for database table and view :
http://dev.mysql.com/doc/refman/5.0/en/replication-features-drop-if-exists.html
yep 2 step process is a good way like gahooa said