If I want to use a query in another query what should I use :
reference
duplicate
and what's the difference between the 2 functions ?
I'm not sure where you see "related" but the other two show up if you right-click a query in the Power Query Editor.
Duplicate is like copy and pasting a query. It creates another query with all of the same steps and code. It's independent and modifying one query doesn't affect the other.
Reference does not reproduce the query. The new query starts exactly where the referenced query ended and is therefore dependent on it. If you change the referenced query, then the new query is affected too.
Duplicate creates a new copy with all the existing steps. The new copy will be isolated from the original query. You can make changes in the original or the new query, and they will NOT affect each other.
Reference, on the other hand, is a new copy with only one single step: getting data from the original query.
Related
Workflow
In a data import workflow, we are creating a staging table using CREATE TABLE LIKE statement.
CREATE TABLE abc_staging (LIKE abc INCLUDING DEFAULTS);
Then, we run COPY to import CSV data from S3 into the staging table.
The data in CSV is incomplete. Namely, there are fields partition_0, partition_1, partition_2 which are missing in the CSV file; we fill them in like this:
UPDATE
abc_staging
SET
partition_0 = 'BUZINGA',
partition_1 = '2018',
partition_2 = '07';
Problem
This query seems expensive (takes ≈20 minutes oftentimes), and I would like to avoid it. That could have been possible if I could configure DEFAULT values on these columns when creating the abc_staging table. I did not find any method as to how that can be done; nor any explicit indication that is impossible. So perhaps this is still possible but I am missing how to do that?
Alternative solutions I considered
Drop these columns and add them again
That would be easy to do, but ALTER TABLE ADD COLUMN only adds columns to the end of the column list. In abc table, they are not at the end of the column list, which means the schemas of abc and abc_staging will mismatch. That breaks ALTER TABLE APPEND operation that I use to move data from staging table to the main table.
Note. Reordering columns in abc table to alleviate this difficulty will require recreating the huge abc table which I'd like to avoid.
Generate the staging table creation script programmatically with proper columns and get rid of CREATE TABLE LIKE
I will have to do that if I do not find any better solution.
Fill in the partition_* fields in the original CSV file
That is possible but will break backwards compatibility (I already have perhaps hundreds thousands of files in there). Harder but manageable.
As you are finding you are not creating a table exactly LIKE the original and Redshift doesn't let you ALTER a column's default value. Your proposed path is likely the best (define the staging table explicitly).
Since I don't know your exact situation other paths might be better so me explore a bit. First off when you UPDATE the staging table you are in fact reading every row in the table, invalidating that row, and writing a new row (with new information) at the end of the table. This leads to a lot of invalidated rows. Now when you do ALTER TABLE APPEND all these invalidated rows are being added to your main table. Unless you vacuum the staging table before hand. So you may not be getting the value you want out of ALTER TABLE APPEND.
You may be better off INSERTing the data onto your main table with an ORDER BY clause. This is slower than the ALTER TABLE APPEND statement but you won't have to do the UPDATE so the overall process could be faster. You could come out further ahead because of reduced need to VACUUM. Your situation will determine if this is better or not. Just another option for your list.
I am curious about your UPDATE speed. This just needs to read and then write every row in the staging table. Unless the staging table is very large it doesn't seem like this should take 20 min. Other activity could be creating this slowdown. Just curious.
Another option would be to change your main table to have these 3 columns last (yes this would be some work). This way you could add the columns to the staging table and things would line up for ALTER TABLE APPEND. Just another possibility.
The easiest solution turned to be adding the necessary partition_* fields to the source CSV files.
After employing that change and removing the UPDATE from the importer pipeline, the performance has greatly improved. Imports now take ≈10 minutes each in total (that encompasses COPY, DELETE duplicates and ALTER TABLE APPEND).
Disk space is no longer climbing up to 100%.
Thanks everyone for help!
In Power query I have a query fetching data from tblSales and performing row filtering on amount>100.
I want to reuse this query for another query. There are 2 options i have: duplicate or reference.
In duplicate the steps from the query are brought into the new query.
Where as in reference, the new query references the original query.
I right clicked on the original query and selected reference. Then added additional row filters. This new query that is generated, does it reuse the data fetched by original query?
Yes it will run the query 'again'.
If you want to only load the data once, you need to wrap your base query in one of the Buffer functions.
Table.Buffer()
Binary.Buffer()
List.Buffer()
See here for more details
In this scenario, the data is loaded twice:
In this Scenario, the data is loaded once ( Table.Buffer() on the base table ):
I have several fields that contain exactly the same sql query! Is it possible to place the sql question centrally in APEX in the same way as list of values or as a function in oracle? I am using APEX 18.2
Here are two extended solutions
Pipelined SQL
https://smart4solutions.nl/blog/on-apex-lovs-and-how-to-define-their-queries/
Dynamic SQL
http://stevenfeuersteinonplsql.blogspot.com/2017/01/learn-to-hate-repetition-lesson-from.html
Call me dense, but I don't think I understand why you'd have multiple fields (presumably on the same form) whose source is the same SQL query.
Are you passing a parameter to the SQL to get a different value for each field?
If you are passing a parameter to the SQL query, why not create a database view to hold the query, then pass the parameter to the view. That way, if you need to change it, it's in one place.
If they really are all the same value from the same query, how about using the SQL for one field/page_item, then make the source for the others be the first page item?
I would create a hidden item with the query in the source as text and use &HIDDEN_ITEM_NAME. to reference its value in the source of any item I was trying to display the query.
Finally isolved it with a function and use the type PL/SQL Function Body returning SQL Query in APEX, then i have all in one place. I created a function in SQL developer that returns a SQL query.
Is there an automated way to convert a Duplicate Table(With all its steps) into a Reference Table preserving all the steps in Query Editor ?
Short answer, not really, but it's possibly trivial to do manually for one query.
Reference Table and Duplicate Table are GUI operations, which like other GUI operations, simply insert M code into the query. You can see the entire query in the Advanced Editor.
Reference Table just inserts the name of the other query; the effect is branching the data processing pipelines. If you change the original query, it affects all downstream queries.
Duplicate Table copies all of the steps; the effect is creating a separate query. You can change them at any point later. There is no link to where the steps came from even if they aren't changed.
So, it seems that you just want to convert duplicated steps to references. There is no automated way of doing it. But if you know two queries start with the same steps, try this: Duplicate to a base query and remove final steps that are not in common. Mark the new query to not load to the report by: Click All Properties; Uncheck Enable load to report. Then you can replace the duplicated initial steps in the other queries with a reference by a step like Source = BaseQuery in the Advanced Editor.
Also, if you find yourself duplicating steps in the middle of a query, you can create a query used as a function.
I am trying to retrieve a single row from a table. This row contains filed that hold foreign keys into another table, which in turns is related to yet another table. I am trying to get just one row returned, yet, the problem is, it returns not only the row but ALL the objects that are jointly related to that table as well. As I have to deal with a fairly large amount of data, the returned object is very cumbersome as it contains all the related data as well. In some cases my script simply times out because there is just far too much data to grab.
My question is; is there a way to retrieve just a single record without the associated fluff with it? I am basically accessing the table via the entityManager from the repository, then trying to get my record by using the ->find($id) method.
I am sure this is something stupidly simple but I can't seem to figure this out. Thanks in advance for any help, it is much appreciated.
Doctrine 2 use "lazy loading", it means that the associated objects are not really retrieved from the database while you don't try to access them.
So the find($id) is just fine.