I have a relatively simple issue, which seems as if it should be achievable, but I have tried absolutely everything with no success.
Here is the situation:
Using Power Query inside Excel I would like to be able to combine multiple queries (lets call these: Query1, Query2 and Query3) into one single query using the Table.Combine function.
The only catch is that the list of queries I will be combining will be dynamic and dependant on another query (lets call this: QueryList)
For example, under certain circumstances QueryList will be:
Query1
Query2
Query3
and under some other condtions QueryList may simply be:
Query1
Query3
What I would like to do is to be able to parse the value of QueryList into the Table.Combine Function:
eg. Table.Combine(#"QueryList")
and thereby allow dynamic consolidation of queries
Whats happening is that I am getting an error that states:
Expression.Error: We cannot convert the value "Query1" to type Table.
Details:
Value=Query1
Type=Type
Update:
I have tried variations of Table.ToList, using { } to create a list, TableFromlist, all with no success (normally errors complain about not being able to comvert from text to list or to table etc.
Thanks in advance for the help.
If your QueryList would be {Query1, Query2} then Table.Combine(QueryList) would work.
Apparently, your QueryList is {"Query1", "Query2"}.
So the strings must be converted to tables, which can be done using Expression.Evaluate. As second parameter, you must supply a record with all possible queries, so the formula becomes, for Query1, Query2, Query3:
= Table.Combine(List.Transform(QueryList, each Expression.Evaluate(_, [Query1 = Query1, Query2 = Query2, Query3 = Query3])))
Related
I am trying to run some dynamoDB operations with AWS.DynamoDB.DocumentClient from aws-sdk module but I am unable to find an easy solution to select items where an attribute is not equals to an array of values.
e.g
attribute <> ["value1", "value2]
This is equivalent to a simple typical SQL operation in the form of:
select * from sometable where attribute not in("value1", "value2"...);
After trying out different ScanFilter and QueryFilter following the documentation here, it seems that the AttributeValueList for NE and NOT_CONTAINS does not accept multiple values.
I wish to arrive at the results as shown below without having to define multiple 'AND' queries
I have since arrived at this solution but it seems clumsy and I would have to write logic to create the filter condition string and ExpressionAttributeValues as the filter condition is dynamic.
FilterExpression: 'answer <> :answer1 AND answer <> :answer2',
ExpressionAttributeValues : {
':answer1' : "test1",
':answer2' : "test2"
}
I have therefore, 2 questions:
Is there a better way of doing this?
Is there a length limit to the string of KeyConditionExpression? I
am very sure there is but I cannot seem to find information with
regards to this.
There is no other way to achieve what you need. If all these values have something in common, and you know it in the write time, you can insert them with some kind of prefix and create GSI where they will be a sort key. In this case you'll be able to query them by prefix in the key condition expression. Otherwise, what you've suggested is your only option.
4KB for all of the expressions combined. As described in the Expression Parameters:
Expression parameters include ProjectionExpression, ConditionExpression, UpdateExpression, and FilterExpression.
The maximum length of any expression string is 4 KB. For example, the size of the ConditionExpression a=b is 3 bytes.
I am trying to insert in database(Oracle) in python with cx_oracle. I need to select from table and insert into another table.
insert_select_string = "INSERT INTO wf_measure_details(PARENT_JOB_ID, STAGE_JOB_ID, MEASURE_VALS, STEP_LEVEL, OOZIE_JOB_ID, CREATE_TIME_TS) \
select PARENT_JOB_ID, STAGE_JOB_ID, MEASURE_VALS, STEP_LEVEL, OOZIE_JOB_ID, CREATE_TIME_TS from wf_measure_details_stag where oozie_job_id = '{0}'.format(self.DAG_id)"
conn.executemany(insert_select_string)
conn.commit()
insert_count = conn.rowcount
But I am getting below error. I do not have select parameter of data as data is getting from select query.
Required argument 'parameters' (pos 2) not found
Please suggest how to solve this
As mentioned by Chris in the comments to your question, you want to use cursor.execute() instead of cursor.executemany(). You also want to use bind variables instead of interpolated parameters in order to improve performance and reduce security risks. Take a look at the documentation. In your case you would want something like this (untested):
cursor.execute("""
INSERT INTO wf_measure_details(PARENT_JOB_ID, STAGE_JOB_ID,
MEASURE_VALS, STEP_LEVEL, OOZIE_JOB_ID, CREATE_TIME_TS)
select PARENT_JOB_ID, STAGE_JOB_ID, MEASURE_VALS, STEP_LEVEL,
OOZIE_JOB_ID, CREATE_TIME_TS
from wf_measure_details_stag
where oozie_job_id = :id""",
id=self.DAG_id)
I have the following query in M:
= Table.Combine({
Table.Distinct(Table.SelectColumns(Tab1,{"item"})),
Table.Distinct(Table.SelectColumns(Tab2,{"Column1"}))
})
Is it possible to get it working without prior changing column names?
I want to get something similar to SQL syntax:
select item from Tab1 union all
select Column1 from Tab2
If you need just one column from each table then you may use this code:
= Table.FromList(List.Distinct(Tab1[item])
& List.Distinct(Tab2[Column1]))
If you use M (like in your example or the append query option) the columns names must be the same otherwise it wont work.
But it works in DAX with the command
=UNION(Table1; Table2)
https://learn.microsoft.com/en-us/dax/union-function-dax
It's not possible in Power Query M. Table.Combine make an union with columns that match. If you want to keep all in the same step you can add the change names step instead of tap2 like you did with Table.SelectColumns.
This comparison of matching names is to union in a correct way.
Hope you can manage in the same step if that's what you want.
I generate a list of ID numbers. I want to execute an insert statement that grabs all records from one table where the ID value is in my list and insert those records into another table.
Instead of running through multiple execute statements (as I know is possible), I found this cx_Oracle function, that supposedly can execute everything with a single statement and list parameter. (It also avoids the clunky formatting of the SQL statement before passing in the parameters) But I think I need to alter my list before passing it in as a parameter. Just not sure how.
I referenced this web page:
https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-executemany.html
ids = getIDs()
print(ids)
[('12345',),('24567',),('78945',),('65423',)]
sql = """insert into scheme.newtable
select id, data1, data2, data3
from scheme.oldtable
where id in (%s)"""
cursor.prepare(sql)
cursor.executemany(None, ids)
I expected the SQL statement to execute as follows:
Insert into scheme.newtable
select id, data1, data2, data3 from scheme.oldtable where id in ('12345','24567','78945','65423')
Instead I get the following error:
ORA-01036: illegal variable name/number
Edit:
I found this StackOverflow: How can I do a batch insert into an Oracle database using Python?
I updated my code to prepare the statement before hand and updated the list items to tuples and I'm still getting the same error.
You use executemany() for batch DML, e.g. when you want to insert a large number of values into a table as an efficient equivalent of running multiple insert statements. There are cx_Oracle examples discussed in https://blogs.oracle.com/opal/efficient-and-scalable-batch-statement-execution-in-python-cx_oracle
However what you are doing with
insert into scheme.newtable
select id, data1, data2, data3
from scheme.oldtable
where id in (%s)
is a different thing - you are trying to execute one INSERT statement using multiple values in an IN clause. You would use a normal execute() for this.
Since Oracle keeps bind data distinct from SQL, you can't pass in multiple values to a single bind parameter because the data is treated as a single SQL entity, not a list of values. You could use %s string substitution syntax you have, but this is open to SQL Injection attacks.
There are various generic techniques that are common to Oracle language interfaces, see https://oracle.github.io/node-oracledb/doc/api.html#sqlwherein for solutions that you can rewrite to Python syntax.
using temporary table to save ids (batch insert)
cursor.prepare('insert into temp_table values (:1)')
dictList = [{'1': x} for x in ids]
cursor.executemany(None, dictList)
then insert selected value into newtable
sql="insert into scheme.newtable (selectid, data1, data2, data3 from scheme.oldtable inner join temp_table on scheme.oldtable.id = temp_table.id)"
cursor.execut(sql,connection)
the script of create temporary table in oracle
CREATE GLOBAL TEMPORARY TABLE temp_table
(
ID number
);
commit
I hope this useful.
How do I do the equivalent of
SELECT *, id=1 AS calc_column FROM table
in Korma? (Obviously, id=1 is just a simple example).
Korma has sqlfn which can be used in fields as show here. If even that doesn't fit your need then you always have the raw function to feed in raw sql in main query.