I am trying to take a list of points, and query a geospatial database, to find all matching rows.
I have a computed SQL statement that looks like this:
cursor = connection.cursor()
cursor.execute(
'''
SELECT g.ident
FROM (VALUES %s) AS v (lon, lat)
LEFT JOIN customers g
ON (ST_Within(ST_SetSRID(ST_MakePoint(v.lon, v.lat), %s), g.poly_home));
''', [AsIs(formatted_points), SRID]
)
Here is an example of what the formatted_points variable looks like:
(-115.062,38.485), (-96.295,43.771)
So, when that is inserted into the SQL expression, then VALUES expression reads:
(VALUES (-115.062,38.485), (-96.295,43.771)) AS v (lon, lat)
So far so good. However, when the list of points is empty, the VALUES expression looks like this:
(VALUES ) AS v (lon, lat)
.. which causes me to get this error:
django.db.utils.ProgrammingError: syntax error at or near ")"
In other words, (VALUES ) is not legal SQL.
Here's the question: How do I represent an empty list using VALUES? I could special case this, and just return an empty list when this function is passed an empty list, but that doesn't seem very elegant.
I have looked at the PostgreSQL manual page for VALUES, but I don't understand how to construct an empty VALUES expression.
If you can put your lons and lats in separate arrays, you could use arrays with unnest:
select * from unnest(ARRAY[1, 2, 3]::int[], ARRAY[4, 5, 6]::int[]) as v(lon, lat);
lon | lat
-----+-----
1 | 4
2 | 5
3 | 6
(3 rows)
select * from unnest(ARRAY[]::int[], ARRAY[]::int[]) as v(lon, lat);
lon | lat
-----+-----
(0 rows)
You'll have to cast the arrays to the appropriate type (probably not int[]). Postgres will guess the type if the arrays aren't empty, but it will throw an error if they are empty and you don't cast them to a specific type.
Related
i have 2 tables and, i would like to check if table 1 (Type_Sorting) == (CCSClassCode_Type) is matched with table 2 (_Type Sorting) == (_CCS Class Type):
for example, you can see vi got the wrong value in table 1 (CCSClassCode_Type)
and, the right value is XLBas you can see in table 2 (_CCS Class Type) not ULM,
the idea of table 2 to check if people type the right values, Please not that table 2 (_CCS Class Type) have duplicate values
thank you in advance :)
You can calculate this like that:
Table 2 =
Var trt =
SELECTCOLUMNS(Table_2, "XX"
, COMBINEVALUES(",",Table_2[_CCS Class Type],Table_2[_Type Sorting]))
return
SUMMARIZECOLUMNS(Table_1[Column1]
, Table_1[CCSClassCode_Type]
, Table_1[Type_Sorting]
, FILTER(ALL(Table_1[CCSClassCode_Type],Table_1[Type_Sorting]), not( COMBINEVALUES(",",Table_1[CCSClassCode_Type],Table_1[Type_Sorting])
in trt )
))
I have a prog that stores data in a sqlite db. Among other tables in the db, I have one created as follows:
conn.execute("CREATE TABLE {tn} ({cn} {ct})".format(tn=test, cn="STEP_NAME", ct="TEXT"))
Therein, the table creates has several columns. One is:
conn.execute("ALTER TABLE {tn} ADD COLUMN '{cn}' {ct} ".format(tn=test, cn=value, ct="TEXT"))
Im trying to save data to it, but it's behaving in a way I can't explain. When I save 270113185308874890 to it, it appears 270113185308874890 when recalled. However, when I save 89014103258771944209 to it, it saves as 8.90141032588e+19.
How can I prevent this? I've tried different column types with no luck and really don't understand why it's converting it.
EDIT:
Code that I'm using to store it
def store_result(conn, table_name, row_name, data):
for k, v in data.iteritems():
if isinstance(v, str):
data[k] = v.replace('"', "'").rstrip(' \t\r\n\0')
keys = data.keys()
vals = data.values()
# add test name column for everything but info call
if table_name != "info":
keys.insert(0, "STEP_NAME")
vals.insert(0, str(row_name))
# Make pretty for sqlite3 and its crazy param rules.
sql_keys = ','.join(str(v) for v in keys)
sql_vals = ','.join(str(v) for v in [x if str.isdigit(str(x)) else '"{}"'.format(x) for x in vals])
# try to write or tell me why not.
try:
conn.execute("""INSERT into {table}({sql_keys}) values ({vals})""".format(table=table_name,
sql_keys=sql_keys,
vals=sql_vals))
conn.commit()
except Exception as e:
logging.warn("DB ERROR:{}_{}_{}".format(e, table_name, row_name))
When you print the values after they are returned from the table, the type of the variable that holds the values affects both how they're printed and they're precision. As an example:
int1 = 270113185308874890;
float1 = 270113185308874890.0;
int2 = 89014103258771944209;
float2 = 89014103258771944209.0;
print 'int1 : ' + str(int1);
print 'float1: ' + str(float1);
print '';
print 'int2 : ' + str(int2);
print 'float2: ' + str(float2);
Will print:
int1 : 270113185308874890
float1: 2.70113185309e+17
int2 : 89014103258771944209
float2: 8.90141032588e+19
It seems likely that in the SQLite table the type is TEXT, as shown in the example from the SQLite website (https://www.sqlite.org/datatype3.html) below. You should use the typeof() function to ensure that you're data is being stored as TEXT.
Finally, you should consider using the INTEGER type rather than TEXT in your SQLite table if all of your numbers will be integers. Also if you are using TEXT to try and preserve precision, make sure you are not limited by the calling code. I.e. unless you are dealing with the Decimal Python type the REAL SQLite type will match the precision of the Float Python type.
2.3 Column Affinity Behavior Example
The following SQL demonstrates how SQLite uses column affinity to do
type conversions when values are inserted into a table.
CREATE TABLE t1(
t TEXT, -- text affinity by rule 2
nu NUMERIC, -- numeric affinity by rule 5
i INTEGER, -- integer affinity by rule 1
r REAL, -- real affinity by rule 4
no BLOB -- no affinity by rule 3 );
-- Values stored as TEXT, INTEGER, INTEGER, REAL, TEXT.
INSERT INTO t1 VALUES('500.0', '500.0', '500.0', '500.0', '500.0');
SELECT typeof(t), typeof(nu), typeof(i), typeof(r), typeof(no) FROM t1;
text|integer|integer|real|text
-- Values stored as TEXT, INTEGER, INTEGER, REAL, REAL.
DELETE FROM t1;
INSERT INTO t1 VALUES(500.0, 500.0, 500.0, 500.0, 500.0);
SELECT typeof(t), typeof(nu), typeof(i), typeof(r), typeof(no) FROM t1;
text|integer|integer|real|real
-- Values stored as TEXT, INTEGER, INTEGER, REAL, INTEGER.
DELETE FROM t1;
INSERT INTO t1 VALUES(500, 500, 500, 500, 500);
SELECT typeof(t), typeof(nu), typeof(i), typeof(r), typeof(no) FROM t1;
text|integer|integer|real|integer
I have a problem. When I pass a Python array:
self.notPermited = [2,3]
This is my procedure
def select_ids_entre_amistades(self,cod_us,ids_not):
lista = []
try:
cursor = self.__cursor.var(cx_Oracle.CURSOR)
print ids_not
data = self.__cursor.arrayvar(cx_Oracle.NUMBER, ids_not)
print data
l_query = self.__cursor.callproc("SCHEMA.PROC_SELECT_IDS_ENT_AMISTADES", [cursor,cod_us,data])
lista = l_query[0]
return lista
except cx_Oracle.DatabaseError as ex:
error, = ex.args
print(error.message)
return lista
The problem is when I call that procedure using this:
self.select_ids_entre_amistades(int_id,self.notPermited)
I visualize in the console the following message:
PLS-00306: wrong number or types of arguments in call to 'PROC
In the database I create the array object like this:
CREATE TYPE SCHEMA.ARRAY_ID_FRIENDS AS TABLE OF INT;
The Oracle stored procedure starts like this:
CREATE OR REPLACE PROCEDURE FACEBOOK.PROC_SELECT_IDS_ENT_AMISTADES
(CONSULTA OUT SYS_REFCURSOR,COD_US IN INT, IDS_FRIEND IN SCHEMA.ARRAY_ID_FRIENDS)
I don't know what the problem is, I believe cx_Oracle.NUMBER is not integer but there aren't other numeric type. Thanks in advance.
Try to use a plsql array in the parameters of the procedure and after that you pass the content of a sql array. The last one will be used to the sql statement into the procedure. It solve my trouble using oracle database 11g because in 12g you don't need to pass the content to an sql array. This could be the code:
def select_ids_entre_amistades(self,cod_us,ids_not):
lista = []
try:
cursor = self.__cursor.var(cx_Oracle.CURSOR)
varray = self.__cursor.arrayvar(cx_Oracle.NUMBER,ids_not)
l_query = self.__cursor.callproc("PACKFACE.P_SELECT_IDBFRIENDS", [cursor, cod_us, varray])
lista = l_query[0]
return lista
except cx_Oracle.DatabaseError as ex:
error, = ex.args
self.guardar_errores('dato ' + str(error.message))
return lista
And the stored procedure like this:
First you create a type
CREATE OR REPLACE TYPE LIST_IDS AS TABLE OF INT;
Second you create your package
CREATE OR REPLACE PACKAGE PACKFACE IS
TYPE LISTADO_IDS IS TABLE OF INT INDEX BY PLS_INTEGER;
PROCEDURE P_SELECT_IDBFRIENDS (CONSULTA OUT SYS_REFCURSOR,COD_US IN INT,IDS_NOT IN LISTADO_IDS);
END;
And finally create the body of the package
CREATE OR REPLACE PACKAGE BODY PACKFACE IS
PROCEDURE P_SELECT_IDBFRIENDS (CONSULTA OUT SYS_REFCURSOR,COD_US IN INT, IDS_NOT IN LISTADO_IDS)
IS
num_array LIST_IDS;
BEGIN
num_array:=LIST_IDS();
for i in 1 .. IDS_NOT.count
loop
num_array.extend(1);
num_array(i) := IDS_NOT(i);
end loop;
OPEN CONSULTA FOR
SELECT * FROM T_TABLE WHERE ID IN (SELECT COLUMN_VALUE FROM TABLE(num_array));
END;
END;
I hope that It helps you.
When you look at the cx_Oracle documentation, it says you can create the arrays like this;
Cursor.arrayvar(dataType, value[, size])
Create an array variable associated with the cursor of the given type and size and return a variable object (Variable Objects). The value is either an integer specifying the number of elements to allocate or it is a list and the number of elements allocated is drawn from the size of the list. If the value is a list, the variable is also set with the contents of the list. If the size is not specified and the type is a string or binary, 4000 bytes (maximum allowable by Oracle) is allocated. This is needed for passing arrays to PL/SQL (in cases where the list might be empty and the type cannot be determined automatically) or returning arrays from PL/SQL.
You may pass your arrays as long as array types are compatible with your PL/SQL procedure's parameter. Here is a simple example to create an array.
>>> myarray=cursor.arrayvar(cx_Oracle.NUMBER,range(0,10))
>>> myarray
<cx_Oracle.NUMBER with value [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]>
Here is a link (belongs to 2005 seems outdated, not sure) showing how to create arrays in PL/SQL side.
EDIT:
I added a complete example below showing how to pass arrayvar and other variable types. I tested the code with Oracle 10g and Python 2.7. I hope this helps.
from __future__ import print_function
import cx_Oracle as cxo
conn = cxo.connect("<YOUR TNS STRING>")
cursor = conn.cursor()
ref_cursor = cursor.var(cxo.CURSOR)
cod_us = cursor.var(cxo.NUMBER, 10)
ids_friend = cursor.arrayvar(cxo.NUMBER, range(0, 10))
ids_friend_sum = cursor.var(cxo.NUMBER)
cursor.execute('''
DECLARE
TYPE REF_CURSOR IS REF CURSOR;
TYPE ARRAY_ID_FRIENDS IS TABLE OF INT INDEX BY BINARY_INTEGER;
FUNCTION test(CONSULTA OUT REF_CURSOR,
COD_US IN INT,
IDS_FRIEND IN ARRAY_ID_FRIENDS) RETURN NUMBER
IS
sum_ NUMBER:=0;
BEGIN
OPEN CONSULTA FOR SELECT 1 FROM DUAL UNION SELECT 2 FROM DUAL;
FOR i in IDS_FRIEND.FIRST..IDS_FRIEND.LAST LOOP
sum_:=sum_+IDS_FRIEND(i);
END LOOP;
RETURN sum_;
END;
BEGIN
:ids_friend_sum:=test(:ref_cursor,:cod_us,:ids_friend);
END;
''', {"ref_cursor": ref_cursor, "cod_us": cod_us, "ids_friend": ids_friend,
"ids_friend_sum": ids_friend_sum})
print("ref cursor=", end=" ")
for rec in ref_cursor.getvalue():
print(rec, end="\t")
print("\nids_friend_sum=", ids_friend_sum.getvalue())
I have three lists that look like this:
age = ['51+', '21-30', '41-50', '31-40', '<21']
cluster = ['notarget', 'cluster3', 'allclusters', 'cluster1', 'cluster2']
device = ['htc_one_2gb','iphone_6/6+_at&t','iphone_6/6+_vzn','iphone_6/6+_all_other_devices','htc_one_2gb_limited_time_offer','nokia_lumia_v3','iphone5s','htc_one_1gb','nokia_lumia_v3_more_everything']
I also have column in a df that looks like this:
campaign_name
0 notarget_<21_nokia_lumia_v3
1 htc_one_1gb_21-30_notarget
2 41-50_htc_one_2gb_cluster3
3 <21_htc_one_2gb_limited_time_offer_notarget
4 51+_cluster3_iphone_6/6+_all_other_devices
I want to split the column into three separate columns based on the values in the above lists. Like so:
age cluster device
0 <21 notarget nokia_lumia_v3
1 21-30 notarget htc_one_1gb
2 41-50 cluster3 htc_one_2gb
3 <21 notarget htc_one_2gb_limited_time_offer
4 51+ cluster3 iphone_6/6+_all_other_devices
First thought was to do a simple test like this:
ages_list = []
for i in ages:
if i in df['campaign_name'][0]:
ages_list.append(i)
print ages_list
>>> ['<21']
I was then going to convert ages_list to a series and combine it with the remaining two to get the end result above but i assume there is a more pythonic way of doing it?
the idea behind this is that you'll create a regular expression based on the values you already have , for example if you want to build a regular expressions that capture any value from your age list you may do something like this '|'.join(age) and so on for all the values you already have cluster & device.
a special case for device list becuase it contains + sign that will conflict with the regex ( because + means one or more when it comes to regex ) so we can fix this issue by replacing any value of + with \+ , so this mean I want to capture literally +
df = pd.DataFrame({'campaign_name' : ['notarget_<21_nokia_lumia_v3' , 'htc_one_1gb_21-30_notarget' , '41-50_htc_one_2gb_cluster3' , '<21_htc_one_2gb_limited_time_offer_notarget' , '51+_cluster3_iphone_6/6+_all_other_devices'] })
def split_df(df):
campaign_name = df['campaign_name']
df['age'] = re.findall('|'.join(age) , campaign_name)[0]
df['cluster'] = re.findall('|'.join(cluster) , campaign_name)[0]
df['device'] = re.findall('|'.join([x.replace('+' , '\+') for x in device ]) , campaign_name)[0]
return df
df.apply(split_df, axis = 1 )
if you want to drop the original column you can do this
df.apply(split_df, axis = 1 ).drop( 'campaign_name', axis = 1)
Here I'm assuming that a value must be matched by regex but if this is not the case you can do your checks , you got the idea
I'm trying to query based on node number and return a list of properties. The following cypher query works in the neo4j browser but when I try to pass the same cypher query via py2neo I get:
"Expected a property container or number here, but got:91"
where "91" is the node number I'm querying on.
an excerpt from my code:
def neighbor_finder(a):
try:
graph_db = neo4j.GraphDatabaseService(url)
query = neo4j.CypherQuery(graph_db,
"""CYPHER 2.0
MATCH (n)-[r]-(m)
WHERE ID(n) = {t}
WITH collect (Distinct m.size) as sizes, collect (Distinct m.weight) as weights, collect (Distinct m.color) as colors, collect (distinct n.label) as node_
RETURN sizes, weights, colors, node_
""")
result = query.execute(t=a)
for r in result:
column = [str(item) for item in r.columns]
value = [str(item) for item in r.values]
db_dict={k: v for k, v in zip(column, value)}
for x, y in db_dict.items():
print x, y
except Exception as e:
print e
Can you provide details of the type and value of the a argument that is passed into this function? Without it I cannot see what py2neo is passing into the query parameters.
argument "a" isn't being read because it needs to be declared as an integer, cypher/py2neo are currently reading it as a string.