Python2.7 & SQLite3: DELETE & SELECT with DATE(SUBSTR()) DO NOT working - python-2.7

Firstly, I have a table in SQLlite3 with two fields CAR (TEXT NOT NULL), checkout (TEXT NOT NULL)
car checkout
red %d%d/%m%m/%Y (for example 27/09/2021)
Second, I wrote a script which the structure is when I run it, all the entries that current date is equal or bigger than checkout to be deleted.
Third, in the same script with SELECT to check if the car is in the list and checkout is bigger than current date exclude from my available cars.
The code snippet makes the first step is the following:
try:
con = lite.connect(DB)
with con:
paper=[]
cur=con.cursor()
cur.execute("DELETE FROM CHECK_TABLE WHERE DATE(substr(checkout,7,4)||substr(checkout,4,2)||substr(checkout,1,2))<=DATE(strftime('%Y%m%d',date('now')))")
con.commit()
print('Entries with old dates deleted.')
except lite.Error as e:
print('Error connection: ',e)
The problem is that is not deleting anything. The strange behaviour is firstly that the SQL query works in DB Browser,
Image: Proof DB Browser in Windows 10 - Python2.7 - SQLite3
the second strange behaviour is that no error is raising and the third strange is that I tested two days ago and it worked normally! I really need your thoughts.
The same logic is in the following code snippet which is the the third step that I described above with SELECT command.
def ReadDateAndCar(car):
try:
con = lite.connect(DB)
with con:
paper=[]
cur=con.cursor()
cur.execute("SELECT DISTINCT car FROM CHECK_TABLE WHERE car='"+car+"' AND DATE(substr(checkout,7,4)||substr(checkout,4,2)||substr(checkout,1,2))<=DATE(strftime('%Y%m%d',date('now')))")
free_cars=cur.fetchall()
return free_cars
except lite.Error as e:
print('Error connection: ',e)
return 0
Exactly the same problems. SQL query works fine, no python error is raising, it worked few days ago. Can someone enlighten me?

Both your queries are wrong and they don't work in DB Browser either.
What you should do is store the dates with the ISO format YYYY-MM-DD, because this is the only text date format compatible with SQLite's datetime functions like date() and strftime() and it is comparable.
If you use any other format the result of these functions is null and this is what happens in your case.
The expressions substr(checkout,7,4)||substr(checkout,4,2)||substr(checkout,1,2) and strftime('%Y%m%d',date('now')) return dates in the format YYYYMMDD and if you use them inside date() or strftime() the result is null.
Since you obtain in both sides of the inequality dates in the format YYYYMMDD then they are directly comparable and you should not use the function date().
The condition should be:
substr(checkout, -4) || substr(checkout, 4, 2) || substr(checkout, 1, 2) <= strftime('%Y%m%d', 'now')

Related

Using Django SQL Explorer how do I correctly set default dates for paramters with PostgreSQL backend?

Our Django application just migrated from MySQL to PostgreSQL, and we are in the process of updating our user queries to use the correct syntax. These queries are stored/executed via Django-SQL-Explorer.
I have a query that for simplicity's sake looks like this:
SELECT * FROM table t WHERE t.scheduled_start_date BETWEEN $$from_date$$ AND $$to_date$$
The query above works, but I would like to set defaults for today & today+30 respectively.
I've tried the following WHERE clauses to no avail:
Works with user entered date, default throws syntax error
WHERE t.scheduled_start_date BETWEEN date('$$from_date:CURRENT_DATE$$') AND date('$$to_date:CURRENT_DATE + INTERVAL \'30 DAY\'$$')
Error using defaults:
syntax error at or near "30" LINE 28: ...urrent_date') AND date('current_date + interval \'30 day\'') ^
Defaults work correctly, but user entered dates do not:
WHERE t.scheduled_start_date BETWEEN date($$from_date:CURRENT_DATE$$) AND date($$to_date:CURRENT_DATE + INTERVAL '30 DAY'$$)
Error with user dates:
function date(integer) does not exist LINE 28: WHERE t.scheduled_start_date BETWEEN date(2019-09-30) AND da... ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts.
The message about casting makes sense to me since the user enters in string data, however I can't seem to get the syntax right that casts to DATE but only when the default is overridden.
Any suggestions would be helpful. Thanks!
Your input is a string value, current_date is a function call. The former needs quotes whereas the latter does not. Therefore your client side substitution will not work. You can move the default logic into the SQL statement using the COALESCE() function doc.
In case the client sends NULL values for an empty user input this would result in the following where clause:
BETWEEN date(COALESCE('$$from_date$$',current_date))
AND date(COALESCE('$$from_date$$',current_date + interval '30 DAYS'))
You added that your client sends an empty string, not a NULL value. So this should be changed to:
BETWEEN COALESCE(NULLIF('$$from_date$$','')::date, current_date)
AND COALESCE(NULLIF('$$from_date$$','')::date, current_date + interval '30 DAYS')

Python + MySQL DB dynamic insert query based on number of columns to insert

I'm pretty novice at programming (recently learned functions), and have found myself re-writing the same "insert into mysql table" function (below) from script to script... mainly to just modify these two section - (name,insert_ts) &&& VALUES (%s, %s)
Is there a good way to re-write the below to accept ANY number of values , based on length of a tuple that contains values as well as inserting the column headers based on 'labels' list? VALUES (%s, %s) and this part (name,insert_ts)
list_of_tuples = [] #list of records to be inserted.
#take a list of dictionaries - and create a list of tuples in proper format/order
for dict1 in output:
one_list = []
one_list.extend((dict1['name'],dict1['insert_ts']))
list_of_tuples.append(tuple(one_list))
labels = ['name', 'insert_ts']
#db_write accepts table name as str, labels as str, and output as list of tuples
def db_write(table,labels,output):
local_cursor.executemany(""" INSERT INTO my_table
(name,insert_ts) #this is pulled from 'labels'
VALUES (%s, %s) #number of %s comes from len(labels)
"""
, list_of_tuples)
local_db.commit()
local_db.close()
#print 'done posting!'
Or, is there a better way to accomplish what I'm trying to do, using mysqldb?
Thank you all in advance!
After a bit of experience (3 months, heh), wanted to update everyone on the solution that seems to work pretty well!
Instead of using mysqldb, I spent some time learning how to use SQL Alchemy python package, and would recommend everyone do the same!
SQL Alchemy allows you to:
1) Define a table within python code (used Excel to come up with column names, etc).
2) Most important! You can pass on a dictionary to SQL Alchemy, and as long as dictionary's key names match the table's key names, everything will magically get posted to your SQL table. If you have 60 columns in your sql table, but your dict has only two keys - BAM, SQL Alchemy will take care of everything and post just the two values, and leave the other values in MySQL as blanks. MAGIC!

Sitecore date comparison doesn't return expected results

I'm trying to return a list of courses using an iqueryable query but am having issues with some date comparisons.
I'm currently using the code
query = query.Where(r => r.EndDate >= DateTime.UtcNow);
which returns courses with dates in the future, however it will not return courses that end on the same day with a time ending later than the time returned by DateTime.UtcNow.
Any ideas what I am doing wrong?
I've just used Luke to check the index and if I use
end_date:[20170531t092205609z TO *]
I get back the exact results I need, however in the logs the actual query uses
+end_date:[20170531t092205609z TO *] +_template:a84b75fccac64eafa746f4b71e628adc - Filter :
I then get more results back including the course I was missing.
a) Why do I get more results back using the second query?
b) Why is it that in my C# code the results returned do not match the search results?
Had a similar issue and described our solution here: https://ggullentops.blogspot.be/2015/12/sitecore-lucene-index-and-datetime.html.
Our problems had 2 reasons:
The first reason is that Sitecore stores its DateTimes in UTC (which was an hour difference with our local time)
Second reason was that Sitecore uses "t" in the dates as lowercase in the query. In my index however they are all uppercase. If I try the query with Luke it does give me the wrong results indeed.. When I alter the query in Luke to use uppercase T it works correctly..
The easiest solution we found was a format attribute in the index config:
<field fieldName="datefrom" storageType="YES" indexType="UNTOKENIZED" vectorType="NO" boost="1f"
format="yyyyMMdd" type="System.DateTime"
settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider"/>
(note the format="...")

Executemany on pyodbc only return result from last parameter

I have a problem when I try to use pyodbc executemany function.
I have an Oracle database and I want to extract data for multiple days.
I cannot use between in my request, because the database is not indexed on the date field and its taking forever.
I want to manually ask all day and process answers.
I cannot thread this part, so I wanted to use executemany to get rows more quickly.
The problem is when I use executemany I only got the result of the last argument asked.
Here is my code:
import pyodbc
conn = pyodbc.connect('DRIVER={Oracle in instantclient_11_2};DBQ=dbname;UID=uid;PWD=pwd')
cursor = conn.cursor()
query = "SELECT date FROM table WHERE date = TO_DATE(?, 'DD/MM/YYYY')"
query_args = (
('29/04/2016',),
('28/04/2016',),
)
cursor.executemany(query, query_args)
rows = cursor.fetchall()
In rows, I can only find rows with (datetime.datetime(2016, 4, 28, 0, 0), ).
Always the last argument.
I am using python 2.7.9 from WinPython on a Oracle database with a client on 11.0.2.
Except this query, every other query is perfectly fine.
I cannot use IN () synthax for 2 reasons:
I want to limit operations on database side, and do most of thing on script side (I've tried but it's way too long)
I might have more than 1000 different dates in the request.
(Right now I'm using IN() OR IN() OR IN()... but if anyone find something better that would be wonderful !)
Am I doing something wrong ?
Thanks for helping.
Your query runs once with one argument. If you want to run for multiple dates either use "IN" clause, this will require to modify query_args a bit.
"SELECT date FROM table WHERE date in (TO_DATE(?, 'DD/MM/YYYY'), TO_DATE(?, 'DD/MM/YYYY'))"
query_args = (
('29/04/2016','28/04/2016'),
)
or cursor through each date argument:
while query_arg in query_args:
cursor.executemany(query, query_arg )
rows = cursor.fetchall()

mysql.connector, multi=True, sql variable assignment not working

SQL code (all in one file that is eventually saved in the python variable "query"):
select #dtmax:=DATE_FORMAT(max(dt), '%Y%m') from table_A;
delete from table_B where DATE_FORMAT(dt, '%Y%m')=#dtmax;
Does mysql-connector allow the use of variable assignment like I've done in the query above. i.e. take the value of max(date) from TABLE_A and delete everything with that date from TABLE_B.
python code:
c = conn.cursor(buffered=True)
c.execute(query, multi=True)
conn.commit()
conn.close()
All I know is that the 2nd SQL statement doesnt execute.
I can copy and paste the SQL code into Toad and run it there without any problems but not through mysql.connector library. I would have used pandas but this is legacy script written by someone else and I don't have time to re-write everything.
I kindly appreciate some help.
When you use multi=True, then execute() will return a generator. You need to iterate over that generator to actually advance the processing to the next sql statement in your multi-statement query:
c = conn.cursor(buffered=True)
results = c.execute(query, multi=True)
for cur in results:
print('cursor:', cur)
if cur.with_rows:
print('result:', cur.fetchall())
conn.commit()
conn.close()
cur.with_rows will be True if there are results to fetch for the current statement.
This is all described in the documentation of MySQLCursor.execute()