I have a problem when I try to use pyodbc executemany function.
I have an Oracle database and I want to extract data for multiple days.
I cannot use between in my request, because the database is not indexed on the date field and its taking forever.
I want to manually ask all day and process answers.
I cannot thread this part, so I wanted to use executemany to get rows more quickly.
The problem is when I use executemany I only got the result of the last argument asked.
Here is my code:
import pyodbc
conn = pyodbc.connect('DRIVER={Oracle in instantclient_11_2};DBQ=dbname;UID=uid;PWD=pwd')
cursor = conn.cursor()
query = "SELECT date FROM table WHERE date = TO_DATE(?, 'DD/MM/YYYY')"
query_args = (
('29/04/2016',),
('28/04/2016',),
)
cursor.executemany(query, query_args)
rows = cursor.fetchall()
In rows, I can only find rows with (datetime.datetime(2016, 4, 28, 0, 0), ).
Always the last argument.
I am using python 2.7.9 from WinPython on a Oracle database with a client on 11.0.2.
Except this query, every other query is perfectly fine.
I cannot use IN () synthax for 2 reasons:
I want to limit operations on database side, and do most of thing on script side (I've tried but it's way too long)
I might have more than 1000 different dates in the request.
(Right now I'm using IN() OR IN() OR IN()... but if anyone find something better that would be wonderful !)
Am I doing something wrong ?
Thanks for helping.
Your query runs once with one argument. If you want to run for multiple dates either use "IN" clause, this will require to modify query_args a bit.
"SELECT date FROM table WHERE date in (TO_DATE(?, 'DD/MM/YYYY'), TO_DATE(?, 'DD/MM/YYYY'))"
query_args = (
('29/04/2016','28/04/2016'),
)
or cursor through each date argument:
while query_arg in query_args:
cursor.executemany(query, query_arg )
rows = cursor.fetchall()
Related
If I have created a Dates column in flask sqlalchemy and also stored some dates in it, how can I check if each and every one of these dates are between to dates that I choose
There are lots of ways to accomplish this. Here's one example.
The goal is to select all rows that have a date outside your desired range. If the result set is empty, all rows have a valid date. For good measure, we'll include any rows that don't have a date value in our "bad rows" query.
from sqlalchemy import select, or_
with Session.begin() as session:
my_start_date = '2022-01-01'
my_end_date = '2022-01-31'
query = select(MyTable).where(
or_(MyTable.date < my_start_date,
MyTable.date > my_end_date,
MyTable.date == Null)
)
results = session.execute(query).all()
Now you can take a look at the results and see what's up.
you can use between in your orm or plain query:
with Session.begin() as session:
my_start_date = '2022-01-01'
my_end_date = '2022-01-31'
q = session.query(table_name).filter(table_name.c.date.between(my_start_date,my_end_date))
.....
or
select(table_name).where(table_name.c.date.between(my_start_date, my_end_date))
Firstly, I have a table in SQLlite3 with two fields CAR (TEXT NOT NULL), checkout (TEXT NOT NULL)
car checkout
red %d%d/%m%m/%Y (for example 27/09/2021)
Second, I wrote a script which the structure is when I run it, all the entries that current date is equal or bigger than checkout to be deleted.
Third, in the same script with SELECT to check if the car is in the list and checkout is bigger than current date exclude from my available cars.
The code snippet makes the first step is the following:
try:
con = lite.connect(DB)
with con:
paper=[]
cur=con.cursor()
cur.execute("DELETE FROM CHECK_TABLE WHERE DATE(substr(checkout,7,4)||substr(checkout,4,2)||substr(checkout,1,2))<=DATE(strftime('%Y%m%d',date('now')))")
con.commit()
print('Entries with old dates deleted.')
except lite.Error as e:
print('Error connection: ',e)
The problem is that is not deleting anything. The strange behaviour is firstly that the SQL query works in DB Browser,
Image: Proof DB Browser in Windows 10 - Python2.7 - SQLite3
the second strange behaviour is that no error is raising and the third strange is that I tested two days ago and it worked normally! I really need your thoughts.
The same logic is in the following code snippet which is the the third step that I described above with SELECT command.
def ReadDateAndCar(car):
try:
con = lite.connect(DB)
with con:
paper=[]
cur=con.cursor()
cur.execute("SELECT DISTINCT car FROM CHECK_TABLE WHERE car='"+car+"' AND DATE(substr(checkout,7,4)||substr(checkout,4,2)||substr(checkout,1,2))<=DATE(strftime('%Y%m%d',date('now')))")
free_cars=cur.fetchall()
return free_cars
except lite.Error as e:
print('Error connection: ',e)
return 0
Exactly the same problems. SQL query works fine, no python error is raising, it worked few days ago. Can someone enlighten me?
Both your queries are wrong and they don't work in DB Browser either.
What you should do is store the dates with the ISO format YYYY-MM-DD, because this is the only text date format compatible with SQLite's datetime functions like date() and strftime() and it is comparable.
If you use any other format the result of these functions is null and this is what happens in your case.
The expressions substr(checkout,7,4)||substr(checkout,4,2)||substr(checkout,1,2) and strftime('%Y%m%d',date('now')) return dates in the format YYYYMMDD and if you use them inside date() or strftime() the result is null.
Since you obtain in both sides of the inequality dates in the format YYYYMMDD then they are directly comparable and you should not use the function date().
The condition should be:
substr(checkout, -4) || substr(checkout, 4, 2) || substr(checkout, 1, 2) <= strftime('%Y%m%d', 'now')
I'm pretty novice at programming (recently learned functions), and have found myself re-writing the same "insert into mysql table" function (below) from script to script... mainly to just modify these two section - (name,insert_ts) &&& VALUES (%s, %s)
Is there a good way to re-write the below to accept ANY number of values , based on length of a tuple that contains values as well as inserting the column headers based on 'labels' list? VALUES (%s, %s) and this part (name,insert_ts)
list_of_tuples = [] #list of records to be inserted.
#take a list of dictionaries - and create a list of tuples in proper format/order
for dict1 in output:
one_list = []
one_list.extend((dict1['name'],dict1['insert_ts']))
list_of_tuples.append(tuple(one_list))
labels = ['name', 'insert_ts']
#db_write accepts table name as str, labels as str, and output as list of tuples
def db_write(table,labels,output):
local_cursor.executemany(""" INSERT INTO my_table
(name,insert_ts) #this is pulled from 'labels'
VALUES (%s, %s) #number of %s comes from len(labels)
"""
, list_of_tuples)
local_db.commit()
local_db.close()
#print 'done posting!'
Or, is there a better way to accomplish what I'm trying to do, using mysqldb?
Thank you all in advance!
After a bit of experience (3 months, heh), wanted to update everyone on the solution that seems to work pretty well!
Instead of using mysqldb, I spent some time learning how to use SQL Alchemy python package, and would recommend everyone do the same!
SQL Alchemy allows you to:
1) Define a table within python code (used Excel to come up with column names, etc).
2) Most important! You can pass on a dictionary to SQL Alchemy, and as long as dictionary's key names match the table's key names, everything will magically get posted to your SQL table. If you have 60 columns in your sql table, but your dict has only two keys - BAM, SQL Alchemy will take care of everything and post just the two values, and leave the other values in MySQL as blanks. MAGIC!
I have this custom date that I created as a measure:
Start Date = DATE(YEAR(MAX(Loss[dte_month_end]))-1,12,31)
So this part looks fine in PowerBI and seems to be the right format.
So now I created a new column where I'm going through my data to check whether a record is equal to my "Start Date" as defined above.
IsStart = IF(Loss[dte_month_end]=[Start Date], TRUE, FALSE)
but the weird thing is that all records are evaluated to false.
I know this is actually not the case in my actual data, and I could find actual records with dte_month_end = 12/31/2017 as shown above.
Can someone help me understand why the IF statement would not be able to evaluate this correctly? I initially thought that this may be a case of the DATETIME format being inconsistent - but I purposefully changed both formats to be the same to no avail.
Thanks.
Edit1----------- FYI:
This is the format that my dte_month_end field has:
Edit2 --
I tried changing the dte_month_end format to Date instead of DateTime, and it still doesn't seem to work:
This is happening because you are using a measure inside of a calculated column. When you do this, the filter context for the measure is the row context in the table.
To fix this, you need to modify the filter context for your measure. For example:
Start Date = DATE(YEAR(CALCULATE(MAX(Loss[dte_month_end]), ALL(Loss))) - 1, 12, 31)
or
Start Date = DATE(YEAR(MAXX(ALL(Loss), Loss[dte_month_end])) - 1, 12, 31)
If you don't do this, the MAX only looks at the current row, rather than all the rows in the table.
class Log:
project = ForeignKey(Project)
msg = CharField(...)
date = DateField(...)
I want to select the four most recent Log entries where each Log entry must have a unique project foreign key. I've tries the solutions on google search but none of them works and the django documentation isn't that very good for lookup..
I tried stuff like:
Log.objects.all().distinct('project')[:4]
Log.objects.values('project').distinct()[:4]
Log.objects.values_list('project').distinct('project')[:4]
But this either return nothing or Log entries of the same project..
Any help would be appreciated!
Queries don't work like that - either in Django's ORM or in the underlying SQL. If you want to get unique IDs, you can only query for the ID. So you'll need to do two queries to get the actual Log entries. Something like:
id_list = Log.objects.order_by('-date').values_list('project_id').distinct()[:4]
entries = Log.objects.filter(id__in=id_list)
Actually, you can get the project_ids in SQL. Assuming that you want the unique project ids for the four projects with the latest log entries, the SQL would look like this:
SELECT project_id, max(log.date) as max_date
FROM logs
GROUP BY project_id
ORDER BY max_date DESC LIMIT 4;
Now, you actually want all of the log information. In PostgreSQL 8.4 and later you can use windowing functions, but that doesn't work on other versions/databases, so I'll do it the more complex way:
SELECT logs.*
FROM logs JOIN (
SELECT project_id, max(log.date) as max_date
FROM logs
GROUP BY project_id
ORDER BY max_date DESC LIMIT 4 ) as latest
ON logs.project_id = latest.project_id
AND logs.date = latest.max_date;
Now, if you have access to windowing functions, it's a bit neater (I think anyway), and certainly faster to execute:
SELECT * FROM (
SELECT logs.field1, logs.field2, logs.field3, logs.date
rank() over ( partition by project_id
order by "date" DESC ) as dateorder
FROM logs ) as logsort
WHERE dateorder = 1
ORDER BY logs.date DESC LIMIT 1;
OK, maybe it's not easier to understand, but take my word for it, it runs worlds faster on a large database.
I'm not entirely sure how that translates to object syntax, though, or even if it does. Also, if you wanted to get other project data, you'd need to join against the projects table.
I know this is an old post, but in Django 2.0, I think you could just use:
Log.objects.values('project').distinct().order_by('project')[:4]
You need two querysets. The good thing is it still results in a single trip to the database (though there is a subquery involved).
latest_ids_per_project = Log.objects.values_list(
'project').annotate(latest=Max('date')).order_by(
'-latest').values_list('project')
log_objects = Log.objects.filter(
id__in=latest_ids_per_project[:4]).order_by('-date')
This looks a bit convoluted, but it actually results in a surprisingly compact query:
SELECT "log"."id",
"log"."project_id",
"log"."msg"
"log"."date"
FROM "log"
WHERE "log"."id" IN
(SELECT U0."id"
FROM "log" U0
GROUP BY U0."project_id"
ORDER BY MAX(U0."date") DESC
LIMIT 4)
ORDER BY "log"."date" DESC