Parent Model:
id = auto_generated
add_me = IntegerField()
Child Model:
id = ForeignKey(Parent)
my_date= DateTime(auto_now_add=True)
and,
today = django.util.timezone.now()
time = today - timedelta(days=10)
now I want to retrieve records as:
child.objects.filter(my_date__lte= time +
(timedelta(days=1)*F('id__add_me')))
Everthing works fine except:
(timedelta(days=1)*F('id__add_me'))
I am using MYSQL:
I get:
django.db.utils.ProgrammingError:
(1064, "You have an error in your SQL syntax;
check the manual that corresponds to your MySQL
server version for the right syntax to use near
'* `parent`.`add_me`))))' at line 1")
Query Generated is like:
SELECT* FROM `child INNER JOIN `parent` ON
(`child`.`id` = `parent`.`id`) WHERE
( `child`.`my_date` <= ((2018-07-30 09:11:18.670900
+ (INTERVAL '86400.000000' SECOND_MICROSECOND * `parent`.`add_me`))))
As you can see * (multiplication) is with INTERVAL. I hard coded query putting the parent.add_me inside the INTERVAL and got it right, how can I do so via Django? i.e. DATE_ADD in django
I want to reduce days from a datetime field, where days are stored in master table field.
Maybe it doesn't work with MYSQL, so please provide an alternative solution to such.
UPDATED:
Somehow I found error by writing MYSQL query as:
SELECT * FROM child INNER JOIN parent ON" \
child.id = parent.id WHERE
child.my_date<= '" + str(time) + \
"' + INTERVAL parent.add_me DAY "
The problem was with single quotes (') i.e. in mysql for <= the datetime shall be within single quotes else won't work. i.e.
2018-07-30 09:11:18.670900 shall be written as
'2018-07-30 09:11:18.670900' if adding with INTERVAL,
But as queries are generated by Django, how do I achieve that?
Related
I have searched for the solution to this problem for a long time, but I haven't got the appropriate method.
Basically All I have is tons of tables, and I want to query value from different tables using raw SQL.
In Django, we need a class representing a table to perform the query, for example:
Routes.objects.raw("SELECT * FROM routes")
In this way, I can only query a table, but what if I want to query different tables based on the user's input?
I'm new to Django, back in ASP.NET we can simply do the following query:
string query = "SELECT * FROM " + county + " ;";
var bus = _context.Database.SqlQuery<keelung>(query).ToList();
Is this case, I can do the query directly on the database instead of the model class, and I can select the table based on the user's selection.
Is there any method to achieve this with Django?
You can run raw queries in Django like this -
From django.db import connection
cursor = connection.cursor()
table = my_table;
cursor.execute("Select * from " + table)
data = cursor.fetchall()
If I have a large number of SQLite databases, all with the same schema, what is the best way to merge them together in order to perform a query on all databases?
I know it is possible to use ATTACH to do this but it has a limit of 32 and 64 databases depending on the memory system on the machine.
To summarize from the Nabble post in DavidM's answer:
attach 'c:\test\b.db3' as toMerge;
BEGIN;
insert into AuditRecords select * from toMerge.AuditRecords;
COMMIT;
detach toMerge;
Repeat as needed.
Note: added detach toMerge; as per mike's comment.
Although a very old thread, this is still a relevant question in today's programming needs. I am posting this here because none of the answers provided yet is concise, easy, and straight-to-point. This is for sake of Googlers that end up on this page. GUI we go:
Download Sqlitestudio
Add all your database files by using the Ctrl + O keyboard shortcut
Double-click each now-loaded db file to open/activate/expand them all
Fun part: simply right-click on each of the tables and click on Copy, and then go to the target database in the list of the loaded database files (or create new one if required) and right-click on the target db and click on Paste
I was wowed to realize that such a daunting task can be solved using the ancient programming skill called: copy-and-paste :)
Here is a simple python code to either merge two database files or scan a directory to find all database files and merge them all together (by simply inserting all data in other files to the first database file found).Note that this code just attaches the databases with the same schema.
import sqlite3
import os
def merge_databases(db1, db2):
con3 = sqlite3.connect(db1)
con3.execute("ATTACH '" + db2 + "' as dba")
con3.execute("BEGIN")
for row in con3.execute("SELECT * FROM dba.sqlite_master WHERE type='table'"):
combine = "INSERT OR IGNORE INTO "+ row[1] + " SELECT * FROM dba." + row[1]
print(combine)
con3.execute(combine)
con3.commit()
con3.execute("detach database dba")
def read_files(directory):
fname = []
for root,d_names,f_names in os.walk(directory):
for f in f_names:
c_name = os.path.join(root, f)
filename, file_extension = os.path.splitext(c_name)
if (file_extension == '.sqlitedb'):
fname.append(c_name)
return fname
def batch_merge(directory):
db_files = read_files(directory)
for db_file in db_files[1:]:
merge_databases(db_files[0], db_file)
if __name__ == '__main__':
batch_merge('/directory/to/database/files')
Late answer, but you can use:
#!/usr/bin/python
import sys, sqlite3
class sqlMerge(object):
"""Basic python script to merge data of 2 !!!IDENTICAL!!!! SQL tables"""
def __init__(self, parent=None):
super(sqlMerge, self).__init__()
self.db_a = None
self.db_b = None
def loadTables(self, file_a, file_b):
self.db_a = sqlite3.connect(file_a)
self.db_b = sqlite3.connect(file_b)
cursor_a = self.db_a.cursor()
cursor_a.execute("SELECT name FROM sqlite_master WHERE type='table';")
table_counter = 0
print("SQL Tables available: \n===================================================\n")
for table_item in cursor_a.fetchall():
current_table = table_item[0]
table_counter += 1
print("-> " + current_table)
print("\n===================================================\n")
if table_counter == 1:
table_to_merge = current_table
else:
table_to_merge = input("Table to Merge: ")
return table_to_merge
def merge(self, table_name):
cursor_a = self.db_a.cursor()
cursor_b = self.db_b.cursor()
new_table_name = table_name + "_new"
try:
cursor_a.execute("CREATE TABLE IF NOT EXISTS " + new_table_name + " AS SELECT * FROM " + table_name)
for row in cursor_b.execute("SELECT * FROM " + table_name):
print(row)
cursor_a.execute("INSERT INTO " + new_table_name + " VALUES" + str(row) +";")
cursor_a.execute("DROP TABLE IF EXISTS " + table_name);
cursor_a.execute("ALTER TABLE " + new_table_name + " RENAME TO " + table_name);
self.db_a.commit()
print("\n\nMerge Successful!\n")
except sqlite3.OperationalError:
print("ERROR!: Merge Failed")
cursor_a.execute("DROP TABLE IF EXISTS " + new_table_name);
finally:
self.db_a.close()
self.db_b.close()
return
def main(self):
print("Please enter name of db file")
file_name_a = input("File Name A:")
file_name_b = input("File Name B:")
table_name = self.loadTables(file_name_a, file_name_b)
self.merge(table_name)
return
if __name__ == '__main__':
app = sqlMerge()
app.main()
SRC : Tool to merge identical SQLite3 databases
If you only need to do this merge operation once (to create a new bigger database), you could create a script/program that will loop all your sqlite databases and then insert the data into your main (big) database.
If you have reached the bottom of this feed and yet didn't find your solution, here is also a way to merge the tables of 2 or more sqlite databases.
First try to download and install DB browser for sqlite database. Then try to open your databases in 2 windows and try merging them by simply drag and drop tables from one to another. But the problem is that you can just drag and drop only one table at a time and therefore its not really a solution for this answer specifically but yet it can used to save some time from further searches if your database is small.
With no offense, just as one developer to another, I'm afraid that your idea seems terribly inefficient.
It seems to me that instead of uniting SQLite databases you should probably be storing several tables within the same Database file.
However if I'm mistaken I guess you could ATTACH the databases and then use a VIEW to simplify your queries. Or make an in-memory table and copy over all the data (but that's even worse performance wise, especially if you have large databases)
I have a problem when I try to use pyodbc executemany function.
I have an Oracle database and I want to extract data for multiple days.
I cannot use between in my request, because the database is not indexed on the date field and its taking forever.
I want to manually ask all day and process answers.
I cannot thread this part, so I wanted to use executemany to get rows more quickly.
The problem is when I use executemany I only got the result of the last argument asked.
Here is my code:
import pyodbc
conn = pyodbc.connect('DRIVER={Oracle in instantclient_11_2};DBQ=dbname;UID=uid;PWD=pwd')
cursor = conn.cursor()
query = "SELECT date FROM table WHERE date = TO_DATE(?, 'DD/MM/YYYY')"
query_args = (
('29/04/2016',),
('28/04/2016',),
)
cursor.executemany(query, query_args)
rows = cursor.fetchall()
In rows, I can only find rows with (datetime.datetime(2016, 4, 28, 0, 0), ).
Always the last argument.
I am using python 2.7.9 from WinPython on a Oracle database with a client on 11.0.2.
Except this query, every other query is perfectly fine.
I cannot use IN () synthax for 2 reasons:
I want to limit operations on database side, and do most of thing on script side (I've tried but it's way too long)
I might have more than 1000 different dates in the request.
(Right now I'm using IN() OR IN() OR IN()... but if anyone find something better that would be wonderful !)
Am I doing something wrong ?
Thanks for helping.
Your query runs once with one argument. If you want to run for multiple dates either use "IN" clause, this will require to modify query_args a bit.
"SELECT date FROM table WHERE date in (TO_DATE(?, 'DD/MM/YYYY'), TO_DATE(?, 'DD/MM/YYYY'))"
query_args = (
('29/04/2016','28/04/2016'),
)
or cursor through each date argument:
while query_arg in query_args:
cursor.executemany(query, query_arg )
rows = cursor.fetchall()
I am using Java DB (Java DB is Oracle's supported version of Apache Derby and contains the same binaries as Apache Derby. source: http://www.oracle.com/technetwork/java/javadb/overview/faqs-jsp-156714.html#1q2).
I am trying to update a column in one table, however I need to join that table with 2 other tables within the same database to get accurate results (not my design, nor my choice).
Below are my three tables, ADSID is a key linking Vehicles and Customers and ADDRESS and ZIP in Salesresp are used to link it to Customers. (Other fields left out for the sake of brevity.)
Salesresp(address, zip, prevsale)
Customers(adsid, address, zipcode)
Vehicles(adsid, selldate)
The goal is to find customers in the SalesResp table that have previously purchased a vehicle before the given date. They are identified by address and adsid in Customers and Vechiles respectively.
I have seen updates to a column with a single join and in fact asked a question about one of my own update/joins here (UPDATE with INNER JOIN). But now I need to take it that one step further and use both tables to get all the information.
I can get a multi-JOIN SELECT statement to work:
SELECT * FROM salesresp
INNER JOIN customers ON (SALESRESP.ZIP = customers.ZIPCODE) AND
(SALESRESP.ADDRESS = customers.ADDRESS)
INNER JOIN vehicles ON (Vehicles.ADSId =Customers.ADSId )
WHERE (VEHICLES.SELLDATE<'2013-09-24');
However I cannot get a multi-JOIN UPDATE statement to work.
I have attempted to try the update like this:
UPDATE salesresp SET PREVSALE = (SELECT SALESRESP.address FROM SALESRESP
WHERE SALESRESP.address IN (SELECT customers.address FROM customers
WHERE customers.adsid IN (SELECT vehicles.adsid FROM vehicles
WHERE vehicles.SELLDATE < '2013-09-24')));
And I am given this error: "Error code 30000, SQL state 21000: Scalar subquery is only allowed to return a single row".
But if I change that first "=" to a "IN" it gives me a syntax error for having encountered "IN" (Error code 30000, SQL state 42X01).
I also attempted to do more blatant inner joins, but upon attempting to execute this code I got the the same error as above: "Error code 30000, SQL state 42X01" with it complaining about my use of the "FROM" keyword.
update salesresp set prevsale = vehicles.selldate
from salesresp sr
inner join vehicles v
on sr.prevsale = v.selldate
inner join customers c
on v.adsid = c.adsid
where v.selldate < '2013-09-24';
And in a different configuration:
update salesresp
inner join customer on salesresp.address = customer.address
inner join vehicles on customer.adsid = vehicles.ADSID
set salesresp.SELLDATE = vehicles.selldate where vehicles.selldate < '2013-09-24';
Where it finds the "INNER" distasteful: Error code 30000, SQL state 42X01: Syntax error: Encountered "inner" at line 3, column 1.
What do I need to do to get this multi-join update query to work? Or is it simply not possible with this database?
Any advice is appreciated.
If I were you I would:
1) Turn off autocommit (if you haven't already)
2) Craft a select/join which returns a set of columns that identifies the record you want to update E.g. select c1, c2, ... from A join B join C... WHERE ...
3) Issue the update. E.g. update salesrep SET CX = cx where C1 = c1 AND C2 = c2 AND...
(Having an index on C1, C2, ... will boost performance)
4) Commit.
That way you don't have worry about mixing the update and the join, and doing it within a txn ensures that nothing can change the result of the join before your update goes through.
I have several database tables with 2 primary keys, id and date. I do not update the records but instead insert a new record with the updated information. This new record has the same id and the date field is NOW(). I will use a product table to explain my question.
I want to be able to request the product details at a specific date. I therefore use the following subquery in DQL, which works fine:
WHERE p.date = (
SELECT MAX(pp.date)
FROM Entity\Product pp
WHERE pp.id = p.id
AND pp.date < :date
)
This product table has some referenced tables, like category. This category table has the same id and date primary key combination. I want to be able to request the product details and the category details at a specific date. I therefore expanded the DQL as shown above to the following, which also works fine:
JOIN p.category c
WHERE p.date = (
SELECT MAX(pp.date)
FROM Entity\Product pp
WHERE pp.id = p.id
AND pp.date < :date
)
AND c.date = (
SELECT MAX(cc.date)
FROM Entity\ProductCategory cc
WHERE cc.id = c.id
AND cc.date < :date
)
However, as you can see, if I have multiple referenced tables I will have to copy the same piece of DQL. I want to somehow add these subqueries to the entities so that every time an entity is called it adds this subquery.
I have thought of adding this in a __construct($date) or some kind of setUp($date) method, but I'm kind of stuck here. Also, would it help to add #Id to Entity\Product::date?
I hope someone can help me. I do not expect a complete solution, one step in a good direction would be very much appreciated.
I think I've found my solution. The trick was (first, to update to Doctrine 2.2 and) using a filter:
namespace Filter;
use Doctrine\ORM\Mapping\ClassMetaData,
Doctrine\ORM\Query\Filter\SQLFilter;
class VersionFilter extends SQLFilter {
public function addFilterConstraint(ClassMetadata $targetEntity, $targetTableAlias) {
$return = $targetTableAlias . '.date = (
SELECT MAX(sub.date)
FROM ' . $targetEntity->table['name'] . ' sub
WHERE sub.id = ' . $targetTableAlias . '.id
AND sub.date < ' . $this->getParameter('date') . '
)';
return $return;
}
}
Add the filter to the configuration:
$configuration->addFilter("version", Filter\VersionFilter");
And enable it in my repository:
$this->_em->getFilters()->enable("version")->setParameter('date', $date);