flask-sqlalchemy column_property value has cache, i how to set to close cache
the code is
class EbayItem(db.Model):
__tablename__ = 'table_name'
id = Column(BigInteger, primary_key=True)
yesterday_sale_quntity = column_property(
select(EbayItemDaily.quantity_sold_d)
.execution_options(stream_results=True)
.select_from(EbayItemDaily)
.where(and_(EbayItemDaily.item_id == id, EbayItemDaily.date == datetime.utcnow().date()))
.correlate_except(EbayItemDaily)
.scalar_subquery()
)
now i need to reload flask to update the yesterday_sale_quntity value.
how to set to close the cache for flask-sqlalchemy column_property value?
Related
I have a question reg. comparing big set of model instances with data in DB.
for example I have a model with 5 fields:
model Foo(models.Model)
fields1 = models.Integerfield()
fields2 = models.Integerfield()
fields3 = models.Integerfield()
fields4 = models.Integerfield()
fields5 = models.Integerfield()
class Meta:
unique_together = (‘field1', 'field2’,)
and I have 400.000 model Foo entries saved in DB.
Also I have 400.000 +/- few instances in memory(in python)of the same model generated from internet CSV file (without pk set).
Question is – what is the most efficient way to do following:
1)If instance in python equal to same instance in DB – keep instance in DB.
2)If instance in python not equal to instance in DB – update instance in DB.
If instance in python does not coincide to any instances in DB – write it to DB.
If no instances in python coincide to particular instance in DB – delete instance from DB.
Its should be bulk operations or RAW SQL as sample size is quite big.
Thank you.
PS. What i do now - delete all, reset counter, write all again to db
Hoping you use Django>=2.2. Try this script,
import csv, os
from your_app.models import Foo
def get_db_object_from_file_object(file_object):
return Foo(
field1=file_object[0],
field2=file_object[1],
field3=file_object[2],
field4=file_object[3],
field5=file_object[4],
)
def update_db_object_from_file_object(db_object, file_object):
db_object.field1 = file_object[0]
db_object.field2 = file_object[1]
db_object.field3 = file_object[2]
db_object.field4 = file_object[3]
db_object.field5 = file_object[4]
with open("<filename.csv>") as f:
reader = csv.reader(f)
eof = os.fstat(f.fileno()).st_size
file_objects = {(each[0], each[1]): each for each in reader}
db_objects = {
(each.field1, each.field2): each
for each in Foo.objects.filter(
field1__in=(each[0] for each in next_lines),
field2__in=(each[1] for each in next_lines)
)
}
to_create = []
to_update = []
for pair, obj in file_objects.items():
if pair not in db_objects:
to_create.append(obj)
else:
update_db_object_from_file_object(db_objects[pair], obj)
to_update.append(db_objects[pair])
del db_objects[pair]
Foo.objects.bulk_create(to_create)
Foo.objects.bulk_update(to_update, ["field1", "field2", "field3", "field4", "field5"])
Foo.objects.filter(id__in=[each.pk for each in db_objects.values()])
I want to have a custom key for the field id, for example, id_user, I've tried the following
class UserModel(db.model, UserMixin)
...
#property
def id(self):
return self.id_user
But couldn't make it work. When I try to login it sends me this message:
{
"message": "You don't have the permission to access the requested resource. It is either read-protected or not readable by the server."
}
I ended up with a nasty solution. I cloned the UserModel object, added a duplicated field for id, with the custom key I needed and told Flask-Security to use that object as the UserModel This is the function code I used:
def clone_model(model):
data = model
attr = getattr(model, "id_user")
setattr(data, "id", attr)
return data
cUserModel = clone_model(UserModel)
user_datastore = SQLAlchemyUserDatastore(db, cUserModel, Roles)
security = Security(app, user_datastore)
Hope someone find it useful
For anybody who is using Flask-Security-Too looking to change the default id column, here is my solution.
Define the User and Role Tables as usual and just define the id column depending on your preference.
class Role(db.Model, FsRoleMixin):
__tablename__ = 'role'
id = Column(String, primary_key=True)
class Users(db.Model, FsUserMixin):
__tablename__ = 'users'
id = Column(String, primary_key=True)
For me, I need to use String primary_key Column. For that to I needed to change the many-to-many relationship table (roles_users). Below is the code snippet for the same.
# Initialize the db object
db = SQLAlchemy()
# import this Class
from flask_security.models.fsqla_v2 import FsModels
# call this after creating the db object
FsModels.db = db
FsModels.user_table_name = 'users' # If you want a different table name than the default
FsModels.role_table_name = 'role'
# Create the relationship table as per your preferences
FsModels.roles_users = db.Table(
"roles_users",
Column("user_id", String, ForeignKey("users.id")),
Column("role_id", String, ForeignKey("role.id")),
)
In my application, a 'set' can have a number of 'products' associated with it. Products listed against a set must have quantities defined. For this many-to-many relationship I have followed the SQLAlchemy documentation to use an association table with an additional column (quantity).
I am trying to create a form where the user can assign products and quantities against a given set. Both the sets and products already exist in the database. The data from the form are:
set.id
product.id
quantity
This works to create a new association (e.g. set 1 is 'linked' to product 3 with quantity=XYZ) but I get an integrity error when I try to update an existing record.
I can manually add a relationship/record (dummy data) or within the Flask view function as follows:
s = Set.query.get(2)
p = Product.query.get(3)
a = Set_Product_Association(set=s, product=p, quantity=23)
db.session.add(a)
db.session.commit()
Updating the record (different quantity) manually as follows works:
s.products[0].quantity = 43
db.session.add(s)
db.session.commit()
However when I use the code from the first block instead (with the aim to update the quantity field for a given, existing set and product ID), i.e.:
a = Set_Product_Association(set=s, product=p, quantity=43)
I get an integrity error
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: set_product_association.set_id, set_product_association.product_id [SQL: 'INSERT INTO set_product_association (set_id, product_id, quantity) VALUES (?, ?, ?)'] [parameters: (2, 3, 43)]
I assume this is to tell me that I'm trying to append a new record rather than updating the existing one.
How should I approach this? The 'manual' method works but relies on working out the correct index in the list (i.e. for the correct product.id).
Curiously, if I use form.popluate_obj(set) in my Flask view function to process the form data as described in my question here, I can update fields but not create new 'associations'. Unfortunately, I don't know what goes on behind the scenes there....
My models are defined like so:
class Set_Product_Association(db.Model):
__tablename__ = 'set_product_association'
set_id = db.Column(db.Integer, db.ForeignKey('sets.id'), primary_key=True)
product_id = db.Column(db.Integer, db.ForeignKey('products.id'), primary_key=True)
quantity = db.Column(db.Integer)
product = db.relationship("Product", back_populates="sets")
set = db.relationship("Set", back_populates="products")
class Set(db.Model):
__tablename__ = 'sets'
id = db.Column(db.Integer, primary_key=True)
products = db.relationship("Set_Product_Association",
back_populates="set")
class Product(db.Model):
__tablename__= 'products'
id = db.Column(db.Integer, primary_key=True)
part_number = db.Column(db.String(100), unique=True, nullable=False)
sets = db.relationship("Set_Product_Association",
back_populates="product")
Edit:
I've also tried reversing the operation as suggested here:
s = Set.query.get(2)
a = Set_Product_Association()
a.quantity = 43
a.product = Product.query.get(3)
a.set = s
db.session.commit()
But I still get an error:
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: set_product_association.set_id, set_product_association.product_id [SQL: 'INSERT INTO set_product_association (set_id, product_id, quantity) VALUES (?, ?, ?)'] [parameters: (2, 3, 43)]
You get an integrity error because you are trying to create a new object with the same primary keys.
This:
a = Set_Product_Association(set=s, product=p, quantity=43)
Does not update, but create.
If you want to update the actual row in the table, you need to update the existing one:
assoc = Set_Product_Association.query.filter_by(set=s, product=p).first()
assoc.quantity = 43
db.session.commit()
Also, from the documentation it is advised to not use a model but an actual table.
I have this query that is trying to find a record given the same day and a status:
ld=LunchDay.query.filter(and_(func.DATE(LunchDay.timestamp == datetime.date.today()), LunchDay.status==1))
The model:
class LunchDay(db.Model):
__tablename__ = 'lunch_day'
id = db.Column(db.Integer, primary_key=True)
user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
user = relationship("User", foreign_keys=[user_id])
timestamp = db.Column(db.DateTime, index=True, default=datetime.today())
status = db.Column(db.Integer)
It runs and doesn't throw an error, yet it seems to not regard date. It will find entries with dates like yesterdays in the DateTimeField of the database:
2018-11-13 00:00:00.000000
Which is yesterdays date, but it is picking it up based just on the status almost just like it is looking at it as an OR. The imports I use are:
from sqlalchemy import func, and_
Print out the actual sql that is being generated by your query to see what is happening. E.g.:
ld=LunchDay.query.filter(and_(func.DATE(LunchDay.timestamp == datetime.date.today()), LunchDay.status==1))
print(ld)
Prints:
SELECT lunch_day.id AS lunch_day_id, lunch_day.timestamp AS lunch_day_timestamp, lunch_day.status AS lunch_day_status
FROM lunch_day
WHERE DATE(lunch_day.timestamp = %(timestamp_1)s) AND lunch_day.status = %(status_1)s
There you can see that the equality of lunch_day.timestamp and the param timestamp_1 is being passed to the DATE function.
Which is actually pretty easy to see in your sqlalchemy query: func.DATE(LunchDay.timestamp == datetime.date.today()). I assume you want to convert LunchDay.timestamp to a date and then compare it to datetime.date.today() which should be db.func.DATE(LunchDay.timestamp) == date.today().
print(LunchDay.query.filter(and_(func.DATE(LunchDay.timestamp) == datetime.date.today(), LunchDay.status == 1)))
prints:
SELECT lunch_day.id AS lunch_day_id, lunch_day.timestamp AS lunch_day_timestamp, lunch_day.status AS lunch_day_status
FROM lunch_day
WHERE DATE(lunch_day.timestamp) = %(DATE_1)s AND lunch_day.status = %(status_1)s
One other thing to note is that multiple statements passed to filter() are automatically treated as an and_(), so you can simplify your query a little by removing that:
LunchDay.query.filter(func.DATE(LunchDay.timestamp) == datetime.date.today(), LunchDay.status == 1)
I have a problem. I have to do this query:
#app.route('/api/subscriptions/<string:id>', methods=('DELETE',))
#decorators.login_required
def delete_subscription(id):
dbsession = DBSession()
session = Session()
favorit = (dbsession.query(StudentsFavorites)
.filter(Exams.number == str(id))
.filter(StudentsFavorites.exam_id)
.filter(Students.id == StudentsFavorites.student_id)
.filter(Students.id == str(session.get_user_id()))
.delete() )
dbsession.flush()
return jsonify(error=False)
But when I do this query I get this exception:
OperationalError: (OperationalError) no such column: exams.number u'DELETE FROM students_favorites WHERE exams.number = ? AND students_favorites.exam_id AND students.id = students_favorites.student_id AND students.id = ?' ('123123123', 'a24213')
The tables are very big and got lots of information, so i can't post all of it. But this query works:
#app.route('/api/subscriptions/<string:id>', methods=('PUT',))
#decorators.login_required
def add_subscription(id):
dbsession = DBSession()
session = Session()
examID = (dbsession.query(Exams.id)
.filter(Exams.number == id).first()
)
favorit=StudentsFavorites(student_id=session.get_user_id(), exam_id=examID.id)
dbsession.add(favorit)
dbsession.flush()
return jsonify(error=False)
Short view to the table:
table: Exams
rows: id, number (number is the id i put into the function)
table: StudentsFavorites
rows: student_id, exams_id
table: Students
rows: id
I really didn't understand, why he didn't find the number row in the exception.
EDIT:
Database StudentsFavorites:
class StudentsFavorites(Base):
"""N:M resolve model for the exams to the semester.
"""
__tablename__ = "students_favorites"
student_id = Column(Unicode(255), ForeignKey("students.id"), primary_key=True, autoincrement=False)
exam_id = Column(Integer, ForeignKey("exams.id"), primary_key=True, autoincrement=False)
created_at = Column(DateTime, nullable = False, default = datetime.now)
student = relationship("Students", uselist = False, lazy="joined")
exam = relationship("Exams", uselist=False, lazy="joined")
Something like this? I tried this:
(dbsession.query(StudentsFavorites)
.filter(StudentsFavorites.exam.id == str(id))
.filter(StudentsFavorites.student.id == str(session.get_user_id()))
.delete()
)
But got the error, that id didn't exist in exams / student
You have two cases of the same problem. Your query has information for StudentFavorites which means it knows about StudentFavorites.student_id and StudentFaovrites.exams_id. It doesn't know anything about Students.id, Exames.id and Exames.number. In order for you to query a StudentFavorites object and have it know about those other values you're going to have to perform a sql join.
Join's can be a bit of a pain in the ass to get working in sqlalchemy (well... in regular sql as well). Since I don't know what your table schema is I can't talk about that but the view should look something like this.
#app.route('/api/subscriptions/<string:id>', methods=('DELETE',))
#decorators.login_required
def delete_subscription(id):
dbsession = DBSession()
session = Session()
favorit = (dbsession.query(StudentsFavorites)
.join(Exames)
.join(students)
.filter(Exams.number == str(id))
.filter(StudentsFavorites.exam_id)
.filter(Students.id == StudentsFavorites.student_id)
.filter(Students.id == str(session.get_user_id()))
.delete() )
dbsession.flush()
return jsonify(error=False)
Alternatively, you can look into setting up Foreign key relationships in your table statements if you use the ORM to create your tables
The reason your second example works is because you're specifying a query over an exam table and only using values found in that table.
Response to Edit:
Right now your table relationships aren't set up correctly. Specifically the sections: Many To Many and Deleting Rows from the Many to Many Table
This example code is explained in much more (and better) detail in the posted link but the basic idea is that you have a associate_table (in your case StudentFavorites) contains foreign keys which have a relationship which is specified in one or more of your other tables. I personally advise that you go with the table example and not the object example.
association_table = Table('association', Base.metadata,
Column('left_id', Integer, ForeignKey('left.id')),
Column('right_id', Integer, ForeignKey('right.id'))
)
class Parent(Base):
__tablename__ = 'left'
id = Column(Integer, primary_key=True)
children = relationship("Child",
secondary=association_table,
backref="parents")
class Child(Base):
__tablename__ = 'right'
id = Column(Integer, primary_key=True)