I have to update 2 tables in database every time user visit a page.
this is working:
order_order = OrderTable.no_objects.get(order_number=order_id)
order_order.status = 'updated status'
order_order.save()
This is not working: (related with first table through foreign key)
order_item = ItemsTable.objects.filter(order_id__order_number=order_id)
for i in range(order_item.count()):
order_item[i].qty = i + 5
order_item[i].qty_nextMonth = i + 30
order_item[i].save()
Can anyone tell what's wrong in 2nd part of code. It's not updating the database.
Each time you write order_item[i], you make a separate fetch, and you return an ItemsTable item from the database. That means that if you set the .qty = ... attribute of that object, it is simply ignored, since the next order_item[i] will trigger a new fetch. Your order_item[i].save() updates the record in the database, but with the values that have been retrieved just from a fetch from the database.
It is better to simply iterate over the queryset, and thus maintain a reference to the same ItemsTable object:
order_items = ItemsTable.objects.filter(order_id__order_number=order_id)
for i, order_item in enumerate(order_items):
order_item.qty = i + 5
order_item.qty_nextMonth = i + 30
order_item.save()
This is more efficient as well, since enumerating over a queryset forces evaluation, and hence you fetch all objects at once.
As of django-2.2, you can use .bulk_update(..) [Django-doc], to update in bulk an iterable of objects.
Related
I get a queryset object every time i want some data from models.
So when i say,
"items = Items.object.get(value=value)"
I get --
"<QuerySet [<Item-name>]>"
I have to iterate through the queryset object to get the data and I do that with
"items[0]"
Is there any way I can avoid this?
Edit: I meant "items = Items.object.filter(value=value)"
first of all items = Items.objects.get(value=value) does not return a queryset,
rather it returns an object of <Items: Items object (1)>
To get the first(or just one result) or last date from the object, do this Items.objects.first() or Items.objects.last()
To get the desired data without using its index position, then you can filter it like this Items.objects.filter(value=value)
You are mistaken. items = Items.object.get(value=value) will not give you a queryset, but an object. items = Items.object.filter(value=value)
would give you a queryset.
Filter method will always give you a queryset, because; in order to minimize the need of database hits, django considers you might add additional filters through your code. So if you not execute that queryset, e.g. by using list(your_queryset) django never hits the database.
# when you are using 'get' in your query, you don't need to iterate, directly get an access to the field values
try:
items = Items.object.get(value=value)
except Items.DoesNotExist:
items = None
if items:
print(items.value)
My question is I have this code.
reset_numbers = Picture.objects.filter(author__id=author.id)
for index, pic in enumerate(reset_numbers):
pic.number = index
pic.save()
I want to reset the ording of Pictures.
How I can do this in one query or less queries to get rid of the for loop?
Is there some way to do this?
pks = Picture.objects.filter(author__id=author.id).values_list('pk', flat=True)
for index, pk in enumrate(pks):
Picture.objects.filter(pk=pk).update(number=index)
This loads only primary keys, and only update number field. But query count is same as the original code.
views.py:
q3=KEBReading.objects.filter(datetime_reading__month=a).filter(datetime_reading__year=selected_year).values("signed")
for item in q3:
item["signed"]="signed"
print item["signed"]
q3.save()
How do I save a field into the database? I'm trying to save the field called "signed" with a value. If I do q3.save() it gives a error as it is a queryset. I'm doing a query from the database and then, based on the result, want to set a value to a field and save it.
prevdate=KEBReading.objects.filter(datetime_reading__lt=date)
i am getting all the rows from the database less than the current date. but i want only the latest record. if im entering 2012-06-03. wen i query i want the date less than this date i.e the date just previous to this. can sumbody help?
q3 = KEBReading.objects.filter(datetime_reading__month=a,
datetime_reading__year=selected_year)
for item in q3:
item.signed = True
item.save()
q3=KEBReading.objects.filter(...)
will return you a list of objects. Any instance of a Django Model is an object and all fields of the instance are attributes of that object. That means, you must use them using dot (.) notation.
like:
item.signed = "signed"
If your object is a dictionary or a class derived from dictionary, then you can use named-index like:
item["signed"] = "signed"
and in your situation, that usage is invalid (because your object's type is not dictionary based)
You can either call update query:
KEBReading.objects.filter(...).update(selected="selected")
or set new value in a loop and then save it
for item in q3:
item.signed="signed"
q3.save()
but in your situation, update query is a better approach since it executes less database calls.
Try using update query:
If signed is a booleanfield:
q3 = KEBReading.objects.filter(datetime_reading__month = a).filter(datetime_reading__year = selected_year).update(signed = True)
If it is a charfield:
q3 = KEBReading.objects.filter(datetime_reading__month = a).filter(datetime_reading__year = selected_year).update(signed = "True")
Update for comments:
If you want to fetch records based datetime_reading month, you can do it by providing month as number. For example, 2 for February:
q3 = KEBReading.objects.filter(datetime_reading__month = 2).order_by('datetime_reading')
And if you to fetch records with signed = True, you can do it by:
q3 = KEBReading.objects.filter(signed = True)
If you want to fetch only records of previous date by giving a date, you can use:
prevdate = KEBReading.objects.filter(datetime_reading = (date - datetime.timedelta(days = 1)))
I'd like to update a table with Django - something like this in raw SQL:
update tbl_name set name = 'foo' where name = 'bar'
My first result is something like this - but that's nasty, isn't it?
list = ModelClass.objects.filter(name = 'bar')
for obj in list:
obj.name = 'foo'
obj.save()
Is there a more elegant way?
Update:
Django 2.2 version now has a bulk_update.
Old answer:
Refer to the following django documentation section
Updating multiple objects at once
In short you should be able to use:
ModelClass.objects.filter(name='bar').update(name="foo")
You can also use F objects to do things like incrementing rows:
from django.db.models import F
Entry.objects.all().update(n_pingbacks=F('n_pingbacks') + 1)
See the documentation.
However, note that:
This won't use ModelClass.save method (so if you have some logic inside it won't be triggered).
No django signals will be emitted.
You can't perform an .update() on a sliced QuerySet, it must be on an original QuerySet so you'll need to lean on the .filter() and .exclude() methods.
Consider using django-bulk-update found here on GitHub.
Install: pip install django-bulk-update
Implement: (code taken directly from projects ReadMe file)
from bulk_update.helper import bulk_update
random_names = ['Walter', 'The Dude', 'Donny', 'Jesus']
people = Person.objects.all()
for person in people:
r = random.randrange(4)
person.name = random_names[r]
bulk_update(people) # updates all columns using the default db
Update: As Marc points out in the comments this is not suitable for updating thousands of rows at once. Though it is suitable for smaller batches 10's to 100's. The size of the batch that is right for you depends on your CPU and query complexity. This tool is more like a wheel barrow than a dump truck.
Django 2.2 version now has a bulk_update method (release notes).
https://docs.djangoproject.com/en/stable/ref/models/querysets/#bulk-update
Example:
# get a pk: record dictionary of existing records
updates = YourModel.objects.filter(...).in_bulk()
....
# do something with the updates dict
....
if hasattr(YourModel.objects, 'bulk_update') and updates:
# Use the new method
YourModel.objects.bulk_update(updates.values(), [list the fields to update], batch_size=100)
else:
# The old & slow way
with transaction.atomic():
for obj in updates.values():
obj.save(update_fields=[list the fields to update])
If you want to set the same value on a collection of rows, you can use the update() method combined with any query term to update all rows in one query:
some_list = ModelClass.objects.filter(some condition).values('id')
ModelClass.objects.filter(pk__in=some_list).update(foo=bar)
If you want to update a collection of rows with different values depending on some condition, you can in best case batch the updates according to values. Let's say you have 1000 rows where you want to set a column to one of X values, then you could prepare the batches beforehand and then only run X update-queries (each essentially having the form of the first example above) + the initial SELECT-query.
If every row requires a unique value there is no way to avoid one query per update. Perhaps look into other architectures like CQRS/Event sourcing if you need performance in this latter case.
Here is a useful content which i found in internet regarding the above question
https://www.sankalpjonna.com/learn-django/running-a-bulk-update-with-django
The inefficient way
model_qs= ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
obj.name = 'foo'
obj.save()
The efficient way
ModelClass.objects.filter(name = 'bar').update(name="foo") # for single value 'foo' or add loop
Using bulk_update
update_list = []
model_qs= ModelClass.objects.filter(name = 'bar')
for model_obj in model_qs:
model_obj.name = "foo" # Or what ever the value is for simplicty im providing foo only
update_list.append(model_obj)
ModelClass.objects.bulk_update(update_list,['name'])
Using an atomic transaction
from django.db import transaction
with transaction.atomic():
model_qs = ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
ModelClass.objects.filter(name = 'bar').update(name="foo")
Any Up Votes ? Thanks in advance : Thank you for keep an attention ;)
To update with same value we can simply use this
ModelClass.objects.filter(name = 'bar').update(name='foo')
To update with different values
ob_list = ModelClass.objects.filter(name = 'bar')
obj_to_be_update = []
for obj in obj_list:
obj.name = "Dear "+obj.name
obj_to_be_update.append(obj)
ModelClass.objects.bulk_update(obj_to_be_update, ['name'], batch_size=1000)
It won't trigger save signal every time instead we keep all the objects to be updated on the list and trigger update signal at once.
IT returns number of objects are updated in table.
update_counts = ModelClass.objects.filter(name='bar').update(name="foo")
You can refer this link to get more information on bulk update and create.
Bulk update and Create
I am using bulk_create to loads thousands or rows into a postgresql DB. Unfortunately some of the rows are causing IntegrityError and stoping the bulk_create process. I was wondering if there was a way to tell django to ignore such rows and save as much of the batch as possible?
This is now possible on Django 2.2
Django 2.2 adds a new ignore_conflicts option to the bulk_create method, from the documentation:
On databases that support it (all except PostgreSQL < 9.5 and Oracle), setting the ignore_conflicts parameter to True tells the database to ignore failure to insert any rows that fail constraints such as duplicate unique values. Enabling this parameter disables setting the primary key on each model instance (if the database normally supports it).
Example:
Entry.objects.bulk_create([
Entry(headline='This is a test'),
Entry(headline='This is only a test'),
], ignore_conflicts=True)
One quick-and-dirty workaround for this that doesn't involve manual SQL and temporary tables is to just attempt to bulk insert the data. If it fails, revert to serial insertion.
objs = [(Event), (Event), (Event)...]
try:
Event.objects.bulk_create(objs)
except IntegrityError:
for obj in objs:
try:
obj.save()
except IntegrityError:
continue
If you have lots and lots of errors this may not be so efficient (you'll spend more time serially inserting than doing so in bulk), but I'm working through a high-cardinality dataset with few duplicates so this solves most of my problems.
(Note: I don't use Django, so there may be more suitable framework-specific answers)
It is not possible for Django to do this by simply ignoring INSERT failures because PostgreSQL aborts the whole transaction on the first error.
Django would need one of these approaches:
INSERT each row in a separate transaction and ignore errors (very slow);
Create a SAVEPOINT before each insert (can have scaling problems);
Use a procedure or query to insert only if the row doesn't already exist (complicated and slow); or
Bulk-insert or (better) COPY the data into a TEMPORARY table, then merge that into the main table server-side.
The upsert-like approach (3) seems like a good idea, but upsert and insert-if-not-exists are surprisingly complicated.
Personally, I'd take (4): I'd bulk-insert into a new separate table, probably UNLOGGED or TEMPORARY, then I'd run some manual SQL to:
LOCK TABLE realtable IN EXCLUSIVE MODE;
INSERT INTO realtable
SELECT * FROM temptable WHERE NOT EXISTS (
SELECT 1 FROM realtable WHERE temptable.id = realtable.id
);
The LOCK TABLE ... IN EXCLUSIVE MODE prevents a concurrent insert that creates a row from causing a conflict with an insert done by the above statement and failing. It does not prevent concurrent SELECTs, only SELECT ... FOR UPDATE, INSERT,UPDATE and DELETE, so reads from the table carry on as normal.
If you can't afford to block concurrent writes for too long you could instead use a writable CTE to copy ranges of rows from temptable into realtable, retrying each block if it failed.
Or 5. Divide and conquer
I didn't test or benchmark this thoroughly, but it performs pretty well for me. YMMV, depending in particular on how many errors you expect to get in a bulk operation.
def psql_copy(records):
count = len(records)
if count < 1:
return True
try:
pg.copy_bin_values(records)
return True
except IntegrityError:
if count == 1:
# found culprit!
msg = "Integrity error copying record:\n%r"
logger.error(msg % records[0], exc_info=True)
return False
finally:
connection.commit()
# There was an integrity error but we had more than one record.
# Divide and conquer.
mid = count / 2
return psql_copy(records[:mid]) and psql_copy(records[mid:])
# or just return False
Even in Django 1.11 there is no way to do this. I found a better option than using Raw SQL. It using djnago-query-builder. It has an upsert method
from querybuilder.query import Query
q = Query().from_table(YourModel)
# replace with your real objects
rows = [YourModel() for i in range(10)]
q.upsert(rows, ['unique_fld1', 'unique_fld2'], ['fld1_to_update', 'fld2_to_update'])
Note: The library only support postgreSQL
Here is a gist that I use for bulk insert that supports ignoring IntegrityErrors and returns the records inserted.
Late answer for pre Django 2.2 projects :
I ran into this situation recently and I found my way out with a seconder list array for check the uniqueness.
In my case, the model has that unique together check, and bulk create is throwing Integrity Error exception because of the array of bulk create has duplicate data in it.
So I decided to create checklist besides bulk create objects list. Here is the sample code; The unique keys are owner and brand, and in this example owner is an user object instance and brand is a string instance:
create_list = []
create_list_check = []
for brand in brands:
if (owner.id, brand) not in create_list_check:
create_list_check.append((owner.id, brand))
create_list.append(ProductBrand(owner=owner, name=brand))
if create_list:
ProductBrand.objects.bulk_create(create_list)
it's work for me
i am use this this funtion in thread.
my csv file contains 120907 no of rows.
def products_create():
full_path = os.path.join(settings.MEDIA_ROOT,'productcsv')
filename = os.listdir(full_path)[0]
logger.debug(filename)
logger.debug(len(Product.objects.all()))
if len(Product.objects.all()) > 0:
logger.debug("Products Data Erasing")
Product.objects.all().delete()
logger.debug("Products Erasing Done")
csvfile = os.path.join(full_path,filename)
csv_df = pd.read_csv(csvfile,sep=',')
csv_df['HSN Code'] = csv_df['HSN Code'].fillna(0)
row_iter = csv_df.iterrows()
logger.debug(row_iter)
logger.debug("New Products Creating")
for index, row in row_iter:
Product.objects.create(part_number = row[0],
part_description = row[1],
mrp = row[2],
hsn_code = row[3],
gst = row[4],
)
# products_list = [
# Product(
# part_number = row[0] ,
# part_description = row[1],
# mrp = row[2],
# hsn_code = row[3],
# gst = row[4],
# )
# for index, row in row_iter
# ]
# logger.debug(products_list)
# Product.objects.bulk_create(products_list)
logger.debug("Products uploading done")```