I need to bulk_create the historical records for each object deleted in a queryset. I've coded it correct I think as follows.
def bulk_delete_with_history(objects, model, batch_size=None, default_user=None, default_change_reason="", default_date=None):
"""
The package `simple_history` logs the deletion one object at a time.
This does it in bulk.
"""
model_manager = model._default_manager
model_manager.filter(pk__in=[obj.pk for obj in objects]).delete()
history_manager = get_history_manager_for_model(model)
history_type = "-"
historical_instances = []
for instance in objects:
history_user = getattr(
instance,
"_history_user",
default_user or history_manager.model.get_default_history_user(
instance),
)
row = history_manager.model(
history_date=getattr(
instance, "_history_date", default_date or timezone.now()
),
history_user=history_user,
history_change_reason=get_change_reason_from_object(
instance) or default_change_reason,
history_type=history_type,
**{
field.attname: getattr(instance, field.attname)
for field in instance._meta.fields
if field.name not in history_manager.model._history_excluded_fields
}
)
if hasattr(history_manager.model, "history_relation"):
row.history_relation_id = instance.pk
historical_instances.append(row)
return history_manager.bulk_create(
historical_instances, batch_size=batch_size
)
The problem though is I need to disconnect the post_delete signal so that a historical record isn't created by simple history before i do it all at once.
I've tried this but it doesn't work.
models.signals.post_delete.disconnect(HistoricalRecords.post_delete, sender=Customer)
Where Customer is just a class I'm using to test this utility function.
Can anybody advise? Thanks in advance.
Asked the question on their github page also - https://github.com/jazzband/django-simple-history/issues/717
I made a classic mistake with disconnecting signals. One must of course disconnect with the object that was connected. This is discussed here in more detail - django signal disconnect not working
This now works because I am disconnecting with reference to the correct object.
receiver_object = models.signals.post_delete._live_receivers(Customer)[0]
models.signals.post_delete.disconnect(receiver_object, sender=Customer)
Still, I think it would be nice if django-simple-history provided a bulk_delete utility function like they provide one for bulk_create and bulk_update.
Related
Can't seem to find much info about this. This is NOT happening in a django test. I'm using DATABASES = { ATOMIC_REQUESTS: True }. Within a method (in mixin I created) called by the view, I'm trying to perform something like this:
def process_valid(self, view):
old_id = view.object.id
view.object.id = None # need a new instance in db
view.object.save()
old_fac = Entfac.objects.get(id=old_id)
new_fac = view.object
old_dets = Detfac.objects.filter(fk_ent__id__exact = old_fac.id)
new_formset = view.DetFormsetClass(view.request.POST, instance=view.object, save_as_new=True)
if new_formset.is_valid():
new_dets = new_formset.save()
new_fac.fk_cancel = old_fac # need a fk reference to initial fac in new one
old_fac.fk_cancel = new_fac # need a fk reference to new in old fac
# any save() action after this crashes with TransactionManagementError
new_fac.save()
I do not understand this error. I already created & saved a new object in db (when I set the object.id to None & saved that). Why would creating other objects create an issue for further saves?
I have tried not instantiating the new_dets objects with the Formset, but instead explicitely defining them:
new_det = Detfac(...)
new_det.save()
But then again, any further save after that raises the error.
Further details:
Essentially, I have an Entfac model, and a Detfac model that has a foreignkey to Entfac. I need to instantiate a new Enfac (distinct in db), as well as corresponding new Detfac for the new Entfac. Then I need to change some values in some of the fields for both new & old objects, and save all that to db.
Ah. The code above is fine.
But turns out, signals can be bad. I had forgotten that upon saving Detfac, there is a signal that goes to another class and that depending on the circumstances, adds a record to another table (sort of an history table).
Since that signal is just a single operation. Something like that:
#receiver(post_save, sender=Detfac)
def quantity_adjust_detfac(sender, **kwargs):
try:
detfac_qty = kwargs["instance"].qte
product = kwargs["instance"].fk_produit
if kwargs["created"]:
initial = {# bunch of values}
adjustment = HistoQuantity(**initial)
adjustment.save()
else:
except TypeError as ex:
logger.error(f"....")
except AttributeError as ex:
logger.error(f"....")
In itself, the fact that THIS wasn't marked as atomic isn't problematic. BUT if one of those exception throws, THEN I get the transactionmanagementerror. I am still not 100% sure why, tough the django docs do mention that when wrapping a whole view in atomic (or any chunk of code for that matter), then try/except within that block can yield unexpected result, because DJango does rely on exception to decide whether or not to commit the transaction as a whole. And the data I was testing with actually threw the exception (type error when creating the HistoQuantity object).
Wrapping the try/exception with a transaction.atomic manager worked however. Guessing that this... removed/handled the throw, thus the outer atomic could work.
I have an object Option which is like a type, it only has a property: name (it could have more properties in the future).
Instances of Option have a unique name.
Many objects may have zero or more of Option instances.
For example:
class Consumer:
options = models.ManyToManyField(Option, blank=True, help_text="the options")
But then, in order to create the Option instances for Consumer's many-to-many options relationship, I need to create a new Option instance and add it to options.
This however "breaks" my uniqueness: Now I have two with the same name! And so forth for every instance of Option I create.for Many-to-Many links. Instead of the 4 I need, I have now 68 in my DB...
I believe I have fundamentally misunderstood Many-To-Many, and / or mis-designed this relationship...
Can anybody help?
EDIT: Here's how I set options in an example:
def enable_option(request, pk=0, option_pk=0, *args, **kwargs):
consumer = get_object_or_404(Consumer, pk=pk)
option = get_object_or_404(Option, pk=option_pk)
new_option = Option()
new_option.name = option.name // I know I am breaking my own rule...but when I read the consumer options, I need the exact same name! Still, I believe I am modeling wrong
new_option.save()
consumer.options.add(new_option)
consumer.save()
return HttpResponse()
I don't really understand why you create a new Option here. You get the existing one; you can just add that to the relationship:
consumer = get_object_or_404(Consumer, pk=pk)
option = get_object_or_404(Option, pk=option_pk)
consumer.options.add(option)
You don't even need to call save, as modifying an m2m does not change the instance itself.
I've been using the get_or_create method with MongoEngine in a Django app. Today, I noticed there were a few duplicate entries. I came across this in the MongoEngine API Reference for get_or_create:
This requires two separate operations and therefore a race condition exists. Because there are no transactions in mongoDB other approaches should be investigated, to ensure you don’t accidentally duplicate data when using this method. This is now scheduled to be removed before 1.0
Should I be using something like this?:
from models import Post
post = Post(name='hello')
try:
Posts.objects.get(name=post.name)
print "exists"
except:
post.save()
print "saved"
Will that solve my problem?
Is there a better way?
To perform an upsert use the following syntax:
Posts.objects(name="hello").update(set__X=Y, upsert=True)
That will add a post with the name "hello" and where X = Y if it doesn't already exist, otherwise it will update an existing post just setting X = Y.
If you need pass a dictionery, can do this:
post = Post.objects(name="hello").first() or Post(name="hello")
then you can update with something like this:
# data = dictionary_with_data
for field, value in data.items():
post[field] = value
post.save()
I'm trying to render a form with a combo that shows related entities. Therefore I'm using a ModelChoiceField.
This approach works well, until I needed to limit which entities to show. If I use a simple query expression it also works well, but things break if I use a raw SQL query.
So my code that works, sets the queryset to a filter expression.
class ReservationForm(forms.Form):
location_time_slot = ModelChoiceField(queryset=LocationTimeSlot.objects.all(), empty_label="Select your prefered time")
def __init__(self,*args,**kwargs):
city_id = kwargs.pop("city_id") # client is the parameter passed from views.py
super(ReservationForm, self).__init__(*args,**kwargs)
# TODO: move this to a manager
self.fields['location_time_slot'].queryset = LocationTimeSlot.objects.filter(city__id = city_id )
BUT, if I change that to a raw query I start having problems. Code that does not work:
class ReservationForm(forms.Form):
location_time_slot = ModelChoiceField(queryset=LocationTimeSlot.objects.all(), empty_label="Select your prefered time")
def __init__(self,*args,**kwargs):
city_id = kwargs.pop("city_id") # client is the parameter passed from views.py
super(ReservationForm, self).__init__(*args,**kwargs)
# TODO: move this to a manager
query = """SELECT ts.id, ts.datetime_to, ts.datetime_from, ts.available_reserves, l.name, l.'order'
FROM reservations_locationtimeslot AS ts
INNER JOIN reservations_location AS l ON l.id = ts.location_id
WHERE l.city_id = %s
AND ts.available_reserves > 0
AND ts.datetime_from > datetime() """
time_slots = LocationTimeSlot.objects.raw(query, [city_id])
self.fields['location_time_slot'].queryset = time_slots
The first error I get when trying to render the widget is: 'RawQuerySet' object has no attribute 'all'
I could solve that one thanks to one of the commets in enter link description here, by doing:
time_slots.all = time_slots.__iter__ # Dummy fix to allow default form rendering with raw SQL
But now I'm getting something similar when posting the form:
'RawQuerySet' object has no attribute 'get'
Is there a proper way to prepare a RawQuerySet to be used by ModelChoiceField?
Thanks!
Are you sure you actually need a raw query there? Just looking at that query, I can't see any reason you can't just do it with filter(location__city=city_id, available_reserves__gte=0, datetime_from__gt=datetime.datetime.now()).
Raw query sets are missing a number of methods that are defined on conventional query sets, so just dropping them in place isn't likely to work without writing your own definitions for all those methods.
I temporarily fixed the problem adding the missing methods.
The way I'm currently using the ModelChoiceField I only needed to add the all() and get() methods, but in different scenarios you might need to add some other methods as well. Also this is not a perfect solution because:
1) Defining the get method this way migth produce incorrect results. I think the get() method is used to validate that the selected option is within the options returned by all(). The way I temporarily implemented it only validates that the id exists in the table.
2) I guess the get method is less performant specified this way.
If anyone can think of a better solution, please let me know.
So my temporary solution:
class LocationTimeSlotManager(models.Manager):
def availableSlots(self, city_id):
query = """SELECT ts.id, ts.datetime_to, ts.datetime_from, ts.available_reserves, l.name, l.'order'
FROM reservations_locationtimeslot AS ts
.....
.....
MORE SQL """
time_slots = LocationTimeSlot.objects.raw(query, [city_id])
# Dummy fix to allow default form rendering with raw SQL
time_slots.all = time_slots.__iter__
time_slots.get = LocationTimeSlot.objects.get
return time_slots
I have a lot of objects to save in database, and so I want to create Model instances with that.
With django, I can create all the models instances, with MyModel(data), and then I want to save them all.
Currently, I have something like that:
for item in items:
object = MyModel(name=item.name)
object.save()
I'm wondering if I can save a list of objects directly, eg:
objects = []
for item in items:
objects.append(MyModel(name=item.name))
objects.save_all()
How to save all the objects in one transaction?
as of the django development, there exists bulk_create as an object manager method which takes as input an array of objects created using the class constructor. check out django docs
Use bulk_create() method. It's standard in Django now.
Example:
Entry.objects.bulk_create([
Entry(headline="Django 1.0 Released"),
Entry(headline="Django 1.1 Announced"),
Entry(headline="Breaking: Django is awesome")
])
worked for me to use manual transaction handling for the loop(postgres 9.1):
from django.db import transaction
with transaction.atomic():
for item in items:
MyModel.objects.create(name=item.name)
in fact it's not the same, as 'native' database bulk insert, but it allows you to avoid/descrease transport/orms operations/sql query analyse costs
name = request.data.get('name')
period = request.data.get('period')
email = request.data.get('email')
prefix = request.data.get('prefix')
bulk_number = int(request.data.get('bulk_number'))
bulk_list = list()
for _ in range(bulk_number):
code = code_prefix + uuid.uuid4().hex.upper()
bulk_list.append(
DjangoModel(name=name, code=code, period=period, user=email))
bulk_msj = DjangoModel.objects.bulk_create(bulk_list)
Here is how to bulk-create entities from column-separated file, leaving aside all unquoting and un-escaping routines:
SomeModel(Model):
#classmethod
def from_file(model, file_obj, headers, delimiter):
model.objects.bulk_create([
model(**dict(zip(headers, line.split(delimiter))))
for line in file_obj],
batch_size=None)
Using create will cause one query per new item. If you want to reduce the number of INSERT queries, you'll need to use something else.
I've had some success using the Bulk Insert snippet, even though the snippet is quite old.
Perhaps there are some changes required to get it working again.
http://djangosnippets.org/snippets/446/
Check out this blog post on the bulkops module.
On my django 1.3 app, I have experienced significant speedup.
bulk_create() method is one of the ways to insert multiple records in the database table. How the bulk_create()
**
Event.objects.bulk_create([
Event(event_name="Event WF -001",event_type = "sensor_value"),
Entry(event_name="Event WT -002", event_type = "geozone"),
Entry(event_name="Event WD -001", event_type = "outage") ])
**
for a single line implementation, you can use a lambda expression in a map
map(lambda x:MyModel.objects.get_or_create(name=x), items)
Here, lambda matches each item in items list to x and create a Database record if necessary.
Lambda Documentation
The easiest way is to use the create Manager method, which creates and saves the object in a single step.
for item in items:
MyModel.objects.create(name=item.name)