Compare set of model instances with their conterparts in DB | Django - django

Question

I have a question reg. comparing big set of model instances with data in DB.
for example I have a model with 5 fields:
model Foo(models.Model)
fields1 = models.Integerfield()
fields2 = models.Integerfield()
fields3 = models.Integerfield()
fields4 = models.Integerfield()
fields5 = models.Integerfield()
class Meta:
unique_together = (‘field1', 'field2’,)
and I have 400.000 model Foo entries saved in DB.
Also I have 400.000 +/- few instances in memory(in python)of the same model generated from internet CSV file (without pk set).
Question is – what is the most efficient way to do following:
1)If instance in python equal to same instance in DB – keep instance in DB.
2)If instance in python not equal to instance in DB – update instance in DB.
If instance in python does not coincide to any instances in DB – write it to DB.
If no instances in python coincide to particular instance in DB – delete instance from DB.
Its should be bulk operations or RAW SQL as sample size is quite big.
Thank you.
PS. What i do now - delete all, reset counter, write all again to db

Answer 1

Hoping you use Django>=2.2. Try this script,
import csv, os
from your_app.models import Foo
def get_db_object_from_file_object(file_object):
return Foo(
field1=file_object[0],
field2=file_object[1],
field3=file_object[2],
field4=file_object[3],
field5=file_object[4],
)
def update_db_object_from_file_object(db_object, file_object):
db_object.field1 = file_object[0]
db_object.field2 = file_object[1]
db_object.field3 = file_object[2]
db_object.field4 = file_object[3]
db_object.field5 = file_object[4]
with open("<filename.csv>") as f:
reader = csv.reader(f)
eof = os.fstat(f.fileno()).st_size
file_objects = {(each[0], each[1]): each for each in reader}
db_objects = {
(each.field1, each.field2): each
for each in Foo.objects.filter(
field1__in=(each[0] for each in next_lines),
field2__in=(each[1] for each in next_lines)
)
}
to_create = []
to_update = []
for pair, obj in file_objects.items():
if pair not in db_objects:
to_create.append(obj)
else:
update_db_object_from_file_object(db_objects[pair], obj)
to_update.append(db_objects[pair])
del db_objects[pair]
Foo.objects.bulk_create(to_create)
Foo.objects.bulk_update(to_update, ["field1", "field2", "field3", "field4", "field5"])
Foo.objects.filter(id__in=[each.pk for each in db_objects.values()])