I'm scripting a migration tool to get my data from an old software to a new version with different data structures.
While looping through the csv-files I'm creating the objects.
The problem is that every object gets created in postgres but django doesn't.
I'm getting errors like:
duplicate key value violates unique constraint "api_order_pkey"
DETAIL: Key (id)=(159865) already exists.
Order.objects.get(pk=159865) does return "Does not exist"
But a SELECT * FROM api_order WHERE pk = 159865 directly in Postgres finds a row.
with open("order.csv","r") as f:
reader = csv.reader(f)
for row in reader:
Order.objects.create(pk = row[0], ordernr= row[1],....)`
thats of course a short version of it.
On the import of about 250.000 rows this problem appears with about 100 rows.
I had the same problem with deleting. I ran Order.objects.all().delete() but in Postgres there where still a few rows left while Order.objects.all() returned an empty Queryset.
Is there something I can do about this?
Has anyone had a similar problem?
Sorry, my fault.
I had a Manager running which was modifying the queryset.
Thats why Django did not show all records.
Related
I´m trying to update the data of an existing model with a csv. I read the file and assign the values with no problem.
If I try `MyModel.update() everything runs with no error but the data is not saved.
with open('Productosold.csv') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
p = ProductosBase.objects.filter(codigo_barra = row['codigo_barra'])
p.region=row['region']
p.producto_ing=row['producto_ing']
p.packaging_ing=row['packaging_ing']
p.precio_compra=row['precio_compra']
p.uom=row['uom']
p.units_inner=row['units_inner']
p.inner_master=row['inner_master']
p.tier=row['tier']
p.precio_lista_internacional=row['precio_lista_internacional']
p.update()
I usualy upload new data using the MyModel.save() method and have no problem.
Now, if I use that I get "Queryset has no attribute save".
p.save()
If I print some of the p.values I can see they are populated correctly from the csv file.
What I´m doing wrong?
Thanks in advance!
.filter always returns a queryset, not a single instance. When you set all those values, you are just setting (previously non-existing) attributes onto that queryset object; you are not setting fields in a model instance. You should use .get to get an instance and save that.
p = ProductosBase.objects.get(codigo_barra = row['codigo_barra'])
p.region=row['region']
...
p.save()
However, since all the columns in your CSV map precisely to fields on the model, you could in fact use filter and update to do the whole thing in one go:
for row in reader:
ProductosBase.objects.filter(codigo_barra=row['codigo_barra']).update(**row)
and no need for any of the rest of the code.
You need filter() whenever you expect more than just one object that matches your criteria. If no item was found matching your criteria, filter() returns am empty queryset without throwing an error.
Also you can use get() but when you expect one (and only one) item that matches your criteria. Get throws an error if the item does not exist or if multiple items exist that match your criteria. You should therefore always use if in a try.. except .. block or with a shortcut function like get_object_or_404 in order to handle the exceptions properly. I'd recommend using get_object_or_404 in this case.
p = get_object_or_404(ProductosBase, codigo_barra=row['codigo_barra'])
How to get the value of the counter of an AutoField, such as the usual id field of most models?
At the moment, I do:
MyModel.objects.latest('id').id
But that does not work when all the objects have been deleted from the database.
Of course, a database-agnostic answer would be best.
EDIT
The accepted answer in Model next available primary key is not very relevant to my question, as I do not intend to use the counter value to create a new object. Also I don't mind if the value I get is not super accurate.
Background.
AFAIK there isn't a database agnostic query. Different databases handle auto increment differently and there rarely is a use case for django to find out what the next possible auto increment ID is.
To elaborate further, in postgresql you could do select nextval('my_sequence') while in mysql you would need to use the last_insert_id() but what this returns is the ID for the last insert and not the next one these two may actually be very different! To get the actual value you would need to use 'SHOW TABLE STATUS'
Solution.
Create a record, save it, inspect it's ID and delete it.
This will change the next id but you have indicated that you need only an approximation.
The alternative is to do a manual transaction with a rollback. This too would alter the next id in case of mysql.
from django.db import transaction
#transaction.atomic
def find_next_val(mymodel):
try:
# ...
obj = mymoel.objects.create(....)
print obj.id
raise IntegrityError
except IntegrityError:
pass
I'm reading django doc and see how django knows to do the update or insert method when calling save(). The doc says:
If the object’s primary key attribute is set to a value that evaluates to True (i.e. a value other than None or the empty string), Django executes an UPDATE.
If the object’s primary key attribute is not set or if the UPDATE didn’t update anything, Django executes an INSERT link.
But in practice, when I create a new instance of a Model and set its "id" property to a value that already exist in my database records. For example: I have a Model class named "User" and have a propery named "name".Just like below:
class User(model.Model):
name=model.CharField(max_length=100)
Then I create a new User and save it:
user = User(name="xxx")
user.save()
now in my database table, a record like id=1, name="xxx" exists.
Then I create a new User and just set the propery id=1:
newuser = User(id=1）
newuser.save()
not like the doc says.when I had this down.I checked out two records in my database table.One is id = 1 ,another is id=2
So, can anyone explain this to me? I'm confused.Thanks!
Because in newer version of django ( 1.5 > ), django does not check whether the id is in the database or not. So this could depend on the database. If the database report that this is duplicate, then it will update and if the database does not report it then it will insert. Check the doc -
In Django 1.5 and earlier, Django did a SELECT when the primary key
attribute was set. If the SELECT found a row, then Django did an
UPDATE, otherwise it did an INSERT. The old algorithm results in one
more query in the UPDATE case. There are some rare cases where the
database doesn’t report that a row was updated even if the database
contains a row for the object’s primary key value. An example is the
PostgreSQL ON UPDATE trigger which returns NULL. In such cases it is
possible to revert to the old algorithm by setting the select_on_save
option to True.
https://docs.djangoproject.com/en/1.8/ref/models/instances/#how-django-knows-to-update-vs-insert
But if you want this behavior, set select_on_save option to True.
You might wanna try force_update if that is required -
https://docs.djangoproject.com/en/1.8/ref/models/instances/#forcing-an-insert-or-update
I have a 2 models with a foreign/Primary key to same model.
model Foo:
FK(Too, pk)
model Coo:
FK(Too, pk)
model Too:
blah = charfield()
In the views I am seeing some very strange behavior. I think I am doing something very wrong.
I want to replicate a object of Too and then save it. For e.g.
too = Too.create(blah="Awesome")
too.save()
foo = Foo.create(too=too)
foo.save()
too.id = None #Copy the original
too.save()
coo = Coo.create(too=too)
coo.save()
print foo.too.id
print coo.too.id
#above 2 print statements give same id
When I check in the admin the both foo and coo have different too object saved. But while printing it is showing the same. Why is that happening. I think I am doing something fundamentally wrong.
Django looks at the primary key to determine uniqueness, so work with that directly:
too.pk = None
too.save()
Setting the primary key to None will cause Django to perform an INSERT, saving a new instance of the model, rather than an UPDATE to the existing instance.
Source: https://stackoverflow.com/a/4736172/1533388
UPDATE: err, using pk and id are interchangeable in this case, so you'll get the same result. My first answer didn't address your question.
The discrepancy here is between what is occurring in python vs. what can be reconstituted from the database.
Your code causes Django to save two unique objects to the database, but you're only working with one python Too instance. When foo.save() occurs, the database entry for 'foo' is created with a reference to the DB entry for the first Too object. When coo.save() occurs, the database entry for 'coo' is created, pointing to the second, unique Too object that was stored via:
too.id = None #Copy the original
too.save()
However, in python, both coo and foo refer to the same object, named 'too', via their respective '.too' attributes. In python, there is only one 'Too' instance. So when you update too.id, you're updating one object, referred to by both coo and foo.
Only when the models are reconstituted from the database (as the admin view does in order to display them) are unique instances created for each foreign key; this is why the admin view shows two unique saved instances.
In MySQL, we can have foreign key relationships between tables in different databases. I am finding it difficult to translate this relationship on the respective Django models.
I have read in the docs that cross-db relationships are not supported, but can we override some property/function so that we can make tables be identified as DB.table rather than table?
For example, there is table table1 in DB1 that gets referenced in some table2 in DB2. Django tries (unsuccessfully) to find table1 in DB2, and raises a DatabaseError
Variable Value
charset 'latin1'
exc <class '_mysql_exceptions.ProgrammingError'>
self <MySQLdb.cursors.Cursor object at 0x2a87ed0>
args (195,)
db <weakproxy at 0x2a95208 to Connection at 0xdad0>
value ProgrammingError(1146, "Table 'DB2.table1' doesn't exist")
query 'SELECT (1) AS `a` FROM `table1` WHERE `table1`.`ndx` = 195 LIMIT 1'
Almost everything works, except the save method. A push in the right direction would help a lot!
You required Manually Selecting a Database.
Looking on the error you gave, you should do something like this:
qs = table1.objects.using('DB1 ').filter(pk=id)
# just an example
In this example we are explicitly telling Django to locate table1 in DB1.
It seems we cannot do anything to get relationships working between two tables in different mysql DBs. This is by design. Ticket 17875 has some info. We need to write code that works around this.