So after spending the better part of my day off trying to wrap my head around data and schema migrations in South, I feel like I'm getting close -- but I'm having some trouble with my datamigration forwards function.
For reference, here was my original model:
class Lead_Contact(models.Model):
...
general_notes = models.TextField(blank=True)
...
I'm attempting to migrate to the following:
class Lead_Contact(models.Model):
...
# General_notes has been removed from here...
...
class General_Note(models.Model):
#...and added as a foreign key here.
...
lead_contact = models.ForeignKey('Lead_Contact', null=True, blank=True)
user = models.CharField(max_length=2, choices=USER_CHOICES)
general_date = models.DateField(blank = True, null=True)
general_time = models.TimeField(blank = True, null=True)
general_message = models.TextField(blank=True)
...
I've followed the steps to convert_to_south my app, as well as followed tutorial #3 to add my table, then create my datamigration, and then remove the old Lead_contact.general_notes field in a second schema migration.
The problem is writing my actual Forwards() method; I'm attempting to write out the data from the old general_notes field into the General_Note table:
class Migration(DataMigration):
def forwards(self, orm):
for doctor in orm.Lead_Contact.objects.all():
orm.General_Note.objects.create(lead_contact=doctor.id, user = "AU", general_date = "2011-03-12", general_time = "09:00:00", general_message = doctor.general_notes)
def backwards(self, orm):
for note in orm.General_Note.objects.all():
new_gn = orm.Lead_Contact.objects.get(id=note.lead_contact)
new_gn.general_notes = note.general_message
new_gn.save()
For reference, I'm using django 1.2, south 0.7, and MySql 5.0.51a.
Edit: Removed the Try/Except bits, and the error message I'm getting is: "ValueError: Cannot assign "158L": "General_Note.lead_contact" must be a "Lead_Contact" instance.
Shouldn't tying General_Note.lead_contact to Doctor.id be an appropriate Lead_Contact instance?
Try changing doctor.id to doctor:
orm.General_Note.objects.create(lead_contact=doctor.id,
user = "AU",
general_date = "2011-03-12",
general_time = "09:00:00",
general_message = doctor.general_notes)
To:
orm.General_Note.objects.create(lead_contact=doctor,
user = "AU",
general_date = "2011-03-12",
general_time = "09:00:00",
general_message = doctor.general_notes)
Related
I was using django-hitcont to count the views on my Post model. I am trying to get the most viewed post in my ListView using this query objects.order_by('hit_count_generic__hits') and it is working fine on SQLite but on PostgreSQL, it is giving me this error :
django.db.utils.ProgrammingError: operator does not exist: integer = text LINE 1: ...R JOIN "hitcount_hit_count" ON ("posts_post"."id" = "hitcoun....
models.py
class Post(models.Model, HitCountMixin):
author = models.ForeignKey(User, related_name='authors', on_delete=models.CASCADE)
title = models.CharField('Post Title', max_length = 150)
description = models.TextField('Description', max_length=1000, blank = True)
date_posted = models.DateTimeField('Date posted', default = timezone.now)
date_modifed = models.DateTimeField('Date last modified', default = timezone.now)
document = models.FileField('Document of Post', upload_to='documents', \
validators=[FileExtensionValidator(allowed_extensions = ['pdf', 'docx']), validate_document_size] \
)
hit_count_generic = GenericRelation(
HitCount,
object_id_field='object_pk',
related_query_name='hit_count_generic_relation'
)
views.py
queryset = Post.objects.order_by('hit_count_generic__hits')
I found this issue on Github related to the problem, but I am still not able to figure out the mentioned workaround.
When comparing different types (in this example integer and text), equals operator throws this exception. To fix that, convert HitCount model pk field to integer and you are good to go. To do that, you need to create and apply migration operation. Django is a really good framework to handle this kind of operations. You just need to check values are not null and are "convertable" to integer. Just change the field type and run two commands below.
python manage.py makemigrations
python manage.py migrate
Before updating your model, I highly recommend you to take a backup in case of failure. This is not an easy operation but you can follow the these links to understand what is going on during this the process.
migrations dump and restore initial data
If you don't care the data on table, just drop table and create a brand new migration file and recreate table.
In this model:
class Rank(models.Model):
User = models.ForeignKey(User)
Rank = models.ForeignKey(RankStructure)
date_promoted = models.DateField()
def __str__(self):
return self.Rank.Name.order_by('promotion__date_promoted').latest()
I'm getting the error:
Exception Value:
'str' object has no attribute 'order_by'
I want the latest Rank as default. How do I set this?
Thanks.
Update #1
Added Rank Structure
class RankStructure(models.Model):
RankID = models.CharField(max_length=4)
SName = models.CharField(max_length=5)
Name = models.CharField(max_length=125)
LongName = models.CharField(max_length=512)
GENRE_CHOICES = (
('TOS', 'The Original Series'),
('TMP', 'The Motion Picture'),
('TNG', 'The Next Generation'),
('DS9', 'Deep Space Nine'),
('VOY', 'VOYAGER'),
('FUT', 'FUTURE'),
('KTM', 'KELVIN TIMELINE')
)
Genre = models.CharField(max_length=3, choices=GENRE_CHOICES)
SPECIALTY_OPTIONS = (
('CMD', 'Command'),
('OPS', 'Operations'),
('SCI', 'Science'),
('MED', 'Medical'),
('ENG', 'Engineering'),
('MAR', 'Marine'),
('FLT', 'Flight Officer'),
)
Specialty = models.CharField(max_length=25, choices=SPECIALTY_OPTIONS)
image = models.FileField(upload_to=image_upload_handler, blank=True)
This is the Rank_structure referenced by Rank in Class Rank.
THe User Foreign key goes to the standard User table.
The reason that you’re getting an error is because self.Rank.Name is not a ModelManager on which you can call order_by. You’ll need an objects in there somewhere if you want to call order_by. We can’t help you with the django formatting for the query you want unless you also post the model definitions as requested by several commenters. That said, I suspect that what you want is something like:
def __str__(self):
return self.objects.filter(Rank_id=self.Rank_id).order_by('date_promoted').latest().User.Name
I am running into a database issue in my unit tests. I think it has something to do with the way I am using TestCase and setUpData.
When I try to set up my test data with certain values, the tests throw the following error:
django.db.utils.IntegrityError: duplicate key value violates unique constraint
...
psycopg2.IntegrityError: duplicate key value violates unique constraint "InventoryLogs_productgroup_product_name_48ec6f8d_uniq"
DETAIL: Key (product_name)=(Almonds) already exists.
I changed all of my primary keys and it seems to be running fine. It doesn't seem to affect any of the tests.
However, I'm concerned that I am doing something wrong. When it first happened, I reversed about an hour's worth of work on my app (not that much code for a noob), which corrected the problem.
Then when I wrote the changes back in, the same issue presented itself again. TestCase is pasted below. The issue seems to occur after I add the sortrecord items, but corresponds with the items above it.
I don't want to keep going through and changing primary keys and urls in my tests, so if anyone sees something wrong with the way I am using this, please help me out. Thanks!
TestCase
class DetailsPageTest(TestCase):
#classmethod
def setUpTestData(cls):
cls.product1 = ProductGroup.objects.create(
product_name="Almonds"
)
cls.variety1 = Variety.objects.create(
product_group = cls.product1,
variety_name = "non pareil",
husked = False,
finished = False,
)
cls.supplier1 = Supplier.objects.create(
company_name = "Acme",
company_location = "Acme Acres",
contact_info = "Call me!"
)
cls.shipment1 = Purchase.objects.create(
tag=9,
shipment_id=9999,
supplier_id = cls.supplier1,
purchase_date='2015-01-09',
purchase_price=9.99,
product_name=cls.variety1,
pieces=99,
kgs=999,
crackout_estimate=99.9
)
cls.shipment2 = Purchase.objects.create(
tag=8,
shipment_id=8888,
supplier_id=cls.supplier1,
purchase_date='2015-01-08',
purchase_price=8.88,
product_name=cls.variety1,
pieces=88,
kgs=888,
crackout_estimate=88.8
)
cls.shipment3 = Purchase.objects.create(
tag=7,
shipment_id=7777,
supplier_id=cls.supplier1,
purchase_date='2014-01-07',
purchase_price=7.77,
product_name=cls.variety1,
pieces=77,
kgs=777,
crackout_estimate=77.7
)
cls.sortrecord1 = SortingRecords.objects.create(
tag=cls.shipment1,
date="2015-02-05",
bags_sorted=20,
turnout=199,
)
cls.sortrecord2 = SortingRecords.objects.create(
tag=cls.shipment1,
date="2015-02-07",
bags_sorted=40,
turnout=399,
)
cls.sortrecord3 = SortingRecords.objects.create(
tag=cls.shipment1,
date='2015-02-09',
bags_sorted=30,
turnout=299,
)
Models
from datetime import datetime
from django.db import models
from django.db.models import Q
class ProductGroup(models.Model):
product_name = models.CharField(max_length=140, primary_key=True)
def __str__(self):
return self.product_name
class Meta:
verbose_name = "Product"
class Supplier(models.Model):
company_name = models.CharField(max_length=45)
company_location = models.CharField(max_length=45)
contact_info = models.CharField(max_length=256)
class Meta:
ordering = ["company_name"]
def __str__(self):
return self.company_name
class Variety(models.Model):
product_group = models.ForeignKey(ProductGroup)
variety_name = models.CharField(max_length=140)
husked = models.BooleanField()
finished = models.BooleanField()
description = models.CharField(max_length=500, blank=True)
class Meta:
ordering = ["product_group_id"]
verbose_name_plural = "Varieties"
def __str__(self):
return self.variety_name
class PurchaseYears(models.Manager):
def purchase_years_list(self):
unique_years = Purchase.objects.dates('purchase_date', 'year')
results_list = []
for p in unique_years:
results_list.append(p.year)
return results_list
class Purchase(models.Model):
tag = models.IntegerField(primary_key=True)
product_name = models.ForeignKey(Variety, related_name='purchases')
shipment_id = models.CharField(max_length=24)
supplier_id = models.ForeignKey(Supplier)
purchase_date = models.DateField()
estimated_delivery = models.DateField(null=True, blank=True)
purchase_price = models.DecimalField(max_digits=6, decimal_places=3)
pieces = models.IntegerField()
kgs = models.IntegerField()
crackout_estimate = models.DecimalField(max_digits=6,decimal_places=3, null=True)
crackout_actual = models.DecimalField(max_digits=6,decimal_places=3, null=True)
objects = models.Manager()
purchase_years = PurchaseYears()
# Keep manager as "objects" in case admin, etc. needs it. Filter can be called like so:
# Purchase.objects.purchase_years_list()
# Managers in docs: https://docs.djangoproject.com/en/1.8/intro/tutorial01/
class Meta:
ordering = ["purchase_date"]
def __str__(self):
return self.shipment_id
def _weight_conversion(self):
return round(self.kgs * 2.20462)
lbs = property(_weight_conversion)
class SortingModelsBagsCalulator(models.Manager):
def total_sorted(self, record_date, current_set):
sorted = [SortingRecords['bags_sorted'] for SortingRecords in current_set if
SortingRecords['date'] <= record_date]
return sum(sorted)
class SortingRecords(models.Model):
tag = models.ForeignKey(Purchase, related_name='sorting_record')
date = models.DateField()
bags_sorted = models.IntegerField()
turnout = models.IntegerField()
objects = models.Manager()
def __str__(self):
return "%s [%s]" % (self.date, self.tag.tag)
class Meta:
ordering = ["date"]
verbose_name_plural = "Sorting Records"
def _calculate_kgs_sorted(self):
kg_per_bag = self.tag.kgs / self.tag.pieces
kgs_sorted = kg_per_bag * self.bags_sorted
return (round(kgs_sorted, 2))
kgs_sorted = property(_calculate_kgs_sorted)
def _byproduct(self):
waste = self.kgs_sorted - self.turnout
return (round(waste, 2))
byproduct = property(_byproduct)
def _bags_remaining(self):
current_set = SortingRecords.objects.values().filter(~Q(id=self.id), tag=self.tag)
sorted = [SortingRecords['bags_sorted'] for SortingRecords in current_set if
SortingRecords['date'] <= self.date]
remaining = self.tag.pieces - sum(sorted) - self.bags_sorted
return remaining
bags_remaining = property(_bags_remaining)
EDIT
It also fails with integers, like so.
django.db.utils.IntegrityError: duplicate key value violates unique constraint "InventoryLogs_purchase_pkey"
DETAIL: Key (tag)=(9) already exists.
UDPATE
So I should have mentioned this earlier, but I completely forgot. I have two unit test files that use the same data. Just for kicks, I matched a primary key in both instances of setUpTestData() to a different value and sure enough, I got the same error.
These two setups were working fine side-by-side before I added more data to one of them. Now, it appears that they need different values. I guess you can only get away with using repeat data for so long.
I continued to get this error without having any duplicate data but I was able to resolve the issue by initializing the object and calling the save() method rather than creating the object via Model.objects.create()
In other words, I did this:
#classmethod
def setUpTestData(cls):
cls.person = Person(first_name="Jane", last_name="Doe")
cls.person.save()
Instead of this:
#classmethod
def setUpTestData(cls):
cls.person = Person.objects.create(first_name="Jane", last_name="Doe")
I've been running into this issue sporadically for months now. I believe I just figured out the root cause and a couple solutions.
Summary
For whatever reason, it seems like the Django test case base classes aren't removing the database records created by let's just call it TestCase1 before running TestCase2. Which, in TestCase2 when it tries to create records in the database using the same IDs as TestCase1 the database raises a DuplicateKey exception because those IDs already exists in the database. And even saying the magic word "please" won't help with database duplicate key errors.
Good news is, there are multiple ways to solve this problem! Here are a couple...
Solution 1
Make sure if you are overriding the class method tearDownClass that you call super().tearDownClass(). If you override tearDownClass() without calling its super, it will in turn never call TransactionTestCase._post_teardown() nor TransactionTestCase._fixture_teardown(). Quoting from the doc string in TransactionTestCase._post_teardown()`:
def _post_teardown(self):
"""
Perform post-test things:
* Flush the contents of the database to leave a clean slate. If the
class has an 'available_apps' attribute, don't fire post_migrate.
* Force-close the connection so the next test gets a clean cursor.
"""
If TestCase.tearDownClass() is not called via super() then the database is not reset in between test cases and you will get the dreaded duplicate key exception.
Solution 2
Override TransactionTestCase and set the class variable serialized_rollback = True, like this:
class MyTestCase(TransactionTestCase):
fixtures = ['test-data.json']
serialized_rollback = True
def test_name_goes_here(self):
pass
Quoting from the source:
class TransactionTestCase(SimpleTestCase):
...
# If transactions aren't available, Django will serialize the database
# contents into a fixture during setup and flush and reload them
# during teardown (as flush does not restore data from migrations).
# This can be slow; this flag allows enabling on a per-case basis.
serialized_rollback = False
When serialized_rollback is set to True, Django test runner rolls back any transactions inserted into the database beween test cases. And batta bing, batta bang... no more duplicate key errors!
Conclusion
There are probably many more ways to implement a solution for the OP's issue, but these two should work nicely. Would definitely love to have more solutions added by others for clarity sake and a deeper understanding of the underlying Django test case base classes. Phew, say that last line real fast three times and you could win a pony!
The log you provided states DETAIL: Key (product_name)=(Almonds) already exists. Did you verify in your db?
To prevent such errors in the future, you should prefix all your test data string by test_
I discovered the issue, as noted at the bottom of the question.
From what I can tell, the database didn't like me using duplicate data in the setUpTestData() methods of two different tests. Changing the primary key values in the second test corrected the problem.
I think the problem here is that you had a tearDownClass method in your TestCase without the call to super method.
In this way the django TestCase lost the transactional functionalities behind the setUpTestData so it doesn't clean your test db after a TestCase is finished.
Check warning in django docs here:
https://docs.djangoproject.com/en/1.10/topics/testing/tools/#django.test.SimpleTestCase.allow_database_queries
I had similar problem that had been caused by providing the primary key value to a test case explicitly.
As discussed in the Django documentation, manually assigning a value to an auto-incrementing field doesn’t update the field’s sequence, which might later cause a conflict.
I have solved it by altering the sequence manually:
from django.db import connection
class MyTestCase(TestCase):
#classmethod
def setUpTestData(cls):
Model.objects.create(id=1)
with connection.cursor() as c:
c.execute(
"""
ALTER SEQUENCE "app_model_id_seq" RESTART WITH 2;
"""
)
So I'm trying to populate a model in django using a postgres (postgis) database. The problem I'm having is inputting the datetimefield. I have written a population script but every time I run it I get the error django.db.utils.IntegrityError: null value in column "pub_date" violates not-null constraint. The code below shows my model and the part of the population script that applies to the table.
The model:
class Article(models.Model):
authors = models.ManyToManyField(Author)
location = models.ForeignKey(Location)
article_title = models.CharField(max_length=200, unique_for_date="pub_date")
pub_date = models.DateTimeField('date published')
article_keywords = ArrayField(ArrayField(models.CharField(max_length=20, blank=True), size=8), size=8,)
title_id = models.CharField(max_length=200)
section_id = models.CharField(max_length=200)
And the population script:
def populate():
add_article(
id = "1",
article_title = "Obama scrambles to get sceptics in Congress to support Iran nuclear deal",
pub_date = "2015-04-06T20:38:59Z",
article_keywords = "{obama, iran, debate, congress, america, un, republican, democrat, nuclear, isreal}",
title_id = "white-house-scrambles-sceptics-congress-iran-nuclear-deal",
section_id = "us-news",
location_id = "1"
)
def add_article(id, article_title, pub_date, article_keywords, title_id, section_id, location_id):
article = Article.objects.get_or_create(article_title=article_title)[0]
article.id
article.article_title
article.pub_date
article.article_keywords
article.title_id
article.section_id
article.location_id
article.save()
return article
if __name__ == '__main__':
print "Starting Newsmap population script..."
populate()
I've searched around for ages but there seems to be no solution to this specific problem. Any help much appreciated!!
The issue is that you do not pass to Article.objects.get_or_create the data needed to create a new object in case none already exists.
What you need to do is (see the documentation for get_or_create):
article = Article.objects.get_or_create(
article_title=article_title,
pub_date=pub_date,
defaults={
'id': id,
'article_keywords': article_keywords,
# etc...
}
)[0]
The data passed using the defaults argument will only be used to create a new object. The data passed using other keyword arguments will be used to check if an existing object matches in the database.
First post to stackoverflow I did do a search and came up dry. I also own
the django book (Forcier,Bissex,Chun) and they don't explain how to do
this. In short I can't figure out how to progmatically add a data via
a python shell script to the ManyToMay model..
from django.db import models
from django.contrib import admin
class Client(models.Model):
client = models.CharField(max_length=256, primary_key=True)
access = models.DateField()
description = models.TextField()
host = models.CharField(max_length=256)
lineEnd = models.CharField(max_length=256)
options = models.TextField()
owner = models.CharField(max_length=100)
root = models.CharField(max_length=256)
submitOptions = models.CharField(max_length=256)
update = models.DateField()
def __unicode__(self):
return str(self.client)
admin.site.register(Client)
class Change(models.Model):
"""This simply expands out 'p4 describe' """
change = models.IntegerField(primary_key=True)
client = models.ManyToManyField(Client)
desc = models.TextField()
status = models.CharField(max_length=128)
def __unicode__(self):
return str(self.change)
admin.site.register(Change)
Here is what I have which works but I don't know how to add the
ManyToMany. I can't seem to figure out how to progmatically call it.
I know the row in SQL exists.
--- massImport.py ---
# Assume the client "clientspec" exists. I know how to create that if
neeeded.
changes = [ { 'change': 123, 'desc': "foobar", status': "foobar",
client': "clientspec", }]
for item in changes:
entry = Change(
change = item['change'],
desc = item['desc'],
status = item['status'],
# client = Client.objects.filter(client=item['client'])
)
entry.save()
Can anyone show me where the error of my ways is. I would really
appreciate it.
Thanks!!
Turns out Tiago was very close..
# Assume the client "clientspec" exists. I know how to create that if
neeeded.
changes = [ { 'change': 123, 'desc': "foobar", status': "foobar",
client': "clientspec", }]
for item in changes:
entry = Change()
entry.change = item['change']
entry.desc = item['desc']
entry.status = item['status']
entry.time = datetime.datetime.fromtimestamp(float(item['time']))
entry.client.add(Client.objects.get(client=item['client']))
entry.save()
So.. I will give props to Tiago
.filter returns a list, when you need is a single object, so you should use .get(client=item['client'])
I tried the code but i got error
ValueError: "<Change: 123 -- foobar>" needs to have a value for field "change" before this many-to-many relationship can be used
Manytomany(entry.client.add) can be used only after saving the field ie entry.save()
There may be a lot of clients so you can use:
changes = [{'change': 123, 'desc': "foobar", 'status': "foobar",
'client': ("client1","client2"),},{......]
for item in changes:
entry = Change(
change = item['change'],
desc = item['desc'],
status = item['status'],)
entry.save()
for c in item['client']:
entry.client.add(Client.objects.get(client=c))