I'm trying to migrate a column from a Char field to a Many-to-Many field running Django 1.8.2. I'm doing a custom Data Migration, to move the data properly. When I try to migrate, I get a database error, can't insert null into the many to many table id column.
My models, simplified:
class LicenseArea(models.Model):
#appraisal_account = models.CharField(max_length=17, null=True, db_index=True)
appraisal_account = models.ManyToManyField(TaxAccount, db_table='LicAreaTaxAccount', related_name='accounts_for_license_area', related_query_name='license_area_for_account', null=True)
class TaxAccount(models.Model):
account = models.CharField(max_length=17, db_index=True)
So I first create TaxAccount objects in a RunPython block, then remove the old field and add the new one, like so:
migrations.RunPython(create_tax_account_objects),
migrations.RemoveField(
model_name='licensearea',
name='appraisal_account',
),
migrations.AddField(
model_name='licensearea',
name='appraisal_account',
field=models.ManyToManyField(related_query_name='license_area_for_account', related_name='accounts_for_license_area', db_table='licenses_LicAreaTaxAccount', to='licenses.TaxAccount'),
),
All that works. My issue comes when I try to migrate the data, relating the LicenseArea object with its corresponding TaxAccount object. In another RunPython block, I try the code shown below (I've tried in both directions; acct.licensearea_set.add indicates that the TaxAccount model has no licensearea_set attribute, while the second option shown below gives me the IntegrityError (ORA-01400) that I can't insert null into the ID column :
for la in LicenseArea.objects.all():
acct = TaxAccount.objects.get(account=la.appraisal_account_temp)
#acct.licensearea_set.add(la)
#la.appraisal_account.add(acct)
How do I solve this? Thanks in advance.
Related
TL\DR version - Find and delete rows in Postgres with same date but different time, leaving one record per date.
Long read:
At some point we've migrated our app's backend to a newer version - this is a Django application - migrated from Python 2, Django 1.8 to Python 3 - Django 4, and with this update we're changed timezone for backend from UTC+2 to UTC+3. And now strange things happens - records which previously successfully have been read from db with queryset StatChildVisit.objects.filter(date=day, garden_group=garden_group) - (day is a python's date only not datetime) after update returns empty queryset, although records for that day are still in db. More so newly created records have different time in them - records created with old timezone looks like 2022-12-28 22:00:00.000000 +00:00 new records looks like 2022-12-28 21:00:00.000000 +00:00
Seems that bug happened because date field in django's model have been declared as DateTimeField -
class StatChildVisit(models.Model):
child = models.ForeignKey(Child, on_delete=models.CASCADE)
date = models.DateTimeField(default=timezone.now)
visit = models.BooleanField(_('Atended'), default=True)
disease = models.BooleanField(_('Sick'), default=False)
other_approved = models.BooleanField(_('Other approved'), default=False)
garden_group = models.ForeignKey(GardenGroup, verbose_name=_('Garden group'), editable=False, blank=True, null=True, on_delete=models.CASCADE)
rossecure_visit = models.ForeignKey('rossecure.Visits', editable=False, null=True, blank=True, on_delete=models.CASCADE)
class Meta:
verbose_name = _('Attendence')
verbose_name_plural = _('Attendence')
index_together = (
('date', 'garden_group'),
)
unique_together = (
('date', 'child'),
)
all records are always being created with date only (not datetime) passing to constructor
So we've decided to migrate this field to DateField, but after migration field type in DB is still 'timestamp with time zone', and besides, because this is a production database users after finding that some data looks like lost partially recreated records.
So now we have multiple records for same day but with different time which need to be deleted and because of constrains table column can not be altered with ALTER TABLE reports_statchildvisit ALTER COLUMN date TYPE date;
Because table have rather large records count (about 4 million) I think that problem should be solved via SQL side, and not Django side. My plan is to delete duplicates and then change column type to date.
I've tried to alter records with
update reports_statchildvisit
set date = date(date) + '21:00:00'::time
but because I've tried that after users created similar records script failed with ERROR: duplicate key value violates unique constraint
UPD: DDL on the SQL side looks like this:
create table public.reports_statchildvisit
(
id serial
primary key,
date timestamp with time zone not null,
visit boolean not null,
disease boolean not null,
child_id integer not null
constraint reports_statchil_child_id_30fbdf92a34d3fea_fk_children_child_id
references public.children_child
deferrable initially deferred,
garden_group_id integer
constraint repor_garden_group_id_ef61dd52421b5d2_fk_project_gardengroup_id
references public.project_gardengroup
deferrable initially deferred,
other_approved boolean not null,
rossecure_visit_id integer
constraint repo_rossecure_visit_id_488614f59207663f_fk_rossecure_visits_id
references public.rossecure_visits
deferrable initially deferred,
constraint reports_statchildvisit_date_3d6916481fe1e727_uniq
unique (date, child_id)
);
alter table public.reports_statchildvisit
owner to django;
create index reports_statchildvisit_10e12719
on public.reports_statchildvisit (garden_group_id);
create index reports_statchildvisit_42d2af72
on public.reports_statchildvisit (rossecure_visit_id);
create index reports_statchildvisit_date_66064e65c46d4137_idx
on public.reports_statchildvisit (date, garden_group_id);
create index reports_statchildvisit_f36263a3
on public.reports_statchildvisit (child_id);
I was using django-hitcont to count the views on my Post model. I am trying to get the most viewed post in my ListView using this query objects.order_by('hit_count_generic__hits') and it is working fine on SQLite but on PostgreSQL, it is giving me this error :
django.db.utils.ProgrammingError: operator does not exist: integer = text LINE 1: ...R JOIN "hitcount_hit_count" ON ("posts_post"."id" = "hitcoun....
models.py
class Post(models.Model, HitCountMixin):
author = models.ForeignKey(User, related_name='authors', on_delete=models.CASCADE)
title = models.CharField('Post Title', max_length = 150)
description = models.TextField('Description', max_length=1000, blank = True)
date_posted = models.DateTimeField('Date posted', default = timezone.now)
date_modifed = models.DateTimeField('Date last modified', default = timezone.now)
document = models.FileField('Document of Post', upload_to='documents', \
validators=[FileExtensionValidator(allowed_extensions = ['pdf', 'docx']), validate_document_size] \
)
hit_count_generic = GenericRelation(
HitCount,
object_id_field='object_pk',
related_query_name='hit_count_generic_relation'
)
views.py
queryset = Post.objects.order_by('hit_count_generic__hits')
I found this issue on Github related to the problem, but I am still not able to figure out the mentioned workaround.
When comparing different types (in this example integer and text), equals operator throws this exception. To fix that, convert HitCount model pk field to integer and you are good to go. To do that, you need to create and apply migration operation. Django is a really good framework to handle this kind of operations. You just need to check values are not null and are "convertable" to integer. Just change the field type and run two commands below.
python manage.py makemigrations
python manage.py migrate
Before updating your model, I highly recommend you to take a backup in case of failure. This is not an easy operation but you can follow the these links to understand what is going on during this the process.
migrations dump and restore initial data
If you don't care the data on table, just drop table and create a brand new migration file and recreate table.
I'm working on a django project where I need a DateField to sometimes be empty. My model looks like this:
#models.py
end = models.DateField(default=None, blank=True)
But when I run python manage.py sql myapp the sql statement always end up being
CREATE TABLE "myapp_date" (
"end" date NOT NULL
);
Therefore my field isn't nullable and I can't understand what I should do to make it so. Any idea would be appreciated !
You should use
end = models.DateField(default=None, blank=True, null=True)
Basically blank allows you to pass it a null value, but null tells the database to accept null values.
I've got a tiny little problem that, unfortunately, is taking all my time.
It is really simple, I already have my database and I created then modified models.py, and admin.py. Some staff users, who will need to enter values in my database, need the simpliest form to do so.
Here is my database :
-- Table NGSdb.line
CREATE TABLE IF NOT EXISTS `NGSdb`.`line` (
`id` INT NOT NULL AUTO_INCREMENT ,
`value` INT NOT NULL ,
PRIMARY KEY (`id`) )
ENGINE = InnoDB;
CREATE UNIQUE INDEX `value_UNIQUE` ON `NGSdb`.`line` (`value` ASC) ;
-- Table NGSdb.run_has_sample_lines
CREATE TABLE IF NOT EXISTS `NGSdb`.`run_has_sample_lines` (
`line_id` INT NOT NULL ,
`runhassample_id` INT NOT NULL ,
PRIMARY KEY (`line_id`, `runhassample_id`) ,
CONSTRAINT `fk_sample_has_line_line1`
FOREIGN KEY (`line_id` )
REFERENCES `NGSdb`.`line` (`id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `fk_sample_has_line_run_has_sample1`
FOREIGN KEY (`runhassample_id` )
REFERENCES `NGSdb`.`run_has_sample` (`id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION)
-- Table NGSdb.run_has_sample
CREATE TABLE IF NOT EXISTS `NGSdb`.`run_has_sample` (
`id` INT NOT NULL AUTO_INCREMENT ,
`run_id` INT NOT NULL ,
`sample_id` INT NOT NULL ,
`dna_quantification_ng_per_ul` FLOAT NULL ,
PRIMARY KEY (`id`, `run_id`, `sample_id`) ,
CONSTRAINT `fk_run_has_sample_run1`
FOREIGN KEY (`run_id` )
REFERENCES `NGSdb`.`run` (`id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `fk_run_has_sample_sample1`
FOREIGN KEY (`sample_id` )
REFERENCES `NGSdb`.`sample` (`id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION)
Here is my models.py :
class Run(models.Model):
id = models.AutoField(primary_key=True)
start_date = models.DateField(null=True, blank=True, verbose_name='start date')
end_date = models.DateField(null=True, blank=True, verbose_name='end date')
project = models.ForeignKey(Project)
sequencing_type = models.ForeignKey(SequencingType)
def __unicode__(self):
return u"run started %s from the project %s" % (self.start_date,self.project)
class Line(models.Model):
id = models.AutoField(primary_key=True)
value = models.IntegerField()
def __unicode__(self):
return u"%s" % str(self.value)
class RunHasSample(models.Model):
id = models.AutoField(primary_key=True)
run = models.ForeignKey(Run)
sample = models.ForeignKey(Sample)
dna_quantification_ng_per_ul = models.FloatField(null=True, blank=True)
lines = models.ManyToManyField(Line)
def __unicode__(self):
return u"Sample %s from run %s" % (self.sample, self.run)
And here is my admin.py :
class RunHasSamplesInLine(admin.TabularInline):
model = RunHasSample
fields = ['sample', 'dna_quantification_ng_per_ul', 'lines']
extra = 6
class RunAdmin(admin.ModelAdmin):
fields = ['project', 'start_date', 'end_date', 'sequencing_type']
inlines = [RunHasSamplesInLine]
list_display = ('project', 'start_date', 'end_date', 'sequencing_type')
As you can see, my samples are displayed in lines in the run form so that the staff can easily fullfill the database.
When I try to fill the database I have this error :
(1054, "Unknown column 'run_has_sample_lines.id' in 'field list'")
Of course, there are no field "lines" in my database ! It is a many to many field so I already created my intermediate table !
Okay okay ! So I tried to create the model for the intermediate table (run_has_sample_lines) and add a "through" to the ManyToManyField in the RunHasSample model. But, as I add manually the "through", I cannot use the ManyToMany field. The only way to add lines to the admin view is to stack them in lines... As you can see the samples are already in lines, it is impossible to put a new "inlines" in the already in lines samples...
Finally, I just tried to see what django had created with the manage.py sqlall.
I see that :
CREATE TABLE `run_has_sample_lines` (
`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
`runhassample_id` integer NOT NULL,
`line_id` integer NOT NULL,
UNIQUE (`runhassample_id`, `line_id`)
)
;
ALTER TABLE `run_has_sample_lines` ADD CONSTRAINT `line_id_refs_id_4f0766aa` FOREIGN KEY (`line_id`) REFERENCES `line` (`id`);
It seems that there are no foreign key on the run_has_sample table whereas I created it in the database in the first place. I guess that the problem is coming from here but I cannot resolve it and I really hope that you can...
Thank you very much !
you may wish to try a 'through' attribute on the many-to-many relationship and declare your intermediate table in Django.
I found where the problem is...
It is not a problem in the ManyToManyField but in the intermediate table. Django refused that my intermediate table doesn't have an unique id !
So, in the sql which created django, it created automatically an unique id named "id", but in my database I didn't create one (because the couple of two foreign key is usually enough).
Next time, I'll be more carefull.
this is a model of the view table.
class QryDescChar(models.Model):
iid_id = models.IntegerField()
cid_id = models.IntegerField()
cs = models.CharField(max_length=10)
cid = models.IntegerField()
charname = models.CharField(max_length=50)
class Meta:
db_table = u'qry_desc_char'
this is the SQL i use to create the table
CREATE VIEW qry_desc_char as
SELECT
tbl_desc.iid_id,
tbl_desc.cid_id,
tbl_desc.cs,
tbl_char.cid,
tbl_char.charname
FROM tbl_desC,tbl_char
WHERE tbl_desc.cid_id = tbl_char.cid;
i dont know if i need a function in models or views or both. i want to get a list of objects from that database to display it. This might be easy but im new at Django and python so i having some problems
Django 1.1 brought in a new feature that you might find useful. You should be able to do something like:
class QryDescChar(models.Model):
iid_id = models.IntegerField()
cid_id = models.IntegerField()
cs = models.CharField(max_length=10)
cid = models.IntegerField()
charname = models.CharField(max_length=50)
class Meta:
db_table = u'qry_desc_char'
managed = False
The documentation for the managed Meta class option is here. A relevant quote:
If False, no database table creation
or deletion operations will be
performed for this model. This is
useful if the model represents an
existing table or a database view that
has been created by some other means.
This is the only difference when
managed is False. All other aspects of
model handling are exactly the same as
normal.
Once that is done, you should be able to use your model normally. To get a list of objects you'd do something like:
qry_desc_char_list = QryDescChar.objects.all()
To actually get the list into your template you might want to look at generic views, specifically the object_list view.
If your RDBMS lets you create writable views and the view you create has the exact structure than the table Django would create I guess that should work directly.
(This is an old question, but is an area that still trips people up and is still highly relevant to anyone using Django with a pre-existing, normalized schema.)
In your SELECT statement you will need to add a numeric "id" because Django expects one, even on an unmanaged model. You can use the row_number() window function to accomplish this if there isn't a guaranteed unique integer value on the row somewhere (and with views this is often the case).
In this case I'm using an ORDER BY clause with the window function, but you can do anything that's valid, and while you're at it you may as well use a clause that's useful to you in some way. Just make sure you do not try to use Django ORM dot references to relations because they look for the "id" column by default, and yours are fake.
Additionally I would consider renaming my output columns to something more meaningful if you're going to use it within an object. With those changes in place the query would look more like (of course, substitute your own terms for the "AS" clauses):
CREATE VIEW qry_desc_char as
SELECT
row_number() OVER (ORDER BY tbl_char.cid) AS id,
tbl_desc.iid_id AS iid_id,
tbl_desc.cid_id AS cid_id,
tbl_desc.cs AS a_better_name,
tbl_char.cid AS something_descriptive,
tbl_char.charname AS name
FROM tbl_desc,tbl_char
WHERE tbl_desc.cid_id = tbl_char.cid;
Once that is done, in Django your model could look like this:
class QryDescChar(models.Model):
iid_id = models.ForeignKey('WhateverIidIs', related_name='+',
db_column='iid_id', on_delete=models.DO_NOTHING)
cid_id = models.ForeignKey('WhateverCidIs', related_name='+',
db_column='cid_id', on_delete=models.DO_NOTHING)
a_better_name = models.CharField(max_length=10)
something_descriptive = models.IntegerField()
name = models.CharField(max_length=50)
class Meta:
managed = False
db_table = 'qry_desc_char'
You don't need the "_id" part on the end of the id column names, because you can declare the column name on the Django model with something more descriptive using the "db_column" argument as I did above (but here I only it to prevent Django from adding another "_id" to the end of cid_id and iid_id -- which added zero semantic value to your code). Also, note the "on_delete" argument. Django does its own thing when it comes to cascading deletes, and on an interesting data model you don't want this -- and when it comes to views you'll just get an error and an aborted transaction. Prior to Django 1.5 you have to patch it to make DO_NOTHING actually mean "do nothing" -- otherwise it will still try to (needlessly) query and collect all related objects before going through its delete cycle, and the query will fail, halting the entire operation.
Incidentally, I wrote an in-depth explanation of how to do this just the other day.
You are trying to fetch records from a view. This is not correct as a view does not map to a model, a table maps to a model.
You should use Django ORM to fetch QryDescChar objects. Please note that Django ORM will fetch them directly from the table. You can consult Django docs for extra() and select_related() methods which will allow you to fetch related data (data you want to get from the other table) in different ways.