I'm using Django 3.1.3 and have a model with a FileField field. When I upload a file with same name, but different case, the logic I have is reporting that it is a duplicate. For example:
Files <QuerySet ['92/565/20191222_152213.jpg', '92/565/cs_731__DSC8110.jpg', '92/565/ADs-2.MP4']>, this 92/565/ADS-2.mp4
The logic is...
other_pieces_in_room = Piece.objects.filter(room__id=self.room_id)
other_files_in_room = other_pieces_in_room.values_list('file', flat=True)
mylog.debug("Files {}, this {}".format(other_files_in_room, file.name))
if file.name in other_files_in_room:
raise forms.ValidationError("...")
Model (relevant fields) is:
class Piece(models.Model):
file = models.FileField(upload_to=media_location)
room = models.ForeignKey(Room, on_delete=models.CASCADE)
Any thoughts as to what is going on?
For my case, the database was created with the default settings. The FileField in Django is stored in the database as a varchar and when doing:
show full columns from gallery_piece;
I could see that the field is using collation utf8_general_ci (which is case-insensitive).
I altered the database and all the char/varchar table fields to use utf8mb4 character set and utf8mb4_bin collation, using commands like:
ALTER TABLE gallery_exhibit MODIFY file varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
Once done, the filename (and other char type fields) are treating files as case sensitive. I did read that on file upload, the validation will be case-sensitive, so will pass, but then a database integrity error will be encountered, when saving an entry with only case differences using the default collation.
For production, when I create the database, I'll need to do:
CREATE DATABASE mydb
CHARACTER SET utf8mb4
COLLATE utf8mb4_bin;
Related
I am trying to use the Django CITextExtension PostgreSQL extension to create case-insensitive text fields in my database.
Concretely, here is how I am trying to use django.contrib.postgres.fields.CICharField in my Expression model to create a case-insensitive CharField in my database:
from django.contrib.postgres.fields import CICharField
class Expression(models.Model):
expression = CICharField(max_length=64, unique=True)
When I open the admin, however, and go into the Expressions area I get:
OperationalError at /admin/dictionary/expression/
no such column: dictionary_expression.expression
I could not find anything at all on the internet about this particular issue. Why is CICharField not creating a column in my database?
It could be because you are using a SQLite database. Make sure you are using a PostgreSQL database.
In my Django app, I want to insert a record with a composite primary key. Apparently this should be possible by making use of "unique_together". I'm quite sure this code was working in the past, but for some reason it does not seem to be working now. This code used to run on a Linux VM, and now I'm hosting it in Google App Engine. However I don't see how this can be the cause for this error.
class TermsAndConditionsDocument(models.Model):
organization = models.ForeignKey(Organization, on_delete=models.CASCADE, verbose_name=_("Organization"))
language = models.CharField(_('Language'),choices=LANGUAGE_CHOICES, max_length=5, help_text=_("The language of the content."))
content = models.TextField()
class Meta:
unique_together = ('organization', 'language')
The error:
IntegrityError at /transactions/settings/terms_and_conditions
null value in column "id" violates not-null constraint
DETAIL: Failing row contains (null, nl-BE, <p>B</p>, 10).
According to what I've read, using "unique_together" should cause Django to not need or include an ID as primary key. I checked the database, and the ID field DOES exist. I do not understand where the database constraint and the ID field are still coming from?
Apparently, as pointed out in the comments, a primary key "id" field is always added, even if you don't need it. It's supposed to get out of your way, so you don't even notice its existence. In my case, it required me to give it a value when I created a new record, which is not how things are supposed to work.
A while back I migrated this database from one Postgres database to another Postgres database. I used an SQL dump and load method for this. Some sequences seem to have been lost during that migration.
Because there are no sequences, some fields now lacked autoincrement capabilities, explaining the IntegrityError on insertion.
In order to fix this, I did the following:
1) Export the current data:
manage.py dumpdata > data.json
2) Drop your database and create a new empty one.
3) Run database migrations:
manage.py migrate
4) Load the data again, excluding some default data already recreated by Django.
manage.py loaddata --exclude auth.permission --exclude contenttypes data.json
This procedure seems to have recreated the sequences while also keeping the data.
The unique_together only creates a DB constraint (https://docs.djangoproject.com/en/2.2/ref/models/options/#unique-together).
You could create a custom primary key with the option primary_key https://docs.djangoproject.com/en/2.2/ref/models/fields/#django.db.models.Field.primary_key but you could only do that for one field.
But I suggest to just keep the auto increment id field, this works better with Django.
For the error are you saving a model? or doing a raw import?
I'm using Django for a project. I have a .sql file containing insert statements. How can I get them into my sqlite3 database and have them work with my models?
The project "fortune" has a models.py file that looks like the following:
class fortune(models.Model):
id = models.IntegerField(primary_key=True)
category = models.CharField(max_length=50)
length = models.IntegerField()
aphorism = models.CharField(max_length=5000)
I have a .sql file with a list of Insert statements like the follwing:
INSERT INTO "fortune_fortune" VALUES(1,'fortunes',127,'Arbitrary Text');
When I run .schema on my db.sqlite3 file which is configured with my project I see:
CREATE TABLE fortune_fortune(id integer, category varchar(50), length integer, aphorism varchar(5000));
I've tried using .read in my sqlite shell with no luck. I've tried typing "sqlite3 file.sqlite3 < file.sql" in bash as well. There is something I'm missing here, but I can't seem to ID the problem.
Thanks for your help.
ok, wait.. normally you dont use sql statements to insert data into db, if you work with django.
to insert data into db, you work with django ORM which is way much fun than these ugly sql statements.
fortune = fortune(category='new cat', length=19, aphorism='i love life')
fortune.save()
then as a result, you will have one new row in fortune table in your db. just read django docs and you feel happy!
and one more thing, class names are always in capital.
to your issue:
Django provides a hook for passing the database arbitrary SQL that’s executed just after the CREATE TABLE statements when you run migrate. You can use this hook to populate default records, or you could also create SQL functions, views, triggers, etc.
The hook is simple: Django just looks for a file called sql/<modelname>.sql, in your app directory, where <modelname> is the model’s name in lowercase.
more in docs
I created this field in my model:
numero_str = models.AutoField(primary_key=True, unique=True, default = 0)
The default value seems to invalidate the auto increment of AutoField but if I take it out I receive an error saying that the field can't be null. What I don't understand is: if I declareted it as a AutoField, isn't it supposed to generate a sequencial integer automatically? Or should I declare something when saving an item?
To be more especific, my app is basically a form that is send by e-mail and saved in a database. The error occur when sending (in case I take out the default value). It says:
IntegrityError at /solicitacaodetreinamento/
str_solicitacao.numero_str may not be NULL
I found a solution: I deleted the DB file and all the migration of South (include the Initial). Then I recreated the database and made the migrations. This time, using the default primary key, i.e., "id". In my case it was simpler because I had no real data at all (it is not in production), otherwise I would have to export the data, recreate the database and then import the data. Thank you all.
I have a fixture (json) which loads in development environment but fails to do so in server environment. The error says: "DatabaseError: value too long for type character varying(50)"
My development environment is Windows & Postgres 8.4. The server runs Debian and Postgres 8.3. Database encoding is UTF8 in both systems.
It is as if unicode markers in the fixture count as chars on the server and they cause some strings to exceed their field's max length. However that does not happen in the dev environment..
Update: the 50 char limit is now 255 in Django 1.8
--
Original answer:
I just encountered this this afternoon, too, and I have a fix (of sorts)
This post here implied it's a Django bug to do with length of the value allowed for auth_permission. Further digging backs up that idea, as does this Django ticket (even though it's initially MySQL-related).
It's basically that a permission name is created based on the verbose_name of a model plus a descriptive permission string, and that can overflow to more than the 50 chars allowed in auth.models.Permission.name.
To quote a comment on the Django ticket:
The longest prefixes for the string value in the column auth_permission.name are "Can change " and "Can delete ", both with 11 characters. The column maximum length is 50 so the maximum length of Meta.verbose_name is 39.
One solution would be to hack that column to support > 50 characters (ideally via a South migration, I say, so that it's easily repeatable) but the quickest, most reliable fix I could think of was simply to make my extra-long verbose_name definition a lot shorter (from 47 chars in the verbose_name to around 20). All works fine now.
Well, what makes the difference is the encoding of the template databases. On the production server they had ascii encoding while on the dev box it is utf-8.
By default postgres creates a database using the template1. My understanding is that if its encoding is not utf-8, then the database you create will have this issue, even though you create it with utf-8 encoding.
Therefore I dropped it and recreated it with its encoding set to UTF8. The snippet below does it (taken from here):
psql -U postgres
UPDATE pg_database SET datallowconn = TRUE where datname = 'template0';
\c template0
UPDATE pg_database SET datistemplate = FALSE where datname = 'template1';
drop database template1;
create database template1 with template = template0 encoding = 'UNICODE';
UPDATE pg_database SET datistemplate = TRUE where datname = 'template1';
\c template1
UPDATE pg_database SET datallowconn = FALSE where datname = 'template0';
Now the fixture loads smoothly.
Get the real SQL query on both systems and see what is different.
Just for information : I also had this error
DatabaseError: value too long for type character varying(10)
It seems that I was writing data over the limit of 10 for a field. I fixed it by increasing the size of a CharField from 10 to 20
I hope it helps
As #stevejalim says, it's quite possible that the column auth_permission.name is the problem with length 50, you verify this with \d+ auth_permission in postgres's shell. In my case this is the problema, thus when I load django models's fixtures I got “DatabaseError: value too long for type character varying(50)”, then change django.contrib.auth's Permission model is complicated, so ... the simple solution was perform a migrate on Permission model, I did this running ALTER TABLE auth_permission ALTER COLUMN name TYPE VARCHAR(100); command in postgres's shell, this works for me.
credits for this comment
You can make Django use longer fields for this model by monkey-patching the model prior to using it to create the database tables. In "manage.py", change:
if __name__ == "__main__":
execute_manager(settings)
to:
from django.contrib.auth.models import Permission
if __name__ == "__main__":
# Patch the field width to allow for our long model names
Permission._meta.get_field('name').max_length=200
Permission._meta.get_field('codename').max_length=200
execute_manager(settings)
This modifies the options on the field before (say) manage.py syncdb is run, so the databate table has nice wide varchar() fields. You don't need to do this when invoking your app, as you never attempt to modify the Permissions table whle running.