Can someone illuminate me as to exactly when an uploaded file is actually written to the location returned by "upload_to" in the FileField, in particular with regards to the order of field, model, and form validation and cleaning?
Right now I have a "clean" method on my model which assumes the uploaded file is in place, so it can do some validation on it. It looks like the file isn't yet saved, and may just be held in a temporary location or in memory. If that is the case, how do I "open" it or find a path to it if I need to execute some external process/program to validate the file?
Thanks,
Ian
The form cleansing has nothing to do with actually saving the file, or with saving any other data for that matter. The file isn't saved until to you run the save() method of the model instance (note that if you use ModelName.objects.create() this save() method is called for you automatically).
The bound form will contain an open File object, so you should be able to do any validation on that object directly. For example:
form = MyForm(request.POST, request.FILES)
if form.is_valid():
file_object = form.cleaned_data['myFile']
#run any validation on the file_object, or define a clean_myFile() method
# that will be run automatically when you call form.is_valid()
model_inst = MyModel('my_file' = file_object,
#assign other attributes here....
)
model_inst.save() #file is saved to disk here
What do you need to do on it? If your validation will work without a temporary file, you can access the data by calling read() on what your file field returns.
def clean_field(self):
_file = self.cleaned_data.get('filefield')
contents = _file.read()
If you do need it on the disk, you know where to go from here :) write it to a temporary location and do some magic on it!
Or write it as a custom form field. This is the basic idea how I go about verification of an MP3 file using the 'mutagen' library.
Notes:
first check the file size then if correct size write to tmp location.
Will write the file to temporary location specified in SETTINGS check its MP3 and then delete it.
The code:
from django import forms
import os
from mutagen.mp3 import MP3, HeaderNotFoundError, InvalidMPEGHeader
from django.conf import settings
class MP3FileField(forms.FileField):
def clean(self, *args, **kwargs):
super(MP3FileField, self).clean(*args, **kwargs)
tmp_file = args[0]
if tmp_file.size > 6600000:
raise forms.ValidationError("File is too large.")
file_path = getattr(settings,'FILE_UPLOAD_TEMP_DIR')+'/'+tmp_file.name
destination = open(file_path, 'wb+')
for chunk in tmp_file.chunks():
destination.write(chunk)
destination.close()
try:
audio = MP3(file_path)
if audio.info.length > 300:
os.remove(file_path)
raise forms.ValidationError("MP3 is too long.")
except (HeaderNotFoundError, InvalidMPEGHeader):
os.remove(file_path)
raise forms.ValidationError("File is not valid MP3 CBR/VBR format.")
os.remove(file_path)
return args
Related
I am looking to attach a file to an email which includes all the content a user inputs from a contact form. I currently refer a PDF which records their inputs, and I attach that PDF from a file destination. However, I do not know how to attach additional files which the user provides on the contact form. In this case, this is represented by "msg.attach_file(upload_file)." My thoughts are:
Have the file be uploaded to a destination; however, it needs to renamed to a uniform name each time so I can refer to it during the attachment process (msg.attach_file).
Figure out a way to use request.FILES to attach it immediately without having to worry about its file name or upload destination (I am not sure if msg.attach_file is a valid command for this method).
Is there a right way to perform this action? I am attempting to perform method 2 with my views.py file which refers to my forms.py file, but it is giving me an error.
Views.py
def quote_req(request):
submitted = False
if request.method == 'POST':
form = QuoteForm(request.POST, request.FILES)
company = request.POST['company']
contact_person = request.POST['contact_person']
upload_file = request.FILES['upload_file']
description = 'You have received a sales contact form'
if form.is_valid():
data_dict = {
'company_': str(company),
'contact_person_': str(contact_person),
}
write_fillable_pdf(INVOICE_TEMPLATE_PATH, INVOICE_OUTPUT_PATH, data_dict)
form.save()
# assert false
msg = EmailMessage('Contact Form', description, settings.EMAIL_HOST_USER, ['sample#mail.com'])
msg.attach_file('/uploads/file.pdf')
msg.attach_file(upload_file)
msg.send(fail_silently=False)
return HttpResponseRedirect('/quote/?submitted=True')
else:
form = QuoteForm()
if 'submitted' in request.GET:
submitted = True
Error Log
TypeError at /quote/
expected str, bytes or os.PathLike object, not InMemoryUploadedFile
Request Method: POST
Request URL: http://www.mytestingwebsitesample.com/quote/
Django Version: 2.1.3
Exception Type: TypeError
Exception Value:
expected str, bytes or os.PathLike object, not InMemoryUploadedFile
Can you try the following? Since InMemoryUploadedFile doesn't work, might have to process it first
upload_file = request.FILES['upload_file']
content = upload_file.read()
attachment = (upload_file.name, content, 'application/pdf')
# . . .
msg.attach(attachment)
upload_file.read() will return bytes. You might want to try using attach instead of attach_file. attach_file requires the file to be saved to your filesystem, while attach can take data. However, I believe that with attach, you should be able to use request.FILES['upload_file'] directly.
https://docs.djangoproject.com/en/2.2/topics/email/#emailmessage-objects
I have resolved my issue by employing a storage.py file that overwrites files with the same name; in my case, I am uploading each file, renaming it to a uniform name, and then having the storage file overwrite it later on rather than Django adding an extension to a file name with the same title.
In my model, I want to use a validator to analyze the content of a file, the thing I can not figure out is how to access the content of the file to parse through it as the file has not yet been saved (which is good) when the validators are running.
I'm not understanding how to get the data from the value passed to the validator into a file (I assume I should use tempfile) so I can then open it and evaluate the data.
Here's a simplified example, in my real code, I want to open the file and evaluate it with csv.
in Models.py
class ValidateFile(object):
....
def __call__(self, value):
# value is the fieldfile object but its not saved
# I believe I need to do something like:
temp_file = tempfile.TemporaryFile()
temp_file.write(value.read())
# Check the data in temp_file
....
class MyItems(models.Model):
data = models.FileField(upload_to=get_upload_path,
validators=[FileExtensionValidator(allowed_extensions=['cv']),
ValidateFile()])
Thanks for the help!
Take a look how this is done in the ImageField implementation:
So your ValidateFile class may be something like this:
from io import BytesIO
class ValidateFile(object):
def __call__(self, value):
if value is None:
#do something when None
return None
if hasattr(value, 'temporary_file_path'):
file = value.temporary_file_path()
else:
if hasattr(value, 'read'):
file = BytesIO(value.read())
else:
file = BytesIO(value['content'])
#Now validate your file
No need for tempfile:
The value passed to a FileField validator is an instance of FieldFile, as already mentioned by the OP.
Under the hood, the FieldFile instance might already use a tempfile.NamedTemporaryFile (source), or it might wrap an in-memory file, but you need not worry about that:
To "evaluate the data" you can simply treat the FieldFile instance as any Python file object.
For example, you could iterate over it:
def my_filefield_validator(value):
# note that value is a FieldFile instance
for line in value:
... # do something with line
The documentation says:
In addition to the API inherited from File such as read() and write(), FieldFile includes several methods that can be used to interact with the underlying file: ...
and the FieldFile class provides
... a wrapper around the result of the Storage.open() method, which may be a File object, or it may be a custom storage’s implementation of the File API.
An example of such an underlying file implementation is the InMemoryUploadedFile docs/source.
Also from the docs:
The File class is a thin wrapper around a Python file object with some Django-specific additions
Also note: class-based validators vs function-based validators
I have written an email form class in Django with a FileField. I want to check the uploaded file for its type via checking its mimetype. Subsequently, I want to limit file types to pdfs, word, and open office documents.
To this end, I have installed python-magic and would like to check file types as follows per the specs for python-magic:
mime = magic.Magic(mime=True)
file_mime_type = mime.from_file('address/of/file.txt')
However, recently uploaded files lack addresses on my server. I also do not know of any method of the mime object akin to "from_file_content" that checks for the mime type given the content of the file.
What is an effective way to use magic to verify file types of uploaded files in Django forms?
Stan described good variant with buffer. Unfortunately the weakness of this method is reading file to the memory. Another option is using temporary stored file:
import tempfile
import magic
with tempfile.NamedTemporaryFile() as tmp:
for chunk in form.cleaned_data['file'].chunks():
tmp.write(chunk)
print(magic.from_file(tmp.name, mime=True))
Also, you might want to check the file size:
if form.cleaned_data['file'].size < ...:
print(magic.from_buffer(form.cleaned_data['file'].read()))
else:
# store to disk (the code above)
Additionally:
Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later).
So you might want to handle it like so:
import os
tmp = tempfile.NamedTemporaryFile(delete=False)
try:
for chunk in form.cleaned_data['file'].chunks():
tmp.write(chunk)
print(magic.from_file(tmp.name, mime=True))
finally:
os.unlink(tmp.name)
tmp.close()
Also, you might want to seek(0) after read():
if hasattr(f, 'seek') and callable(f.seek):
f.seek(0)
Where uploaded data is stored
Why no trying something like that in your view :
m = magic.Magic()
m.from_buffer(request.FILES['my_file_field'].read())
Or use request.FILES in place of form.cleaned_data if django.forms.Form is really not an option.
mime = magic.Magic(mime=True)
attachment = form.cleaned_data['attachment']
if hasattr(attachment, 'temporary_file_path'):
# file is temporary on the disk, so we can get full path of it.
mime_type = mime.from_file(attachment.temporary_file_path())
else:
# file is on the memory
mime_type = mime.from_buffer(attachment.read())
Also, you might want to seek(0) after read():
if hasattr(f, 'seek') and callable(f.seek):
f.seek(0)
Example from Django code. Performed for image fields during validation.
You can use django-safe-filefield package to validate that uploaded file extension match it MIME-type.
from safe_filefield.forms import SafeFileField
class MyForm(forms.Form):
attachment = SafeFileField(
allowed_extensions=('xls', 'xlsx', 'csv')
)
In case you're handling a file upload and concerned only about images,
Django will set content_type for you (or rather for itself?):
from django.forms import ModelForm
from django.core.files import File
from django.db import models
class MyPhoto(models.Model):
photo = models.ImageField(upload_to=photo_upload_to, max_length=1000)
class MyForm(ModelForm):
class Meta:
model = MyPhoto
fields = ['photo']
photo = MyPhoto.objects.first()
photo = File(open('1.jpeg', 'rb'))
form = MyForm(files={'photo': photo})
if form.is_valid():
print(form.instance.photo.file.content_type)
It doesn't rely on content type provided by the user. But
django.db.models.fields.files.FieldFile.file is an undocumented
property.
Actually, initially content_type is set from the request, but when
the form gets validated, the value is updated.
Regarding non-images, doing request.FILES['name'].read() seems okay to me.
First, that's what Django does. Second, files larger than 2.5 Mb by default
are stored on a disk. So let me point you at the other answer
here.
For the curious, here's the stack trace that leads to updating
content_type:
django.forms.forms.BaseForm.is_valid: self.errors
django.forms.forms.BaseForm.errors: self.full_clean()
django.forms.forms.BaseForm.full_clean: self._clean_fields()
django.forms.forms.BaseForm._clean_fiels: field.clean()
django.forms.fields.FileField.clean: super().clean()
django.forms.fields.Field.clean: self.to_python()
django.forms.fields.ImageField.to_python
I'm experimenting with a site that will allow users to upload audio files. I've read every doc that I can get my hands on but can't find much about validating files.
Total newb here (never done any file validation of any kind before) and trying to figure this out. Can someone hold my hand and tell me what I need to know?
As always, thank you in advance.
You want to validate the file before it gets written to disk. When you upload a file, the form gets validated then the uploaded file gets passed to a handler/method that deals with the actual writing to the disk on your server. So in between these two operations, you want to perform some custom validation to make sure it's a valid audio file
You could:
check if the the file is less then a certain size (good practice)
then check if the submitted file has a certain content type (i.e. an audio file)
this is pretty useless as someone could easily spoof it
then check that the file ends in a certain extension (or extensions)
this is also pretty useless
try read the file and see if it's actually audio
(I haven't tested this code)
models.py
class UserSong(models.Model):
title = models.CharField(max_length=100)
audio_file = models.FileField()
forms.py
class UserSongForm(forms.ModelForm):
# Add some custom validation to our file field
def clean_audio_file(self):
file = self.cleaned_data.get('audio_file',False):
if file:
if file._size > 4*1024*1024:
raise ValidationError("Audio file too large ( > 4mb )")
if not file.content-type in ["audio/mpeg","audio/..."]:
raise ValidationError("Content-Type is not mpeg")
if not os.path.splitext(file.name)[1] in [".mp3",".wav" ...]:
raise ValidationError("Doesn't have proper extension")
# Here we need to now to read the file and see if it's actually
# a valid audio file. I don't know what the best library is to
# to do this
if not some_lib.is_audio(file.content):
raise ValidationError("Not a valid audio file")
return file
else:
raise ValidationError("Couldn't read uploaded file")
views.py
from utils import handle_uploaded_file
def upload_file(request):
if request.method == 'POST':
form = UserSongForm(request.POST, request.FILES)
if form.is_valid():
# If we are here, the above file validation has completed
# so we can now write the file to disk
handle_uploaded_file(request.FILES['file'])
return HttpResponseRedirect('/success/url/')
else:
form = UploadFileForm()
return render_to_response('upload.html', {'form': form})
utils.py
# from django's docs
def handle_uploaded_file(f):
ext = os.path.splitext(f.name)[1]
destination = open('some/file/name%s'%(ext), 'wb+')
for chunk in f.chunks():
destination.write(chunk)
destination.close()
https://docs.djangoproject.com/en/dev/topics/http/file-uploads/#file-uploads
https://docs.djangoproject.com/en/dev/ref/forms/fields/#filefield
https://docs.djangoproject.com/en/dev/ref/files/file/#django.core.files.File
I've got a model like this:
class Talk(BaseModel):
title = models.CharField(max_length=200)
mp3 = models.FileField(upload_to = u'talks/', max_length=200)
seconds = models.IntegerField(blank = True, null = True)
I want to validate before saving that the uploaded file is an MP3, like this:
def is_mp3(path_to_file):
from mutagen.mp3 import MP3
audio = MP3(path_to_file)
return not audio.info.sketchy
Once I'm sure I've got an MP3, I want to save the length of the talk in the seconds attribute, like this:
audio = MP3(path_to_file)
self.seconds = audio.info.length
The problem is, before saving, the uploaded file doesn't have a path (see this ticket, closed as wontfix), so I can't process the MP3.
I'd like to raise a nice validation error so that ModelForms can display a helpful error ("You idiot, you didn't upload an MP3" or something).
Any idea how I can go about accessing the file before it's saved?
p.s. If anyone knows a better way of validating files are MP3s I'm all ears - I also want to be able to mess around with ID3 data (set the artist, album, title and probably album art, so I need it to be processable by mutagen).
You can access the file data in request.FILES while in your view.
I think that best way is to bind uploaded files to a form, override the forms clean method, get the UploadedFile object from cleaned_data, validate it anyway you like, then override the save method and populate your models instance with information about the file and then save it.
a cleaner way to get the file before be saved is like this:
from django.core.exceptions import ValidationError
#this go in your class Model
def clean(self):
try:
f = self.mp3.file #the file in Memory
except ValueError:
raise ValidationError("A File is needed")
f.__class__ #this prints <class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
processfile(f)
and if we need a path, ther answer is in this other question
You could follow the technique used by ImageField where it validates the file header and then seeks back to the start of the file.
class ImageField(FileField):
# ...
def to_python(self, data):
f = super(ImageField, self).to_python(data)
# ...
# We need to get a file object for Pillow. We might have a path or we might
# have to read the data into memory.
if hasattr(data, 'temporary_file_path'):
file = data.temporary_file_path()
else:
if hasattr(data, 'read'):
file = BytesIO(data.read())
else:
file = BytesIO(data['content'])
try:
# ...
except Exception:
# Pillow doesn't recognize it as an image.
six.reraise(ValidationError, ValidationError(
self.error_messages['invalid_image'],
code='invalid_image',
), sys.exc_info()[2])
if hasattr(f, 'seek') and callable(f.seek):
f.seek(0)
return f