Hello I'm having trouble here with multiple image with same field.
As far as I know in django tutorial they telling this.
for f in request.FILES.getlist('files'):
# do something (validate here maybe)
in which I don't quite get it. Like do i do manual validation? If so why?
Anyway there is another approach they give
files = forms.FileField(widget=ClearableFileInput(attrs={'multiple': True})
This one does not work in the way I want. It's self.cleaned_data['files'] only gives one output (There is a similar problem here) and django/multiupload was having a bug on my experience and sadly it was too slow to fix :(.
What I want was to validate each file and give errors to each via ImageField because I like it was validating a file versus I code it myself.
Thus I made a prototype code.
forms.py
class ImageForm(forms.Form):
# validate each image here
image = forms.ImageField()
class BaseForm(forms.Form):
# first form
ping = forms.CharField()
#images = SomeThingMultipleFileField that will raise multiple errors each validate image.
# since no option I decided to do that. below.
# so for decoration that images is required.
images = forms.ImageField()
def handle(self, request, *args, **kwargs):
#custom function
image_list = []
errors = []
# validate each image in images via another form
# if there is errors e.g this field is required will be append to errors = []
for image in request.FILES.getlist('images'):
data = ImageForm(image)
if data.is_valid():
image_list.append(data.cleaned_data['image'])
else:
errors.append(data.errors)
if errors:
# raise errors
# return the data
views.py
def base(request):
# this is an api
# expected input should be from the code or format
# {'ping': 'test', 'images': 1.jpg, 'images': 2.jpg}
# This is not the actual view code.
data = forms.BaseForm(request.POST, request.FILES)
if data.is_valid():
value = data.handle(request)
return JSONResponse({'data': value})
return JSONResponse({'errors': data.errors})
Not elegant to be honest but having trouble now and no more options I can think off but that.
The problem in my code is that
data = ImageForm(image)
does not reading the file thus image_list is always empty
So anyone can help me here?. Im stuck
Any better approach?
I wonder also for a general error like if one image is not valid it triggers like {'files': 'One of the images is not valid.'}
so far, I tested again so my bad
it seems it requires the format of data, files in ordinary forms.
in order to do so.
forms.py
... # previous code
# data = ImageForm(image) , old code
data = ImageForm({}, {'image': image})
in this way,it fills up the default QueryDict: {}, MultiValueDict in the args
Number 3 can be answered.
instead of
# previous code
else:
errors.append(error)
now should be
raise ValidationError(_('Your error'))
Any Better approach?
Not much I can think off sadly. So anyone stumble here. feel free to comment. Much appreciated for the help.
Related
I would like to test a file upload from my view with the following function:
def test_post(self):
with open("path/to/myfile/test_file.txt") as file:
post_data = {
'description': "Some important file",
'file': file,
}
response = self.client.post(self.test_url, post_data)
self.assertEqual(response.status_code, 302)
document = Document.objects.first()
self.assertEqual(document.description, "My File")
self.assertEqual(document.filename, 'test_file.txt')
When I test the file upload on the actual website, it works. But when I run this test, I get the following error:
django.core.exceptions.SuspiciousFileOperation: Storage can not find
an available filename for "WHJpMYuGGCMdSKFruieo/Documents/eWEjGvEojETTghSVCijsaCINZVxTvzpXNRBvmpjOrYvKAKCjYu/test_file_KTEMuQa.txt".
Please make sure that the corresponding file field allows sufficient
"max_length".
Here is my form save method:
def save(self, commit=True):
instance = super(DocumentForm, self).save(commit=False)
instance.filename = self.cleaned_data['file'].name
if commit:
instance.save() # error occurs here
return instance
Seeing as it works on the actual website, I suspect it has something to do with the way I setup the file in the test; probably something small.
For the sake of brevity, I had removed irrelevant model fields from my original question. But when Ahtisham requested to see the upload_to attribute (which had a custom function), I removed those irrelevant fields and it worked!
So this is my original code (which didn't work) with the irrelevant fields:
def documents_path(instance, filename):
grant = instance.grant
client = grant.client
return '{0}/Grant Documents/{1}/{2}'.format(client.folder_name, grant.name, filename)
....
file = models.FileField(upload_to=documents_path)
But this works:
def documents_path(instance, filename):
return 'Documents/{0}'.format(filename)
The reason why it worked on the actual website is because it wasn't using long characters from the test fixtures. Seems like those fields that I thought were irrelevant for the question were actually quite important!
TL;DR I decreased the length of the custom document path.
I had the same error with an ImageField. The solution was adding max_length=255 in models.py:
image = models.ImageField(upload_to=user_path, max_length=255)
Then running python manage.py makemigrations and python manage.py migrate.
I have something like this:
def upload_something(request):
data = {}
if request.FILES:
raw_file = request.FILES['myfile'].read()
else:
raw_file = request.raw_post_data
I can't seem to be able to write a unit-test that populates raw_post_data, how would I go about doing that? I basically just want to send an image file. I'm trying to create a test case for when I read raw_post_data and it errors with:
You cannot access raw_post_data after reading from request's data stream
I'm assuming you have figured this out by now, but as the answers are almost out of date with the deprecation of raw_post_data I thought i'd post.
def test_xml_payload(self):
data = '<?xml version="1.0" encoding="UTF-8"?><blah></blah>'
response = self.client.post(reverse('my_url'),
data=data,
content_type='application/xml')
def my_view(request):
xml = request.body
You can use mocking. Some examples available here and in docs here
Updated
Kit, I think it's very depends on your test case. But in general you shouldn't use raw_post_data directly. Instead it's have to be patched like in example below:
from mock import Mock, MagicMock
class SomeTestCase(TestCase):
def testRawPostData(self):
...
request = Mock(spec=request)
request.raw_post_data = 'myrawdata'
print request.raw_post_data # prints 'myrawdata'
file_mock = MagicMock(spec=file)
file_mock.read.return_value = 'myfiledata'
request.FILES = {'myfile': file_mock}
print request.FILES['myfile'].read() # prints 'myfiledata'
The error message the interpreter is giving is correct. After you access the POST data via if request.FILES, you can no longer access the raw_post_data. If in your actual code (not the tests) you hit that line, it would error with the same message. Basically, you need two separate views for form-based POSTS and direct file POSTS.
I took this listing here
c = Client()
f = open('wishlist.doc')
c.post('/customers/wishes/', {'name': 'fred', 'attachment': f})
f.close()
Client is a special class for testing your views. This is the example of posting files to your view. It's part of Django testing framework.
My goal is to limit a FileField on a Django ModelForm to PDFs and Word Documents. The answers I have googled all deal with creating a separate file handler, but I am not sure how to do so in the context of a ModelForm. Is there a setting in settings.py I may use to limit upload file types?
Create a validation method like:
def validate_file_extension(value):
if not value.name.endswith('.pdf'):
raise ValidationError(u'Error message')
and include it on the FileField validators like this:
actual_file = models.FileField(upload_to='uploaded_files', validators=[validate_file_extension])
Also, instead of manually setting which extensions your model allows, you should create a list on your setting.py and iterate over it.
Edit
To filter for multiple files:
def validate_file_extension(value):
import os
ext = os.path.splitext(value.name)[1]
valid_extensions = ['.pdf','.doc','.docx']
if not ext in valid_extensions:
raise ValidationError(u'File not supported!')
Validating with the extension of a file name is not a consistent way. For example I can rename a picture.jpg into a picture.pdf and the validation won't raise an error.
A better approach is to check the content_type of a file.
Validation Method
def validate_file_extension(value):
if value.file.content_type != 'application/pdf':
raise ValidationError(u'Error message')
Usage
actual_file = models.FileField(upload_to='uploaded_files', validators=[validate_file_extension])
An easier way of doing it is as below in your Form
file = forms.FileField(widget=forms.FileInput(attrs={'accept':'application/pdf'}))
Django since 1.11 has a FileExtensionValidator for this purpose:
class SomeDocument(Model):
document = models.FileFiled(validators=[
FileExtensionValidator(allowed_extensions=['pdf', 'doc'])])
As #savp mentioned, you will also want to customize the widget so that users can't select inappropriate files in the first place:
class SomeDocumentForm(ModelForm):
class Meta:
model = SomeDocument
widgets = {'document': FileInput(attrs={'accept': 'application/pdf,application/msword'})}
fields = '__all__'
You may need to fiddle with accept to figure out exactly what MIME types are needed for your purposes.
As others have mentioned, none of this will prevent someone from renaming badstuff.exe to innocent.pdf and uploading it through your form—you will still need to handle the uploaded file safely. Something like the python-magic library can help you determine the actual file type once you have the contents.
For a more generic use, I wrote a small class ExtensionValidator that extends Django's built-in RegexValidator. It accepts single or multiple extensions, as well as an optional custom error message.
class ExtensionValidator(RegexValidator):
def __init__(self, extensions, message=None):
if not hasattr(extensions, '__iter__'):
extensions = [extensions]
regex = '\.(%s)$' % '|'.join(extensions)
if message is None:
message = 'File type not supported. Accepted types are: %s.' % ', '.join(extensions)
super(ExtensionValidator, self).__init__(regex, message)
def __call__(self, value):
super(ExtensionValidator, self).__call__(value.name)
Now you can define a validator inline with the field, e.g.:
my_file = models.FileField('My file', validators=[ExtensionValidator(['pdf', 'doc', 'docx'])])
I use something along these lines (note, "pip install filemagic" is required for this...):
import magic
def validate_mime_type(value):
supported_types=['application/pdf',]
with magic.Magic(flags=magic.MAGIC_MIME_TYPE) as m:
mime_type=m.id_buffer(value.file.read(1024))
value.file.seek(0)
if mime_type not in supported_types:
raise ValidationError(u'Unsupported file type.')
You could probably also incorporate the previous examples into this - for example also check the extension/uploaded type (which might be faster as a primary check than magic.) This still isn't foolproof - but it's better, since it relies more on data in the file, rather than browser provided headers.
Note: This is a validator function that you'd want to add to the list of validators for the FileField model.
I find that the best way to check the type of a file is by checking its content type. I will also add that one the best place to do type checking is in form validation. I would have a form and a validation as follows:
class UploadFileForm(forms.Form):
file = forms.FileField()
def clean_file(self):
data = self.cleaned_data['file']
# check if the content type is what we expect
content_type = data.content_type
if content_type == 'application/pdf':
return data
else:
raise ValidationError(_('Invalid content type'))
The following documentation links can be helpful:
https://docs.djangoproject.com/en/3.1/ref/files/uploads/ and https://docs.djangoproject.com/en/3.1/ref/forms/validation/
I handle this by using a clean_[your_field] method on a ModelForm. You could set a list of acceptable file extensions in settings.py to check against in your clean method, but there's nothing built-into settings.py to limit upload types.
Django-Filebrowser, for example, takes the approach of creating a list of acceptable file extensions in settings.py.
Hope that helps you out.
I am trying to follow the code listed on https://github.com/alex/django-ajax-validation/blob/master/ajax_validation/views.py
I have been able to understand a small chunk of it. I have added comments stating my understanding of what is happening.
I would really appreciate some assistance on questions I listed in comments next to the lines I couldn't quite follow.
def validate(request, *args, **kwargs):
# I thing it is some sort of initializations but I cannot really understand what's happening
form_class = kwargs.pop('form_class')
defaults = {
'data': request.POST
}
extra_args_func = kwargs.pop('callback', lambda request, *args, **kwargs: {})
kwargs = extra_args_func(request, *args, **kwargs)
defaults.update(kwargs)
form = form_class(**defaults)
if form.is_valid(): #straightforward, if there is no error then the form is valid
data = {
'valid': True,
}
else:
# if we're dealing with a FormSet then walk over .forms to populate errors and formfields
if isinstance(form, BaseFormSet): #I cannot really understand what is BaseFromSet
errors = {}
formfields = {}
for f in form.forms: # I am guessing that this is for when there are multiple form submitted for validation
for field in f.fields.keys(): # I think he is looping over all fields and checking for error. what does add_prefix () return? and what is formfields[]?
formfields[f.add_prefix(field)] = f[field]
for field, error in f.errors.iteritems():
errors[f.add_prefix(field)] = error
if form.non_form_errors():
errors['__all__'] = form.non_form_errors() # what is the '__all__'?
else:
errors = form.errors
formfields = dict([(fieldname, form[fieldname]) for fieldname in form.fields.keys()])
# if fields have been specified then restrict the error list
if request.POST.getlist('fields'): # I am having a hard time understanding what this if statement does.
fields = request.POST.getlist('fields') + ['__all__']
errors = dict([(key, val) for key, val in errors.iteritems() if key in fields])
final_errors = {} # here the author of this code totally lost me.
for key, val in errors.iteritems():
if '__all__' in key:
final_errors[key] = val
elif not isinstance(formfields[key].field, forms.FileField):
html_id = formfields[key].field.widget.attrs.get('id') or formfields[key].auto_id
html_id = formfields[key].field.widget.id_for_label(html_id)
final_errors[html_id] = val
data = {
'valid': False or not final_errors,
'errors': final_errors,
}
json_serializer = LazyEncoder() # Why does the result have to be returned in json?
return HttpResponse(json_serializer.encode(data), mimetype='application/json')
validate = require_POST(validate) # a decorator that requires a post to submit
LazyEncoder
class LazyEncoder(JSONEncoder):
def default(self, obj):
if isinstance(obj, Promise):
return force_unicode(obj)
return obj
form_class = kwargs.pop('form_class')
This is simply pulling the keyword argument, form_class, that was passed in via the URL conf.
(r'^SOME/URL/$', 'ajax_validation.views.validate',
{'form_class': ContactForm}, # this keyword argument.
'contact_form_validate')
BaseFormSet is simply the formset class doing the work behind the scenes. When you don't know, search the source! grep -ri "baseformset" . It's an invaluable tool.
Take a look at at django.forms.formsets to see how formset_factory produces new "formset" classes based on the BaseFormSet, hence the factory part!
I am guessing that this is for when there are multiple form submitted for validation
Yes, that's exactly what a formset is for (dealing with multiple forms)
I think he is looping over all fields and checking for error. what does add_prefix () return? and what is formfields[]?
Yes, that would be looping through the field names.
add_prefix() is for prefixing form field names with a specific form. Because a formset repeats form elements multiple times, each field needs a unique prefix, such as 0-field1, 1-field1, etc.
formfields is just an empty dictionary defined a few lines above.
what is the 'all'?
__all__ is defined at the top of django.forms.forms
NON_FIELD_ERRORS = '__all__'
It's just what non field specific errors (such as constraints across 2 fields) are stored under in the errors dictionary as opposed to errors[fieldname].
I am having a hard time understanding what this if statement does.
The author has left a note:
# if fields have been specified then restrict the error list
if request.POST.getlist('fields'):
It's checking if you specified any specific fields to validate in your URLConf, this is not django but ajax_validation.
You can see that he's overwriting his errors dictionary based on only the fields specified, thus passing on the validation only for those fields.
errors = dict([(key, val) for key, val in errors.iteritems() if key in fields])
here the author of this code totally lost me.
The author has mapped a custom errors and fields dictionary to specific field names with prefixes, (as opposed to the usual FormSet with each form having its own errors dictionary, unaware of the formset itself) which he presumably uses in the AJAX response to validate all fields.
Normally, you can iterate over a formset and go through the errors on a form by form basis, but not so if you need to validate all of them through ajax.
The line pulling html_id should be straight forward most of the time, but it's there because form widgets CAN add interesting things to the end of the ID's based on whether or not the widget is a radio select for example.
From source comments :
# RadioSelect is represented by multiple <input type="radio"> fields,
# each of which has a distinct ID. The IDs are made distinct by a "_X"
# suffix, where X is the zero-based index of the radio field. Thus,
# the label for a RadioSelect should reference the first one ('_0').
Why does the result have to be returned in json?
Because it's an ajax request and javascript easily eats json.
2- could you go through these lines of code...
extra_args_func = kwargs.pop('callback', lambda request, *args, **kwargs: {})
Either return a keyword argument named 'callback' (which if passed in, is supposed to be a function that accepts request and return a dictionary), and if it wasn't, return a lambda function that only returns an empty dictionary.
I'm not sure what the specific use is for the extra context. You could use it to run arbitrary snippets of code without modifying or subclassing ajax_validation...
It might help you to run this code, and put a debugger breakpoint in somewhere so you can step through and examine the variables and methods. You can do this by simply putting this line where you want to break:
import pdb; pdb.set_trace()
and you will be dumped into the debugger in the console.
I've got a model like this:
class Talk(BaseModel):
title = models.CharField(max_length=200)
mp3 = models.FileField(upload_to = u'talks/', max_length=200)
seconds = models.IntegerField(blank = True, null = True)
I want to validate before saving that the uploaded file is an MP3, like this:
def is_mp3(path_to_file):
from mutagen.mp3 import MP3
audio = MP3(path_to_file)
return not audio.info.sketchy
Once I'm sure I've got an MP3, I want to save the length of the talk in the seconds attribute, like this:
audio = MP3(path_to_file)
self.seconds = audio.info.length
The problem is, before saving, the uploaded file doesn't have a path (see this ticket, closed as wontfix), so I can't process the MP3.
I'd like to raise a nice validation error so that ModelForms can display a helpful error ("You idiot, you didn't upload an MP3" or something).
Any idea how I can go about accessing the file before it's saved?
p.s. If anyone knows a better way of validating files are MP3s I'm all ears - I also want to be able to mess around with ID3 data (set the artist, album, title and probably album art, so I need it to be processable by mutagen).
You can access the file data in request.FILES while in your view.
I think that best way is to bind uploaded files to a form, override the forms clean method, get the UploadedFile object from cleaned_data, validate it anyway you like, then override the save method and populate your models instance with information about the file and then save it.
a cleaner way to get the file before be saved is like this:
from django.core.exceptions import ValidationError
#this go in your class Model
def clean(self):
try:
f = self.mp3.file #the file in Memory
except ValueError:
raise ValidationError("A File is needed")
f.__class__ #this prints <class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
processfile(f)
and if we need a path, ther answer is in this other question
You could follow the technique used by ImageField where it validates the file header and then seeks back to the start of the file.
class ImageField(FileField):
# ...
def to_python(self, data):
f = super(ImageField, self).to_python(data)
# ...
# We need to get a file object for Pillow. We might have a path or we might
# have to read the data into memory.
if hasattr(data, 'temporary_file_path'):
file = data.temporary_file_path()
else:
if hasattr(data, 'read'):
file = BytesIO(data.read())
else:
file = BytesIO(data['content'])
try:
# ...
except Exception:
# Pillow doesn't recognize it as an image.
six.reraise(ValidationError, ValidationError(
self.error_messages['invalid_image'],
code='invalid_image',
), sys.exc_info()[2])
if hasattr(f, 'seek') and callable(f.seek):
f.seek(0)
return f