I am learning VXML and Django. I am trying to find out how to cleanly retrieve a recording from some voice-xml (vxml) browser and pass it to the server side where I use django to further handle the passed information. Then I want to store the file somewhere in a .wav file to replay it later. I have the following code snippets:
In the VXML file:
<record name="recording" />
[here i record the recording]
<filled>
<submit next="/url/" method="post" namelist="recording"/>
</filled>
In the urls.py of django, I would have
url(r'^url$', view.index, name='index')
The views.index definition
def index(request):
_recording = [..retrieve .wav from request here]
_modelObject = ModelObject(recording= _recording)
_modelObject.save() #store recording in some database
return render(request, 'genericfile.xml', content_type='text/xml')
In the model.py I'd guess I would have a class like:
from django.db import model
class ModelObject(model.Models)
recording = [declare type of .wav file here]
How would I go about completing the steps in the [..] in a clean manner?
I didn't work with vxml before but look like you want to store both .xml format and .wav format.
So here is my solution in this case:
from django.db import model
class ModelObject(model.Models)
# Define a text filed or anything that can store long string
# of _recording var above.
recording = models.TextField()
def save(self, *args, **kwargs):
if self.recording:
# Convert vxml to wav and store to a file
pass
super(ModelObject, self).save(*args, **kwargs)
#property
def recording_wav(self):
if not self.recording:
return None
return 'path/to/file.wav'
Remember use post_delete signal to remove file.wav once an instance of ModelObject is deleted.
Related
I have two models,Foto and FotoMetadata. Foto just has one property called upload, that is an upload field. FotoMetadata has a few properties and should receive metadata from the foto uploaded at Foto. This can be done manually at the admin interface, but I want to do it automatically, i.e: when a photo is uploaded through admin interface, the FotoMetadata is automatically filled.
In my model.py I have a few classes, including Foto and FotoMetadata:
class Foto(models.Model):
upload = models.FileField(upload_to="fotos")
def __str__(self):
return '%s' %(self.upload)
class FotoMetadata(models.Model):
image_formats = (
('RAW', 'RAW'),
('JPG', 'JPG'),
)
date = models.DateTimeField()
camera = models.ForeignKey(Camera, on_delete=models.PROTECT)
format = models.CharField(max_length=8, choices=image_formats)
exposure = models.CharField(max_length=8)
fnumber = models.CharField(max_length=8)
iso = models.IntegerField()
foto = models.OneToOneField(
Foto,
on_delete=models.CASCADE,
primary_key=True,
)
When I login at the admin site, I have an upload form related to the Foto, and this is working fine. My problem is that I can't insert metadata at FotoMetadata on the go. I made a function that parse the photo and give me a dictionary with the info I need. This function is called GetExif is at a file called getexif.py. This will be a simplified version of it:
def GetExif(foto):
# Open image file for reading (binary mode)
f = open(foto, 'rb')
# Parse file
...
<parsing code>
...
f.close()
#create dictionary to receive data
meta={}
meta['date'] = str(tags['EXIF DateTimeOriginal'].values)
meta['fnumber'] = str(tags['EXIF FNumber'])
meta['exposure'] = str(tags['EXIF ExposureTime'])
meta['iso'] = str(tags['EXIF ISOSpeedRatings'])
meta['camera'] =str( tags['Image Model'].values)
return meta
So, basically, what I'm trying to do is use this function at admin.py to automatically populate the FotoMetadata when uploading a photo at Foto, but I really couldn't figure out how to make it. Does any one have a clue?
Edit 24/03/2016
Ok, after a lot more failures, I'm trying to use save_model in admin.py:
from django.contrib import admin
from .models import Autor, Camera, Lente, Foto, FotoMetadata
from fotomanager.local.getexif import GetExif
admin.site.register(Autor)
admin.site.register(Camera)
admin.site.register(Lente)
admin.site.register(FotoMetadata)
class FotoAdmin(admin.ModelAdmin):
def save_model(self, request, obj, form, change):
# populate the model
obj.save()
# get metadata
metadados = GetExif(obj.upload.url)
# Create instance of FotoMetadata
fotometa = FotoMetadata()
# FotoMetadata.id = Foto.id
fotometa.foto = obj.pk
# save exposure
fotometa.exposure = metadados['exposure']
admin.site.register(Foto, FotoAdmin)
I thought it would work, or that I will have problems saving data to the model, but actually I got stucked before this. I got this error:
Exception Type: FileNotFoundError
Exception Value:
[Errno 2] No such file or directory: 'http://127.0.0.1:8000/media/fotos/IMG_8628.CR2'
Exception Location: /home/ricardo/Desenvolvimento/fotosite/fotomanager/local/getexif.py in GetExif, line 24
My GetExif function can't read the file, however, the file path is right! If I copy and paste it to my browser, it downloads the file. I'm trying to figure out a way to correct the address, or to pass the internal path, or to pass the real file to the function instead of its path. I'm also thinking about a diferent way to access the file at GetExif() function too. Any idea of how to solve it?
Solution
I solved the problem above! By reading the FileField source, I've found a property called path, which solve the problem. I also made a few other modifications and the code is working. The class FotoAdmin, at admin.py is like this now:
class FotoAdmin(admin.ModelAdmin):
def save_model(self, request, obj, form, change):
# populate the model
obj.save()
# get metadata
metadados = GetExif(obj.upload.path)
# Create instance of FotoMetadata
fotometa = FotoMetadata()
# FotoMetadata.id = Foto.id
fotometa.foto = obj
# set and save exposure
fotometa.exposure = metadados['exposure']
fotometa.save()
I also had to set null=True at some properties in models.py and everything is working as it should.
I guess you want to enable post_save a signal
read : django signals
Activate the post_save signal - so after you save a FOTO you have a hook to do other stuff, in your case parse photometa and create a FotoMetadata instance.
More, if you want to save the foto only if fotometa succeed , or any other condition you may use , pre_save signal and save the foto only after meta foto was saved.
In Django REST Framework (DRF), how do I support de-Serializing base64 encoded binary data?
I have a model:
class MyModel(Model):
data = models.FileField(...)
and I want to be able to send this data as base64 encoded rather than having to multi-part form data or a "File Upload". Looking at the Parsers, only FileUploadParser and MultiPartParser seem to parse out the files.
I would like to be able to send this data in something like JSON (ie send the binary data in the data rather than the files:
{
'data':'...'
}
I solved it by creating a new Parser:
def get_B64_JSON_Parser(fields):
class Impl(parsers.JSONParser):
media_type = 'application/json+b64'
def parse(self, *args, **kwargs):
ret = super(Impl, self).parse(*args, **kwargs)
for field in fields:
ret[field] = SimpleUploadedFile(name=field, content=ret[field].decode('base64'))
return ret
return Impl
which I then use in the View:
class TestModelViewSet(viewsets.ModelViewSet):
parser_classes = [get_B64_JSON_Parser(('data_file',)),]
This is an old question, but for those looking for an up-to-date solution, there is a plugin for DRF (drf_base64) that handles this situation. It allows reading files encoded as base64 strings in the JSON request.
So given a model like:
class MyModel(Model):
data = models.FileField(...)
and an expected json like:
{
"data": "data:image/png;base64,iVBORw0KGgoA ....",
...
}
The (des) serialization can be handled just importing from drf_base modules instead of the drf itself.
from drf_base64.serializers import ModelSerializer
from .models import MyModel
class MyModel(ModelSerializer):
class Meta:
model = MyModel
Just remember that is posible to get a base64 encoded file in javascript with the FileReader API.
There's probably something clever you can do at the serialiser level but the first thing that comes to mind is to do it in the view.
Step 1: Write the file. Something like:
fh = open("/path/to/media/folder/fileToSave.ext", "wb")
fh.write(fileData.decode('base64'))
fh.close()
Step 2: Set the file on the model. Something like:
instance = self.get_object()
instance.file_field.name = 'folder/fileToSave.ext' # `file_field` was `data` in your example
instance.save()
Note the absolute path at Step 1 and the path relative to the media folder at Step 2.
This should at least get you going.
Ideally you'd specify this as a serialiser field and get validation and auto-assignment to the model instance for free. But that seems complicated at first glance.
I have a web app that needs to do the following:
Present a form to request a client side file for CSV import.
Validate the data in the CSV file or ask for another filename.
At one point, I was doing the CSV data validation in the view, after the form.is_valid() call from getting the filename (i.e. I have the imported CSV file into memory in a dictionary using csv.DictReader). After running into problems trying to pass errors back to the original form, I'm now trying to validate the CONTENTS of the CSV file in the form's clean() method.
I'm currently stumped on how to access the in memory file from clean() as the request.FILES object isn't valid. Note that I have no problems presenting the form to the client browser and then manipulating the resulting CSV file. The real issue is how to validate the contents of the CSV file - if I assume the data format is correct I can import it to my models. I'll post my forms.py file to show where I currently am after moving the code from the view to the form:
forms.py
import csv
from django import forms
from io import TextIOWrapper
class CSVImportForm(forms.Form):
filename = forms.FileField(label='Select a CSV file to import:',)
def clean(self):
cleaned_data = super(CSVImportForm, self).clean()
f = TextIOWrapper(request.FILES['filename'].file, encoding='ASCII')
result_csvlist = csv.DictReader(f)
# first line (only) contains additional information about the event
# let's validate that against its form definition
event_info = next(result_csvlist)
f_eventinfo = ResultsForm(event_info)
if not f_eventinfo.is_valid():
raise forms.ValidationError("Error validating 1st line of data (after header) in CSV")
return cleaned_data
class ResultsForm(forms.Form):
RESULT_CHOICES = (('Won', 'Won'),
('Lost', 'Lost'),
('Tie', 'Tie'),
('WonByForfeit', 'WonByForfeit'),
('LostByForfeit', 'LostByForfeit'))
Team1 = forms.CharField(min_length=10, max_length=11)
Team2 = forms.CharField(min_length=10, max_length=11)
Result = forms.ChoiceField(choices=RESULT_CHOICES)
Score = forms.CharField()
Event = forms.CharField()
Venue = forms.CharField()
Date = forms.DateField()
Div = forms.CharField()
Website = forms.URLField(required=False)
TD = forms.CharField(required=False)
I'd love input on what's the "best" method to validate the contents of an uploaded CSV file and present that information back to the client browser!
I assume that when you want to access that file is in this line inside the clean method:
f = TextIOWrapper(request.FILES['filename'].file, encoding='ASCII')
You can't use that line because request doesn't exist but you can access your form's fields so you can try this instead:
f = TextIOWrapper(self.cleaned_data.get('filename'), encoding='ASCII')
Since you have done super.clean in the first line in your method, that should work. Then, if you want to add custom error message to you form you can do it like this:
from django.forms.util import ErrorList
errors = form._errors.setdefault("filename", ErrorList())
errors.append(u"CSV file incorrect")
Hope it helps.
How can I restrict FileField to only accept a certain type of file (video, audio, pdf, etc.) in an elegant way, server-side?
One very easy way is to use a custom validator.
In your app's validators.py:
def validate_file_extension(value):
import os
from django.core.exceptions import ValidationError
ext = os.path.splitext(value.name)[1] # [0] returns path+filename
valid_extensions = ['.pdf', '.doc', '.docx', '.jpg', '.png', '.xlsx', '.xls']
if not ext.lower() in valid_extensions:
raise ValidationError('Unsupported file extension.')
Then in your models.py:
from .validators import validate_file_extension
... and use the validator for your form field:
class Document(models.Model):
file = models.FileField(upload_to="documents/%Y/%m/%d", validators=[validate_file_extension])
See also: How to limit file types on file uploads for ModelForms with FileFields?.
Warning
For securing your code execution environment from malicious media files
Use Exif libraries to properly validate the media files.
Separate your media files from your application code
execution environment
If possible use solutions like S3, GCS, Minio or
anything similar
When loading media files on client side, use client native methods (for example if you are loading the media files non securely in a
browser, it may cause execution of "crafted" JavaScript code)
Django in version 1.11 has a newly added FileExtensionValidator for model fields, the docs is here: https://docs.djangoproject.com/en/dev/ref/validators/#fileextensionvalidator.
An example of how to validate a file extension:
from django.core.validators import FileExtensionValidator
from django.db import models
class MyModel(models.Model):
pdf_file = models.FileField(
upload_to="foo/", validators=[FileExtensionValidator(allowed_extensions=["pdf"])]
)
Note that this method is not safe. Citation from Django docs:
Don’t rely on validation of the file extension to determine a file’s
type. Files can be renamed to have any extension no matter what data
they contain.
There is also new validate_image_file_extension (https://docs.djangoproject.com/en/dev/ref/validators/#validate-image-file-extension) for validating image extensions (using Pillow).
A few people have suggested using python-magic to validate that the file actually is of the type you are expecting to receive. This can be incorporated into the validator suggested in the accepted answer:
import os
import magic
from django.core.exceptions import ValidationError
def validate_is_pdf(file):
valid_mime_types = ['application/pdf']
file_mime_type = magic.from_buffer(file.read(1024), mime=True)
if file_mime_type not in valid_mime_types:
raise ValidationError('Unsupported file type.')
valid_file_extensions = ['.pdf']
ext = os.path.splitext(file.name)[1]
if ext.lower() not in valid_file_extensions:
raise ValidationError('Unacceptable file extension.')
This example only validates a pdf, but any number of mime-types and file extensions can be added to the arrays.
Assuming you saved the above in validators.py you can incorporate this into your model like so:
from myapp.validators import validate_is_pdf
class PdfFile(models.Model):
file = models.FileField(upload_to='pdfs/', validators=(validate_is_pdf,))
You can use the below to restrict filetypes in your Form
file = forms.FileField(widget=forms.FileInput(attrs={'accept':'application/pdf'}))
There's a Django snippet that does this:
import os
from django import forms
class ExtFileField(forms.FileField):
"""
Same as forms.FileField, but you can specify a file extension whitelist.
>>> from django.core.files.uploadedfile import SimpleUploadedFile
>>>
>>> t = ExtFileField(ext_whitelist=(".pdf", ".txt"))
>>>
>>> t.clean(SimpleUploadedFile('filename.pdf', 'Some File Content'))
>>> t.clean(SimpleUploadedFile('filename.txt', 'Some File Content'))
>>>
>>> t.clean(SimpleUploadedFile('filename.exe', 'Some File Content'))
Traceback (most recent call last):
...
ValidationError: [u'Not allowed filetype!']
"""
def __init__(self, *args, **kwargs):
ext_whitelist = kwargs.pop("ext_whitelist")
self.ext_whitelist = [i.lower() for i in ext_whitelist]
super(ExtFileField, self).__init__(*args, **kwargs)
def clean(self, *args, **kwargs):
data = super(ExtFileField, self).clean(*args, **kwargs)
filename = data.name
ext = os.path.splitext(filename)[1]
ext = ext.lower()
if ext not in self.ext_whitelist:
raise forms.ValidationError("Not allowed filetype!")
#-------------------------------------------------------------------------
if __name__ == "__main__":
import doctest, datetime
doctest.testmod()
First. Create a file named formatChecker.py inside the app where the you have the model that has the FileField that you want to accept a certain file type.
This is your formatChecker.py:
from django.db.models import FileField
from django.forms import forms
from django.template.defaultfilters import filesizeformat
from django.utils.translation import ugettext_lazy as _
class ContentTypeRestrictedFileField(FileField):
"""
Same as FileField, but you can specify:
* content_types - list containing allowed content_types. Example: ['application/pdf', 'image/jpeg']
* max_upload_size - a number indicating the maximum file size allowed for upload.
2.5MB - 2621440
5MB - 5242880
10MB - 10485760
20MB - 20971520
50MB - 5242880
100MB 104857600
250MB - 214958080
500MB - 429916160
"""
def __init__(self, *args, **kwargs):
self.content_types = kwargs.pop("content_types")
self.max_upload_size = kwargs.pop("max_upload_size")
super(ContentTypeRestrictedFileField, self).__init__(*args, **kwargs)
def clean(self, *args, **kwargs):
data = super(ContentTypeRestrictedFileField, self).clean(*args, **kwargs)
file = data.file
try:
content_type = file.content_type
if content_type in self.content_types:
if file._size > self.max_upload_size:
raise forms.ValidationError(_('Please keep filesize under %s. Current filesize %s') % (filesizeformat(self.max_upload_size), filesizeformat(file._size)))
else:
raise forms.ValidationError(_('Filetype not supported.'))
except AttributeError:
pass
return data
Second. In your models.py, add this:
from formatChecker import ContentTypeRestrictedFileField
Then instead of using 'FileField', use this 'ContentTypeRestrictedFileField'.
Example:
class Stuff(models.Model):
title = models.CharField(max_length=245)
handout = ContentTypeRestrictedFileField(upload_to='uploads/', content_types=['video/x-msvideo', 'application/pdf', 'video/mp4', 'audio/mpeg', ],max_upload_size=5242880,blank=True, null=True)
Those are the things you have to when you want to only accept a certain file type in FileField.
after I checked the accepted answer, I decided to share a tip based on Django documentation. There is already a validator for use to validate file extension. You don't need to rewrite your own custom function to validate whether your file extension is allowed or not.
https://docs.djangoproject.com/en/3.0/ref/validators/#fileextensionvalidator
Warning
Don’t rely on validation of the file extension to determine a file’s
type. Files can be renamed to have any extension no matter what data
they contain.
I think you would be best suited using the ExtFileField that Dominic Rodger specified in his answer and python-magic that Daniel Quinn mentioned is the best way to go. If someone is smart enough to change the extension at least you will catch them with the headers.
You can define a list of accepted mime types in settings and then define a validator which uses python-magic to detect the mime-type and raises ValidationError if the mime-type is not accepted. Set that validator on the file form field.
The only problem is that sometimes the mime type is application/octet-stream, which could correspond to different file formats. Did someone of you overcome this issue?
Additionally i Will extend this class with some extra behaviour.
class ContentTypeRestrictedFileField(forms.FileField):
...
widget = None
...
def __init__(self, *args, **kwargs):
...
self.widget = forms.ClearableFileInput(attrs={'accept':kwargs.pop('accept', None)})
super(ContentTypeRestrictedFileField, self).__init__(*args, **kwargs)
When we create instance with param accept=".pdf,.txt", in popup with file structure as a default we will see files with passed extension.
Just a minor tweak to #Thismatters answer since I can't comment. According to the README of python-magic:
recommend using at least the first 2048 bytes, as less can produce incorrect identification
So changing 1024 bytes to 2048 to read the contents of the file and get the mime type base from that can give the most accurate result, hence:
def validate_extension(file):
valid_mime_types = ["application/pdf", "image/jpeg", "image/png", "image/jpg"]
file_mime_type = magic.from_buffer(file.read(2048), mime=True) # Changed this to 1024 to 2048
if file_mime_type not in valid_mime_types:
raise ValidationError("Unsupported file type.")
valid_file_extensions = [".pdf", ".jpeg", ".png", ".jpg"]
ext = os.path.splitext(file.name)[1]
if ext.lower() not in valid_file_extensions:
raise ValidationError("Unacceptable file extension.")
I've got a model like this:
class Talk(BaseModel):
title = models.CharField(max_length=200)
mp3 = models.FileField(upload_to = u'talks/', max_length=200)
seconds = models.IntegerField(blank = True, null = True)
I want to validate before saving that the uploaded file is an MP3, like this:
def is_mp3(path_to_file):
from mutagen.mp3 import MP3
audio = MP3(path_to_file)
return not audio.info.sketchy
Once I'm sure I've got an MP3, I want to save the length of the talk in the seconds attribute, like this:
audio = MP3(path_to_file)
self.seconds = audio.info.length
The problem is, before saving, the uploaded file doesn't have a path (see this ticket, closed as wontfix), so I can't process the MP3.
I'd like to raise a nice validation error so that ModelForms can display a helpful error ("You idiot, you didn't upload an MP3" or something).
Any idea how I can go about accessing the file before it's saved?
p.s. If anyone knows a better way of validating files are MP3s I'm all ears - I also want to be able to mess around with ID3 data (set the artist, album, title and probably album art, so I need it to be processable by mutagen).
You can access the file data in request.FILES while in your view.
I think that best way is to bind uploaded files to a form, override the forms clean method, get the UploadedFile object from cleaned_data, validate it anyway you like, then override the save method and populate your models instance with information about the file and then save it.
a cleaner way to get the file before be saved is like this:
from django.core.exceptions import ValidationError
#this go in your class Model
def clean(self):
try:
f = self.mp3.file #the file in Memory
except ValueError:
raise ValidationError("A File is needed")
f.__class__ #this prints <class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
processfile(f)
and if we need a path, ther answer is in this other question
You could follow the technique used by ImageField where it validates the file header and then seeks back to the start of the file.
class ImageField(FileField):
# ...
def to_python(self, data):
f = super(ImageField, self).to_python(data)
# ...
# We need to get a file object for Pillow. We might have a path or we might
# have to read the data into memory.
if hasattr(data, 'temporary_file_path'):
file = data.temporary_file_path()
else:
if hasattr(data, 'read'):
file = BytesIO(data.read())
else:
file = BytesIO(data['content'])
try:
# ...
except Exception:
# Pillow doesn't recognize it as an image.
six.reraise(ValidationError, ValidationError(
self.error_messages['invalid_image'],
code='invalid_image',
), sys.exc_info()[2])
if hasattr(f, 'seek') and callable(f.seek):
f.seek(0)
return f