Error when testing the view with a file upload - django

I would like to test a file upload from my view with the following function:
def test_post(self):
with open("path/to/myfile/test_file.txt") as file:
post_data = {
'description': "Some important file",
'file': file,
}
response = self.client.post(self.test_url, post_data)
self.assertEqual(response.status_code, 302)
document = Document.objects.first()
self.assertEqual(document.description, "My File")
self.assertEqual(document.filename, 'test_file.txt')
When I test the file upload on the actual website, it works. But when I run this test, I get the following error:
django.core.exceptions.SuspiciousFileOperation: Storage can not find
an available filename for "WHJpMYuGGCMdSKFruieo/Documents/eWEjGvEojETTghSVCijsaCINZVxTvzpXNRBvmpjOrYvKAKCjYu/test_file_KTEMuQa.txt".
Please make sure that the corresponding file field allows sufficient
"max_length".
Here is my form save method:
def save(self, commit=True):
instance = super(DocumentForm, self).save(commit=False)
instance.filename = self.cleaned_data['file'].name
if commit:
instance.save() # error occurs here
return instance
Seeing as it works on the actual website, I suspect it has something to do with the way I setup the file in the test; probably something small.

For the sake of brevity, I had removed irrelevant model fields from my original question. But when Ahtisham requested to see the upload_to attribute (which had a custom function), I removed those irrelevant fields and it worked!
So this is my original code (which didn't work) with the irrelevant fields:
def documents_path(instance, filename):
grant = instance.grant
client = grant.client
return '{0}/Grant Documents/{1}/{2}'.format(client.folder_name, grant.name, filename)
....
file = models.FileField(upload_to=documents_path)
But this works:
def documents_path(instance, filename):
return 'Documents/{0}'.format(filename)
The reason why it worked on the actual website is because it wasn't using long characters from the test fixtures. Seems like those fields that I thought were irrelevant for the question were actually quite important!
TL;DR I decreased the length of the custom document path.

I had the same error with an ImageField. The solution was adding max_length=255 in models.py:
image = models.ImageField(upload_to=user_path, max_length=255)
Then running python manage.py makemigrations and python manage.py migrate.

Related

Dajngo CSV FIle not download ? When we have a large CSV file download its takes some time?Django 502 bad gateway nginx error Django

How can I download a large CSV file that shows me a 502 bad gateway error?
I get this solution I added in below.
Actually, in this, we use streaming references. In this concept for example we download a movie it's will download in the browser and show status when complete this will give the option to show in a folder same as that CSV file download completely this will show us.
There is one solution for resolving this error to increase nginx time but this is will affect cost so better way to use Django streaming. streaming is like an example when we add a movie for download it's downloading on the browser. This concept is used in Django streaming.
Write View for this in Django.
views.py
from django.http import StreamingHttpResponse
503_ERROR = 'something went wrong.'
DASHBOARD_URL = 'path'
def get_headers():
return ['field1', 'field2', 'field3']
def get_data(item):
return {
'field1': item.field1,
'field2': item.field2,
'field3': item.field3,
}
class CSVBuffer(object):
def write(self, value):
return value
class Streaming_CSV(generic.View):
model = Model_name
def get(self, request, *args, **kwargs):
try:
queryset = self.model.objects.filter(is_draft=False)
response = StreamingHttpResponse(streaming_content=(iter_items(queryset, CSVBuffer())), content_type='text/csv', )
file_name = 'Experience_data_%s' % (str(datetime.datetime.now()))
response['Content-Disposition'] = 'attachment;filename=%s.csv' % (file_name)
except Exception as e:
print(e)
messages.error(request, ERROR_503)
return redirect(DASHBOARD_URL)
return response
urls.py
path('streaming-csv/',views.Streaming_CSV.as_view(),name = 'streaming-csv')
For reference use the below links.
https://docs.djangoproject.com/en/4.0/howto/outputting-csv/#streaming-large-csv-files
GIT.
https://gist.github.com/niuware/ba19bbc0169039e89326e1599dba3a87
GIT
Adding rows manually to StreamingHttpResponse (Django)

Django: How to attach a file without referencing a file destination?

I am looking to attach a file to an email which includes all the content a user inputs from a contact form. I currently refer a PDF which records their inputs, and I attach that PDF from a file destination. However, I do not know how to attach additional files which the user provides on the contact form. In this case, this is represented by "msg.attach_file(upload_file)." My thoughts are:
Have the file be uploaded to a destination; however, it needs to renamed to a uniform name each time so I can refer to it during the attachment process (msg.attach_file).
Figure out a way to use request.FILES to attach it immediately without having to worry about its file name or upload destination (I am not sure if msg.attach_file is a valid command for this method).
Is there a right way to perform this action? I am attempting to perform method 2 with my views.py file which refers to my forms.py file, but it is giving me an error.
Views.py
def quote_req(request):
submitted = False
if request.method == 'POST':
form = QuoteForm(request.POST, request.FILES)
company = request.POST['company']
contact_person = request.POST['contact_person']
upload_file = request.FILES['upload_file']
description = 'You have received a sales contact form'
if form.is_valid():
data_dict = {
'company_': str(company),
'contact_person_': str(contact_person),
}
write_fillable_pdf(INVOICE_TEMPLATE_PATH, INVOICE_OUTPUT_PATH, data_dict)
form.save()
# assert false
msg = EmailMessage('Contact Form', description, settings.EMAIL_HOST_USER, ['sample#mail.com'])
msg.attach_file('/uploads/file.pdf')
msg.attach_file(upload_file)
msg.send(fail_silently=False)
return HttpResponseRedirect('/quote/?submitted=True')
else:
form = QuoteForm()
if 'submitted' in request.GET:
submitted = True
Error Log
TypeError at /quote/
expected str, bytes or os.PathLike object, not InMemoryUploadedFile
Request Method: POST
Request URL: http://www.mytestingwebsitesample.com/quote/
Django Version: 2.1.3
Exception Type: TypeError
Exception Value:
expected str, bytes or os.PathLike object, not InMemoryUploadedFile
Can you try the following? Since InMemoryUploadedFile doesn't work, might have to process it first
upload_file = request.FILES['upload_file']
content = upload_file.read()
attachment = (upload_file.name, content, 'application/pdf')
# . . .
msg.attach(attachment)
upload_file.read() will return bytes. You might want to try using attach instead of attach_file. attach_file requires the file to be saved to your filesystem, while attach can take data. However, I believe that with attach, you should be able to use request.FILES['upload_file'] directly.
https://docs.djangoproject.com/en/2.2/topics/email/#emailmessage-objects
I have resolved my issue by employing a storage.py file that overwrites files with the same name; in my case, I am uploading each file, renaming it to a uniform name, and then having the storage file overwrite it later on rather than Django adding an extension to a file name with the same title.

Populate model with metadata of file uploaded through django admin

I have two models,Foto and FotoMetadata. Foto just has one property called upload, that is an upload field. FotoMetadata has a few properties and should receive metadata from the foto uploaded at Foto. This can be done manually at the admin interface, but I want to do it automatically, i.e: when a photo is uploaded through admin interface, the FotoMetadata is automatically filled.
In my model.py I have a few classes, including Foto and FotoMetadata:
class Foto(models.Model):
upload = models.FileField(upload_to="fotos")
def __str__(self):
return '%s' %(self.upload)
class FotoMetadata(models.Model):
image_formats = (
('RAW', 'RAW'),
('JPG', 'JPG'),
)
date = models.DateTimeField()
camera = models.ForeignKey(Camera, on_delete=models.PROTECT)
format = models.CharField(max_length=8, choices=image_formats)
exposure = models.CharField(max_length=8)
fnumber = models.CharField(max_length=8)
iso = models.IntegerField()
foto = models.OneToOneField(
Foto,
on_delete=models.CASCADE,
primary_key=True,
)
When I login at the admin site, I have an upload form related to the Foto, and this is working fine. My problem is that I can't insert metadata at FotoMetadata on the go. I made a function that parse the photo and give me a dictionary with the info I need. This function is called GetExif is at a file called getexif.py. This will be a simplified version of it:
def GetExif(foto):
# Open image file for reading (binary mode)
f = open(foto, 'rb')
# Parse file
...
<parsing code>
...
f.close()
#create dictionary to receive data
meta={}
meta['date'] = str(tags['EXIF DateTimeOriginal'].values)
meta['fnumber'] = str(tags['EXIF FNumber'])
meta['exposure'] = str(tags['EXIF ExposureTime'])
meta['iso'] = str(tags['EXIF ISOSpeedRatings'])
meta['camera'] =str( tags['Image Model'].values)
return meta
So, basically, what I'm trying to do is use this function at admin.py to automatically populate the FotoMetadata when uploading a photo at Foto, but I really couldn't figure out how to make it. Does any one have a clue?
Edit 24/03/2016
Ok, after a lot more failures, I'm trying to use save_model in admin.py:
from django.contrib import admin
from .models import Autor, Camera, Lente, Foto, FotoMetadata
from fotomanager.local.getexif import GetExif
admin.site.register(Autor)
admin.site.register(Camera)
admin.site.register(Lente)
admin.site.register(FotoMetadata)
class FotoAdmin(admin.ModelAdmin):
def save_model(self, request, obj, form, change):
# populate the model
obj.save()
# get metadata
metadados = GetExif(obj.upload.url)
# Create instance of FotoMetadata
fotometa = FotoMetadata()
# FotoMetadata.id = Foto.id
fotometa.foto = obj.pk
# save exposure
fotometa.exposure = metadados['exposure']
admin.site.register(Foto, FotoAdmin)
I thought it would work, or that I will have problems saving data to the model, but actually I got stucked before this. I got this error:
Exception Type: FileNotFoundError
Exception Value:
[Errno 2] No such file or directory: 'http://127.0.0.1:8000/media/fotos/IMG_8628.CR2'
Exception Location: /home/ricardo/Desenvolvimento/fotosite/fotomanager/local/getexif.py in GetExif, line 24
My GetExif function can't read the file, however, the file path is right! If I copy and paste it to my browser, it downloads the file. I'm trying to figure out a way to correct the address, or to pass the internal path, or to pass the real file to the function instead of its path. I'm also thinking about a diferent way to access the file at GetExif() function too. Any idea of how to solve it?
Solution
I solved the problem above! By reading the FileField source, I've found a property called path, which solve the problem. I also made a few other modifications and the code is working. The class FotoAdmin, at admin.py is like this now:
class FotoAdmin(admin.ModelAdmin):
def save_model(self, request, obj, form, change):
# populate the model
obj.save()
# get metadata
metadados = GetExif(obj.upload.path)
# Create instance of FotoMetadata
fotometa = FotoMetadata()
# FotoMetadata.id = Foto.id
fotometa.foto = obj
# set and save exposure
fotometa.exposure = metadados['exposure']
fotometa.save()
I also had to set null=True at some properties in models.py and everything is working as it should.
I guess you want to enable post_save a signal
read : django signals
Activate the post_save signal - so after you save a FOTO you have a hook to do other stuff, in your case parse photometa and create a FotoMetadata instance.
More, if you want to save the foto only if fotometa succeed , or any other condition you may use , pre_save signal and save the foto only after meta foto was saved.

Tastypie not saving uploaded file

My application has a tastypie endpoint that received post requests with a file(not big...40-70kb) and is supposed to save the file and then process it. However, it seems the file never gets saved after upload and thus trying to retrieve the file after save fails. I have read in many places about it but it seems tastypie has no one standard way of doing it. Here are my models and resources:
models
class Tag(models.Model):
track = models.FileField(upload_to='tags/', max_length=250)
def __unicode__(self):
return self.track.url
api_resource
class TagResource(MultipartResource, ModelResource):
track = fields.FileField(attribute="track", null=True, blank=True)
class Meta:
queryset = Tag.objects.all()
resource_name = 'tag'
authorization = Authorization()
object_class = Tag
always_return_data = True
def dehydrate(self, bundle):
bundle.obj.save()
#the processing operation on the saved file
result = recognize(bundle.obj.track)
bundle.data['tag'] = result
return bundle
When I post a file with Curl:
curl -F "track=/path/to/track/track.mp3" http://127.0.0.1:8000/api/tag/
I get an error:
SuspiciousFileOperation: Attempted access to '/path/to/track/track.mp3'
and upon further investigation, i realized the error is being caused by the intended operation working on the url of the source file instead of the uploaded file's url.
What i'm I doing wrong??
I finally got it to work. Seems like I was not posting correctly using CURL. I now use this to upload the file:
curl -F "track=#/path/to/file.mp3" http://1**.**.***.**:8000/api/tag/
The '#' before the path was missing before. Adding that fixed it.

Processing file uploads before object is saved

I've got a model like this:
class Talk(BaseModel):
title = models.CharField(max_length=200)
mp3 = models.FileField(upload_to = u'talks/', max_length=200)
seconds = models.IntegerField(blank = True, null = True)
I want to validate before saving that the uploaded file is an MP3, like this:
def is_mp3(path_to_file):
from mutagen.mp3 import MP3
audio = MP3(path_to_file)
return not audio.info.sketchy
Once I'm sure I've got an MP3, I want to save the length of the talk in the seconds attribute, like this:
audio = MP3(path_to_file)
self.seconds = audio.info.length
The problem is, before saving, the uploaded file doesn't have a path (see this ticket, closed as wontfix), so I can't process the MP3.
I'd like to raise a nice validation error so that ModelForms can display a helpful error ("You idiot, you didn't upload an MP3" or something).
Any idea how I can go about accessing the file before it's saved?
p.s. If anyone knows a better way of validating files are MP3s I'm all ears - I also want to be able to mess around with ID3 data (set the artist, album, title and probably album art, so I need it to be processable by mutagen).
You can access the file data in request.FILES while in your view.
I think that best way is to bind uploaded files to a form, override the forms clean method, get the UploadedFile object from cleaned_data, validate it anyway you like, then override the save method and populate your models instance with information about the file and then save it.
a cleaner way to get the file before be saved is like this:
from django.core.exceptions import ValidationError
#this go in your class Model
def clean(self):
try:
f = self.mp3.file #the file in Memory
except ValueError:
raise ValidationError("A File is needed")
f.__class__ #this prints <class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
processfile(f)
and if we need a path, ther answer is in this other question
You could follow the technique used by ImageField where it validates the file header and then seeks back to the start of the file.
class ImageField(FileField):
# ...
def to_python(self, data):
f = super(ImageField, self).to_python(data)
# ...
# We need to get a file object for Pillow. We might have a path or we might
# have to read the data into memory.
if hasattr(data, 'temporary_file_path'):
file = data.temporary_file_path()
else:
if hasattr(data, 'read'):
file = BytesIO(data.read())
else:
file = BytesIO(data['content'])
try:
# ...
except Exception:
# Pillow doesn't recognize it as an image.
six.reraise(ValidationError, ValidationError(
self.error_messages['invalid_image'],
code='invalid_image',
), sys.exc_info()[2])
if hasattr(f, 'seek') and callable(f.seek):
f.seek(0)
return f