I'm using Imagekit to resize pictures on a website i'm developing with Django, I've used Imagekit before without any issues but I have a strange problem.
I am using S3 for Media/Static files. When I upload an image in admin and refresh the template serving the images i get ValueError at / I/O operation on closed file error page. Then if I refresh the page it loads fine without a problem.
I am using an ImageSpec field to resize images and using {{ image.thumbnail.url }} in my template.
I am very much a beginner with Django so looking for some direction. Why will it work on a page reload and not when initially uploaded?
Here are the relevant fields in my model.
class Media(models.Model):
image = models.ImageField(upload_to="media")
thumbnail = ImageSpecField([Adjust(sharpness=1.1), ResizeToFill(800, 650)],
source='image', format='JPEG', options={'quality': 100})
I am assuming it's a problem with S3. It's trying to access the resized image but the Imagekit thumbnail isn't created on fileupload, only when the page is loaded?
Ok, after some searching, this is the fix (and all now working) to solve the issue with storages which is the source of the problem.
from storages.backends.s3boto3 import S3Boto3Storage
from tempfile import SpooledTemporaryFile
class CustomS3Boto3Storage(S3Boto3Storage):
location = 'media'
file_overwrite = False
"""
This is our custom version of S3Boto3Storage that fixes a bug in
boto3 where the passed in file is closed upon upload.
From:
https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
https://github.com/boto/boto3/issues/929
https://github.com/matthewwithanm/django-imagekit/issues/391
"""
def _save(self, name, content):
"""
We create a clone of the content file as when this is passed to
boto3 it wrongly closes the file upon upload where as the storage
backend expects it to still be open
"""
# Seek our content back to the start
content.seek(0, os.SEEK_SET)
# Create a temporary file that will write to disk after a specified
# size. This file will be automatically deleted when closed by
# boto3 or after exiting the `with` statement if the boto3 is fixed
with SpooledTemporaryFile() as content_autoclose:
# Write our original content into our copy that will be closed by boto3
content_autoclose.write(content.read())
# Upload the object which will auto close the
# content_autoclose instance
return super(CustomS3Boto3Storage, self)._save(name, content_autoclose)```
Related
I cant seem to understand how is it possible that for GCS the authenticated URL shows a different image then the public URL ?
Im uploading the images via a python django script
def upload_to_cloud(blob_name, file_obj):
file_type = imghdr.what(file_obj)
blob_name = str(blob_name) + '.' + file_type # concatenate string to create 'file_name.format'
stats = storage.Blob(bucket=bucket, name=blob_name).exists(client) # check if logo with the same reg.nr exists
if stats is True: # if exists then delete before uploading new logo
storage.Blob(bucket=bucket, name=blob_name).delete()
blob = bucket.blob(blob_name)
blob.upload_from_file(file_obj=file_obj, content_type=f'image/{file_type}')
path = blob.public_url
return path
class CompanyProfile(SuccessMessageMixin, UpdateView): # TODO why company logo differs from the one in ads_list?
model = Company
form_class = CompanyProfileCreationForm
def form_valid(self, form):
"""
Check if user uploaded a new logo. If yes
then upload the new logo to google cloud
"""
if 'logo' in self.request.FILES:
blob_name = self.request.user.company.reg_nr # get company registration number
file_obj = self.request.FILES['logo'] # store uploaded file in variable
form.instance.logo_url = upload_to_cloud(blob_name, file_obj) # update company.logo_url with path to uploaded file
company = Company.objects.get(pk=self.request.user.company.pk)
company.save()
return super().form_valid(form)
else:
return super().form_valid(form)
Any ideas on what Im doing wrong and how its even possible? The file that I actually uploaded is the one under authenticated url. The file thats under public url is a file that I uploaded for a different blob
EDIT
Im adding screenshot of the different images because after some time the images appear to be the same as they should be. Some people are confused by this and comment that the images are the same after all
Public URL
Authenticated URL
Note that caching issue is ruled out since I sent the public URL to my friend and he also saw that the image is the HTML text although the image in the authenticated URL (the correct image) was a light bulb. He also noted that the URL preview in fb messenger showed the light bulb image but when he actually opened the URL the HTML text image appeared
This problem persists in case a file is uploaded with the same blob name. This happens regardless if its overwritten by gcs or if I previously execute blob delete function and then create a new file with the same name as the deleted blob.
In general the same object will be served by storage.googleapis.com and storage.cloud.google.com.
The only exception is if there is some caching (either in your browser, in a proxy, with Cloud CDN or in GCS). If you read the object via storage.cloud.google.com before uploading a new version, then reading after by storage.cloud.google.com may serve the old version while storage.googleapis.com returns the new one. Caching can also be location dependent.
If you can't allow an hour of caching, set Cache control to no-cache.
when running the Django server and hitting the URL http://127.0.0.1:8000/media/pictures/h2.jpg, I was getting the requested image (jpg).
Now I exchange the jpg by a file, which is also called h2.jpg but when I call the same URL again
it still shows the old picture.
How to handle that?
I need to do it automatically by the backend or somehow --> without user action
Django version 2.1.7
You can use this middleware
from django.utils.cache import add_never_cache_headers
class NoCachingMiddleware(object):
def process_response(self, request, response):
add_never_cache_headers(response)
return respons
from this question:
https://stackoverflow.com/a/13489175/11027652
So, now the new file has a timestamp included in the filename. By this I can first read all files available in the folder and then take the first one to create a NEW dynamic filepath.
I have an ImageField in one of my models so that users can upload an image. When a user submits an upload form, I want to verify that the file in question is a fully valid and displayable image.
I tried using PIL to verify that the image was in fact authentic, but using
from PIL import Image
Image.open(model.file)
Image.verify()
No matter what file I give it though, it always throws an exception.
Anyone know of an easy way to verify the file?
Good news, you don't need to do this:
class ImageField(upload_to=None, height_field=None, width_field=None, max_length=100, **options)
Inherits all attributes and methods from FileField, but also validates that the uploaded object is a valid image.
https://docs.djangoproject.com/en/1.10/ref/models/fields/#django.db.models.ImageField
Also, you should use verify() as follows:
from PIL import Image
im = Image.open(model.file)
im.verify()
you can use a 'Pillow' with 'try,except 'block, before insert image/data to a database or use it where you want,
like my following example for submit ticket support form , 'view.py' file :
from PIL import Image
if request.method=='POST':
# check if attachment file is not empty inside try/except to pass django error.
try:
ticket_attachmet_image = request.FILES["q_attachment_image"]
except:
ticket_attachmet_image = None
# check if uploaded image is valid (for example not video file ) .
if not ticket_attachmet_image == None:
try:
Image.open(ticket_attachmet_image)
except:
messages.warning(request, 'sorry, your image is invalid')
return redirect('your_url_name')
#done.
if 'image' in request.FILES['image'].content_type:
# Some code
else:
# the file is not image
The ImageField doesn't work when the form is not created by the form.py
In fact, if we upload a file, that is not an image, and save it in the image field, it wouldn't raise any error so, the content_type of the file must be checked before saving.
I'm developing a web application in Django, and one of its features is adding new articles with a photo.
My Article model class contains models.FileField. I use S3BotoStorage as DEFAULT_FILE_STORAGE (Amazon S3).
Users can add a photo to an article in two ways:
1) Upload a photo from disk (by using input type=file)
2) Paste URL to existing photo online
If users uses option 1), everything works. I get uploaded file in view from request.FILES dictionary and assing it to FileField in Article object. The photo is uploaded to S3.
But when user pastes URL to a photo, the first thing I have to do in view is to download this photo. I do it by using function:
def downloadPhotoFromURL(url):
try:
img = urllib.urlretrieve(url)[0]
return img
except Exception:
return None
Then I save this image to FileField in model, so my whole code responsible for downloading image, and uploading it to S3 is like this:
articleImg = downloadPhotoFromURL(url)
f = File(open(articleImg), 'rb')
newArticle.image.save('tmp', f)
In this situation, I cannot upload it to S3 and after 2 minutes I'm receiving BotoServerError: 400 Bad Request. Unfortunately I don't have any other information why this request is bad. Any idea what can be going wrong? By the time I saved an image to model, I had saved a model, so model exists when I try to upload photo to S3.
I want to store some mp3s in a folder which is not public, can't be directly accessed through the web and allow users to hear/download the songs with a browser only if they are logged in.
How can I do that?
I do my web development with Django, but if I know how it works is enough.
You first need to setup authentication. The django tutorials thoroughly explore this.
You don't' link the mp3's directly, You link to a django script that checks the auth, then reads the mp3 and serves it to the client with a mp3 content type header.
http://yourserver.com/listen?file=Fat+Boys+Greatest+Hits
I assume you use django. Then you can try something like this:
from django.conf import settings
from django.contrib.auth.decorators import login_required
from django.http import HttpResponse
#login_required
def listen(request, file_name):
# note that MP3_STORAGE should not be in MEDIA_ROOT
file = open("%smp3/%s" % (settings.MP3_STORAGE, file_name))
response = HttpResponse(file.read(), mimetype="audio/mpeg")
return response
Note that you will get dramatic speed decrease. Using generator to read file in blocks may help to save memory.
Lazy Method for Reading Big File in Python?
File outside of public access (not in
MEDIA_URL folders)
Check if user logged in
Serve files only via a view, with
unique links for every user
Pseudocode:
class Mp3(models.Model):
file = models.FileField(upload_to=path_outside_of_public_access)
hash = models.CharField(unique=True)
def generate_link_hash(request, file):
return hashlib.md5("%s_%i_%s_%s" % (request.session.session_key, file.id, str(file.date_added), file.hash)) # or however u like
def files_list(request)
""" view to show files list """
for file in files:
file.link_hash = generate_link_hash(request, file)
#login_required
def download_file(request, file_hash, link_hash):
""" view to download file """
file = Mp3.objects.get(hash=file_hash)
if link_hash == generate_link_hash(request, file):
file = open(file.file)
return HttpResponse(file.read(), mimetype="audio/mpeg")
else:
raise Http404
Should do the job enough, but remember - what is once accessed, you have no control where it goes from now on. And that every file download needs reading the file through the app (it's not given statically), which will affect the performance of your app.