Renaming django FileField files - django

Some additional features were added to a django application, and as a result the upload_to function also expanded.
Since django by default stores filenames in the database, and files on disk, no harm has been done - new files are named using a new upload_to function, while old files continue to work.
However, this is messy - we end up in a situation with files like
/media/userID/oldfilename.pdf
and
/media/app/userID/projectID/newfilename.pdf
Is there a way to bulk rename those files? I guess it could be done by iterating through the database, checking if the path in FileField matches the result of current upload_to, and if not, rename.. it seems like a common problem, so perhaps there is a more generic way out there?

The simple solution is to write a custom Django management command. You can run the command using Django's standard manage.py.
Something like this:
from django.core.management.base import BaseCommand, CommandError
from example.models import YourModel
class Command(BaseCommand):
args = ''
help = ''
def handle(self, *args, **options):
# Get all objects
objects = YourModel.objects.all()
for object in objects: # For each object
# If old file path:
if not 'userID/projectID' in objects.filefield.name:
# Move the file, eg usign shutil http://docs.python.org/library/shutil.html#shutil.move
# Update filefield
# save object

Related

Dynamic image upload/browsing path for django-tinymce

I would like to use tinyMCE as the editor for my django application, but have run into some trouble. I have everything setup, but it appears there is no way to specify the upload path for the image insert/upload function. I have two specific scenarios where this becomes a problem:
File browser for user submitted content should not show files uploaded by other users. I'd like to "jail" them to a specific directory by user id.
I need a way to link uploaded files with the object they are related to. That way I can prune those files in the future if the object no longer exists, and I can show only images that are related to that object in the filebrowser.
django-tinymce-filebrowser automatically sets the upload path to mce_filebrowser/%Y/%m/%d. There doesn't appear to be an option to change this path in any way.
django-filebrowser has options for setting the upload directory in settings.py, but I haven't been able to find any record of someone overriding that path for a specific modelform instance. The closest I found was Django filebrowser, model specific directory parameter for FileBrowserField, but I think the solution there isn't applicable to my situation.
Is anyone using another filebrowser for django-tinymce? Did you have a similar problem and find a fix for it? I'd appreciate any points in the right direction.
I used a similar approach but instead of modifying the django-filebrowser code I ended up extending the browse() method in a subclass of FileBrowserSite and making the modification there:
from django.core.files.storage import DefaultStorage
from filebrowser.sites import FileBrowserSite
class FileBrowserSite(FileBrowserSite):
def browse(self, request):
self.directory = self.directory + str(request.user) + '/'
return super(FileBrowserSite, self).browse(request)
storage = DefaultStorage()
site = FileBrowserSite(name='file', storage=storage)
site.directory = "content/"
I put this piece of code on a file called filebrowser.py and the then on my urls.py I did:
from .filebrowser import site
urlpatterns = [
url(r'^admin/content/file/', include(site.urls)),
]
I think is a much cleaner approach than modifying the source code, and is working like charm on my project.
I've extended a little bit the answer by Erasmo. Generally, it works great. Thanks! However, as OriolJ pointed out, every user needs a created directory to use the filebrowser. It is recommended to avoid using signals, so I added the functionality to the custom FileBrowserSite.
filebrowser.py
import os
from django.conf import settings
from django.core.files.storage import DefaultStorage
from filebrowser.sites import FileBrowserSite
class FileBrowserSite(FileBrowserSite):
def browse(self, request):
# get directory path from settings to avoid recursion
self.directory = settings.DIRECTORY + str(request.user) + '/'
# create a directory for a user if it does not already exist
full_path = self.storage.location + '/' + self.directory
if not os.path.exists(full_path):
os.makedirs(full_path)
return super().browse(request)
storage = DefaultStorage()
site = FileBrowserSite(name='file', storage=storage)
settings.py
FILEBROWSER_DIRECTORY = 'user_content/'
DIRECTORY = ''
urls.py
from .filebrowser import site
urlpatterns = [
url(r'^admin/content/file/', include(site.urls)),
]
Hope this slight update will save someone a couple of minutes.
I've, somewhat hacked this. I'm on grappelli 2.7.2, and django-filebrowser 3.6.1.
I've just added a line to the django-filebrowser file: sites.py:
on the method browse() of the class FileBrowserSite, line ~273, I've added:
self._directory_set(DIRECTORY + str(request.user) + "/")
It's important though that there is already a directory created for every user, you can make a post_save signal to create a directory every time a user is created. This solution will not relate the files with the object, but I think that this may be a good start for you and it will isolate files of different users.
For me, this is currently working just fine, I can imagine that this is not the most perfect way to do it, feedback is very welcome.

Django running wrong view method

I have to view files stored in mysite folder. one is named as views.py and other is named as request_view.py. In urls.py, I have used 'answer' method for views.py and 'display_meta' method for request_view.py.
(django version: 1.5 and python version: 2.7.3)
this is the url pattern:
url(r'^twitter/$', answer), url(r'request/$', display_meta)
when I call http:/127.0.0.1:8000/request/, then also first view(i.e. /twitter/) is called!
any help?
one more thing. In my view.py, I have some unbounded code (i.e. the code which is neither present in a method or class). can this be the cause of the problem?
l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
keyword = input('enter the keyword you want to search for?')
stream.filter(track = [keyword])
apart from this code, evry code is either in the class or method.
One thing that I noticed is that first the code of the view.py runs, then display_meta runs.
Thanks in advance.
SOLVED
The problem was with the import function that I was using. since my code was unbounded in one of the views, the import function always import that regardless of the url that I choose.
Suggestion
Always use the nomenclature mentioned in the this example. In many books it has been suggested that we should import the views, but it might cause an error if you have unbounded code like I had.
I don't know exactly why /twitter/ view is called, but I can see two things to change:
You should use a string as the second parameter for url(), as you can see in this example [1]. You can use 'myapp.views.my_method' nomenclature.
You forgot to start the request URL with ^ that indicates the start of the URL.
About the unbounded code, I don't know if that could be causing the problem. But I can't see why are you putting that code unbounded. I am not sure when that code would be executed, I guess the first time you call a view in that file and Django loads the file (I'm guessing, I don't know exactly), but I don't think that would be a good way to do that. Think when do you want to execute that code, put it in a method, and call it.
[1] https://docs.djangoproject.com/en/1.5/topics/http/urls/#example
HI hemant i am wondering why you have written request_view.py.
Please see the django docs.
what you can do is .
Create two function in your views.py like
def answer(request):
do some stuffs.
render_to_response(template.html)
and on the same page write another
def display_meta(request):
# do your studd
render_to_response(some.html)
YOU NEED NOT TO CREATE TWO SEPERATE VIEWS.PY
I dont know what this code does
l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
keyword = input('enter the keyword you want to search for?')
stream.filter(track = [keyword])
But if you want to use StdOutListener inside your function you can call in your view
like
def display_meta(request):
stobject = StdOutListener() # use parameters if you have here
# do your studd
render_to_response(some.html)
Organizing views into a python package could solve this problem. So if you have a structure like this...
# views.py
def SomeView(request):
return HttpResponse('SomeView')
# another_view.py
def AnotherView(request):
return HttpResponse('AnotherView')
Your can reorganize these separate view files into a views package. That is...
# views
# __init__.py
from views import SomeView
from another_view import AnotherView
# views.py
def SomeView(request):
return HttpResponse('SomeView')
# another_view.py
def AnotherView(request):
return HttpResponse('AnotherView')
And now, everything can be called in a django-standard way:
url(r'^url-to-some-view/$', 'views.SomeView'),
url(r'^url-to-another-view/$', 'views.AnotherView'),
UPDATED:
To make a 'python package'...
Create a views directory at the same level as the view.py file [mkdir views]
Create a __init.py__ file inside the views directory # this is what makes a directory a 'python package'
Move views.py into the views directory.
Move your request_view.py into the views directory.
Edit the __init__.py file with the necessary import statements. In this case:
from views import answer
from request_view import display_meta
What this does is replace a file with a directory. By importing everything into the __init__.py file, this directory looks like a large file to your code, rather than another module.

Preventing leftover image files when testing models with ImageFields in Django 1.3

Django 1.3 changed the behavior of models with FileFields such that when the instance is deleted the actual file(s) remain. For most purposes, this is fine. However, I'm testing an application that makes heavy use of ImageFields and I'm ending up with hundreds of useless leftover images in my development upload directory. Since it's development, it's not a huge deal, but I'd prefer to clean up after myself in the tests.
What's the best way to make these images not hang around when testing. I emphasize that part because I don't want to modify this behavior across the board, using a delete override, etc.
I broke down and compromised a bit. I decided to go ahead and use an overridden delete, but I made deleting the actual image files dependent on passing a include_images=True kwarg.
class Photo(models.Model):
...
def delete(self, *args, **kwargs):
if kwargs.pop('include_images', False):
for field in self._meta.fields:
if type(field) == models.ImageField:
image = self.__getattribute__(field.name)
if image.name != '':
image.storage.delete(image.name)
super(Photo, self).delete(*args, **kwargs)
Then, I modified the tearDown method in my test cases like so:
def tearDown(self):
for p in Photo.objects.all():
p.delete(include_images=True)
I'm not entirely happy with this solution, but it's the closest I could get to what I consider ideal. This should really all be handled by the tests themselves.
A cron that call to django command
I had the issue and found this post in Google. I much prefer to use a command instead - so here is some code which might be useful for others.That's in Django 1.4 but should work in any version. I use WebTest:
In the base test class:
def get_test_image_input(self):
return (
'_test-image.jpg',
file(os.path.join(settings.TEST_FILES_PATH, "images", "test.jpg")).read()
)
In any test:
form['image'] = self.get_test_image_input()
To clean up afterwards, I simply delete all files starting with _test-image in a command:
from django.core.management.base import BaseCommand
from fabric.operations import local
class Command(BaseCommand):
def handle(self, *args, **options):
# test an app call "tests"
local("python manage_test.py test tests -v 2")
# could use call_command(...) instead
# remove all test images
local("sudo rm -rf `find . -name _test-image*`")

Allow users to play mp3 files but don't expose them directly on the web

I want to store some mp3s in a folder which is not public, can't be directly accessed through the web and allow users to hear/download the songs with a browser only if they are logged in.
How can I do that?
I do my web development with Django, but if I know how it works is enough.
You first need to setup authentication. The django tutorials thoroughly explore this.
You don't' link the mp3's directly, You link to a django script that checks the auth, then reads the mp3 and serves it to the client with a mp3 content type header.
http://yourserver.com/listen?file=Fat+Boys+Greatest+Hits
I assume you use django. Then you can try something like this:
from django.conf import settings
from django.contrib.auth.decorators import login_required
from django.http import HttpResponse
#login_required
def listen(request, file_name):
# note that MP3_STORAGE should not be in MEDIA_ROOT
file = open("%smp3/%s" % (settings.MP3_STORAGE, file_name))
response = HttpResponse(file.read(), mimetype="audio/mpeg")
return response
Note that you will get dramatic speed decrease. Using generator to read file in blocks may help to save memory.
Lazy Method for Reading Big File in Python?
File outside of public access (not in
MEDIA_URL folders)
Check if user logged in
Serve files only via a view, with
unique links for every user
Pseudocode:
class Mp3(models.Model):
file = models.FileField(upload_to=path_outside_of_public_access)
hash = models.CharField(unique=True)
def generate_link_hash(request, file):
return hashlib.md5("%s_%i_%s_%s" % (request.session.session_key, file.id, str(file.date_added), file.hash)) # or however u like
def files_list(request)
""" view to show files list """
for file in files:
file.link_hash = generate_link_hash(request, file)
#login_required
def download_file(request, file_hash, link_hash):
""" view to download file """
file = Mp3.objects.get(hash=file_hash)
if link_hash == generate_link_hash(request, file):
file = open(file.file)
return HttpResponse(file.read(), mimetype="audio/mpeg")
else:
raise Http404
Should do the job enough, but remember - what is once accessed, you have no control where it goes from now on. And that every file download needs reading the file through the app (it's not given statically), which will affect the performance of your app.

How do you put a file in a fixture in Django?

I can easily fill the field of a FileField or ImageField in a Django fixture with a file name, but that file doesn't exist and when I try to test my application it fails because that file doesn't exist.
How do I correctly populate a FileField or Imagefield in a Django fixture so that the file itself is available too?
I'm afraid the short answer is that you can't do this using the FileField or ImageField classes; they just store a file path and have no real concept of the file's actual data. The long answer, however, is that anything is possible if you leverage the Django API for writing your own custom model fields.
At a minimum, you'll want to implement the value_to_string method to convert the data for serialization (there's an example in the django docs at the link above). Note that the examples at the URL link above also include mention of subclassing FileField and ImageField, which is helpful for your situation!
You'll also have to decide if the data should therefore be stored in the database, or on the file system. If the former, you will have to implement your custom class as a Blob field, including customization for every DB you wish to support; you'll also have to provide some support for how the data should be returned to the user out of the database when the HTML requests a .gif/.jpg/.png/.whatever url. If the latter, which is the smarter way to go IMHO, you'll have to implement methods for serializing, de-serializing binary data to the filesystem. Either way, if you implement these as subclasses of FileField and ImageField, you should still be able to use the Admin tools and other modules that expect such django features.
If and only if you elect to use the more involved blob approach, here's a snippet of code from an old project of mind (back when I was learning Django) that handles blob for MySQL and PostgreSQL; you'll probably be able to find a number of improvements as I haven't touched it since :-) It does not handle serialization, though, so you'll have to add that using the method above.
from django.db import models
from django.conf import settings
class BlobValueWrapper(object):
"""Wrap the blob value so that we can override the unicode method.
After the query succeeds, Django attempts to record the last query
executed, and at that point it attempts to force the query string
to unicode. This does not work for binary data and generates an
uncaught exception.
"""
def __init__(self, val):
self.val = val
def __str__(self):
return 'blobdata'
def __unicode__(self):
return u'blobdata'
class BlobField(models.Field):
"""A field for persisting binary data in databases that we support."""
__metaclass__ = models.SubfieldBase
def db_type(self):
if settings.DATABASE_ENGINE == 'mysql':
return 'LONGBLOB'
elif settings.DATABASE_ENGINE == 'postgresql_psycopg2':
return 'bytea'
else:
raise NotImplementedError
def to_python(self, value):
if settings.DATABASE_ENGINE == 'postgresql_psycopg2':
if value is None:
return value
return str(value)
else:
return value
def get_db_prep_save(self, value):
if value is None:
return None
if settings.DATABASE_ENGINE =='postgresql_psycopg2':
return psycopg2.Binary(value)
else:
return BlobValueWrapper(value)
There's no way to "include" the files in the serialized fixture. If creating a test fixture, you just need to do it yourself; make sure that some test files actually exist in locations referenced by the FileField/ImageField values. The values of those fields are paths relative to MEDIA_ROOT: if you need to, you can set MEDIA_ROOT in your test setUp() method in a custom test_settings.py to ensure that your test files are found wherever you put them.
EDIT: If you want to do it in your setUp() method, you can also monkeypatch default_storage directly:
from django.core.files.storage import default_storage
class MyTest(TestCase):
def setUp(self):
self._old_default_storage_location = default_storage.location
default_storage.location = '/some/other/place'
def tearDown(self):
default_storage.location = self._old_default_storage_location
That seems to work. default_storage is a documented public API, so this should be reliable.