I have many files under a folder(for example, 'datasets') and these files are separated in many different sub–dictionaries.
I want to let user specify the folder 'datasets' in a form and upload all the files in Django. After upload, The Django view function will extract some pieces information from each files and save into database. how can I do this.
The following is the structure of my files to be uploaded:
datasets
- subfolder 1
- file1
- file2
- subfolder 1a
- file3
- subfolder 2
- file4
- file5
The FileField.upload_to is defined as follows
This attribute provides a way of setting the upload directory and file
name, and can be set in two ways. In both cases, the value is passed
to the Storage.save() method. ... upload_to may also be a callable,
such as a function. This will be called to obtain the upload path,
including the filename. This callable must accept two arguments and
return a Unix-style path (with forward slashes) to be passed along to
the storage system. The two arguments are:
So what you need to do is to create a function that will check the filename or it's content and decide where the file should be saved
def get_upload_path(instance, filename):
if filename .... :
return 'path1'
else :
return 'path2'
And your model will change to
image = models.FileField(upload_to=get_upload_path)
Related
When upload image, vtiger add prefix to filename.
Befor upload: IMG_NAME.png.
After upload: 26540_IMG_NAME.png.
How I can remove '26540_' prefix?
Its not recommended to change the standard of storing files with the name. Because the prefix ('26540_' in your case) is the unique identifier which will add before the filename. And if we upload same file with the same name vTiger treat as a different file.
But still if you dont want to prefix added then customize the code as per below:
Open \data\CRMEntity.php
Search function uploadAndSaveFile(
Comment the line
$upload_status = move_uploaded_file($filetmp_name, $upload_file_path .$current_id . "_" . $binFile);
Add (Removed $current_id)
$upload_status = move_uploaded_file($filetmp_name, $upload_file_path . $binFile);
Save the script and test. Cheers!
I would like to keep the original file name of an UploadedFile in Django that has its location stored in a FileField. Right now I am observing that if two files have the same name, the first file uploaded keeps its original name but the second time a file with that name is uploaded, it has a random string appended to make the file name unique. One solution is to add an additional field to the model: Django: How to save original filename in FileField? or Saving Original File Name in Django with FileField but these solutions seem suboptimal as they require changing the Model fields.
An alternative would be to prepend a random directory path to the front of the file make sure that in a given directory the file name is unique and allowing the basename to remain unchanged. One way to do this would be to pass in a callable upload_to that does just that. Another option would be to subclass FileField and override get_filename to not strip the input filename to the basename allowing the caller to pass in a filename with a prepended path. The latter option is not ideal if I want to use an ImageField as I would have to subclass that as well.
In looking at the code that actually generates the unique filename by appending the random string, it looks like the best solution to this problem might be to subclass the Storage class in-use and override get_available_name method to create unique filenames by prepending a directory rather than post-pending the string to the base name.
Sorry for the quick answere, here is another approach to your question :
The idea here is to create an unique folder for each uploaded file.
# in your settings.py file
MY_FILE_PATH = 'stored_files/'
The path were your files will be stored : /public/media/stored_files
# somewhere in your project create an utils.py file
import random
try:
from hashlib import sha1 as sha_constructor
except ImportError:
from django.utils.hashcompat import sha_constructor
def generate_sha1(string, salt=None):
"""
Generates a sha1 hash for supplied string.
:param string:
The string that needs to be encrypted.
:param salt:
Optionally define your own salt. If none is supplied, will use a random
string of 5 characters.
:return: Tuple containing the salt and hash.
"""
if not isinstance(string, (str, unicode)):
string = str(string)
if isinstance(string, unicode):
string = string.encode("utf-8")
if not salt:
salt = sha_constructor(str(random.random())).hexdigest()[:5]
hash = sha_constructor(salt+string).hexdigest()
return (salt, hash)
In your models.py
from django.conf import settings
from utils.py import generate_sha1
def upload_to_unqiue_folder(instance, filename):
"""
Uploads a file to an unique generated Path to keep the original filename
"""
salt, hash = generate_sha1('{}{}'.format(filename, get_datetime_now().now))
return '%(path)s%(hash_path)s%(filename)s' % {'path': settings.MY_FILE_PATH,
'hash_path': hash[:10],
'filename': filename}
#And then add in your model fileField the uplaod_to function
class MyModel(models.Model):
file = models.FileField(upload_to=upload_to_unique_folder)
The file will be uploaded to this location :
public/media/stored_file_path/unique_hash_folder/my_file.extention
Note : I got the code from Django userena sources, and adapted it to my needs
Note2 : For more informations take a look at this greate post on Django File upload : File upload example
Have a good day.
Edit : Trying to provide a working solution :)
To my understanding, during the form submission/file upload process, you can add form validation functions.
During the validation and cleaning process, you could check that the database does not already have a duplicate name (ie. query to see if that file name exists).
If it is duplicate, you could just rename it xyz_1, xyz_2, etc
I have an API endpoint with Django Rest Framework to upload an image.
class MyImageSerializer(serializers.ModelSerializer):
image = serializers.ImageField(source='image')
I can upload images but they are saved with the filename that is sent from the client which can result to collisions. I would like instead to upload the file to my CDN with a timestamp filename.
Generating the filename is not the problem, just saving the image with it.
Any one knows how to do that?
Thanks.
If your image is of type ImageField from django, then you don't really have to do anything, not even declare it in your serializer like you did. It's enough to add it in the fields attribute and django will handle collisions. This means django will add _index on each new file which might generate a collision, so if you upload a file named 'my_pic.jpg' 5 times, you will actually have files 'my_pic.jpg', 'my_pic_1.jpg', 'my_pic_2.jpg', 'my_pic_3.jpg', 'my_pic_4.jpg' on your server.
Now, this is done using django's implementation for FileSystemStorage (see here), but if you want it to append a timestamp to your filename, all you have to do is write a storage class where you overwrite the get_available_name(name) method. Example:
class MyFileSystemStorage(FileSystemStorage):
def get_available_name(self, name):
''' name is the current file name '''
now = time.time()
stamp = datetime.datetime.fromtimestamp(now).strftime('%Y-%m-%d-%H-%M-%S')
return '{0}_{1}'.format(name, str(stamp))
And the image field in your model:
image = models.ImageField(upload_to='your upload dir', storage= MyFileSystemStorage)
Important update
As of August 20, 2014 this is no longer an issue, since Django found a vulnerability related to this behaviour (thanks #mlissner for pointing it out) . Important excerpt :
We’ve remedied the issue by changing the algorithm for generating file
names if a file with the uploaded name already exists.
Storage.get_available_name() now appends an underscore plus a random 7
character alphanumeric string (e.g. "_x3a1gho"), rather than iterating
through an underscore followed by a number (e.g. "_1", "_2", etc.).
I am doing migration from html to Drupal. Using migrate module.
Here in our custom migration script i need to match all .html files from all the folder except images folder.
pass this regex to $list_files = new MigrateListFiles([],[],$regex)
Below is format of html files
/magazines/sample.html
/test/index.html
/test/format_ss1.html
/test/folder/newstyle_1.html
/images/two.html
i need to get only first 2 html files i.e., we are exluding files which ends with '_[0-9]' and '_ss[0-9]' and .hmtl files in images folder.
i have successfully done by excluding 3 and 4 but i can't able to excule .html files in images folder.
$regex = '/[a-zA-Z0-9\-][^_ss\d][^_\d]+\.html/'; //this will do for 3 and 4 files
but i need to exlude images folder..
i have tried like
$regex = '/[^images\/][a-zA-Z0-9\-][^_ss\d][^_\d]+\.html/'; // not working
Where In PHP script it will work
$regex = '~^(?!/images/)[a-zA-Z0-9/-]+(?!_ss\d|\d)\.html$~' //works in php script
can some one help me out this..
Try
/((?!images)[0-9a-zA-Z])+/[^_]*[^\d]+\.html
Matches:
/magazines/sample.html
/test/index.html
/test/folder/newstyle.html
/test/format_ss.html
Does not match:
/test/format_ss1.html
/test/folder/newstyle_1.html
/images/two.html
/images/1.html
/test/folder/newstyle1.html
/test/folder/newstyle_12.html
is this acceptable?
It's a Drupal/Migrate specific issue - the regex is only a regex for the filename (not the directory) as eventually it gets passed to https://api.drupal.org/api/drupal/includes%21file.inc/function/file_scan_directory/7
file_scan_directory($dir, $mask, $options = array(), $depth = 0)
$mask: The preg_match() regular expression of the files to find.
I think the only way to exclude certain directories is to throw a false in the prepareRow() function if the row has a path you don't require.
function prepareRow($row)
The prepareRow() method is called by the source class next() method, after loading the data row. The argument $row is a stdClass object containing the raw data as provided by the source. There are two primary reasons to implement prepareRow():
To modify the data row before it passes through any further methods and handlers: for example, fetching related data, splitting out source fields, combining or creating new source fields based on some logic.
To conditionally skip a row (by returning FALSE).
https://www.drupal.org/node/1132582
I'd like to store uploaded files into a specific directory that depends on the URI of the POST request. Perhaps, I'd also like to rename the file to something fixed (the name of the file input for example) so I have an easy way to grep the file system, etc. and also to avoid possible security problems.
What's the preferred way to do this in Django?
Edit: I should clarify that I'd be interested in possibly doing this as a file upload handler to avoid writing a large file twice to the file system.
Edit2: I suppose one can just 'mv' the tmp file to a new location. That's a cheap operation if on the same file system.
Fixed olooney example. It is working now
#csrf_exempt
def upload_video_file(request):
folder = 'tmp_dir2/' #request.path.replace("/", "_")
uploaded_filename = request.FILES['file'].name
BASE_PATH = '/home/'
# create the folder if it doesn't exist.
try:
os.mkdir(os.path.join(BASE_PATH, folder))
except:
pass
# save the uploaded file inside that folder.
full_filename = os.path.join(BASE_PATH, folder, uploaded_filename)
fout = open(full_filename, 'wb+')
file_content = ContentFile( request.FILES['file'].read() )
try:
# Iterate through the chunks.
for chunk in file_content.chunks():
fout.write(chunk)
fout.close()
html = "<html><body>SAVED</body></html>"
return HttpResponse(html)
except:
html = "<html><body>NOT SAVED</body></html>"
return HttpResponse(html)
Django gives you total control over where (and if) you save files. See: http://docs.djangoproject.com/en/dev/topics/http/file-uploads/
The below example shows how to combine the URL and the name of the uploaded file and write the file out to disk:
def upload(request):
folder = request.path.replace("/", "_")
uploaded_filename = request.FILES['file'].name
# create the folder if it doesn't exist.
try:
os.mkdir(os.path.join(BASE_PATH, folder))
except:
pass
# save the uploaded file inside that folder.
full_filename = os.path.join(BASE_PATH, folder, uploaded_filename)
fout = open(full_filename, 'wb+')
# Iterate through the chunks.
for chunk in fout.chunks():
fout.write(chunk)
fout.close()
Edit: How to do this with a FileUploadHandler? It traced down through the code and it seems like you need to do four things to repurpose the TemporaryFileUploadHandler to save outside of FILE_UPLOAD_TEMP_DIR:
extend TemporaryUploadedFile and override init() to pass through a different directory to NamedTemporaryFile. It can use the try mkdir except for pass I showed above.
extend TemporaryFileUploadHandler and override new_file() to use the above class.
also extend init() to accept the directory where you want the folder to go.
Dynamically add the request handler, passing through a directory determined from the URL:
request.upload_handlers = [ProgressBarUploadHandler(request.path.replace('/', '_')]
While non-trivial, it's still easier than writing a handler from scratch: In particular, you won't have to write a single line of error-prone buffered reading. Steps 3 and 4 are necessary because FileUploadHandlers are not passed request information by default, I believe, so you'll have to tell it separately if you want to use the URL somehow.
I can't really recommend writing a custom FileUploadHandler for this. It's really mixing layers of responsibility. Relative to the speed of uploading a file over the internet, doing a local file copy is insignificant. And if the file's small, Django will just keep it in memory without writing it out to a temp file. I have a bad feeling that you'll get all this working and find you can't even measure the performance difference.