I have checked several other threads but I am still having a problem. I have a model that includes a FileField and I am generating semi-random instances for various purposes. However, I am having a problem uploading the files.
When I create a new file, it appears to work (the new instance is saved to the database), a file is created in the appropriate directory, but the file's content is missing or corrupt.
Here is the relevant code:
class UploadedFile(models.Model):
document = models.FileField(upload_to=PATH)
from django.core.files import File
doc = UploadedFile()
with open(filepath, 'wb+') as doc_file:
doc.documen.save(filename, File(doc_file), save=True)
doc.save()
Thank you!
Could it be as simple as the opening of the file. Since you opened the file in 'wb+' (write, binary, append) the handle is at the end of the file. try:
class UploadedFile(models.Model):
document = models.FileField(upload_to=PATH)
from django.core.files import File
doc = UploadedFile()
with open(filepath, 'rb') as doc_file:
doc.document.save(filename, File(doc_file), save=True)
doc.save()
Now its open at the beginning of the file.
Related
I am using Django default_storage API to save my media files.
I am able to save the file, and open the file for writing. But I am not able to download the file.
I used the code below to save the file:
default_storage.save(filename, ContentFile(str(a).encode()))
Is there any way to download the file in the same way?
I used the code below to download the file, but it is not either downloading or not throwing any error:
with default_storage.open(filepath, 'rb') as fh:
response = HttpResponse(fh.read(), content_type="application/vnd.ms-excel")
response['Content-Disposition'] = 'inline ; filename=' +os.path.basename(filepath)
return response
raise Http404
You are on the right path.
with default_storage.open(filepath, 'rb') as fh:
with open('my_local_file','wb') as wh:
data = fh.read() # You may want to split this into chunks..
wh.write(data)
I have a project which reads the database composition into a compose environment variable file .env. I use django to read it and generate it.
Part of the code is as follows:
from django.http import HttpResponse
content = "Stringtest"
filename = ".env"
response = HttpResponse(content, content_type='text/plain')
response['Content-Disposition'] = 'attachment; filename={0}'.format(filename)
return response
When I download the file using brower, the preceding comma is automatically removed and the file name is changed to env instead of the required .env
How can I change the file name to remain the same?
I'm building a Django app that enables users to upload a CSV via a form using a FormField. Once the CSV is imported I use the Pandas read_csv(filename) command to read in the CSV so I can do some processing on the CSV using Pandas.
I've recently started learning the really useful Dask library because the size of the uploaded files can be larger than memory. Everything works fine when using Pandas pd.read_csv(filename) but when I try and use Dask dd.read_csv(filename) I get the error "'InMemoryUploadedFile' object has no attribute 'startswith'".
I'm pretty new to Django, Pandas and Dask. I've searched high and low and can't seem to find this error when associated with Dask anywhere on Google.
Here is the code I'm trying to use below (just the relevant bits... I hope):
Inside forms.py I have:
class ImportFileForm(forms.Form):
file_name = forms.FileField(label='Select a csv',validators=[validate_file_extension, file_size])
Inside views.py
import pandas as pd
import codecs
import dask.array as da
import dask.dataframe as dd
from dask.distributed import Client
client = Client()
def import_csv(request):
if request.method == 'POST':
form = ImportFileForm(request.POST, request.FILES)
if form.is_valid():
utf8_file = codecs.EncodedFile(request.FILES['file_name'].open(),"utf-8")
# IF I USE THIS PANDAS LINE IT WORKS AND I CAN THEN USE PANDAS TO PROCESS THE FILE
#df_in = pd.read_csv(utf8_file)
# IF I USE THIS DASK LINE IT DOES NOT WORK AND PRODUCES THE ERROR
df_in = dd.read_csv(utf8_file)
And here is the error output I'm getting:
AttributeError at /import_data/import_csv/
'InMemoryUploadedFile' object has no attribute 'startswith'
/home/username/projects/myproject/import_data/services.py in save_imported_doc
df_in = dd.read_csv(utf8_file) …
▶ Local vars
/home/username/anaconda3/lib/python3.7/site-packages/dask/dataframe/io/csv.py in read
**kwargs …
▶ Local vars
/home/username/anaconda3/lib/python3.7/site-packages/dask/dataframe/io/csv.py in read_pandas
**(storage_options or {}) …
▶ Local vars
/home/username/anaconda3/lib/python3.7/site-packages/dask/bytes/core.py in read_bytes
fs, fs_token, paths = get_fs_token_paths(urlpath, mode="rb", storage_options=kwargs) …
▶ Local vars
/home/username/anaconda3/lib/python3.7/site-packages/fsspec/core.py in get_fs_token_paths
path = cls._strip_protocol(urlpath) …
▶ Local vars
/home/username/anaconda3/lib/python3.7/site-packages/fsspec/implementations/local.py in _strip_protocol
if path.startswith("file://"): …
▶ Local vars
/home/username/anaconda3/lib/python3.7/codecs.py in __getattr__
return getattr(self.stream, name)
I finally got it working. Here's a Django specific solution building on the answer from #mdurant who thankfully pointed me in the right direction.
By default Django stores files under 2.5MB in memory and so Dask isn't able to access it in the way Pandas does as Dask asks for a location in actual storage. However, when the file is over 2.5MB Django stores the file in a temp folder which can then be located with the Django command temporary_file_path(). This temp file path can then be used directly by Dask. I found some really useful information about how Django actually handles files in the background in their docs: https://docs.djangoproject.com/en/3.0/ref/files/uploads/#custom-upload-handlers.
In case you can't predict in advance your user uploaded file sizes (as is in my case) and you happen to have a file less than 2.5MB you can change FILE_UPLOAD_HANDLERS in your Django settings file so that it writes all files to a temp storage folder regardless of size so it can always be accessed by Dask.
Here is how I changed my code in case this is helpful for anyone else in the same situation.
In views.py
def import_csv(request):
if request.method == 'POST':
form = ImportFileForm(request.POST, request.FILES)
if form.is_valid():
# the temporary_file_path() shows Dask where to find the file
df_in = dd.read_csv(request.FILES['file_name'].temporary_file_path())
And in settings.py adding in the setting as below makes Django always write an uploaded file to temp storage whether the file is under 2.5MB or not so it can always be accessed by Dask
FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.TemporaryFileUploadHandler',]
It seems you are not passing a file on disc, but some django-specific buffer object. Since you are expecting large files, you probably want to tell django to stream the uploads directly to disc and give you the filename for dask; i.e., is request.FILES['file_name'] actually somewhere in your storage? The error message seems to suggest not, in which case you need to configure django (sorry, I don't know how).
Note that Dask can deal with in-memory file-like objects such as io.BytesIO, using the MemoryFileSystem, but this isn't very typical, and won't help with your memory issues.
I want to encrypt user uploaded files in django before saving them.
When user send files through POST requests, I get a "InMemoryUploadedFile" type object.
How can I encrypt the files before saving them ? I currently use pyAesCrypt to encrypt files but I can't manage to pass in it the "InMemoryUploadedFile" objects. I manage to only encrypt them after they are saved with :
import pyAesCrypt
with open("*FileName*", "rb") as InputFile:
with open("*OutputFileName*", "wb+") as OutputFile:
pyAesCrypt.encryptStream(InputFile, OutputFile, Password, BufferSize)
I recently asked this questions and a user told me to use a package with better community support. It is pyca/cryptography. I was stuck in the same thing and I found a solution. Mind that, I use Django Rest Framework.
from cryptography.fernet import Fernet
# Generate a key and store safely
key = Fernet.generate_key()
f = Fernet(key)
I'll take an excel file for example but you could actually use any file.
import pandas as pd
import io
from django.core.files.uploadedfile import SimpleUploadedFile
# Request file from user and load the file into a dataframe
df = pd.read_excel(request.FILES('file_name'))
# Create BytesIO
output = io.BytesIO()
# Output df to BytesIO
df.to_excel(output, index=False)
# Encrypt data (BytesIO)
encrypted_out = f.encrypt(output.getvalue())
# Export encrypted file
output_file = SimpleUploadedFile('<some_file_name.extension>',encrypted_out)
# Typically you would pass this through a serializer.
To decrypt the file before you can serve the user. Read the file and write it to BytesIO and then you can serve the file to the user.
I want to upload files to the database that Django use, I know that I can do it
through forms, but I want to read the files in my files system get the path of the docx or pdf and uploaded it into the database, how can I do that
Here is the code that i use to get the path of the files in my filedsystem
for dir_, _, files in os.walk(superpath):
for fileName in files:
print fileName
if fileName.find('~$')==-1:
relDir = os.path.relpath(dir_, superpath)
if relDir=='.':
relFile =os.path.join(superpath, fileName)
else:
relFile = os.path.join(superpath,os.path.join(relDir, fileName))
path.append(relFile)
You can use ContentFile for this.
from django.core.files.base import ContentFile
f = open(file_path, 'r')
object.image.save('image.jpg', ContentFile(f.read()))