I have a Django app that will be placed in a Docker container.
The app prepares data in Dataframe format. I would like to allow the user to download the data to his/her local drive as excel file.
I have used df.to_excel in the past, but this won't work in this case.
Please advise best way to do this.
As of pandas-0.17, you can let Django write to a BytesIO directly, like:
from django.http import HttpResponse
from io import BytesIO
def some_view(request):
with BytesIO() as b:
# Use the StringIO object as the filehandle.
writer = pd.ExcelWriter(b, engine='xlsxwriter')
df.to_excel(writer, sheet_name='Sheet1')
writer.save()
# Set up the Http response.
filename = 'django_simple.xlsx'
response = HttpResponse(
b.getvalue(),
content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
)
response['Content-Disposition'] = 'attachment; filename=%s' % filename
return response
You might need to install an Excel writer module (like xlsxwriter, or openpyxl).
I think it can be even simpler and more concise these days. You can just pass the http response directly to the Excel writer. The following works for me:
from django.http import HttpResponse
import pandas as pd
# df = CREATE YOUR OWN DATAFRAME
response = HttpResponse(content_type='application/xlsx')
response['Content-Disposition'] = f'attachment; filename="FILENAME.xlsx"'
with pd.ExcelWriter(response) as writer:
df.to_excel(writer, sheet_name='SHEET NAME')
return response
Related
I'm trying to write an AWS Lambda service using Python 2.7 that will generate an In-Memory CSV file and email it as an attachment. I feel like I'm close with this script based on what I've learned but I'm not quite there.
# Import smtplib for the actual sending function
import smtplib
import sys
import csv
import cStringIO
from os.path import basename
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.application import MIMEApplication
# Import the email modules we'll need
server = smtplib.SMTP('smtp.postmarkapp.com', 587)
server.starttls()
server.login('.....','.....')
list = []
row1 = ["One","Two","Three"]
list.append(row1)
msg = MIMEMultipart()
msg['To'] = "daniel#mydomain.com"
msg['From'] = "noreply#mydomain.com"
msg['Subject'] = "DG Test subject"
msg.attach(MIMEText("Test Message"))
csv_buffer = cStringIO.StringIO()
writer = csv.writer(csv_buffer, lineterminator='\n')
writer.writerow(["1","2","3"])
for row in list:
writer.writerow(row)
print(csv_buffer.getvalue())
msg.attach(csv_buffer)
try:
response = server.sendmail(msg['From'], ["daniel#mydomain.com"],msg.as_string())
server.quit()
except AttributeError as error:
print(error)
else:
print(response)
This gives me the following error:
1,2,3
One,Two,Three
'cStringIO.StringO' object has no attribute 'get_content_maintype'
Basically it comes down to not being sure how to use the csv_buffer object. Assuming I just need to add that attribute to the object somehow but I'm not quite sure how. If I try to add any additional arguments to the .attach() line, it complains that I have too many arguments.
Thanks!
I figured it out, thanks to stitching together a few SO posts.
import cStringIO
import csv
csv_buffer = cStringIO.StringIO()
writer = csv.writer(csv_buffer, delimiter=',', quoting=csv.QUOTE_ALL)
writer.writerow(["1","2","3"])
for row in list:
writer.writerow(row)
print(csv_buffer.getvalue())
# new lines
csv_file = MIMEText(csv_buffer.getvalue())
attachment = csv_file.add_header('Content-Disposition', 'attachment', filename="csv_file.csv")
msg.attach(csv_file)
I am trying to use the call_command method to call the dumpdata command. Manually, I use it as follows to save the data to a file.
python manage.py dumpdata appname_one appname_two > /path/to/save/file.json
and it saves the json file. Now, I am in a situation where I need to call this command using the call_command method.
I am able to print out the json from the command using the following:
from django.core.management import call_command
call_command('dumpdata', 'appname_one', 'appname_two')
Is there a way I can save the given data to a file like we do it from the command line?
had to redirect sys.stdout to the file in order to achieve the above. Something like.
import sys
from django.core.management import call_command
sysout = sys.stdout
sys.stdout = open('filename.json', 'w')
call_command('dumpdata', 'appname_one', 'appname_two')
sys.stdout = sysout
An even better way is to use Django's built-in stdout redirection for their command modules. See docs here.
If you want to manipulate the stream before sending it to a file, you can also pass it a StringIO buffer:
import os
from cStringIO import StringIO
from django.core import management
def create_fixture(app_name, filename):
buf = StringIO()
management.call_command('dumpdata', app_name, stdout=buf)
buf.seek(0)
with open(filename, 'w') as f:
f.write(buf.read())
I am using Django fixture magic https://github.com/davedash/django-fixture-magic and need to dump a custom fixture. I tried several ways but ended up using Amyth's answer becuase it was the only way that worked.
Here is my admin action that works with fixture magic
def export_survey(modeladmin, request, queryset):
sysout = sys.stdout
survey = queryset[0]
fname = "%s.json" %(survey.slug)
response = HttpResponse(mimetype='application/json')
response['Content-Disposition'] = 'attachment; filename=%s' %(fname)
sys.stdout = response
call_command('custom_dump', 'complete_survey', survey.id)
sys.stdout = sysout
return response
export_survey.short_description = "Exports a single survey as a .json file"
DB fixtures typically compress well, and loaddata can read compressed fixtures. To write a .bz2 compressed fixture directly:
import bz2
with bz2.BZ2File('db.json.bz2', 'w', buffering=1024) as f:
django.core.management.call_command('dumpdata', stdout=f)
This one help for multiple dump data into json file
from django.core.management import call_command
import sys
sys.stdout = open('app_one/fixtures/apple.json', 'w')
call_command('dumpdata', 'app_one.apple')
sys.stdout = open('app_two/fixtures/banana.json', 'w')
call_command('dumpdata', 'app_two.banana')
In a view I create new file with:
sys.stdout = open(backup_name, 'w')
call_command('dumpdata')
How can I now return this file to user?
I tried to change mimetype in HttpResponse to 'application/json' but how can I add file content to response?
Or maybe there is other way to return file?
.
I know it's a bit late, but I found this a useful starting point so I thought others could benefit from what I found too.
For a small file, if you place the json file in a template folder, django can find it and you can return it with render_to_response:
return render_to_response(data_file,mimetype='application/json')
I found this to be problematic for large datasets on certain browsers. I would get the error An existing connection was forcibly closed by the remote host. An alternative approach fixed this.
First you must create full path to your file. Use the PROJECT_ROOT variable (defined by PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__)) in settings.py). To access this and the os methods you must import settings, os in views.py. Once you have this file location you can return it using the code below:
backup_path = os.path.join(settings.PROJECT_ROOT, "templates", "json_dumps", "large_file.json")
return HttpResponse(open(backup_path, 'r'),content_type = 'application/json; charset=utf8')
I found this worked well for even very large files.
OK I have it:
response = HttpResponse(open(backup_path, "r"), mimetype='application/json', )
response['Content-Disposition'] = "filename=%s" % backup_name"
After saving file on disc I open it for reading and set file name in response.
Anyone has another idea?
I was trying to return a dictionary as a json file. Here is my solution:
import json
import cStringIO as StringIO
from wsgiref.util import FileWrapper
from django.http import HttpResponse
data_string = json.dumps(data)
json_file = StringIO.StringIO()
json_file.write(data_string)
json_file.seek(0)
wrapper = FileWrapper(json_file)
response = HttpResponse(wrapper, content_type='application/json')
response['Content-Disposition'] = 'attachement; filename=dump.json'
return response
Just copy/link/call the dumpdata code related to model serialization, and dump it directly into the response, so you avoid permission problems and filesystem pollution. Content-disposition and mimetype still applies.
Remember anyway that dumpdata can be a lenghty process, so you are exposed to timeouts.
My final solution is (thanks to saverio):
response = HttpResponse(mimetype='application/json', )
response['Content-Disposition'] = "filename=%s" % backup_name
sys.stdout = response
call_command('dumpdata')
I have pisa producing .pdfs in django in the browser fine, but what if I want to automatically write the file to disk? What I want to do is to be able to generate a .pdf version file at specified points in time and save it in a uploads directory, so there is no browser interaction. Is this possible?
Yes it is possible. for example, using code from Greg Newman as a starter:
from django.template.loader import get_template
from django.template import Context
import ho.pisa as pisa
import cStringIO as StringIO
import cgi
def write_pdf(template_src, context_dict, filename):
template = get_template(template_src)
context = Context(context_dict)
html = template.render(context)
result = open(filename, 'wb') # Changed from file to filename
pdf = pisa.pisaDocument(StringIO.StringIO(
html.encode("UTF-8")), result)
result.close()
You just need to call write_pdf with a template, data in a dict and a file name.
I need to return css files and js files according to specific logic. Clearly, static serve does not perform what I need. I have a view, whose render method uses logic to find the proper file, but then I have to return it. Technically, I can just read the file and stuff it into a HttpResponse object with the proper mime type, but I was wondering if there was a better strategy. (like fpassthru() in php)
This is what I used:
test_file = open('/home/poop/serve/test.pdf', 'rb')
response = HttpResponse(content=test_file)
response['Content-Type'] = 'application/pdf'
response['Content-Disposition'] = 'attachment; filename="%s.pdf"' \
% 'whatever'
return response
What webserver software are you using?
At least for Apache and NginX, there is a module enabling you to use the X-SendFile HTTP header. The NginX website says Lighty can do this, too.
In your wrapper view:
...
abspath = '/most_secret_directory_on_the_whole_filesystem/protected_filename.css'
response = HttpResponse()
response['X-Sendfile'] = abspath
response['Content-Type'] = 'mimetype/submimetype'
# or let your webserver auto-inject such a header field
# after auto-recognition of mimetype based on filename extension
response['Content-Length'] = <filesize>
# can probably be left out if you don't want to hassle with getting it off disk.
# oh, and:
# if the file is stored via a models.FileField, you just need myfilefield.size
response['Content-Disposition'] = 'attachment; filename=%s.css' \
% 'whatever_public_filename_you_need_it_to_be'
return response
Then you can connect the view via http://mysite.com/url_path/to/serve_hidden_css_file/.
You can use it anytime you need to do something upon a file being requested that should not be directly accessible to users, like limiting who can access it, or counting requests to it for stats, or whatever.
For Apache: http://tn123.ath.cx/mod_xsendfile/
For NginX: http://wiki.nginx.org/NginxXSendfile
Why don't you use Django staticfiles inside your view
from django.contrib.staticfiles.views import serve
...
def view_function(request):
return serve(request, 'absolute_path_to_file_name')
Why not return an HttpResponseRedirect to the location of the correct static file?
Serving files directly from a view is very slow. If you are looking for normal file serving see this question: Having Django serve downloadable files
To very easily serve files through a view (for debug purposes, for example) keep reading.
# In your urls.py:
url(r'^test-files/(?P<name>.+)/$', views.test_files, name='test_files'),
# In your views.py:
from django.http.response import HttpResponse
from django.views.decorators.csrf import csrf_exempt
#csrf_exempt # (Allows file download with POST requests, can be omitted)
def test_files(request, name):
if name == "myxml":
fsock = open("/djangopath/data/static/files/my.xml", "rb")
return HttpResponse(fsock)
This allows you to download the file from: http://127.0.0.1:8080/app/test-files/myxml/
Pass an iterator (such as the result of open()) to the HttpResponse constructor.
you can use below code in your view:
Note:in this function I return images but you can return every thing based your need and set your context_type
from django.http import HttpResponse,Http404
import os
def img_finder(request, img_name):
try:
with open(os.path.dirname(os.path.abspath(__file__)) + '/static/img/' + img_name, 'rb') as f:
return HttpResponse(f.read(), content_type="image/jpeg")
except IOError:
raise Http404
Here the most simple and efficient way to do this.
app/urls.py
from django.urls import re_path
from app import views
urlpatterns = [
re_path(r'^(?P<public_url>.*)$', views.public, name="public"),
]
Warning : put the URL pattern at the end
app/views.py
import os
from django.conf import settings
from django.views.static import serve
def public(request, public_url):
public_folder = os.path.join(str(settings.BASE_DIR), 'folder_path')
return serve(request, public_url, document_root=public_folder)
It should be wasteful to use django to serve static content (not to mention, several orders of magnitude slower).
I'd rather convert the view into a context processor and use the variables in templates to find what blocks to include.