Heroku worker does not work - django

I am following Heroku's documentation about workers but never succeeded. There is a module util.py which will be run as a background job:
util.py:
import requests
def count_words_at_url(url):
resp = requests.get(url)
return len(resp.text.split())
And to put utils.py to the worker's queue:
from utils import count_words_at_url
from rq import Queue
from worker import conn
q = Queue(connection=conn)
result = q.enqueue(count_words_at_url, 'http://heroku.com')
Question1: I would like to know how q.enqueue(...) works. I suppose that the first argument (count_words_at_url is the function name and the second argument ('http://heroku.com') will be used as the count_words_at_url function's argument.
If I am correct, why is resp = requests.get(url) necessary to get that argument? Why not just return len(url.split())?
EDIT:
Question 2: In order to be able to upload large files (Heroku always terminate such a request due to TIMEOUT), I would like to pass an HTTP request to a function which will be run as a background job:
urls.py:
urlpatterns = [
url(r'upload/', views.upload),
]
views.py
def upload(request):
if request.method=='GET':
return render(request, 'upload/upload.html')
# POST
q = Queue(connection=conn)
q.enqueue(uploadFile, request)
return render(request, 'upload/upload.html')
def uploadFile(request):
# Upload files to Google Storage
fileToUpload = request.FILES.get('fileToUpload')
cloudFilename = 'uploads/' + fileToUpload.name
conn = boto.connect_gs(gs_access_key_id=GS_ACCESS_KEY,
gs_secret_access_key=GS_SECRET_KEY)
bucket = conn.get_bucket(GS_BUCKET_NAME)
fpic = boto.gs.key.Key(bucket)
fpic.key = cloudFilename
# Public read:
fpic.set_contents_from_file(fileToUpload, policy='public-read')
upload.html:
...
<form method=post action="/upload/" enctype=multipart/form-data>
{% csrf_token %}
<input type=file name=fileToUpload><br><br>
<input type=submit value=Upload>
</form>
...
I got the error message: Cannot serialize socket object
Turn out that I cannot pass a socket object to a forked process. Is there any other solution for large file uploading? Thanks.

The clue to 1 is in the name of the function. It doesn't say "count words in URL", it says "count words at URL". In other words, it needs to go and get the page that is referenced by the URL, then count the words on it. That's what requests.get() does.
I don't understand what you're asking in point 2.

Related

How to Output More than Once In frontend In DJango

I am building a web app that makes use of the HackerNews API and I keep running into the same error.
I am trying to Output the 10 requested result from API but everytime i use the return response it only outputs the first result.
I want it to output 10 articles from the API instead of just the One.
This is my code:
from operator import itemgetter
import requests
from django.shortcuts import render
# Make an API call and store the response.
def home(request):
url = 'https://hacker-news.firebaseio.com/v0/topstories.json'
r = requests.get(url)
print(f"Status code: {r.status_code}")
# Process information about each submission.
submission_ids = r.json()
submission_dicts = []
for submission_id in submission_ids[:10]:
# Make a separate API call for each submission.
url = f"https://hacker-news.firebaseio.com/v0/item/{submission_id}.json"
r = requests.get(url)
print(f"id: {submission_id}\tstatus: {r.status_code}")
response_dict = r.json()
# Build a dictionary for each article.
submission_dict = {
'title': response_dict['title'],
'hn_link': f"http://news.ycombinator.com/item?id={submission_id}",
# 'comments': response_dict['descendants'],
}
submission_dicts.append(submission_dict)
# submission_dicts = sorted(submission_dicts, key=itemgetter('comments'),
# reverse=True)
for submission_dict in submission_dicts:
print(f"\nTitle: {submission_dict['title']}")
print(f"Discussion link: {submission_dict['hn_link']}")
# print(f"Comments: {submission_dict['comments']}")
count = 0
if count < 10:
return render(request, "news_api/home.html", submission_dict)
You're only getting one result because your return statement is calling "submission_dict". You probably wanted to call "submission_dict(s)" or something similar but it's hard to tell based on what has been commented out in your code. I think that's your primary problem. Without seeing your template, I don't know what your context object is supposed to look like.
But try this:
# views.py
def hacker_news_api_selector():
url = 'https://hacker-news.firebaseio.com/v0/topstories.json'
r = requests.get(url)
submission_list = r.json()
context = {}
context['objects'] = []
# Process information about each submission.
for submission_id in submission_list[:10]:
# Make a separate API call for each submission.
url = f"https://hacker-news.firebaseio.com/v0/item/{submission_id}.json"
r = requests.get(url)
response_dict = r.json()
context['objects'].append(response_dict)
return context
def home(request):
context = hacker_news_api_selector()
return render(request, "news_api/home.html", context)
And in the template...
# news_api/home.html
<html>
...
{% for x in objects %}
<li>{{ x.title }}</li>
{% endfor %}
</html>

Django Sitetree returning "expected str instance, LazyTitle found" in {% sitetree_page_title from menu %}

Django version: 3.2.9 & 4.0.4
Python version:3.8.10
OS: Ubuntu 20.04lts
I have a sitetree that is using a context variable in the title. When going to that path and loading the template with {% sitetree_page_title from menu %} it returns sequence item 1: expected str instance, LazyTitle found.
I didn't find any information on this exact issue, but it seems that when using context data in the title a LazyTitle object is generated. The object is not being converted to a string so the template engine is bombing.
I will include the relevant code below. I was able to get around the issue by editing sitetreeapp.py and wrapping the return of get_current_page_title() in str(), but that feels excessive.
The app in question is a small test app I'm working to layout an example CRUD app with the features our company needs. It does nothing fancy. The app isn't 100% complete yet so if anything looks out of place, it could be I haven't got to that point, but this portion should be working fine.
The sitetree in question is loaded dynamically via the config.ready method. Menu items without context variables are working without issue. I have verified that the context variable in question is available within the template. Hopefully it's something simple that I am overlooking. Any input is appreciated. I should also note that, while I have used StackOverflow for many years, I haven't posted much so please forgive my post formatting.
sitetree.py - note insert and list are working fine
from sitetree.utils import tree, item
sitetrees = [[
# setup the base tree
tree('app_model_test', items=[
# Then define items and their children with `item` function.
item('App Test', 'test_app:app_test_home',
hint='Test CRUD App',
children=[
item('Search Results', 'test_app:app_test_list',
in_menu=False, in_sitetree=False),
item('Insert Record', 'test_app:app_test_insert',
hint="Insert a new record",
),
item('Update Record {{ object.ssn }}', 'test_app:app_test_update object.id',
in_menu=False, in_sitetree=False),
])
]),
], ]
urls.py
app_name = 'test_app'
urlpatterns = [
path('insert/', views.app_test_form, name='app_test_insert'),
path('update/<int:id>/', views.app_test_form, name='app_test_update'),
path('list/', views.appTestListView.as_view(), name='app_test_list'),
]
views.py - relevant pieces
def app_test_form(request, action='add', id=0):
if id > 0:
test_obj = get_object_or_404(appTestModel, id=id)
form = appTestForm(instance=test_obj)
if request.method == 'GET':
if id == 0:
form = appTestForm()
test_obj = None
else:
if id == 0:
form = appTestForm(request.POST)
if form.is_valid():
form.save()
return redirect('/list')
# this will have to be somewhere else and contain whatever app apps are
# installed to get all the menu entries
menus = ['app_model_test']
context = {'form': form,
'title': 'app Test',
'action': action,
'menus': menus,
'object': test_obj}
return render(request, 'CRUD_base.html', context)
dyn_tree_register.py
from sitetree.sitetreeapp import register_dynamic_trees, compose_dynamic_tree
from sitetree.utils import tree, item
from . import sitetree as app_test_tree
register_dynamic_trees(
[
compose_dynamic_tree(*app_test_tree.sitetrees),
],
# Line below tells sitetree to drop and recreate cache, so that all newly registered
# dynamic trees are rendered immediately.
reset_cache=True
)
apps.py
from django.apps import AppConfig
class AppModelTestConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'app_model_test'
def ready(self):
from . import dyn_tree_register
and lastly the relevant template portion:
{% load sitetree %}
{% for menu in menus %}
<h1 class="title" id="page-title">{% sitetree_page_title from menu %}</h1>
{% endfor %}
As I mentioned, I can make this work by modifying sitetreeapp.py and wrapping the return of get_current_page_title in str():
def get_current_page_title(self, tree_alias: str, context: Context) -> str:
"""Returns resolved from sitetree title for current page.
:param tree_alias:
:param context:
"""
return str(self.get_current_page_attr('title_resolved', tree_alias, context))
Edit - 2022-04-13
As a temporary workaround I created a wrapper template tag that calls the sitetree_page_title and then renders it and wraps the output in str(). This seems hackish to me so I appreciate any insight. Short term this will get me by but I would rather not put something like this into production as I feel there's a bug on my side rather than within sitetree otherwise more people would have run into this.
custom tag and template.Node class
#register.tag()
def app_str_sitetree_page_title(parser, token):
'''
There is an issue causing sitetree's page title function to return a LazyPageTitle object which is not being converted to a string.
This function is a temporary fix until the issue is resolved to force the return to a string.
'''
from sitetree.templatetags import sitetree
ret = sitetree.sitetree_page_title(parser, token)
return StrSitetreePageTitle(ret)
class StrSitetreePageTitle(template.Node):
'''
This is the render wrapper to ensure a string is returned from sitetree_page_title
Like app_str_sitetree_page_title this is temporary
'''
def __init__(self, title_obj) -> None:
self.title_obj = title_obj
def render(self, context):
title_render = self.title_obj.render(context)
return str(title_render)

Django - Get response to return image without saving file to models

I am using the tmdb api to return movie info as well as images.
Steps below of Get logic
Api request is made which provides movie info as well as "backdrop_path"
I then use this path to make another request for the jpg related to that movie.
Blocker
I'm unable to then output that jpg. It currently returns a url path as below.
Views.py
from django.shortcuts import render
from django.views.generic import TemplateView
import requests
import urllib
# Create your views here.
def index(request):
# Query API with user input
if 'movie' in request.GET:
api_key = 'api'
id = request.GET['movie']
url = 'https://api.themoviedb.org/3/search/movie?api_key={}&language=en-US&query={}&include_adult=false'
response = requests.get(url.format(api_key,id))
# successful request
if response.status_code == 200:
# Parse json output for key value pairs
tmdb = response.json()
# save image jpg
backdrop_path = tmdb['results'][0]['backdrop_path']
url = 'https://image.tmdb.org/t/p/original/{}'
gg = urllib.request.urlretrieve(url.format(backdrop_path), 'test.jpg')
context = {
'title': tmdb['results'][0]['original_title'],
'overview': tmdb['results'][0]['overview'],
'release_date': tmdb['results'][0]['release_date'],
'vote_average': tmdb['results'][0]['vote_average'],
'vote_count': tmdb['results'][0]['vote_count'],
'backdrop_path' : tmdb['results'][0]['backdrop_path'],
'jpg' : gg
}
return render(request, 'home.html', {'context': context})
else: # returns homepage if invalid request
return render(request, 'home.html')
else: # Homepage without GET request
return render(request, 'home.html')
urlretrieve doesn't return the image ready to be used, instead it returns a tuple with the local name of the file and the headers (as an HTTPMessage object as you can see in your example), which is what you're seeing.
However, I don't think returning a file in your response is ideal, nor it would work in your scenario. Since you seem to be using templates, what I would do is return the image url and use it in an image HTML tag, like this <img src="{{ jpg }}"/>

Upload zip files to Google App Engine using Django

In a nutshell, we are trying to implement the following example from Google using Django:
from google.appengine.ext import blobstore
from google.appengine.ext.webapp import blobstore_handlers
class MainHandler(webapp2.RequestHandler):
def get(self):
upload_url = blobstore.create_upload_url('/upload')
self.response.out.write('<html><body>')
self.response.out.write('<form action="%s" method="POST" enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File: <input type="file" name="file"><br> <input type="submit"
name="submit" value="Submit"> </form></body></html>""")
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
upload_files = self.get_uploads('file') # 'file' is file upload field in the form
blob_info = upload_files[0]
self.redirect('/serve/%s' % blob_info.key())
Our app is written in Django 1.x. We use urls.py, views.py and models.py to define our url scheme, views and data models respectively.
In our database, we need to keep track of the data file (or at least, a reference to where the file exists on disk). Currently, our data model is as follows:
class FileData(models.Model):
token = models.CharField(max_length=10)
file = models.FileField(upload_to=get_upload_file_name, null=True, default=None, blank=True)
The field 'token' contains a token that client apps will use to request the file for download. The field 'file' is intended to contain the data. However, if we need to make it a reference to a blob location, it won't be a problem. We just don't know what to do at this point. What is the right solution?
Our urls pattern is as follows:
urlpatterns = patterns('',
url(r'^upload/', 'views.upload', name='upload'),
url(r'^/serve/([^/]+)?', 'views.servehandler', name='servehandler'),
)
Our form is as follows.
class DataUploadForm(forms.Form):
"""
FileData form
"""
token = forms.CharField(max_length=10)
file = forms.FileField()
class Meta:
model = FileData
fields = ['token', 'file', ]
def __init__(self, *args, **kwargs):
super(DataUploadForm, self).__init__(*args, **kwargs)
self.helper = FormHelper()
self.helper.form_tag = False
self.helper.layout = Layout(
'token',
'file',
ButtonHolder(
Submit('submit', 'Submit', css_class='btn btn-success btn-block'),
),
)
Our view is as follows.
def upload(request):
args = {}
args.update(csrf(request))
if request.method == 'POST':
print "In /upload"
form = DataUploadForm(request.POST, request.FILES)
if form.is_valid():
file_data = FileData()
file_data.token = request.POST.get('token')
file_data.file = request.FILES.get('file')
print "===========> Uploading:" + file_data.token
file_data.save()
return render(request, 'upload_success.html', args)
form = DataUploadForm()
args['form'] = form
return render(request, 'upload.html', args)
When we execute, we get:
Exception Value:
[Errno 30] Read-only file system: u
What are we doing wrong? How can we improve our code? How do we get Google App Engine to save our file, perhaps in the blobstore, and give us the reference to where the file exists for use in downloading it later?
Please be complete. Describe changes needed for urls.py, models.Model, views.py and yaml files (if necessary). Address how to upload large files (i.e., files greater than 40 MB each).
Please don't send us any more links. We have searched and seen a lot of postings-- none of which answers this question. If you can, please post a code snippets that answer the question -- not links.
The reason you have this Read-only file system error is that the default models.FileField save uploaded file to the MEDIA_ROOT folder and it's read-only for Google Appengine apps.
You have to use a third party FileField to save the uploaded files to services like Google Cloud Storage or Amazon S3.
I did a quick search and found:
https://github.com/viadanna/django-gcsengine
https://django-storages.readthedocs.org
And for the upload file size limit issue, Google Appengine has a size limit for request which is 32M. To support larger files, you have to make the user upload their files directly to BlobStore or Google Cloud Storage and build a callback handler to link the uploaded file with your model.
Check these links:
https://cloud.google.com/appengine/docs/python/blobstore/
Upload images/video to google cloud storage using Google App Engine

Creating PDFs with django (wkhtmltopdf)

Could someone please provide me with a comprehensive example of how to get a view in django to return a PDF using wkhtmltopdf. There are limited number of examples that come with django-wkhtmltopdf and they presume a level of knowledge I just don't have. I have looked through the source code but I can't make heads or tails of how to use it (for example whats the difference between PDFTemplateView and PDFTemplateResponse?!?)
I would be very grateful for any help.
BTW(I'm using templates for the header and footer as well)
EDIT
def some_view(request,sID):
something = get_object_or_404(Something,id=sID)
return render_to_response('something.html', {'something':something}, context_instance=RequestContext(request))
How would I get the following simple view to provide me with a pdf instead of an html file?
EDIT 2
I am currently playing around with:
def pdf_view(request,sID):
template = 'pdf.html'
something = get_object_or_404(Something,id=sID)
context = {
'something' : Something,
'object_for_header_and_footer': something.object_for_header_and_footer,
}
cmd_options = settings.WKHTMLTOPDF_CMD_OPTIONS
return PDFTemplateResponse(request=request,
context=context,
template=template,
filename='something',
header_template='header.html',
footer_template='footer.html',
cmd_options=cmd_options)
but I am getting 'str' object has no attribute 'update' in /usr/local/lib/python2.7/dist-packages/wkhtmltopdf/utils.py in wkhtmltopdf, line 74. I don't know whether to starting hacking wkhtmltopdf?!?!
The difference between PDFTemplateView and PDFTemplateResponse is that the view is a class-based view. And PDFTemplateResponse renders the pdf data and sets the right response headers. To add header and footer:
# urls.py
from django.conf.urls.defaults import *
from wkhtmltopdf.views import PDFTemplateView
urlpatterns = patterns('',
...
url(r'^pdf/$', PDFTemplateView.as_view(template_name='my_template.html',
filename='my_pdf.pdf',
header_template='my_header_template.html',
footer_template='my_footer_template.html',
...
), name='pdf'),
)
Opening pdf/ in your browser will start a download of my_pdf.pdf based on the my_template.html, my_header_template.html and my_footer_template.html.
The advanced example shows how to subclass PDFTemplateView extending and changing the logic of PDFTemplateView. To understand what happens read Using class based views.
Like header_template and footer_template you can define a response_class. Because PDFTemplateResponse is the default, you don't have to define it.
EDIT
The following simple view provides you with a pdf instead of an html. This is not using django-wkhtmltopdf. You could use wkhtmltopdf in your html2pdf function.
def some_view(request):
t = loader.get_template('myapp/template.html')
c = RequestContext(request, {'foo': 'bar'})
html = t.render(c)
pdf_data = html2pdf(html) # Your favorite html2pdf generator
response = HttpResponse(pdf_data, content_type='application/pdf')
response['Content-Disposition'] = 'attachment; filename="some_filename.pdf"'
return response
EDIT 2
A simple view with context:
template.html
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>Untitled</title>
</head>
<body>
<h1>{{ title }}</h1>
</body>
</html>
urls.py
from views import MyPDFView
urlpatterns = patterns('',
(r'^pdf/', MyPDFView.as_view()),
)
views.py
from django.views.generic.base import View
from wkhtmltopdf.views import PDFTemplateResponse
class MyPDFView(View):
template='template.html'
context= {'title': 'Hello World!'}
def get(self, request):
response = PDFTemplateResponse(request=request,
template=self.template,
filename="hello.pdf",
context= self.context,
show_content_in_browser=False,
cmd_options={'margin-top': 50,},
)
return response
EDIT 3
If you use a DetailView, you can add the object to context:
url(r'^books/(?P<pk>\d+)/$', MyPDFView.as_view(), name='book-detail'),
class MyPDFView(DetailView):
template='pdftestapp/template.html'
context= {'title': 'Hello World!'}
model = Book
def get(self, request, *args, **kwargs):
self.context['book'] = self.get_object()
response=PDFTemplateResponse(request=request,
template=self.template,
filename ="hello.pdf",
context=self.context,
show_content_in_browser=False,
cmd_options={'margin-top': 50,}
)
return response
Hmm the error indicates that you're passing string somewhere that you shouldn't.
After checking its source code, I guess in settings.py you have WKHTMLTOPDF_CMD_OPTIONS as a string, something like
WKHTMLTOPDF_CMD_OPTIONS = 'some_option_here'
But you should assign a dict there:
WKHTMLTOPDF_CMD_OPTIONS = {
'quiet': True,
}
Otherwise your code should work fine.