I am new to scrapy and in general web tech.
While working on a scrapy example to perform auto login. I came across 1 field , referrer url . I am wondering when do i need to this.
return scrapy.FormRequest.from_response(
response,
url='www.myreferrer.com', #when do i need this ???
formnumber=1,
formdata=self.data['formdata'],
callback=self.after_login
)
I tested with and without it and it works in both instances.
I understand that referrer url is for security but how do i determine from html code that i need or dont need this ?
ADDON
The following html form required the url to be defined :
<form id="login" enctype="multipart/form-data" method="post" action="https:///myshop.com/login/index.php?route=account/login">
I am a returning customer.<br>
<br>
<b>E-Mail Address:</b><br>
<input type="text" name="email">
<br>
<br>
<b>Password:</b><br>
<input type="password" name="password">
<br>
Forgotten Password<br>
<div style="text-align: right;"><a class="button" onclick="$('#login').submit();"><span>Login</span></a></div>
</form>`
class FormRequest(Request):
# delete some code here
#classmethod
def from_response(cls, response, formname=None, formid=None, formnumber=0, formdata=None,
clickdata=None, dont_click=False, formxpath=None, formcss=None, **kwargs):
url = _get_form_url(form, kwargs.pop('url', None))
def _get_form_url(form, url):
if url is None:
return urljoin(form.base_url, form.action)
return urljoin(form.base_url, url)
if the url is empty, it uses form tag's action attribute to get the URL.
if the url is not empty, then it use the URL you give to it.
the base_url comes from the response.
def _get_form(response, formname, formid, formnumber, formxpath):
"""Find the form element """
root = create_root_node(response.text, lxml.html.HTMLParser,
base_url=get_base_url(response))
so, when the action attribute does not exist or the login requests is not sent to the action URL, you need to pass the argument.
Related
I have the code below in my Python script:
def cmd_wui(argv, path_to_tx):
"""Run a web UI."""
from flask import Flask, flash, jsonify, render_template, request
import webbrowser
app = Flask(__name__)
#app.route('/tx/index/')
def index():
"""Load start page where you select your project folder
or load history projects from local DB."""
from txclib import get_version
txc_version = get_version()
prj = project.Project(path_to_tx)
# Let's create a resource list from our config file
res_list = []
prev_proj = ''
for idx, res in enumerate(prj.get_resource_list()):
hostname = prj.get_resource_host(res)
username, password = prj.getset_host_credentials(hostname)
return render_template('init.html', txc_version=txc_version, username=username)
Also, I have an HTML form in init.html:
<form>
<input type="text" id="projectFilepath" size="40" placeholder="Spot your project files">
<input type="button" id="spotButton" value="Spot">
</form>
How can I pass the user input from "projectFilepath" when a user clicks "spotButton" on a variable in my python script?
I'm new in Python and Flask, so forgive me if I make any mistakes.
The form tag needs some attributes set:
action: The URL that the form data is sent to on submit. Generate it with url_for. It can be omitted if the same URL handles showing the form and processing the data.
method="post": Submits the data as form data with the POST method. If not given, or explicitly set to get, the data is submitted in the query string (request.args) with the GET method instead.
enctype="multipart/form-data": When the form contains file inputs, it must have this encoding set, otherwise the files will not be uploaded and Flask won't see them.
The input tag needs a name parameter.
Add a view to handle the submitted data, which is in request.form under the same key as the input's name. Any file inputs will be in request.files.
#app.route('/handle_data', methods=['POST'])
def handle_data():
projectpath = request.form['projectFilepath']
# your code
# return a response
Set the form's action to that view's URL using url_for:
<form action="{{ url_for('handle_data') }}" method="post">
<input type="text" name="projectFilepath">
<input type="submit">
</form>
You need a Flask view that will receive POST data and an HTML form that will send it.
from flask import request
#app.route('/addRegion', methods=['POST'])
def addRegion():
...
return (request.form['projectFilePath'])
<form action="{{ url_for('addRegion') }}" method="post">
Project file path: <input type="text" name="projectFilePath"><br>
<input type="submit" value="Submit">
</form>
i am trying to build a django search functionality for my app but the input form keeps returning a none
views.py
def search(request):
if request.method == 'POST':
query = request.POST.get('text')
houses = Product.objects.filter(name__contains='query')
context = {
'houses':houses,
}
return render (request, 'searchresult.html', context)
search.html
<form>
<input type='text' placeholder='search houses>
<button type='submit'>Search</button>
</form>
First off, your python indentation is invalid, and your HTML is also invalid on the input line. I will assume this is a typo in the question, but if not, you have issues there.
Your main problem is the filter for houses:
houses = Product.objects.filter(name__contains='query')
is looking for a name containing the string "query". You need the variable you've just defined.
houses = Product.objects.filter(name__contains=query)
You have an indentation issue in the code you have posted.
You need to add action and method in your Form.
<form action="/url_of_search/" method="post">
Missing quote in input line.
<input type='text' placeholder='search houses'>
You need to use query instead of 'query' in the filter.
Product.objects.filter(name__contains=query)
Things missing in html code:
form action attribute
form method attribute
input field name attribute
<!-- add form attributes method and action -->
<form method="POST" action="{% url '<url_name>' %}">
<!-- add input attribute name to identify the field and pass the value in request body -->
<input type='text' placeholder='search houses' name='search_text'>
<button type='submit'>Search</button>
</form>
update views for search
def search(request):
if request.method == 'POST':
# use input field name to get the search text
query = request.POST.get('search_text')
houses = Product.objects.filter(name__contains=query)
context = {
'houses':houses,
}
return render (request, 'searchresult.html', context)
I would like to redirect from my page to external site. This external site address is defined and never changes... And I can implement it... The challenge comes from:
I have a form, which posts Q into my view function, which gives final string of parameters to pass into the pattern of url supposedly attached to the main address of this external web site after the ? mark... I just cannot find the information how to pass and attach this string to a given url. Somehow it's difficult for me, please don't judge too harsh. Parameter is only one, it is a string, which formed in view function.
I use RedirectView in urls.py:
path('do_request/found/', RedirectView.as_view(url='https://docs.djangoproject.com/'), name='found_objects'),
In views.py (there are probably some mistakes, but it's just for the purpose of the question):
class UserFoundRequestView(TemplateView):
model=ForQuery
template_name='found_objects.html'
def get_results(self, request):
our_queries = ForQuery.objects.all()
query=self.request.GET.get("q")
if query:
our_object=ForQuery.objects.filter(query_id__iexact=query)
for x in our_queries:
if x.query_id == our_object.query_id:
name = x.dep_station
#context = {
# 'name' : name,
#}
return name
def get_string(request):
base_url = 'https://docs.djangoproject.com/'
query_name = get_results()
query_string = urlencode(query_name)
url = '{}?{}'.format(base_url, query_string)
return redirect(url)
Either to append to url or just bring formed url to patterns->path...
Thank you for the help!
my template:
{% extends "base_generic.html" %}
{% block content %}
<div>
<p>Paste value to the field</p>
<form method="get" action="{% url 'found_objects' %}">
<input type="text" name="q" placeholder="search by value" class="m-1 p-1"><button class=" btn btn-outline-primary m-1 p-1" type="Submit">Search</button>
</form>
</div>
{% endblock %}
You have to trigger the view by submitting the form and the view itself will return the according redirect. Between is the url mapping that links the form submit to the according view like so:
urls.py
from your_app.views import redirect_view
path('do_request/found/', redirect_view, name='found_objects'),
views.py
def redirect_view(request):
our_queries = ForQuery.objects.all()
query = request.GET.get("q")
if query:
our_object=ForQuery.objects.filter(query_id__iexact=query)
for x in our_queries:
if x.query_id == our_object.query_id:
name = x.dep_station
base_url = 'https://docs.djangoproject.com/'
query_string = urlencode(name)
url = '{}?{}'.format(base_url, query_string)
return redirect(url)
I am trying to create a simple search form(search by zip code), but am struggling to pass the user's input to a view:
<form action="{% url 'search_results' query %}" method="post">
<div>
{% csrf_token %}
<input type = 'text' name = 'query' placeholder="Zip Code" />
<button type="submit" name="button">Find Jobs</button>
</div>
</form>
urls.py:
path('search_results/<str:query>', job_views.search_results, name = 'search_results'),
views.py:
def search_results(request, query):
query = request.GET['query']
return HttpResponse(query) # just trying to see if this view has access to 'query'
I'm not sure what is going on here. This returns
raise NoReverseMatch(msg)
django.urls.exceptions.NoReverseMatch: Reverse for 'search_results' with a
rguments '('',)' not found. 1 pattern(s) tried: ['search_results\\/(?P<que
ry>[^/]+)$']
Thanks for any help!
In your form, at the line: action="{% url 'search_results' query %}", you are trying to pass query to the url tag, which would be fine if query had a value, but in your case query isn't defined.
Instead, you're form should look like this:
<form action="{% url 'search_results' %}" method="GET">
<div>
<input type = "text" name = "zip_code" placeholder="Zip Code" />
<button type="submit" name="button">Find Jobs</button>
</div>
</form>
Notice that I removed query from your url tag, and changed the method to GET, see this question for details.
Now change your path:
path('search_results', job_views.search_results, name = 'search_results')
You don't need to add query to the path, because Django will do it for you in your view:
def search_results(request):
zip_code = request.GET.get('zip_code')
print(zip_code)
...
That's not how templates work.
Templates are used to produce a response, which gets returned to the requester. In this case, your main view produces an HTML document which will be passed to a browser. That document contains results of the url template tag, which is determined before the user even sees it. Template tags are not a way to refer to the state of DOM objects!
Instead, you should usually have a view with a stable URL, not varying with the query. Typically you'll then extract the query text from the form data representation - here, it'd be the request.POST dictionary-like object or (usually better) a Django form that you bind to the POST data, because your form uses method="post".
It would be possible to use Javascript to edit the DOM and change your form action prior to submitting it to include the query text, but that's not the usual pattern for working with forms. And template tags will never do that - they're only used to generate the response.
Summary: I am trying to build a job site. On index.html the user enters a zip code into a form to see jobs in that zip code, this form is handled with the job_query view. This brings them to another page(search.html) where at first you only see jobs in that specific zip code but I am trying to add a filter that lets the user see jobs within X miles. How can I pass the zip code value entered in the from on index.html to the next page?
index.html:
<h2>Find a Job</h2>
<!--Search Bar-->
<form method = "GET" action = "{% url 'search' %}" >
<div id = "form_grid">
<input name="query" type="text" placeholder="Zip Code">
<button type="submit">Search</button>
</div>
</form>
search.html:
<form method = "GET" action = "{% url 'search' %}" >
<input class="search_bar" name="query" type="text" placeholder="Zip Code">
<button class="search_btn btn btn-outline-success " type="submit">Find Jobs</button>
</form>
<form id="within_miles_form" method = "GET" action = "{% url 'within_miles' %}" >
<input class="search_bar" name="miles" type="text" placeholder="Within X miles of Zip Code">
<button type="submit">Filter</button>
</form>
<!--code to display jobs-->
views.py:
def job_query(request):
if request.method == "GET":
query = request.GET.get('query')
jobs_matching_query = Job.objects.filter(zip_code__iexact = query) | Job.objects.filter(city__iexact=query) | Job.objects.filter(state__iexact=query)
number_of_results = 0
for job in jobs_matching_query:
number_of_results = number_of_results + 1
return render(request, 'core/search.html', {'query': query ,'jobs_matching_query': jobs_matching_query, 'number_of_results': number_of_results})
def within_miles(request):
miles = request.GET['miles']
#how can i use value of the zip code entered?
urls.py:
url(r'^search$', views.job_query, name="search"),
url(r'within_miles', views.within_miles, name="within_miles"),
I think I included all the relevant info but if I am missing something please let me know, thanks in advance for any help.
You can encode the entered ZIP in a URL, pass it through cookies, store it in the session variables, or use a (hidden) input element that forces the browser to pass it through a GET and POST request.
Encode it in the URL
In that case we can rewrite the URL to:
url(r'^within_miles/(?P<zip>[0-9]{5})/$', views.within_miles, name="within_miles"),
So now one can no longer fetch your.domain.com/within_miles, but your.domain.com/within_miles/12345. It makes it easy for a user to "manipulate" the URL, but since the user can probably provide any ZIP, there is probably not much gain to protect that.
In the form, the URL that is generated is thus:
{% url 'within_miles' zip=query %}
(you can use another variable that is more strictly a ZIP code)
You should thus ensure that query is here a five digit string (or otherwise change the expression in the url(..) part such that it allows all possible queries).
Using hidden form elements
We can also encode content in hidden form elements, for example here we can create an element in the form:
<form id="within_miles_form" method = "GET" action = "{% url 'within_miles' %}" >
<input class="search_bar" name="miles" type="text" placeholder="Within X miles of Zip Code">
<input type="hidden" name="zip_code" value="{{ query }}">
<button type="submit">Filter</button>
</form>
We thus add a form element, fill it with some data, and let the browser submit the value again to the next view. Note that again it is the browser that does this, so a user can inspect the DOM (most browsers allow that, and subsequently edit it).
Using session variables and/or cookies
You can also decide to use session variables (stored at server side, so "secure") or cookies (stored at client side, can be tampered with). A potential problem however is that these are stored in the browser, and changes to the cookies in one tab page, thus can have effect in the other tab page. Furthermore cookies and sessions will "die" after the request, and thus can create a lot of trouble in future views.
You can set a session variable in the view with:
request.session['zip_code'] = query
This will thus store an entry at the server side such that another call can retrieve that value again. The request.session acts like a dictionary that keeps some sort of state per session.
setting and obtaining session variables
In another view, you can thus query the request.session, like:
zip_code = request.session.get('zip_code')
setting and obtaining cookies
We can use a similar approach with cookies. A browser however might reject cookies, or manipulate them, so there are not that much guarantees that there is no tampering with the data (in fact there are none). You can set a cookie with:
response = render(request, 'core/search.html', {'query': query ,'jobs_matching_query': jobs_matching_query, 'number_of_results': number_of_results})
response.set_cookie('zip_code', query)
return response
Before we thus return the result of render(..), we call .set_cookie(..) on the result.
We can - for example in a later view - retrieve the content with:
zip_code = request.COOKIES.get('zip_code')
Improving the job_query view
The job_query view however looks a bit strange: it uses all kinds of "uncommon" code practices. For example the number of elements is calculated by iterating over it, instead of taking the len(..). This also looks basically like a ListView [Django-doc] and we can make the query more elengant by using Q-objects [Django-doc]. The listview then looks like:
def JobListView(ListView):
model = Job
context_object_name = 'jobs_matching_query'
template_name = 'core/search.html'
def get_context_data(self, **kwargs):
kwargs = super(JobListView, self).get_context_data(**kwargs)
kwargs.update(
number_of_results=len(kwargs['object_list'],
query = self.request.GET.get('query')
)
return kwargs
In the view, you then not pass the JobListView, but JobListView.as_view() result as a reference.