Removing query string from url in django while keeping GET information - django

I am working on a Django setup where I can receive a url containining a query string as part of a GET. I would like to be able to process the data provided in the query string and return a page that is adjusted for that data but does not contain the query string in the URL.
Ordinarily I would just use reverse(), but I am not sure how to apply it in this case. Here are the details of the situation:
Example URL: .../test/123/?list_options=1&list_options=2&list_options=3
urls.py
urlpatterns = patterns('',
url(r'test/(P<testrun_id>\d+)/'), views.testrun, name='testrun')
)
views.py
def testrun(request, testrun_id):
if 'list_options' in request.GET.keys():
lopt = request.GET.getlist('list_options')
:
:
[process lopt list]
:
:
:
:
[other processing]
:
:
context = { ...stuff... }
return render(request, 'test_tracker/testview.html', context)
When the example URL is processed, Django will return the page I want but with the URL still containing the query string on the end. The standard way of stripping off the unwanted query string would be to return the testrun function with return HttpResponseRedirect(reverse('testrun', args=(testrun_id,))). However, if I do that here then I'm going to get an infinite loop through the testrun function. Furthermore, I am unsure if the list_options data that was on the original request will still be available after the redirect given that it has been removed from the URL.
How should I work around this? I can see that it might make sense to move the parsing of the list_options variable out into a separate function to avoid the infinite recursion, but I'm afraid that it will lose me the list_options data from the request if I do it that way. Is there a neat way of simultaneously lopping the query string off the end of the URL and returning the page I want in one place so I can avoid having separate things out into multiple functions?
EDIT: A little bit of extra background, since there have been a couple of "Why would you want to do this?" queries.
The website I'm designing is to report on the results of various tests of the software I'm working on. This particular page is for reporting on the results of a single test, and often I will link to it from a bigger list of tests.
The list_options array is a way of specifying the other tests in the list I have just come from. This allows me to populate a drop-down menu with other relevant tests to allow me to easily switch between them.
As such, I could easily end up passing in 15-20 different values and creating huge URLs, which I'd like to avoid. The page is designed to have a default set of other tests to fill in the menu in question if I don't suggest any others in the URL, so it's not a big deal if I remove the list_options. If the user wishes to come back to the page directly he won't care about the other tests in the list, so it's not a problem if that information is not available.

First a word of caution. This is probably not a good idea to do for various reasons:
Bookmarking. Imagine that .../link?q=bar&order=foo will filter some search results and also sort the results in particular order. If you will automatically strip out the querystring, then you will effectively disallow users to bookmark specific search queries.
Tests. Any time you add any automation, things can and will probably go wrong in ways you never imagined. It is always better to stick with simple yet effective approaches since they are widely used thus are less error-prone. Ill give an example for this below.
Maintenance. This is not a standard behavior model therefore this will make maintenance harder for future developers since first they will have to understand first what is going on.
If you still want to achieve this, one of the simplest methods is to use sessions. The idea is that when there is a querystring, you save its contents into a session and then you retrieve it later on when there is no querystring. For example:
def testrun(request, testrun_id):
# save the get data
if request.META['QUERY_STRING']:
request.session['testrun_get'] = request.GET
# the following will not have querystring hence no infinite loop
return HttpResponseRedirect(reverse('testrun', args=(testrun_id,)))
# there is no querystring so retreive it from session
# however someone could visit the url without the querystring
# without visiting the querystring version first hence
# you have to test for it
get_data = request.session.get('testrun_get', None)
if get_data:
if 'list_options' in get_data.keys():
...
else:
# do some default option
...
context = { ...stuff... }
return render(request, 'test_tracker/testview.html', context)
That should work however it can break rather easily and there is no way to easily fix it. This should illustrate the second bullet from above. For example, imagine a user wants to compare two search queries side-by-side. So he will try to visit .../link?q=bar&order=foo and `.../link?q=cat&order=dog in different tabs of the same browser. So far so good because each page will open correct results however as soon as the user will try to refresh the first opened tab, he will get results from the second tab since that is what is currently stored in the session and because browser will have a single session token for both tabs.
Even if you will find some other method to achieve what you want without using sessions, I imagine that you will encounter similar issues because HTTP is stateless hence you will have to store state on the server.
There is actually a way to do this without breaking much of the functionality - store state on client instead of server-side. So you will have a url without a querystring and then let javascript query some API for whatever you will need to display on that page. That however will force you to make some sort of API and use some javascript which does not exactly fall into the scope of your question. So it is possible to do cleanly however that will involve more than just using Django.

Related

Is there an HTML sanitizer library for django to block injection attacks?

I'm trying to find something that will return an exception upon finding anything that even remotely looks like HTML or Javascript. I've figured out how to do it for individual views, but it's not a scalable solution, and ultimately I need to prevent code from being saved to the database no matter what view gets targeted by the injection attack.
Here is the functionality I'm looking for.
ILLEGAL_CHARS = '<>[]{}():;,'.split()
# bunch of code in between
for value in [company_name, url, status, information, lt_type, company_source]:
if any(char in value for char in ILLEGAL_CHARS):
raise Exception(f"You passed one of several illegal characters: {ILLEGAL_CHARS}")
I'm using django rest framework so I have to handle it on the backend. Thanks.
actually you don't nead to sanitize any user input because when you show them int the template the jinja {{object}} will make sure that no html or java script will be executed until you mark them as safe {{object|safe}} but if you want want not to save them in database that might help Sanitizing HTML in submitted form data

How to get data by selecting a value from a drop-down option without using selenium

I need to fetch all URLs from this page -
http://www.questdiagnostics.com/testcenter/BUSearch.action?submitValue=BUSearch&keyword=Toxoplasma+Abs+IgG+%2F+IgM
whenever I am selecting a value from a drop down and click on go button.
I selected a value from dropdown option by using xpath. But i can't able to click on go button.
My code is:
import requests
from lxml.html import fromstring
req = requests.get('http://www.questdiagnostics.com/testcenter/BUSearch.action?submitValue=BUSearch&keyword=Toxoplasma+Abs+IgG+%2F+IgM')
hdoc = lxml.html.fromstring(req.content)
hdoc.xpath('//select[#id="labs"]/option/text()')
How to get all links without using selenium?
Normal Use Case
lxml is a great library, and it has decent support for filling out and submitting forms, as documented here. The real challenge for this particular use case is rooted in the way the form works.
The regional laboratory select box is not part of the form; its value is submitted with a cookie instead. This makes things a little more difficult.
If this wasn't the case, you could just issue your GET, pull the form out of it, change the values you're interested in, submit it, and examine the links that come back. That script might look something like this:
req = requests.get('http://www.questdiagnostics.com/testcenter/BUSearch.action?submitValue=BUSearch&keyword=Toxoplasma+Abs+IgG+%2F+IgM')
hdoc = lxml.html.fromstring(req.content)
form = hdoc.forms[1]
# Set form inputs using `form.fields = dict(...)`
form.action = "http://www.questdiagnostics.com" + form.action
submitResult = lxml.html.parse(lxml.html.submit_form(form)).getroot()
links = submitResult.xpath('//*[#id="maincolumn"]/ol/li/a[#class="title"]/#href')
While you can add arbitrary request parameters when calling lxml.html.submit_form(), I don't see a way to add arbitrary cookies.
This Use Case
That said, since this form essentially works by redirecting back to itself (with an additional cookie to identify the lab), you could simulate this behavior by just adding the cookie to your initial GET. You might not need to mess around with a form submission at all. This script will show the first ten links for the SKB lab:
cookies = dict(TC11SelectedLabCode='SKB')
req = requests.get('http://www.questdiagnostics.com/testcenter/BUSearch.action?submitValue=BUSearch&keyword=Toxoplasma+Abs+IgG+%2F+IgM', cookies=cookies)
hdoc = lxml.html.fromstring(req.content)
links = hdoc.xpath('//*[#id="maincolumn"]/ol/li/a[#class="title"]/#href')
print(links)
You could take this a step further, and issue a GET with no cookies to obtain the list of labs, and then iterate over that list, calling requests.get() on each one, sending the appropriate TC11SelectedLabCode cookie to simulate the form submission.
Notes
Note that while lxml has decent form submission support, you're not actually clicking anything. There is nothing "breathing life into" the DOM.
None of the javascript on the page is running.
To illustrate why this is important, consider this example. If you wanted to verify the links on page 2 of the results, I can't say how you'd accomplish that. If your tests need to exercise javascript on the page, I think you'll need more than requests and lxml.

CFWheels: Redirect to URL with Params Hidden

I am using redirectTo() function with params to redirect to another pages with a query string in the url. For security purpose this does not look appealing because the user can change the parameters in the url, thus altering what is inserted into the database.
My code is:
redirectTo(action="checklist", params="r=#r#&i=#insp#&d=#d#");
Is there anyway around this? I am not using a forms, I just wish to redirect and I want the destination action/Controller to know what I am passing but not display it in the url.
You can obfuscate the variables in the URL. CfWheels makes this really easy.
All you have to do is call set(obfuscateURLs=true) in the config/settings.cfm file to turn on URL obfuscation.
I am sure this works with linkTo() function. I hope it works with RedirectTo() funcation as well. I do not have a set up to check it now. But if doesn't work for RedirectTo(), you can obfuscateParam() and deObfuscateParam() functions to do job for you.
Caution: This will only make harder for user to guess the value. It doesn't encrypt value.
To know more about this, Please read the document configuration and defaults and obfuscating url
A much better approach to this particular situation is to write params to the [flash].1 The flash is exactly the same thing as it is in Ruby on Rails or the ViewBag in ASP.Net. It stores the data in a session or cookie variable and is deleted at the end of the next page's load. This prevents you from posting back long query strings like someone that has been coding for less than a year. ObfuscateParam only works with numbers and is incredibly insecure. Any power user can easily deobfuscate, even more so with someone that actually makes a living stealing data.

Ember dynamic query parameters

I have what I believe to be common but complicated problem to model. I've got a product configurator that has a series of buttons. Every time the user clicks on a button (corresponding to a change in the product configuration), the url will change, essentially creating a bookmarkable state to that configuration. The big caveat: I do not get to know what configuration options or values are until after app initialization.
I'm modeling this using EmberCLI. After much research, I don't think it's a wise idea to try to fold these directly into the path component, and I'm looking into using the new Ember query string additions. That should work for allowing bookmarkability, but I still have the problem of not knowing what those query parameters are until after initialization.
What I need is a way to allow my Ember app to query the server initially for a list of parameters it should accept. On the link above, the documentation uses the parameter 'filteredArticles' for a computed property. Within the associated function, they've hard-coded the value that the computed property should filter by. Is it a good idea to try to extend this somehow to be generalizable, with arguments? Can I even add query parameters on the fly? I was hoping for an assessment of the validity of this approach before I get stuck down the rabbit hole with it.
I dealt with a similar issue when generating a preview popup of a user's changes. The previewed model had a dynamic set of properties that could not be predetermined. The solution I came up with was to base64 encode a set of data and use that as the query param.
Your url would have something like this ?filter=ICLkvaDlpb0iLAogICJtc2dfa3
The query param is bound to a 2-way computed that takes in a base64 string and outputs a json obj,
JSON.parse(atob(serializedPreview));
as well as doing the reverse: take in a json obj and output a base64 string.
serializedPreview = btoa(JSON.stringify(filterParams));
You'll need some logic to prevent empty json objects from being serialized. In that case, you should just set the query param as null, and remove it from your url.
Using this pattern, you can store just about anything you want in your query params and still have the url as shareable. However, the downside is that your url's query params are obfuscated from your users, but I imagine that most users don't really read/edit query params by hand.

Mediawiki mass user delete/merge/block

I have 500 or so spambots and about 5 actual registered users on my wiki. I have used nuke to delete their pages but they just keep reposting. I have spambot registration under control using reCaptcha. Now, I just need a way to delete/block/merge about 500 users at once.
You could just delete the accounts from the user table manually, or at least disable their authentication info with a query such as:
UPDATE /*_*/user SET
user_password = '',
user_newpassword = '',
user_email = '',
user_token = ''
WHERE
/* condition to select the users you want to nuke */
(Replace /*_*/ with your $wgDBprefix, if any. Oh, and do make a backup first.)
Wiping out the user_password and user_newpassword fields prevents the user from logging in. Also wiping out user_email prevents them from requesting a new password via email, and wiping out user_token drops any active sessions they may have.
Update: Since I first posted this, I've had further experience of cleaning up large numbers of spam users and content from a MediaWiki installation. I've documented the method I used (which basically involves first deleting the users from the database, then wiping out up all the now-orphaned revisions, and finally running rebuildall.php to fix the link tables) in this answer on Webmasters Stack Exchange.
Alternatively, you might also find Extension:RegexBlock useful:
"RegexBlock is an extension that adds special page with the interface for blocking, viewing and unblocking user names and IP addresses using regular expressions."
There are risks involved in applying the solution in the accepted answer. The approach may damage your database! It incompletely removes users, doing nothing to preserve referential integrity, and will almost certainly cause display errors.
Here a much better solution is presented (a prerequisite is that you have installed the User merge extension):
I have a little awkward way to accomplish the bulk merge through a
work-around. Hope someone would find it useful! (Must have a little
string concatenation skills in spreadsheets; or one may use a python
or similar script; or use a text editor with bulk replacement
features)
Prepare a list of all SPAMuserIDs, store them in a spreadsheet or textfile. The list may be
prepared from the user creation logs. If you do have the
dB access, the Wiki_user table can be imported into a local list.
The post method used for submitting the Merge & Delete User form (by clicking the button) should be converted to a get method. This
will get us a long URL. See the second comment (by Matthew Simoneau)
dated 13/Jan/2009) at
http://www.mathworks.com/matlabcentral/newsreader/view_thread/242300
for the method.
The resulting URL string should be something like below:
http: //(Your Wiki domain)/Special:UserMerge?olduser=(OldUserNameHere)&newuser=(NewUserNameHere)&deleteuser=1&token=0d30d8b4033a9a523b9574ccf73abad8%2B\
Now, divide this URL into four sections:
A: http: //(Your Wiki domain)/Special:UserMerge?olduser=
B: (OldUserNameHere)
C: &newuser=(NewUserNameHere)&deleteuser=1
D: &token=0d30d8b4033a9a523b9574ccf73abad8%2B\
Now using a text editor or spreadsheet, prefix each spam userIDs with part A and Suffix each with Part C and D. Part C will include the
NewUser(which is a specially created single dummy userID). The Part D,
the Token string is a session-dependent token that will be changed per
user per session. So you will need to get a new token every time a new
session/batch of work is required.
With the above step, you should get a long list of URLs, each good to do a Merge&Delete operation for one user. We can now create a
simple HTML file, view it and use a batch downloader like DownThemAll
in Firefox.
Add two more pieces " Linktext" to each line at
beginning and end. Also add at top and at
bottom and save the file as (for eg:) userlist.html
Open the file in Firefox, use DownThemAll add-on and download all the files! Effectively, you are visiting the Merge&Delete page for
each user and clicking the button!
Although this might look a lengthy and tricky job at first, once you
follow this method, you can remove tens of thousands of users without
much manual efforts.
You can verify if the operation is going well by opening some of the
downloaded html files (or by looking through the recent changes in
another window).
One advantage is that it does not directly edit the
MySQL pages. Nor does it require direct database access.
I did a bit of rewriting to the quoted text, since the original text contains some flaws.