I'm trying to make a simple webscaper using Python and the requests library.
r=requests.get(https://nustar.newcastle.edu.au/psp/CS9PRD/EMPLOYEE/HRMS/c/MANAGE_ACADEMIC_RECORDS.STDNT_ACTIVATION.GBL?FolderPath=PORTAL_ROOT_OBJECT.HCSR_RECORDS_AND_REGISTRATION.HCSR_STUDENT_TERM_INFORMATION.HC_STDNT_ACTIVATION_GBL&IsFolder=false&IgnoreParamTempl=FolderPath%2cIsFolder
I would like to POST a search input into this URL, but I'm struggling to work out how.
This is the search box code from the website:
<input id="STDNT_SRCH_EMPLID" class="PSEDITBOX" type="text" maxlength="11" style="width:140px; " value="" tabindex="13" name="STDNT_SRCH_EMPLID"></input>
I assume I have to somehow change value = "" to value = "foo".
Any help will appreciated, thanks.
See request's quick start here.
import requests
value1='foo'
payload = {'STDNT_SRCH_EMPLID': value1} # 'key2': 'value2' and so on (comma delimited)
r = requests.post("http://yourUrl.org/", data=payload)
print(r.text)
Do a network analysis in the developer tool of your browser and copy the curl command of the POST package.
Then you surf to [curl.trillworks.com][1] and convert the curl command by pasting it into a Python POST request.
Inside of your python request you can modify the values.
Related
I spent a lot of time trying to get data from POST request in my django 1.8 application and it doesn't work whatever I do. Hopping that somebody more skillful suggest the solution.
I need to get data from the request:
curl.exe -X POST --data "Status=OK" http://localhost:8000/postback
relevant content of urls.py :
url(r'^postback', 'app.views.process_postback'),
relevant content of views.py :
#csrf_exempt
def process_postback(request):
if request.method == 'POST':
dict = request.POST
print(dict)
return HttpResponse("OK")
I always get empty value in dict :
My question how to get Status from post to inside my app
Here is few more details:
I process postbacks from third party website, so I cannot change content of POST request
I don't need(want) any forms/models for the request, I just need to collect value of Status.
Thanks for help.
I finally figured out what's was wrong. I use VS 2013 with PTVS (Python Tool for Visual Studio) as my development environment. And it appears that if project launched under debugger then http request gets screwed up. If I run project without debugger then everything works fine. I tried to look at the debugger code (C:\Program Files\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\2.2\visualstudio_py_debugger.py) but didn't find some obvious things that would cause such behavior. If anybody came across it and knows how to fix it would be great to know.
Following the https://superuser.com/questions/149329/what-is-the-curl-command-line-syntax-to-do-a-post-request
Remove -X POST option from Your command
curl.exe --data "Status=OK" http://localhost:8000/postback
Output:
<QueryDict: {u'Status': [u'OK']}>
[04/Dec/2015 21:03:15] "POST /postback HTTP/1.1" 200 2
pip freeze output:
Django==1.8.6
I used django 1.10.so i only can give some advice.when i want to get the value from the form ,no matter GET or POST,i useddict=request.POST.get('dict')
The second 'dict' is your form's input name .Such as in html , it's should be `
<form name="input" method="post">
<input type="text" name="dict" >
<input type="submit" value="submit">
</form>`
I do a web crawling use scrapy. currently, it can extract the start url but not crawl later.
start_urls = ['https://cloud.cubecontentgovernance.com/retention/document_types.aspx']
allowed_domains = ['cubecontentgovernance.com']
rules = (
Rule(LinkExtractor(allow=("document_type_retention.aspx?dtid=1054456",)),
callback='parse_item', follow=True),
)
And the link i want to extract in the develop tool is:<a id="ctl00_body_ListView1_ctrl0_hyperNameLink" href="document_type_retention.aspx?dtid=1054456"> pricing </a>
the corresponding url is https://cloud.cubecontentgovernance.com/retention/document_type_retention.aspx?dtid=1054456
so what the allow field should be? thanks a lot
When I try to open the site of your start URL I get a login window.
Did you try to print response.body in the simple parse method for your start URL? I guess your Scrapy instance gets the same login window which does not have the URL you want to extract with the LinkExtractor.
How can I retrieve the number of Likes or Dislikes of a video using Python?
The entry.rating element will show me:
<ns0:rating xmlns:ns0="http://schemas.google.com/g/2005" average="4.936976" max="5" min="1" numRaters="101501" rel="http://schemas.google.com/g/2005#overall" />
Which according to developers.google.com, is a deprecated element
<gd:rating>
but I don’t know how to use the new element
<yt:rating>.
Can someone help me?
Thank you.
you can use pafy api
http://np1.github.io/pafy/
import pafy
url = "https://www.youtube.com/watch?v=bMt47wvK6u0"
video = pafy.new(url)
print video.likes
print video.dislikes
You can use Data API v3 instead.
videos->rate and videos->getRating are your calls.
You can use Python library and checkout Python code samples.
My company gave me the task of resolving all security issues with a particular application. The security tream reported a cross site scripting error. The error lies in the following input field:
<input type="hidden" name="eventId" value="${param.eventId}"/>
The report from security wasn't very detailed, but the say they can make a POST request to the page that has the above tag including the following malicious code:
eventId=%22%3e%3csCrIpT%3ealert(83676)%3c%2fsCrIpT%3e
And that when the page reloads, it will have the following:
<input type="hidden" name="eventId" value=""><sCrIpt>alert(83676)</sCrIpt></value>
I am trying to "be the hacker" and show the vulnerability. But I can't figure out how they manage to get that script in there. I am guessing they include it as a URL parameter in the GET request for the form, but when I try to do it myself I get a 403 error. Does anyone know how the vulnerability can be shown?
I know there is a number of XSS questions on the site, but none seem to hit this topic.
So, I am not sure why, but my original hunch was correct. The script can be put on as a URL parameter. For some reason though, this was not working with our staging site. Only with running the application locally. I am not sure why, but this works (only locally):
http://localhost:8080/myUrl/MyAction.do?eventId=%22%3e%3csCrIpT%3ealert(83676)%3c%2fsCrIpT%3e
Doing that, you see an alert box pop up. I am planning to fix it using JSTL functions.
<%# taglib uri="http://java.sun.com/jsp/jstl/functions" prefix="fn" %>
...
<input type="hidden" name="eventId" value="${fn:escapeXml(param.eventId)}"/>
Install [TamperData][1] add-on in firefox browser which let you edit the data before submitting. Doesn't matter if it's in POST or GET.
By using this hidden fields can be edited.
What you want to do to fix the problem, is to HTMLAttributeEncode the value before putting it inside the value-attribute. See OWASP ESAPI or MS AntiXSS for methods for doing HTML attribute encoding.
Seeing how the attack string is URL encoding, I think you guess about including it as a GET parameter seems reasonable.
I used the OWASP ESAPI API as the legacy jsp's didn't have JSTL available. This is what I used:
<input type="hidden" name="dataValue" value="<%=ESAPI.encoder().encodeForHTMLAttribute(dataValue)%>">
You can also use the API to filter request.Parameter() which I also needed, as in:
String userURL = request.getParameter( "userURL" )
boolean isValidURL = ESAPI.validator().isValidInput("URLContext", userURL, "URL", 255, false);
if (isValidURL) {
link
}
and:
String name = (String) request.getParameter("name");
name = ESAPI.validator().getValidInput("name ", name , "SafeString", 35, true);
I have a problem with the jquery-autocomplete pluging and my django script. I want an easy to use autocomplete plugin. And for what I see this (http://code.google.com/p/jquery-autocomplete/) one seems very usefull and easy. For the django part I use this (http://code.google.com/p/django-ajax-selects/) I modified it a little, because the out put looked a little bit weired to me. It had 2 '\n' for each new line, and there was no Content-Length Header in the response. First I thought this could be the problem, because all the online examples I found had them. But that was not the problem.
I have a very small test.html with the following body:
<body>
<form action="" method="post">
<p><label for="id_tag_list">Tag list:</label>
<input id="id_tag_list" name="tag_list" maxlength="200" type="text" /> </p>
<input type="submit" value="Submit" />
</form>
</body>
And this is the JQuery call to add autocomplete to the input.
function formatItem_tag_list(row) {
return row[2]
}
function formatResult_tag_list(row) {
return row[1]
}
$(document).ready(function(){
$("input[id='id_tag_list']").autocomplete({
url:'http://gladis.org/ajax/tag',
formatItem: formatItem_tag_list,
formatResult: formatResult_tag_list,
dataType:'text'
});
});
When I'm typing something inside the Textfield Firefox (firebug) and Chromium-browser indicates that ther is an ajax call but with no response. If I just copy the line into my browser, I can see the the response. (this issue is solved, it was a safety feature from ajax not to get data from another domain)
For example when I am typing Bi in the textfield, the url "http://gladis.org/ajax/tag?q=Bi&max... is generated. When you enter this in your browser you get this response:
4|Bier|Bier
43|Kolumbien|Kolumbien
33|Namibia|Namibia
Now my ajax call get the correct response, but there is still no list showing up with all the possible entries. I tried also to format the output, but this doesn't work either. I set brakepoints to the function and realized that they won't be called at all.
Here is a link to my minimum HTML file http://gladis.org/media/input.html
Has anybody an idea what i did wrong. I also uploaded all the files as a small zip at http://gladis.org/media/example.zip.
Thank you for your help!
[Edit]
here is the urls conf:
(r'^ajax/(?P<channel>[a-z]+)$', 'ajax_select.views.ajax_lookup'),
and the ajax lookup channel configuration
AJAX_LOOKUP_CHANNELS = {
# the simplest case, pass a DICT with the model and field to search against :
'tag' : dict(model='htags.Tag', search_field='text'),
}
and the view:
def ajax_lookup(request,channel):
""" this view supplies results for both foreign keys and many to many fields """
# it should come in as GET unless global $.ajaxSetup({type:"POST"}) has been set
# in which case we'll support POST
if request.method == "GET":
# we could also insist on an ajax request
if 'q' not in request.GET:
return HttpResponse('')
query = request.GET['q']
else:
if 'q' not in request.POST:
return HttpResponse('') # suspicious
query = request.POST['q']
lookup_channel = get_lookup(channel)
if query:
instances = lookup_channel.get_query(query,request)
else:
instances = []
results = []
for item in instances:
results.append(u"%s|%s|%s" % (item.pk,lookup_channel.format_item(item),lookup_channel.format_result(item)))
ret_string = "\n".join(results)
resp = HttpResponse(ret_string,mimetype="text/html")
resp['Content-Length'] = len(ret_string)
return resp
You probably need a trailing slash at the end of the URL.
Also, your jQuery selector is wrong. You don't need quotes within the square brackets. However, that selector is better written like this anyway:
$("input#id_tag_list")
or just
$("#id_tag_list")
Separate answer because I've just thought of another possibility: is your static page being served from the same domain as the Ajax call (gladis.org)? If not, the same-domain policy will prevent Ajax from being loaded.
As an aside, assuming your document.ready is in your Django template, it would be a good idea to utilize the {% url %} tag rather than hardcoding your URL.
$(document).ready(function(){
$("input[id='id_tag_list']").autocomplete({
url:'{% url my_tag_lookup %}',
dataType:'text'
});
});
This way the JS snippet will be rendered with the computed URL and your code will remain portable.
I found a solution, but well I still don't know why the first approach didn't worked out. I just switched to a different library. I choose http://bassistance.de/jquery-plugins/jquery-plugin-autocomplete/. This one is actually promoted by jQuery and it works ;)