Python 2.7 posting, and getting result from web site - python-2.7

I appreciate the help in advance. I am trying to write a python script that posts an IP address to a site referenced below, and get the results printed out in the terminal or file, and then read the file immediately after.
Here is my script:
#!/usr/bin/env python
import requests
IP = raw_input("Enter IP address here: ")
Alert_URL = 'http://www.blacklistalert.org'
def submit_form():
"""Submit a form"""
payload = IP
# make a get request
resp = requests.get(Alert_URL)
print "Response to GET request: %s" % resp.content
# send POST request
resp = requests.post(Alert_URL, payload)
print "Headers from a POST request response: %s" % resp.headers
# print "HTML Response: %s" %resp.read()
if __name__ == '__main__':
submit_form()
The site has section to input IP addresses on the web page, and inspecting the site I found lines to input as follows:
<form method=POST onsubmit="document.forms[0].submit.disabled='true';">
IP or Domain <input onclick="this.value='';" name=q value=11.11.154.23>
I would like to post an IP address that I want to check to the site using the input section above somehow. For instance using raw_input to post into the 'value=' section, and get the result.
Thanks for the help.

You need to parse the PHPSESSID and post:
import requests
from bs4 import BeautifulSoup
ip = raw_input("Enter IP address here: ")
data = {"q": ip} # ip goes here
url = "http://www.blacklistalert.org/"
with requests.Session() as s:
# get the page first to parse
soup = BeautifulSoup(s.get(url).content)
# extract and add the PHPSESSID
PHPSESSID = soup.select_one("input[name=PHPSESSID]")["value"]
data["PHPSESSID"] = PHPSESSID
# finally post
res = s.post(url, data=data)
print(res)
print(res.content)

Related

Python Requests: How can I properly submit a multipart/form POST using a file name

I have taken a look at other questions related to multipart/form POST requests in Python but unfortunately, they don't seem to address my exact question. Basically, I normally use CURL in order to hit an API service that allows me to upload zip files in order to create HTML5 assets. The CURL command I use looks like this:
curl -X POST -H "Authorization: api: 222111" --form "type=html" --form "file=Folder1/Folder2/example.zip" "https://example.api.com/upload?ins_id=123"
I am trying to use a python script to iterate through a folder of zip files in order to upload all of these files and receive a "media ID" back. This is what my script looks like:
import os
import requests
import json
ins_id = raw_input("Please enter your member ID: ")
auth = raw_input("Please enter your API authorization token: ")
for filename in os.listdir("zips"):
if filename.endswith(".zip"):
file_path = os.path.abspath(filename)
url = "https://example.api.com/upload?
ins_id="+str(ins_id)
header = {"Authorization": auth}
response = requests.post(url, headers=header, files={"form_type":
(None, "html"), "form_file_upload": (None, str(file_path))})
api_response = response.json()
print api_response
This API service requires the file path to be included when submitting the POST. However, when I use this script, the response indicates that "file not provided". Am I including this information correctly in my script?
Thanks.
Update:
I think I am heading in the right direction now (thanks to the answer provided) but now, I receive an error message stating that there is "no such file or directory". My thinking is that I am not using os.path correctly but even if I change my code to use "relpath" I still get the same message. My script is in a folder and I have a completely different folder called "zips" (in the same directory) which is where all of my zip files are stored.
To upload files with the request library, you can include the file handler directly in the JSON as described in the documentation. This is the corresponding example that I have taken from there:
url = 'http://httpbin.org/post'
files = {'file': open('path_to_your_file', 'rb')}
r = requests.post(url, files=files)
If we integrate this in your script, it would look as follows (I also made it slightly more pythonic):
import os
import requests
import json
folder = 'zips'
ins_id = raw_input("Please enter your member ID: ")
auth = raw_input("Please enter your API authorization token: ")
url = "https://example.api.com/upload?"
header = {"Authorization": auth}
for filename in os.listdir(folder):
if not filename.endswith(".zip"):
continue
file_path = os.path.abspath(os.path.join(folder, filename))
ins_id="+str(ins_id)"
response = requests.post(
url, headers=header,
files={"form_type": (None, "html"),
"form_file_upload": open(file_path, 'rb')}
)
api_response = response.json()
print api_response
As I don't have the API end point, I can't actually test this code block - but it should be something along these lines.

fetch website with python include j.s css

i'm trying to fetch a whole website include the JavaScript and css file while using python.
The script get a "GET" request and send back the website (local proxy).
here is my code :
class myHandler(BaseHTTPRequestHandler):
# Handler for the GET requests
def do_GET(self):
opener = urllib.FancyURLopener({})
f = opener.open("http://www.ynet.co.il")
self.wfile.write(f.read())
return
try:
# Create a web server and define the handler to manage the
# incoming request
server = HTTPServer(('', PORT_NUMBER), myHandler)
print 'Started httpserver on port ', PORT_NUMBER
# Wait forever for incoming htto requests
server.serve_forever()
except KeyboardInterrupt:
print '^C received, shutting down the web server'
server.socket.close()
The result for this code is only the html is present to the client.
Thanks a lot for the help, i'm Trying to solve that for few days with no result any .

Python requests 503 erros when trying to access localhost:8000

I am facing a bit of a situation,
Scenario: I got a django rest api running on my localhost:8000 and I want to access the api using my command line. I have tried urllib2 and python requests libs to talk to the api but failed(i'm getting a 503 error). But when I pass google.com as the url, I am getting the expected response. So I believe my approach is correct but I'm doing something wrong. please see the code below :
import urllib, urllib2, httplib
url = 'http://localhost:8000'
httplib.HTTPConnection.debuglevel = 1
print "urllib"
data = urllib.urlopen(url);
print "urllib2"
request = urllib2.Request(url)
opener = urllib2.build_opener()
feeddata = opener.open(request).read()
print "End\n"
Envioroments:
OS Win7
python v2.7.5
Django==1.6
Markdown==2.3.1
colorconsole==0.6
django-filter==0.7
django-ping==0.2.0
djangorestframework==2.3.10
httplib2==0.8
ipython==1.0.0
jenkinsapi==0.2.14
names==0.3.0
phonenumbers==5.8b1
requests==2.1.0
simplejson==3.3.1
termcolor==1.1.0
virtualenv==1.10.1
Thanks
I had a similar problem, but found that it was the company's proxy that was preventing from pinging myself.
503 Reponse when trying to use python request on local website
Try:
>>> import requests
>>> session = requests.Session()
>>> session.trust_env = False
>>> r = session.get("http://localhost:5000/")
>>> r
<Response [200]>
>>> r.content
'Hello World!'
If you are registering your serializers with DefaultRouter then your api will appear at
http://localhost:8000/api/ for an html view of the index
http://localhost:8000/api/.json for a JSON view of the index
http://localhost:8000/api/appname for an html view of the individual resource
http://localhost:8000/api/appname/.json for a JSON view of the individual resource
you can check the response in your browser to make sure your URL is working as you expect.

XMLRPC (client/proxy) use behind a firewall - Python 2.7

I'm new to XMLRPC but I need to use it (xmlrpclib in Python 2.7) to communicate with a server (www.neos-server.org) which accepts xml files. I'm behind a firewall that severely restricts outgoing and incoming traffic, but I'm able to browse the web mostly unimpeded using a webproxy.
Test 1 and 2 (below) work, but Test 3 results in an HTTP 502 error (cantconnect)
import urllib2
import xmlrpclib
import httplib
class ProxyTransport(xmlrpclib.Transport):
def request(self, host, handler, request_body, verbose):
self.verbose = verbose
url = 'http://' + host + handler
if self.verbose: "ProxyTransport URL: [%s]" % url
request = urllib2.Request(url)
request.add_data(request_body)
request.add_header("User-Agent", self.user_agent)
request.add_header("Content-Type", "text/xml")
proxy_handler = urllib2.ProxyHandler({"http":"MYPROXY:8080"})
opener = urllib2.build_opener(proxy_handler)
f = opener.open(request)
return(self.parse_response(f))
# TEST 1 - HTML fetching
def test1():
html = urllib2.urlopen("http://www.google.com").read() # note no proxy setup here
print html
# TEST 2 - XMLRPC sample server fetching
def test2():
p = ProxyTransport()
test_url = "http://betty.userland.com"
#test_server = xmlrpclib.Server(test_url) # gives <ProtocolError for betty.userland.com/RPC2: 403 WebTrap>
test_server = xmlrpclib.Server(test_url, transport=p)
test_api = "examples.getStateName(9)"
print "API: %s" % test_api
r = eval("test_server.%s" % test_api)
print "Result: ", r
# TEST 3 - XMLRPC server (NEOS)
def test3():
# Setup proxy and server
p = ProxyTransport()
NEOS_HOST = "www.neos-server.org"
NEOS_PORT = 3332
neos = xmlrpclib.Server("http://%s:%d" % (NEOS_HOST, NEOS_PORT), transport = p)
# Talk
print "Ping Neos..."
neos.ping()
test1()
test2()
test3()
I've tried a couple different solutions (https://gist.github.com/nathforge/980961, https://mail.python.org/pipermail/python-list/2006-July/367049.html) but they don't seem to work. I assume I need to be able to ping before I can send/receive xml files and results. What should I try next?

How to merge few python scripts into one?

I am newbie when it comes to programming and python.
Therefore I've got a question. With my fellow students we have created few python scripts but now we are stuck and have no more ideas. We need to merge few python scripts into one working scripts. Could anyone help us with that, please?
Scripts:
# Script: webpage_get.py
# Desc: Fetches data from a webpage, and parses out hyperlinks.
# Author: Wojciech Kociszewski
# Created: Nov, 2013
#
import sys, urllib
def wget(url):
''' Try to retrieve a webpage via its url, and return its contents'''
print '[*] wget()'
#open file like url object from web, based on url
url_file = urllib.urlopen(url)
# get webpage contents
page = url_file.read()
return page
def main():
#temp testing url argument
sys.argv.append('http://www.soc.napier.ac.uk/~cs342/CSN08115/cw_webpage/index.html')
#check args
if len(sys.argv) != 2:
print '[-] Usage: webpage_get URL'
return
#Get and analyse web page
print wget(sys.argv[1])
if __name__ == '__main__':
main()
# Script: webpage_getlinks.py
# Desc: Basic web site info gathering and analysis script. From a URL gets
# page content, parsing links out.
# Author: Wojciech Kociszewski
# Created: Nov, 2013
#
import sys, re
import webpage_get
def print_links(page):
''' find all hyperlinks on a webpage passed in as input and print '''
print '[*] print_links()'
# regex to match on hyperlinks, returning 3 grps, links[1] being the link itself
links = re.findall(r'(\<a.*href\=.*)(http\:.+)(?:[^\'" >]+)', page)
# sort and print the links
links.sort()
print '[+]', str(len(links)), 'HyperLinks Found:'
for link in links:
print link[1]
def main():
# temp testing url argument
sys.argv.append('http://www.soc.napier.ac.uk/~cs342/CSN08115/cw_webpage/index.html')
# Check args
if len(sys.argv) != 2:
print '[-] Usage: webpage_getlinks URL'
return
# Get the web page
page = webpage_get.wget(sys.argv[1])
# Get the links
print_links(page)
if __name__ == '__main__':
main()
# Script: webpage_getemails.py
# Desc: Basic web site info gathering and analysis script. From a URL gets
# page content, parsing emails out.
# Author: Wojciech Kociszewski
# Created: Nov, 2013
#
import sys, re
import webpage_get
def print_emails(page):
''' find all emails on a webpage passed in as input and print '''
print '[*] print_emails()'
# regex to match on emails
emails = re.findall(r'([\d\w\.-_]+#[\w\d\.-_]+\.\w+)', page)
# sort and print the emails
emails.sort()
print '[+]', str(len(emails)), 'Emails Found:'
for email in emails:
print email
def main():
# temp testing url argument
sys.argv.append('http://www.soc.napier.ac.uk/~cs342/CSN08115/cw_webpage/index.html')
# Check args
if len(sys.argv) != 2:
print '[-] Usage: webpage_getemails'
return
# Get the web page
page = webpage_get.wget(sys.argv[1])
# Get the emails
print_emails(page)
if __name__ == '__main__':
main()
Analyse your scripts and find the common code
Convert the common code into a module
Rewrite the individual programs with the common code
If you then wish to make the individual programs into one big program it will be much easier