I'm developing a Django server. I need to get some data from an external service here.
Here is the JavaScript code that I'm using:
function getDeclination(latitude, longitude) {
var url = 'http://www.ngdc.noaa.gov/geomag-web/calculators/calculateDeclination';
url += '?';
url += 'lat1=' + latitude;
url += '&'
url += 'lon1=' + longitude;
url += '&';
url += 'resultFormat=xml';
var xmlHttp = null;
xmlHttp = new XMLHttpRequest();
xmlHttp.open( "GET", url, false );
xmlHttp.send();
xml = xmlHttp.responseXML;
var temp_d = xml.getElementsByTagName('declination');
var string = temp_d[0].childNodes[0].nodeValue;
string = string.substring(1, string.length - 2);
return parseFloat(string);
}
Using this code for a call like this getDeclination(46.0815605783, 13.2158580422) rise this error:
XMLHttpRequest cannot load
http://www.ngdc.noaa.gov/geomag-web/calculators/calculateDeclination?lat1=46.0815605783&lon1=13.2158580422&resultFormat=xml.
Origin http://127.0.0.1:8000 is not allowed by Access-Control-Allow-Origin
How can I get rid of this problem and return to testing?
EDIT:
So I installed the middleware and the JS, but I can't get results.
Perhaps I have done some error during the installation of the middleware, it's the first time for me.
I know that Django found it, and that it's udsing it, but I don't know if I have done the setup correctly.
XS_SHARING_ALLOWED_ORIGINS = "http://127.0.0.1:8000"
XS_SHARING_ALLOWED_METHODS = ['POST','GET','OPTIONS', 'PUT', 'DELETE']
I have put this code in settings.py, is this correct? Because I still get the same error.
Use this middleware while you're developing, and disable it when you put everything into production.
Edit:
As for the Javascript side of things, I've just tested this and it works ok. It does require jQuery and the middleware to be installed though.
function getDeclination(latitude, longitude){
var url = "http://www.ngdc.noaa.gov/geomag-web/calculators/calculateDeclination";
url += "?";
url += "lat1=" + latitude;
url += "&"
url += "lon1=" + longitude;
url += "&";
url += "resultFormat=xml";
$.ajax({
url: url,
success: function(data){
var declination = parseFloat($(data).find("declination").text());
alert(declination);
}
});
}
In the end I implemented the code to get the declination from a server call.
For sake of completeness here is the code:
import urllib2
from xml.dom.minidom import parseString
# build the url
def buildUrl(latitude, longitude):
return settings.MAGNETIC_URL + '?' + 'lat1=' + str(latitude) + '&' + 'lon1=' + str(longitude) + '&' + 'resultFormat=xml'
# load the page from the address
def retrieveWeb(address):
try:
web_handle = urllib2.urlopen(address)
except urllib2.HTTPError, e:
error_desc = BaseHTTPServer.BaseHTTPRequestHandler.responses[e.code][0]
print "Cannot retrieve URL: HTTP Error Code", e.code, "Error: ", error_desc
sys.exit(1)
except urllib2.URLError, e:
print "Cannot retrieve URL: " + e.reason[1]
sys.exit(1)
except:
print "Cannot retrieve URL: unknown error"
sys.exit(1)
return web_handle
# main function to get declination
def retrieveDeclination(lat, lng):
# build url
address = buildUrl(lat, lng)
# get the page
page = retrieveWeb(address)
# get xml
xml = page.read()
page.close()
# inizio il parsin
dom = parseString(xml)
#retrieve the first xml tag (<tag>data</tag>) that the parser finds with name tagName:
xmlTag = dom.getElementsByTagName('declination')[0].toxml()
#strip off the tag (<tag>data</tag> ---> data):
xmlData=xmlTag.replace('<declination>','').replace('</declination>','')
# parse the output as float
return float(xmlData)
Related
I am sending a PATCH request to my DRF server in Postman and it works perfect
However when I do the same in Python I get:
<Response [405]> http://127.0.0.1:8000/api/title/8174/
b'{"detail":"Method \\"PATCH\\" not allowed."}'
Method Not Allowed
My function that sends data:
ss_token = os.getenv('SS_TOKEN')
headers = {
'Authorization': 'Token ' + ss_token,
}
source = Source.objects.all().first()
url = source.url + str(self.ss_id) + '/'
response = requests.patch(source.url, headers=headers, data={'key':'value'})
print(response, url)
print(response.content)
print(response.reason)
return True
Do I have to send other headers to the API to make the PATCH work?
Ah looks like I made a mistake. Forgot to replace source.url with the new url variable called 'url' variable. Because that add the 'ss_id' at the url' so it becomes 'api/title/ID/' instead of just 'api/title'
url = source.url + str(self.ss_id) + '/'
response = requests.patch(url, headers=headers, data={'key':'value'})
When I use urllib, urllib2, or requests on Python 2.7, neither one ends up at the same URL as I do when I copy and paste the starting URL into Chrome or FireFox for Mac.
EDIT: I suspect this is because one has to be signed in to vk.com to be redirected. If this is the case, how do I add the sign-in to my script? Thanks!
Starting URL: https://oauth.vk.com/authorize?client_id=PRIVATE&redirect_uri=https://oauth.vk.com/blank.html&scope=friends&response_type=token&v=5.68
Actual final (redirected) URL: https://oauth.vk.com/blank.html#access_token=PRIVATE_TOKEN&expires_in=86400&user_id=PRIVATE
PRIVATE, PRIVATE_TOKEN = censored information
The following is one of several attempts at this:
import requests
APPID = 'PRIVATE'
DISPLAY_OPTION = 'popup' # or 'window' or 'mobile' or 'popup'
REDIRECT_URL = 'https://oauth.vk.com/blank.html'
SCOPE = 'friends' # https://vk.com/dev/permissions
RESPONSE_TYPE = 'token' # Documentation is vague on this. I don't know what
# other options there are, but given the context, i.e. that we want an
# "access token", I suppose this is the correct input
URL = 'https://oauth.vk.com/authorize?client_id=' + APPID + \
'&display='+ DISPLAY_OPTION + \
'&redirect_uri=' + REDIRECT_URL + \
'&scope=' + SCOPE + \
'&response_type=' + RESPONSE_TYPE + \
'&v=5.68'
# with requests
REQUEST = requests.get(URL)
RESPONSE_URL = REQUEST.url
I hope you notice whatever it is that's wrong with my code.
Extra info: I need the redirect because the PRIVATE_TOKEN value is necessary for further programming.
I tried some debugging but neither the interpreter nor IPython print out the debugging info.
Thanks!
The problem is the result of not being signed in in the Python environment.
Solution:
Use twill to create browser in Python and sign in.
Code:
from twill.commands import *
BROWSER = get_browser()
BROWSER.go(URL) # URL is the URL concatenated in the question
RESPONSE_URL = BROWSER.get_url()
I'm looking for a way to add 2 custom cookies to every http request.
The browsermob proxy (https://github.com/lightbody/browsermob-proxy) has removeHeaders() and addHeader() methods, but what can I do to keep existing cookies in request, but add 2 more cookies?
Thanks!
You can use this method to invoke your custom js code in each request/response
https://github.com/lightbody/browsermob-proxy#http-request-manipulation
Some example in Python
def response_interceptor(self, js):
"""
Executes the javascript against each response
:param js: the javascript to execute
"""
r = requests.post(url='%s/proxy/%s/interceptor/response' % (self.host, self.port),
data=js,
headers={'content-type': 'x-www-form-urlencoded'})
return r.status_code
def request_interceptor(self, js):
"""
Executes the javascript against each request
:param js: the javascript to execute
"""
r = requests.post(url='%s/proxy/%s/interceptor/request' % (self.host, self.port),
data=js,
headers={'content-type': 'x-www-form-urlencoded'})
return r.status_code
and test:
def test_request_interceptor_with_parsing_js(self):
"""
/proxy/:port/interceptor/request
"""
js = 'alert("foo")'
status_code = self.client.request_interceptor(js)
assert(status_code == 200)
As I answered above, you could use proxy's REST API to set custom js handler on every request, made through the proxy.
For example you could add any custom cookies to every request:
curl -X POST -H 'Content-Type: text/plain' -d 'js code' http://10.100.100.20:8080/proxy/8081/interceptor/request
In php it would look like:
/**
* #param Proxy $proxyObject
* #param array $cookiesArray
*/
protected function _setRequestCookies(Proxy $proxyObject, array $cookiesArray)
{
foreach ($cookiesArray as $nameString => $valueString) {
$cookiesArray[$nameString] = $nameString . '=' . $valueString;
}
$jsHandlerString = sprintf(
'var c = request.getMethod().getFirstHeader("Cookie") ? request.getMethod().getFirstHeader("Cookie").getValue() : ""; request.getMethod().setHeader("Cookie", c + "; %s");',
implode('; ', $cookiesArray)
);
$urlString = sprintf('%sproxy/%u/interceptor/request', $this->_hubUrlString, $proxyObject->getPort());
$this->_requesterObject->makeRequest($urlString, Requester::REQUEST_METHOD_POST, $jsHandlerString);
}
Can I force scrapy to request an URL including commas without encoding it into %2C? The site (phorum) I want to crawl does not accept encoded URLs and redirecting me into root.
So, for example, I have site to parse: example.phorum.com/read.php?12,8
The url is being encoded into: example.phorum.com/read.php?12%2C8=
But when try to request this url, every time, I'm redirected into page with list of topics:
example.phorum.com/list.php?12
In those example URLs 12 is category number, 8 is topic number.
I tried to disable redirecting by disabling RedirectMiddleware:
DOWNLOADER_MIDDLEWARES = {
'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': None,
}
and in spider:
handle_httpstatus_list = [302, 403]
Moreover I tried to rewrite this URL and request it by sub parser:
Rules = [Rule(RegexLinkExtractor(allow=[r'(.*%2C.*)']), follow=True, callback='prepare_url')]
def prepare_url(self, response):
url = response.url
url = re.sub(r'%2C', ',', url)
if "=" in url[-1]:
url = url[:-1]
yield Request(urllib.unquote(url), callback = self.parse_site)
Where parse_site is target parser, which still calls using encoded URL.
Thanks in advance for any feedback
You can try canonicalize=False. Example iPython session:
In [1]: import scrapy
In [2]: from scrapy.contrib.linkextractors.regex import RegexLinkExtractor
In [3]: hr = scrapy.http.HtmlResponse(url="http://example.phorum.com", body="""link""")
In [4]: lx = RegexLinkExtractor(canonicalize=False)
In [5]: lx.extract_links(hr)
Out[5]: [Link(url='http://example.phorum.com/list.php?1,2', text=u'link', fragment='', nofollow=False)]
Please can anyone help out? i am trying to POST a SOAP request in python but i am getting the error Response 403: Forbidden. my code looks like below:
i am using the python imports:
import httplib
import base64
import string
#the message
message = """<soap:Envelope ...rest message </soap:Envelope>"""
host = "host.test.com"
url = 'https://server.etc.com' #End point url
i need to use the Basic Authentication too so i need the username and password in the http header
username = 'username'
password = 'password'
webSoapAction = 'urn:etc-com:document...'
#the Authentication in base64 encoding form for Basic Authentication
auth = 'Basic' + string.strip(base64.encodestring(username +'.'+ password))
webservice = httplib.HTTP(host) #connect to the server(host)
here i try to build the header:
webservice.putrequest("POST", url)
webservice.putheader("Host", host)
webservice.putheader("User-Agent", "Python http auth")
webservice.putheader("Content-Type:", "text/xml; charset=\"UTF-8\"")
webservice.putheader("Content-length", "%d" % len(message))
webservice.putheader("SOAPAction",webSoapAction)
webservice.putheader('Authorization', auth)
webservice.endheaders()
webservice.send(message)
i should get the response here
#get the response
statuscode, statusmessage, header = webservice.getreply()
print "Response: ", statuscode, statusmessage
print "Headers: ",header
res = webservice.getfile().read()
print 'Content: ', res
Two things regard to your basic auth header construction:
Put a single space between "Basic" and your secret
Use ':' instead of '.' in between username and password
So it should looks like:
#the Authentication in base64 encoding form for Basic Authentication
auth = 'Basic ' + string.strip(base64.encodestring(username +':'+ password))