I've got the following snip of code the has the audacity to tell me it is "FAIL to load undefnied" (the nerve...) I'm trying to pass my authenticated session to a system call that uses javascript.
import requests
from requests_ntlm import HttpNtlmAuth
from subprocess import call
# THIS WORKS - 200 returned
s = requests.Session()
r = s.get("http://example.com",auth=HttpNtlmAuth('domain\MyUserName','password'))
call(["phantomjs", "yslow.js", r.url])
The issue is when "calL" gets called - all I get is the following
FAIL to load undefined.
Im guessing that just passing the correct authenticated session should work - but the question is how do I do it such that I can extract the info I want. Out of all the other attempts this has been the most fruitful. Please help - thanks!
There seem to be couple things going on here so I'll address them one-by-one.
The subprocess module in python is meant to be used to call out to the system as if you were using the command line. It knows nothing of "authenticated session"s and the command line (or shell) has no knowledge of how to use a python object, like a session, to work with phantomjs.
phantomjs has python bindings since version 1.8 so I would expect this might be made easier by using them. I have not used them, however, so I can not tell you with certainty that they will be helpful.
I looked at yslow's website and there appears to be no way to pass it the content that you are downloading with requests. Even then, the content would not have everything (for example: any externally hosted javascript that would be loaded by selenium/phantomjs or a browser, is not loaded by requests)
yslow seems as though it normally just downloads the URL for you and performs its analysis. When the website is behind NTLM, however, it first sends the client a 401 response which should indicate to the client that it must authenticate. Further, information is sent to the client that tells it how to authenticate and provides it parameters to use when authenticating for NTLM. This is how requests_ntlm works with requests. The first request is made and generates a 401 response, then the authentication handler generates the proper header(s) and re-sends the request which is why you see the 200 response bound to r.
yslow accepts a JSON representation of the headers you want to send so you can try to use the headers found in r.request.headers but I doubt they will work.
In short, this is not a question that the people who normally follow the requests tag can help you with. And looking at the documentation for yslow it seems that it (technically) does not support authentication of any type. yslow developers might argue though that it supports Basic Authentication because it allows you to specify headers.
Related
I'm currently developing an ckan extension, where i need to redirect to a url on a different domain.
In my plugin i defined a custom action function:
#side_effect_free
def download_json(context, data_dict):
toolkit.redirect_to('http://my.json-builder.com?id=1234')
But when i call this endpoint i just get following response:
response screenshot
So i assume that the action function is called, but the redirect_to call does not redirect to the url i defined.
Thanks for your help!
Florian
It's a bit hard to figure out what you're trying to accomplish but here's a few things I hope will help.
Short Answer:
No, you can't redirect from an API endpoint in CKAN. The endpoint response in CKAN is built up and expects certain things from your action. Your action should return some kind of result. In your case it's returning nothing but trying to redirect. A logic action function with IActions is not the same as a Blueprint or pylons controller action.
See Making an API request docs, specifically the breakdown of an API response in CKAN. Also, you can review the pylons implementation that builds up the API response or the flask blueprints implementation.
More Info to help with your approach:
You say you are trying to call an endpoint that redirects a user to a different domain url. Based on this consider the following:
The first thing I thought you wanted was to have a url that someone goes to through the web interface of your site and are redirected to another site. In this case your example code of toolkit.redirect_to('http://my.json-builder.com?id=1234') makes sense and works for a custom controller action using/implemented with IRoutes or if you're using flask then IBlueprint. A User would go to a URL on your site such as http://localhost.com/download_json and be redirected to the new URL/site in their browser.
If you are intending this to be an API call for other users this starts to feel a little bit odd. If a user is using your API, they would expect to get results from your site in JSON CKAN's API is designed to return JSON. Someone consuming your API endpoint would not expect to be redirected to another site e.g. if I called http://localhost.com/api/3/action/download_json I would expect to get a JSON object like
{
help: "http://localhost/api/3/action/help_show?name=download_json",
success: true,
result: {
...
}
}
They would look for success to make sure the call worked and then they would use the result to keep moving forward with their desired processes. If you do want someone via an API to get redirect info I'd likely return the redirect url as the result e.g. result: {'redirect_url': 'http://my.json-builder.com?id=1234'} and document this well in your extension's API docs (e.g. why you're returning this endpoint, what you expect someone to do with it, etc).
If this is an API call for your own extension I'm guessing what you are trying to do is use my.json-builder.com to build a json of something (a dataset maybe?) and return that json as the result at your endpoint or maybe even consume the result to make something else? If that's the case, then in your function you could make the call to my.json-builder.com, process the results and return the results to the user. In this case, you're not actually wanting to redirect a user to a new site but instead make a call to the new site to get some results. If you actually want the results for your extension you don't need an additional endpoint. You could make the call from your extension, consume the results and return the desired object you're trying to create.
Hope this helps and sorry if I've miss-understood completely.
I am trying to setup a simple API test against a local endpoint. I have create the sample API (phone number lookup) and that works fine.
http://192.168.1.11:8080/api/simpleTest is my endpoint and the WSO2 service also runs on 192.168.1.11 ... but when I test it in 'publisher', it always fails. This is a simple GET with no parameters.
I can run it from a browser or CURL (outside of WSO2) and it works fine.
Thanks.
I assume you talk about clicking the Test button when providing Backend Endpoint in API publisher.
The way that Test button works at the moment (as far as I understand) is that it invokes HTTP HEAD method on the endpoint provided (because according to RFC 2616, "This method is often used for testing hypertext links for validity, accessibility, and recent modification.")
Then it checks response. If response is valid or 405 (method not allowed), then the URL is marked as Valid.
Thus sometimes, if backend is not properly following RFC, you might get otherwise working URLs declared as Invalid during the test because of that improper HEAD response evaluation. Obviously, this is just a check for your convenience and you can ignore the check if you know the endpoint works for the methods and resources you need it to work.
P.S. Checked it on API Cloud but behavior is identical to downloadable API Manager.
I am trying to create and feed requests from code (server initialization) to JSPs using Jetty 9.2. This is mainly for warmup/precompilation but also other reasons. I do not have an ongoing request (this is happening at startup) and I would like to avoid the workarounds that actually send requests using the network.
Problem I face right now is that the following line (83) in Jetty's Dispatcher.java fails with a null pointer exception:
Request baseRequest=(request instanceof Request)?((Request)request):HttpChannel.getCurrentHttpChannel().getRequest();
So, how does one properly feed requests code-created requests into embedded Jetty? Pleas help.
Dispatcher requires a valid request.
As the dispatched call requires a bunch of information from the Request (such as method, headers, content encodings, what sort of response encodings are valid, uri, context path, sessions, cookies, security contexts, principals, etc. the list is long)
I've been looking on the web regarding CORS, and I wanted to confirm if whatever I made of it is, what it actually is.
Mentioned below is a totally fictional scenario.
I'll take an example of a normal website. Say my html page has a form that takes a text field name. On submitting it, it sends the form data to myPage.php. Now, what happens internally is that, the server sends the request to www.mydomain.com/mydirectory/myPage.php along with the text fields. Now, the server sees that the request was fired off from the same domain/port/protocol
(Question 1. How does server know about all these details. Where does it extract all these details froms?)
Nonetheless, since the request is originated from same domain, it server the php script and returns whatever is required off it.
Now, for the sake of argument, let's say I don't want to manually fill the data in text field, but instead I want to do it programmatically. What I do is, I create a html page with javascript and fire off a POST request along with the parameters (i.e. values of textField). Now since my request is not from any domain as such, the server disregards the service to my request. and I get cross domain error?
Similarly, I could have written a Java program also, that makes use of HTTPClient/Post request and do the same thing.
Question 2 : Is this what the problem is?
Now, what CORS provide us is, that the server will say that 'anyone can access myPage.php'.
From enable cors.org it says that
For simple CORS requests, the server only needs to add the following header to its response:
Access-Control-Allow-Origin: *
Now, what exactly is the client going to do with this header. As in, the client anyway wanted to make call to the resources on server right? It should be upto server to just configure itself with whether it wants to accept or not, and act accordingly.
Question 3 : What's the use of sending a header back to client (who has already made a request to the server)?
And finally, what I don't get is that, say I am building some RESTful services for my android app. Now, say I have one POST service www.mydomain.com/rest/services/myPost. I've got my Tomcat server hosting these services on my local machine.
In my android app, I just call this service, and get the result back (if any). Where exactly did I use CORS in this case. Does this fall under a different category of server calls? If yes, then how exactly.
Furthermore, I checked Enable Cors for Tomcat and it says that I can add a filter in my web.xml of my dynamic web project, and then it will start accepting it.
Question 4 : Is that what is enabling the calls from my android device to my webservices?
Thanks
First of all, the cross domain check is performed by the browser, not the server. When the JavaScript makes an XmlHttpRequest to a server other than its origin, if the browser supports CORS it will initialize a CORS process. Or else, the request will result in an error (unless user has deliberately reduced browser security)
When the server encounters Origin HTTP header, server will decide if it is in the list of allowed domains. If it is not in the list, the request will fail (i.e. server will send an error response).
For number 3 and 4, I think you should ask separate questions. Otherwise this question will become too broad. And I think it will quickly get close if you do not remove it.
For an explanation of CORS, please see this answer from programmers: https://softwareengineering.stackexchange.com/a/253043/139479
NOTE: CORS is more of a convention. It does not guarantee security. You can write a malicious browser that disregards the same domain policy. And it will execute JavaScript fetched from any site. You can also create HTTP headers with arbitrary Origin headers, and get information from any third party server that implements CORS. CORS only works if you trust your browser.
For question 3, you need to understand the relationship between the two sites and the client's browser. As Krumia alluded to in their answer, it's more of a convention between the three participants in the request.
I recently posted an article which goes into a bit more detail about how CORS handshakes are designed to work.
Well I am not a security expert but I hope, I can answer this question in one line.
If CORS is enabled then server will just ask browser if you are calling the request from [xyz.com]? If browser say yes it will show the result and if browser says no it is from [abc.com] it will throw error.
So CORS is dependent on browser. And that's why browsers send a preflight request before actual request.
In my case I just added
.authorizeRequests().antMatchers(HttpMethod.OPTIONS, "/**").permitAll()
to my WebSecurityConfiguration file issue is resolved
Because I don't exactly know how any auth method works I want to write my own.
So, what I want to do is the following.
A client sends over HTTPs username+password(or SHA1(username+password))
the server gets the username+password and generates a big random number and stores it in
a table called TOKENS(in some database) along with his IP,
then it give the client that exact number.
From now on, all the requests made by the client are accompanied by that TOKEN
and if the TOKEN is not in the table TOKENS then any such request will fail.
If the user hasn't made any requests in 2 hours the TOKEN will expire.
If the user wants to log out he makes a request '/logout' to the server and the server
deletes from the table TOKENS the entry containing his token but ONLY if the request to
'/logout' originates from his IP.
Maybe I am reinventing the wheel... this wouldn't be very good so my question is if there
is some auth system that already works like this , what is it's name , does it have any OSS C++ libraries or Python libraries available ?
I am not sure if finding such an auth system and configuring it would take longer than
writing it myself, on the other hand I know security is a delicate problem so I am
approaching this with some doubt that I am capable of writing something secure enough.
Also, is there a good OSS C++ HTTP library ? I'm planning to write a RESTful Desktop
client for a web app. Depending on the available libraries I will choose if I'll write it
in C++ or Python.
If you are implementing such authentication system over ordinary HTTP, you are vulnerable to replay attacks. Attacker could sniff out the SHA1(username+password) and just resend it every time he/she wants to log in. To make such authentication system work, you will need to use a nonce.
You might want to look at HTTP Digest authentication for tips.
Because I don't exactly know how any auth method works I want to write my own
How could you ever write something you don't understand? Learn at least one, the underlaying concepts are similar in every library.
Python has repoze.what.
I would highly recommend OAuth here, for which many open source libraries are available.