I want to automate document retrieval from a specific website.
1-I need to send a request (GET) to the site.
2-Then submit a filled form (POST).
3-Then obtain all the files (multiple GETS?)
4-Then repeat again using new form information.
Is this all possible with POSTMAN? are there tools better suited for this job?
You can order those requests in a folder and use Collection Runner to automate the requests. The postman will run those requests in the order you have aligned the requests.
Postman is made to make requests.
To "find out the POST request", you need to use another tool to identify these calls, like Fiddler (free), or the Debuging/developer tools of any major browsers.
Related
I have done a few mobile apps using django as my backend, and now I am working on a chrome extension. I want my users to be able to send POST requests up to the server using the app/extension but is there a way to do it without first asking the server for a CSRF token? It just saves the app from making two requests every time we want to submit data. For example, I want to update my profile on my social media app or update a wallet from a chrome extension. It would be nice to open up the profile view input the data and push it to the server. It's less sleek if I have to open the profile, then wait for it to grab a token from the server and then I can submit the data. Is there another way to do this? Or am I stuck making multiple requests every time I want to submit data?
Also, a little clarification, CSRF prevents sites from submitting forms with user's data. But what is to stop me from making a site that uses ajax or something to grab the real site and steal the CSRF token and then paste that into my cross site request form? I feel like there is a loophole here. I know that I am not quite understanding this all the way.
You can, and should, make any API endpoint CSRF exempt.
Django offers the csrf_exempt decorator for exactly this, see https://docs.djangoproject.com/en/dev/ref/contrib/csrf/#django.views.decorators.csrf.csrf_exempt.
Also CSRF is intended to prevent unintended actions being performed via GET request forgeries. It is not intended to make it impossible for an automated system to submit forms, there are captchas for that.
As for what prevents you from using AJAX to grab the whole site and extract the token is something called the Same-Origin Policy. This is implemented by the browser and prevents any AJAX call from returning data when the target of the AJAX call is a different domain without the correct headers set. (I'm not entirely sure what sandboxing is applied to chrome extensions concerning this). As such it will, or at least should, fail to get data via AJAX for normal websites, e.g. a profile page. If you want to interact with third party websites you should look into whether or not they offer an API.
As the title implies,
I need to fetch data from certain website which need logins to use.
The login procedure might need cookies, or sessions.
Do I need QtWebkit, or can I get away with just QNetworkAccessManager?
I have no experience at both, and will start learning as I go.
So please save me a bit of time of comparing both ^^
Thank you in advance,
Evan
Edit: Having read some related answers,
I'll add some clarifications:
The website in concern does not have an API. So I will need to scrape web elements for the data myself.
Can I do that with just QNetworkAccessManager?
No, in most cases you don't need a full simulated web browser. In most cases, just performing the same web requests like a web browser would do is enough.
Try to record the web requests in your browser, using a plugin like "HTTP Live Headers" or "Firebug" in Firefox. I think Chrome provides a similar tool out of the box. These tools record the GET and POST requests done by the website when you send a form in the webpage.
Another option is to inspect the HTML code of the login page. Find the <form> tag and its fields. Put them together in a GET / POST request in your application to simulate the same form.
Remember that some pages use randomized "tokens" in their forms, some set the tokens as cookies. In such cases, you need to request the login page itself in your application first (before sending the filled in form). Both QWebView and QNetworkAccessManager have cookie support.
To sum things up, I think QWebView provides a far more elegant way to simulate user interaction with a web page. The manual way is, however, more "lightweight", as you don't need Webkit and your application might be faster (because only the HTML page is loaded, without any linked resources like images, CSS, javascript files).
QWebView as class name states is a view, so it views something (in this case web pages). If you don't need to display loaded page, then you don't need a view. QNetworkAccessManager may do the work, but you need some knowledge about HTTP protocol, and also anything about target site: how does it hande logins, what type of request you have to send to login etc.
I have a website built in Django. One feature of this website is booking hotels. Now, my client has many agents who have websites. We want to give the facility of booking on these websites. So, I am thinking to use iframe on these websites( with the consent of the respective agent owners), which will point to the booking page of our website. Once booking is done, we will return success message and email the user. IS this a viable solution? Or, are there any other options?
thanks
One way of doing this it's creating a rest api using a helper that works with django like django-piston so your agents could work with them to perform booking.
Should be easy to add a form to any agents site with this way. Or even using ajax if the uri returns a json.
Iframes is not the way you should go I think... I also think that you need to define the API on your website and then you just do requests from the client website to your backend.
I'm trying to write a simple GUI application using Qt framework.
The purpose of this app is to retrieve data from my isp and parse them for presentation.
How do i authenticate my user/password with the webserver and retrieve the html page in question?
Are there any utility libs that make this task trivial?
I figure i need to interact with the server php script and simulate a form input somehow.
Am i on the right track?
You're on the right track, I suggest taking a look at curl.
That should make it alot easier.
edit: Hm, thought it did more than just file-transfer.
Otherwise here's a load more of interesting lib's
The way to authenticate depends completely on the authentication method used by the server. If it's some form to log in you need to retrieve that and send the correct data to the forms action target (usually as POST request). You could do this by constructing your request using QHttpRequestHeader and then simply sending it to the server. If you even know about the form you might even not need to retrieve the login page. If the website uses HTTP authentication you should be able using QAuthenticator.
Description of how a webhook works from http://webhooks.pbwiki.com/ -
How do they work?
By letting the user specify a URL for various events, the application will POST data to those URLs when the events occur...Among other things, you can:
create notifications to you or anybody via email, IRC, Jabber, ...
put the data in another app (real-time data synchronization)
process the data and repost it using the app's API
validate the data and potentially prevent it from being used by the app
Who is using web hooks?
DevjaVu, BitBucket, GitHub, Shopify, Versionshelf, PayPal (IPN), Jott (Links), IMified, PBwiki, Facebook (Platform, sort of), Mailhook.org, SMTP2Web, Astrotrain, Notifixious, Assembla, ZenDesk, Google Code
Do you know of any good uses of webhooks?
AlertGrid is the webhook consumer. You can configure it to accept http calls from ANY source and raise alert (email, sms, phone) to a specified person or group of people (works worldwide!) whenever the parameters in the http callback meet your criteria or when the http call was expected but it didn't occur (kind of 'heartbeat' monitoring). There is a visual editor for you to easily create rules.
Apart from notifying people by sms or email it can also notify existing applications by sending the http requests to their APIs.
It can also visualise data received in http callbacks and show the history.
Unfortunately, the wiki is not the most up to date list of known implementations. I have my own list that I'll put on the wiki when I get around to reorganizing it. Some not mentioned in the current list:
Dropbox
Gnip
Google Code (Project Hosting)
Checkout by Amazon (both for notifications and as actual callbacks with return data)
Hubilicious
Beanstalk
Google Checkout
MailChimp
SurveyGizmo
Hey!Watch
MySpace (for app developers)
I know shopify is using webhooks quite successfully now. By extension so is fetchapp uses them as well. You either are sending an xml file, or receiving one and doing your own processing logic on it.
Oh and shopify's wiki in the link has a whole write up about how to implement it in your app.
OfficeAutopilot has an interesting version of webhooks.. they use their rule interface to trigger API posts. Can trigger in response to any system event.. email opens, clicks, page visits, purchases, etc, etc.
Kiln 1.2 uses webhooks much like GitHub, BitBucket, etc.
(Disclaimer: I'm a Kiln/FogBugz dev.)
Say for example you want to get data from any API( eg. twitter, facebook etc.,). Instead of you polling the data for every few minutes/seconds, it POSTS the data to the specified URL, whenever it is available.
By using this, you will avoid unnecessary polling like say you poll and data is not there yet.
StorageRoom is a JSON-based CMS that supports webhooks, so that you can notify other services or kick of some manual processing on your own servers.
(Please note: I created the service myself)
If you want to connect one service that supports webhooks to another service's API, you can check out IronWorker's webhook support. Here's a blog post that walks through connecting github webhooks to HipChat:
http://blog.iron.io/2012/04/one-webhook-to-rule-them-all-one-url.html
There are some other examples here too, one that takes a chargify callback and posts to Campfire.