As the title implies,
I need to fetch data from certain website which need logins to use.
The login procedure might need cookies, or sessions.
Do I need QtWebkit, or can I get away with just QNetworkAccessManager?
I have no experience at both, and will start learning as I go.
So please save me a bit of time of comparing both ^^
Thank you in advance,
Evan
Edit: Having read some related answers,
I'll add some clarifications:
The website in concern does not have an API. So I will need to scrape web elements for the data myself.
Can I do that with just QNetworkAccessManager?
No, in most cases you don't need a full simulated web browser. In most cases, just performing the same web requests like a web browser would do is enough.
Try to record the web requests in your browser, using a plugin like "HTTP Live Headers" or "Firebug" in Firefox. I think Chrome provides a similar tool out of the box. These tools record the GET and POST requests done by the website when you send a form in the webpage.
Another option is to inspect the HTML code of the login page. Find the <form> tag and its fields. Put them together in a GET / POST request in your application to simulate the same form.
Remember that some pages use randomized "tokens" in their forms, some set the tokens as cookies. In such cases, you need to request the login page itself in your application first (before sending the filled in form). Both QWebView and QNetworkAccessManager have cookie support.
To sum things up, I think QWebView provides a far more elegant way to simulate user interaction with a web page. The manual way is, however, more "lightweight", as you don't need Webkit and your application might be faster (because only the HTML page is loaded, without any linked resources like images, CSS, javascript files).
QWebView as class name states is a view, so it views something (in this case web pages). If you don't need to display loaded page, then you don't need a view. QNetworkAccessManager may do the work, but you need some knowledge about HTTP protocol, and also anything about target site: how does it hande logins, what type of request you have to send to login etc.
Related
I am creating an ecommerce webapp and I want to update my admin page when user place order. After research i found out about Django signals, but it seems pretty confusing. The idea behind this is that, when a user places an order I want the admin page to be refreshed and show the latest updates. I tried ajax but javascript can only work with the current open page. Can anyone help me with usibg django signals this way?
I think you misunderstand a little, if your idea is that the admin page viewed by the user in a browser is to be refreshed. The page viewed by a site visitor is retrieved on demand of the browser. The user could refresh a page, or a page script may auto-refresh on a timer (this is not a very good solution, but it is easy).
Django can't make the browser update the page.
Signals can cause django to do something in the backend, but they can't solve the problem that the browser is in charge.
There is one technology designed to allow the server to push content to the browser: websockets.
If you use websockets (Django's fairly recent built-in support is called Channels (https://channels.readthedocs.io/en/stable/)), then you can push content to the browser, where a receiving script on the page will do something with it. This is a very powerful technique, but there is a learning curve of some hours if you are starting from scratch.
Recently I came across this front-end library which tries to make this easy as
far as the browser goes: https://htmx.org/docs/
But you still have to deal with running a websocket server and learning how to send messages to a websocket. You will however feel like a superhero at the end of it, so there's that.
I am fairly new in web development and I decided to handle a user's availability to send a POST request to server. However, I do not know even whether it is possible or not but when a user close my Django site without using logout button (For example close the browser or tab), in order to understand the user is offline or online, I want to send a request to server. As a result, when the server does not get an answer from the user for a while, it automatically logout the user.
Can you tell me is it a good way to handle a user's availability and first of all is it a realistic solution? If it is, can you suggest me a document or example that helps me please.
I agree to to the answer of #Mounir. That's not related to django, if you want to know when a user is "disconnected"(close the tab or window) you need to use javascript, in concrete you need to use Sockets.
I recommend you this ones:
http://socket.io/
https://github.com/centrifugal/centrifugo
I'm using centrifugo for one project right now. In concrete, for chat and inbox messaging. It's very simple to make it run with Django.
Greetings
For logging out user you can use the Session expiration, but for all other staff you want to achieve I don't see any thin really related to Django itself, everything depend on your requirements and is more Browser/javascript related than Django.
I have done a few mobile apps using django as my backend, and now I am working on a chrome extension. I want my users to be able to send POST requests up to the server using the app/extension but is there a way to do it without first asking the server for a CSRF token? It just saves the app from making two requests every time we want to submit data. For example, I want to update my profile on my social media app or update a wallet from a chrome extension. It would be nice to open up the profile view input the data and push it to the server. It's less sleek if I have to open the profile, then wait for it to grab a token from the server and then I can submit the data. Is there another way to do this? Or am I stuck making multiple requests every time I want to submit data?
Also, a little clarification, CSRF prevents sites from submitting forms with user's data. But what is to stop me from making a site that uses ajax or something to grab the real site and steal the CSRF token and then paste that into my cross site request form? I feel like there is a loophole here. I know that I am not quite understanding this all the way.
You can, and should, make any API endpoint CSRF exempt.
Django offers the csrf_exempt decorator for exactly this, see https://docs.djangoproject.com/en/dev/ref/contrib/csrf/#django.views.decorators.csrf.csrf_exempt.
Also CSRF is intended to prevent unintended actions being performed via GET request forgeries. It is not intended to make it impossible for an automated system to submit forms, there are captchas for that.
As for what prevents you from using AJAX to grab the whole site and extract the token is something called the Same-Origin Policy. This is implemented by the browser and prevents any AJAX call from returning data when the target of the AJAX call is a different domain without the correct headers set. (I'm not entirely sure what sandboxing is applied to chrome extensions concerning this). As such it will, or at least should, fail to get data via AJAX for normal websites, e.g. a profile page. If you want to interact with third party websites you should look into whether or not they offer an API.
I would like to track what websites my site's visitors go to after they leave.
Would it be possible to place a cookie on their browser when they visit my site, and then later if they go to Facebook.com or stackoverflow.com, my cookie will retreive the browser's URL data and send it back to my server.
I could then look at this data and know that my visitors had gone to Facebook.com and stackoverflow.com after they left my site.
Is this possible using cookies?
Thanks for the help.
No. Cookies are not executed or anything. They are just dumb bits of data.
You would need to be able to execute code on the page they are visiting afterwards.
What I presume you are trying to ask, is that you want to track your outbound links.
This is mainly done with Javascript: You need to intercept click events from outbound anchor links, and send an event notification as described here, or using the hitCallbackmethod prior to completing the redirection to the external website. For Google Analytics see documentation. Or you could do via a custom JS implementation sending the info back to your server instead.
Alternatively your could replace all outbound links on the server side in your html source, and have all links pointed to your server first, and redirected to the external sites. But using redirects for this purpose is not really a good recommendation, unless you are an ad networks or a search engine company requiring such method.
Lastly, there is an alternative method using the HTML5 'ping' attribute. But the feature has been either removed and/or not yet fully implemented across all browsers as of this writing.
But you can't track where your visitors go beyond the 1st level outbound links of your site.
I'm trying to write a simple GUI application using Qt framework.
The purpose of this app is to retrieve data from my isp and parse them for presentation.
How do i authenticate my user/password with the webserver and retrieve the html page in question?
Are there any utility libs that make this task trivial?
I figure i need to interact with the server php script and simulate a form input somehow.
Am i on the right track?
You're on the right track, I suggest taking a look at curl.
That should make it alot easier.
edit: Hm, thought it did more than just file-transfer.
Otherwise here's a load more of interesting lib's
The way to authenticate depends completely on the authentication method used by the server. If it's some form to log in you need to retrieve that and send the correct data to the forms action target (usually as POST request). You could do this by constructing your request using QHttpRequestHeader and then simply sending it to the server. If you even know about the form you might even not need to retrieve the login page. If the website uses HTTP authentication you should be able using QAuthenticator.