I'm trying to test the mobile version of a webpage on my desktop. It uses WURFL for mobile detection, which I thought could be tricked by changing my user agent.
This has no impact on what page is displayed to me..
Any ideas how I can trick the page into showing me the mobile view?
Depending on the caching strategy that the website is using, you may need to clear your cookies in order to test both versions of the site. Spoofing your User Agent is the correct method in order to "trick" the website into serving the mobile version.
In addition to what Elliot said, the fact that the mobile version of your page is not shown may depend on how you implemented the routing of your website, so check that, after the device has been detected, the request is properly forwarded to the mobile version.
Related
You can disable cookies, change your ip 500 times but can’t anyone just track you through fingerprinting?
You could disable Java and Flash. Though that would break the page and make you stand out anyway.
You could use Tor but I think if you use Tor you get blacklisted from some sites instantly.
What’s the workaround? Using Chrome is a big nono. Internet explorer maybe and firefox perhaps…
Are there any apps that deal with this? Or just design a good web scraper, have an ip and cross your fingers.
I realize the average site is not going to implement all these features, but I am how one would workaround a site that was extremely vigilant.
There are two types of browser fingerprinting:
1. static fingerprinting - can identify browsers (and probably operating systems) just based on details of their requests. That's the order and capitalization of http headers, browser specific headers etc.
One small aspect is described here: https://gwillem.gitlab.io/2017/05/02/http-header-order-is-important/
As this can be done without any javascript, I guess scrapy is identifyable this way.
How to get around this?
As mentioned in the above article you need to exactly emulate a particular browser's fingerprint by emulating its headers' order and capitalization (and it has to match the user agent, of course)
2. dynamic fingerprinting - uses Javascript to collect data on installed plugins, plugin versions etc ... As Granitosaurus wrote, that won't be triggered by scrapy. But sites that use fingerprinting for scraping protection will block the scraper if it doesn't get any data from its fingerprinting module.
As this type of fingerprinting yields much more dimensions it can be used to identify particular users with a high reliability (over 90%)
You can find a good example how this is done here: https://github.com/Valve/fingerprintjs2
How to get around this?
use a lot of different real browsers for scraping (for example through selenium, no phantomjs, it can be detected)
randomize these browsers' settings and installed plugins (ideally using different versions)
when scraping rotate these browser instances instead of rotating IPs (each browser instance should keep its IP over its livetime)
If one of the instances is "burnt" replace it with a new instance that has a fresh IP and randomized browser fingerprint
... as you'll need many browsers this has to be done in an automated way, of course.
Resetting cookies sounds like a good idea at first, but if the fingerprinting system is worth its salt it won't need cookies to identify each of these machines reliably.
We are currently on Kentico EMS V.7 but when our site(s) was created there was not any mobile device layouts setup. an out entire site is build around ASPX+Portal templates so we don't have the edit layout option on our pages. What would you recommend an origination like ours do in this case? Is there an import or migration tool that can be used to move the pages into a responsive design template (are these available out of the box or do they all need to be built?)
has anyone else come against this challenge with their site when trying to a mobile accessible site, if so what approach did you take to move your site(s) over?
I know Stackoverflow doesn't like to answer broad questions, but I'll save you some trouble and hopefully at least point you in the right direction. I use to be in your shoes and I remember how frustrating it was trying to find resources to learn all of this.
You've got some options, but unfortunately none of them are magic bullets. All of these options involve design and development time. If you haven't already done the research to consider if optimizing your site for mobile users is a good business decision, then I would suggest starting there first.
Mobile Device Detection
Kentico offers device layouts that allow you to change the layout of your site depending on which device a user is browsing your site with. The technique it uses is called device detection, which relies on reading the incoming user agent string of the user's http request on your server and using that information to determine what to send back to the user's browser.
This process happens on the server side of your application and also relies on comparing the user agent string against a list of known user agent strings. Kentico in particular uses the 51degrees library to accomplish this. This has the drawback of not always having the most up-to-date user agent strings, so new devices won't be included in your list unless you keep it updated.
A big problem with this approach is that you wind up maintaining multiple sites and/or layouts. If you uses Kentico's mobile device detection, then any time you need to make changes to a page template you will also have to change each of the different layouts for that template.
You don't have to utilize Kentico's device layouts functionality to do mobile device detection. You could just redirect users to a different site entirely. In which case you can still very easily run into the problem of redundant maintenance.
Yes, there are ways of mitigating these issues, but most web devs agree that responsive design is usually the way to go if you want to cater to mobile users.
Responsive Design
This is a client-side design paradigm that relies on CSS media queries. I'm not going to explain how it works since googling "responsive design" will net you plenty of research material. The short version is your user's browser handles the adjustment of the layout by interpreting the CSS of your site. This means you maintain one CSS file for one site and the client handles the adjustments for you.
Now the down side to this is that it takes a takes a talented web designer to do it properly. There are many responsive frameworks out there that can help you out with this, but their appearance is pretty generic and will likely still require customization to fit your particular brand requirements. Some more popular ones are bootstrap, semantic-ui, and foundation.
I recently helped convert a large website in Kentico with a static design into a responsive design by rebuilding all of the page templates using Bootstrap and merging their stylesheets together so that the desktop version still looked the same and the mobile versions had a consistent appearance with the desktop version. This process took about two months and required a lot of UX and content strategy in addition to visual design and coding skills to accomplish correctly. It's not the hardest job in the world, but requires quite a bit of skill and time to do.
In my humble opinion there is no tool which will make (magically) your site responsive. AFAIK the only way is to redesign your site manually (new CSSs - maybe any UI framework, JS etc...)
Maybe you can still use out of the box features like device profiles and mobile pages. (But I am not sure how it works and if it is supported while using ASPX + Portal templates development models).
I've designed a desktop app using PyQt GUI toolkit and now I need to embed this app on my Django website. Do I need to clone it using django's own logic or is there a way to get it up on website using some interface. Coz I need this to work on my website same way it works as desktop. Do I need to find out packages in django to remake it over the web or is there way to simplify the task?
Please help.
I'm not aware of any libraries to port a PyQT desktop app to a django webapp. Django certainly does nothing to enable this one way or another. I think, you'll find that you have to rewrite it for the web. Django is a great framework and depending on the complexity of your app, it might not be too difficult. If you haven't done much with web development, there is a lot to learn!
If it seemed like common sense to you that you should be able to run a desktop app as a webapp, consider this:
Almost all web communication that you likely encounter is done via HTTP. HTTP is a protocol for passing data between servers and clients (often, browsers). What this means is that any communication that takes place must be resolved into discrete chunks. Consider an example flow:
You go to google in your browser.
Your browser then hits a DNS server (or cache) that resolves the name google.com to some IP address.
Cool, now your browser makes a request to that IP address and says "get me some stuff".
Google decides to send you back a minimal amount of HTML and lots of minified JavaScript in the page.
Your browser realizes that there are some image links in the HTML and so it makes additional requests to google to get each of the images so that it can display them.
Now all the content is loaded on your browser so it starts to execute the JavaScript code, and that code needs some more data from google so it starts sending requests to google too.
This is just a small example of how fundamentally different a web application operates than how a desktop application does. On a desktop app you have the added convenience that any operation doesn't need to be "packaged up" and sent, then have an action taken, etc (unless you're using a messaging architecture, but that's relatively uncommon outside of enterprise apps).
I have recently found those two look-alike solutions/IDE for cross-mobile development: Appcelerator and Rhomobile (I know there are more) and I have questions regarding those two platform:
1) I believe the only way to build the view is using HTML, which I like alot the ideas. But, does that mean the application itself isn't available if the mobile is offline?
2) Do you guys know if it's possible to publish the application to the App Store and Google Store?
3) Are there any simulator for different mobile and do they support all those slide/tab events?
4) And finally, are there a way to transfert the App on your mobile phone without having to publish it anywhere.
Please note that I have no knowledge at all about mobile app dev and those two solutions (Appcelerator, Rhomobile) would be perfect for me as I am familiar with Javascript and HTML.
Thank you!
Ok I have only used appcelerator but:
1) a webview is like a browser without the address bar, it simply parses HTML, where it gets it from is up to you. If you write the HTML and pass in a file well then yes it can be offline, if it is used to parse a response from a webpage well then no as it needs to send a http request to the webpage.
As many people seem to mistake (for a reason unknown to me as all the documentation states other wise), appcelerator is not the same as phonegap, appcelerator uses its own javascript based API to allow developers to make native apps, it is NOT a webview wrapper. It is offline by default and allows you to send http requests if you need something online.
2) yes you can publish to the app store and the google store from appcelerator, the documentation walks you through the process.
3) Appcelerator requires you to download either the IOS sdk or Android SDK which come with simulators, appcelerator / the emulators support the standard events found on these devices.
4) With Android to can build a .apk file and distribute however you wish, with IOS the only way is to publish to the app store. the only other way is to make a mobile website instead of an application
One problem I've been toying with off and on is a service that requires my server to produce a screenshot of a webpage at a given url. The problem is that I don't have any idea how I would accomplish this. I mostly use a LAMP software stack, so answers that were given with that in mind would be the most helpful. Again the basic requirements are: Given a url, the server needs to produce an image file of the rendered web page at that url. Thanks in advance!
You might also want to take a look at webkit, it's known for being easier to embed (used by Adobe for AIR, by Google for Chrome, by Apple for the iPhone...) then other rendering engines. This might take a little more work to setup, but it would be a lot more stable than some hack that launched a webbrowser and took a screenshot.
IF your server is a Mac, then I recommend webkit2png, which is a short python program that leverages WebKit's Objective-C API to render an URL. Personally, I use it in combination with WWW::Mechanize to walk my development site and make screenshots of every page -- useful for testing functionality, showing clients and keeping screenshots up-to-date. The resulting screenshot is perfect, but sometimes very tall for long, scrolling pages.
IF your server has a non-bare-bones Linux distro with KDE installed, then you might try khtml2png. I have not tried that myself, but saw it mentioned on the webkit2png page.
PhantomJS is a headless (commandline) WebKit-based browser which can be easily scripted to save a screenshot of webpage.
You actually need to have the server launch the web browser in question and take a screenshot of the application with the appropriate libraries. Apache will not render the page for you so you have to have software that will.
Yes, that is what is needed. I do this in asp.net, and I actually create a WebBrowser object that is avaialable in the .Net framework class libraries to generate the screenshot.
I use the http://webthumb.bluga.net service for thumbnail generation. Robust, powerful, easy to use, and very reasonable rates. I have a high traffic production website using this service and it works very well. Given the difficulty of creating a robust web screenshot service, it's nice to have someone else do the hard work.
A non-free solution for Java is WebRenderer. Interesting feature: it can emulate Safari, IE or Firefox browsers when rendering. They have a desktop version and a headless server version. Also they have example code showing how to render a screenshot image of a webpage.
virtual framebuffer X server
I would rather recommend XVFB (virtual framebuffer X server) is the best solution for taking screenshots of a headless server. Virtual framebuffer X server xvfb provides an X server that can run on machines with no display hardware and no physical input devices.
I am using that on my server for testing URLs and taking its screenshot. We are using Ubuntu & XVFB + FIREFOX. It is working fine. Modify according to your needs.Take a look on these articles. It might be use full for you.
http://www.semicomplete.com/blog/geekery/xvfb-firefox.html
http://linux.about.com/cs/linux101/g/xvfb.htm
http://www.xfree86.org/4.0.1/Xvfb.1.html