I am a very new developer who can write basic HTML (yes I know it's English), and is looking for a solution to a problem that I basically invented.
I recently bought a product called Doorbird, a very cool and well engineered IP Doorbell. The reason I bought it, is because it has an App, and integrates with IP phones and support video, and has a few bells and whistles, if you're not familiar with this product, I highly recommend looking at it.
The "problem" is that it does not integrate with Google Home, or Amazon Echo. However, it does have an API that allows you to go to the URL, login and unlock the door. I was looking into creating my own action that would have the Google Home login to the Doorbell and go to the unlock URL.
My issue is, I have no experience with this, but I pick things up very quickly, and believe in myself!
I already looked at the api.ai site from Google, and have signed up. It looked simple, but upon further investigation looks very complicated.
If anyone would be able to point me in the right direction I would be greatly appreciative.
Thanks in advance,
-Jack G
You need them to publish their REST api so that you can call the 'action' to unlock directly vs. having to go to a login page. You would probably set up oauth to have your server talk to theirs or use web headers. Ask their tech support if they support direct REST api using a security model.
You can disable cookies, change your ip 500 times but can’t anyone just track you through fingerprinting?
You could disable Java and Flash. Though that would break the page and make you stand out anyway.
You could use Tor but I think if you use Tor you get blacklisted from some sites instantly.
What’s the workaround? Using Chrome is a big nono. Internet explorer maybe and firefox perhaps…
Are there any apps that deal with this? Or just design a good web scraper, have an ip and cross your fingers.
I realize the average site is not going to implement all these features, but I am how one would workaround a site that was extremely vigilant.
There are two types of browser fingerprinting:
1. static fingerprinting - can identify browsers (and probably operating systems) just based on details of their requests. That's the order and capitalization of http headers, browser specific headers etc.
One small aspect is described here: https://gwillem.gitlab.io/2017/05/02/http-header-order-is-important/
As this can be done without any javascript, I guess scrapy is identifyable this way.
How to get around this?
As mentioned in the above article you need to exactly emulate a particular browser's fingerprint by emulating its headers' order and capitalization (and it has to match the user agent, of course)
2. dynamic fingerprinting - uses Javascript to collect data on installed plugins, plugin versions etc ... As Granitosaurus wrote, that won't be triggered by scrapy. But sites that use fingerprinting for scraping protection will block the scraper if it doesn't get any data from its fingerprinting module.
As this type of fingerprinting yields much more dimensions it can be used to identify particular users with a high reliability (over 90%)
You can find a good example how this is done here: https://github.com/Valve/fingerprintjs2
How to get around this?
use a lot of different real browsers for scraping (for example through selenium, no phantomjs, it can be detected)
randomize these browsers' settings and installed plugins (ideally using different versions)
when scraping rotate these browser instances instead of rotating IPs (each browser instance should keep its IP over its livetime)
If one of the instances is "burnt" replace it with a new instance that has a fresh IP and randomized browser fingerprint
... as you'll need many browsers this has to be done in an automated way, of course.
Resetting cookies sounds like a good idea at first, but if the fingerprinting system is worth its salt it won't need cookies to identify each of these machines reliably.
We are currently on Kentico EMS V.7 but when our site(s) was created there was not any mobile device layouts setup. an out entire site is build around ASPX+Portal templates so we don't have the edit layout option on our pages. What would you recommend an origination like ours do in this case? Is there an import or migration tool that can be used to move the pages into a responsive design template (are these available out of the box or do they all need to be built?)
has anyone else come against this challenge with their site when trying to a mobile accessible site, if so what approach did you take to move your site(s) over?
I know Stackoverflow doesn't like to answer broad questions, but I'll save you some trouble and hopefully at least point you in the right direction. I use to be in your shoes and I remember how frustrating it was trying to find resources to learn all of this.
You've got some options, but unfortunately none of them are magic bullets. All of these options involve design and development time. If you haven't already done the research to consider if optimizing your site for mobile users is a good business decision, then I would suggest starting there first.
Mobile Device Detection
Kentico offers device layouts that allow you to change the layout of your site depending on which device a user is browsing your site with. The technique it uses is called device detection, which relies on reading the incoming user agent string of the user's http request on your server and using that information to determine what to send back to the user's browser.
This process happens on the server side of your application and also relies on comparing the user agent string against a list of known user agent strings. Kentico in particular uses the 51degrees library to accomplish this. This has the drawback of not always having the most up-to-date user agent strings, so new devices won't be included in your list unless you keep it updated.
A big problem with this approach is that you wind up maintaining multiple sites and/or layouts. If you uses Kentico's mobile device detection, then any time you need to make changes to a page template you will also have to change each of the different layouts for that template.
You don't have to utilize Kentico's device layouts functionality to do mobile device detection. You could just redirect users to a different site entirely. In which case you can still very easily run into the problem of redundant maintenance.
Yes, there are ways of mitigating these issues, but most web devs agree that responsive design is usually the way to go if you want to cater to mobile users.
Responsive Design
This is a client-side design paradigm that relies on CSS media queries. I'm not going to explain how it works since googling "responsive design" will net you plenty of research material. The short version is your user's browser handles the adjustment of the layout by interpreting the CSS of your site. This means you maintain one CSS file for one site and the client handles the adjustments for you.
Now the down side to this is that it takes a takes a talented web designer to do it properly. There are many responsive frameworks out there that can help you out with this, but their appearance is pretty generic and will likely still require customization to fit your particular brand requirements. Some more popular ones are bootstrap, semantic-ui, and foundation.
I recently helped convert a large website in Kentico with a static design into a responsive design by rebuilding all of the page templates using Bootstrap and merging their stylesheets together so that the desktop version still looked the same and the mobile versions had a consistent appearance with the desktop version. This process took about two months and required a lot of UX and content strategy in addition to visual design and coding skills to accomplish correctly. It's not the hardest job in the world, but requires quite a bit of skill and time to do.
In my humble opinion there is no tool which will make (magically) your site responsive. AFAIK the only way is to redesign your site manually (new CSSs - maybe any UI framework, JS etc...)
Maybe you can still use out of the box features like device profiles and mobile pages. (But I am not sure how it works and if it is supported while using ASPX + Portal templates development models).
I want to start work on a new web application, but I really don't want to re-invent the wheel. I want to make a simple web application that provides some basic functionality, and looks really good. Does anybody know of anything (even if it isn't free) where I could get something that has a good framework laid down already? Where I can have a login/sign up function already built in and ready for use? I have seen a few ecommerce sites that fit the build, but Id have to do a ton of clean-up. I was hoping to find something that was basic, and ready for adding to right away.
I would think that word press or drupal would have something, but I've been looking and haven't been able to find anything.
Thanks!
Try out Thin PHP Framework, it has OAuth built in allow you to login with popular services Google, Twitter, Facebook... or you can use its simple username, password login. Check out the demo here:
http://demo.thinphp.com
website:
http://thinphp.com
Choose your language, there are a lot of different, good frameworks, but you will do best with the language you already know. Wordpress, Drupal, Django, Plone, Joomla, Typo3 are great, and of course they have user authentication.
Looking good is something different, that's design and it is done in CSS. And every CMS or web framework usually has good default CSS templates as a starting point.
I've worked with XAMPP, WAMPP, MAMPP, etc and am starting to look at Django.
A majority of the work we do is very CMS orientated; although we've been told not to use third-party CMS' (mainly because of user's find them hard to use, and other issues), I've found that I can code a very simple CMS using Cake, CodeIgniter or one of the other PHP frameworks.
And yet, I'm getting increasingly frustrated with the amount of coding I need to do just to get something up and running, and I've been told that Django is a good Python framework to use. It also seems to get a lot of buzz from reddit.
I have some concerns and queries about moving from XAMPP to Django.
1) Security
Any web app should be coded defensively. Over the past few years we've seen a movement towards protecting against XSS, SQL injections, Cross site forgeries, session fixation, session hi-jacking, cookie hi-jacking; the amount of security one needs can be overwhelming.
What things does Django do to prevent/limit XSS, SQL injections, Javascript injections, and santizing input; one normally associates with securing PHP web apps? Is it something I need to worry about, or does Django do all this stuff out of the box.
2) What goes in the /www/ public folder?
In a manual I read it said not to put manage.py or the other .py stuff in the main webroot, so this means I put everything outside of the webroot; so what goes in there?
Do I put the /templates/ directory inside the webroot? How does the server know what to run?
3) Can I still use .htaccess on Django projects? I am familiar with Apache and often use it to do routing, or blocking off bad bots, but will using .htaccess still work?
4) Cronjobs
Do cronjobs still work with Python/Django projects?
5) Running Third party perl/other scripts
In PHP you can use other libraries such as the curl library, ffmpeg, ImageMagik as well as many others; can I still use these libraries with Python/Django?
6) Admin screen
Django gives you an out-of-the-box admin screen; is this only for development purposes or can it put live? I am concerned about any the security of the admin screen.
7) Integration with Discuss, Facebook, Twitter, OpenID, captcha, etc.
There are libraries in PHP that help integrate DisQuss, Facebook, Twitter; but is it relatively easy to do an integration with these and other third party apps?
8) E-commerce, SSL
Are there many e-commerce sites that use Django? I've seen a lot of CMS/Blog type software but not many e-commerce sites. By which I mean, shopping card, Protx/Paypal or Worldpay integration.
That's another thing; there are sandboxes for Protx, Paypal, Worldpay etc for PHP -- but are there any for Django?
9) Is it worth it?
Is it worth moving to Django from an XAMPP background? Will it really make things faster, or is that just marketing hype?
Thanks.
Security. The Django core team are very security-conscious, and have taken great care to make things like SQL injection impossible. The next version, 1.2, includes a whole new cross-site request forgery protection library. Obviously, you still need to be aware of these when developing your application, but Django does a lot to help you.
What goes under /www/public: Nothing. Django doesn't work via the normal Apache serving mechanism: it hooks into (preferably) mod_wsgi, which needs a single file which then tells it to run the rest of the code. The templates can go anywhere, and are pointed to by your Django settings file, but again aren't served directly by Apache.
.htaccess: You don't really need it, because of point 2: you're not serving things in a filesystem hierarchy. The best way to do it is to set up vhosts and manage things that way.
Cron jobs: Absolutely. Django is just Python, and you can easily run Python scripts via cron. Django allows you to set up custom command scripts which initialise the ORM and give you access to anything you would need.
Libraries: Again, because Django is Python, you get access to the huge amount of Python libraries that are out there. For curl, Python has urllib; for ImageMagick, it has PIL; and no doubt there are equivalents of ffmpeg too.
Admin: Again, security has been thought of from the beginning. Opinions differ as to whether you should use the admin only for your expert users, or customise it and allow access for all users; I've had a lot of success using it as the basis for my custom CMS interfaces.
Facebook, etc: Yes, there are libraries for all of these.
E-commerce: There is a whole e-commerce project, Satchmo, written in Django. Libraries exist to interface with all the payment providers.
Is it worth it? Only you can tell. My experience working alongside a range of developers who have moved from PHP is that they've enjoyed the experience and became much more productive.
On SQL Injections: Django uses an ORM, which takes care of SQL injection protection, and you will rarely write you own SQL. If you do, just follow the instructions on how to pass parameters to raw queries and prevent SQL Injections.
There is an entire chapter on the django book about security that should answer all your questions.
On what goes into /www/: anything that is not code? The concern is to not put the python code there.
On .htaccess: Yes, it should still work (for any non Django resources as Daniel points out).
On cronjobs: what do you mean?
On Libraries: Python - the language you will use with Django - is rich in libraries that probably provide the same functionality you are used to. This is a key point: you will need to learn Python well to benefit the most from Django.
On the admin interface: This is actually the thing that will probably help you the most, judging from your question. They are customizable (within some limits) and they really give the staff (it is not intended for public users, but for staff users) the basics of CRUD for your database models. It is a time saver. You might need to write your own templates for advanced functionality, but for most simple CRUD aimed at staff (which is usually the point of a CMS) it is very useful and easy to set up.
On integration: Check Pinax for a group of applications that provide extra functionality. There is a rich and diverse universe of integration solutions out there. It is not unusual to find questions here in SO about django + facebook and others.
On E-commerce: Check Satchmo out.
Is it worth it: Now, I have no experience with XAMPP. I know that I like Python better than both Perl and PHP (and Java, for that matter). I know that as a framework Django is simpler to use, faster to deploy than anything I used before.
My suggestion is the age old: go build a simple project and make up your own mind. You are the only one in position to decide if Django is the framework for you.
An older question on SO discusses some Django limitations. My answer to that might be helpful to you too.
I recently moved to developing any new projects in Django, coming from a PHP background. Here are my thoughts on your questions.
1) Security
Strings sent to templates is escaped by default, which takes care of most of that. Since you're using an ORM, SQL injection shouldn't be an issue unless you build raw queries for some reason.
2) What goes in the /www/ public folder?
Django doesn't use a file hierarchy for URLs like a typical PHP setup. The server knows what to run from your urls.py and settings.py pointer to the template folder.
3) Can I still use .htaccess on Django projects? I am familiar with Apache and often use it to do routing, or blocking off bad bots, but will using .htaccess still work?
As noted above, it works for static content just the same. For dynamic pages, you'd want to implement some other form of authentication or redirection for clients you want to block, as far as I know.
4) Cronjobs
There's no reason why you can't use cron for whatever, as you still have a normal Linux system.
5) Running Third party perl/other scripts
You'll want to use the Python versions of those libraries, of course. For instance
FFMpeg
PythonMagick
I replaced most of my need for Curl with the built-in urllib and urrlib2 libraries, but there is also PyCurl if you need it.
6) Admin screen
The Admin screen is intended to be used by your own admins, i.e. site staff. It may be possible to do so, but it's not supposed to be the scaffolding on which you build your public facing project.
7) Integration with Discuss, Facebook, Twitter, OpenID, captcha, etc.
There are a lot of people out there using Python and Django, and I haven't had any problem finding libraries. In my experience there is a bit less support for something than PHP, but what is there is often higher quality.
8) E-commerce, SSL
I haven't tried payment integration, so I can't say. Not sure about the other sites, but the Paypal Sandbox is run by Paypal, isn't it? I don't think it's related to what you're using on the server, so sure, you can access it like normal.
9) Is it worth it? Is it worth moving to Django from an XAMPP background? Will it really make things faster, or is that just marketing hype?
I moved to Django because Python is truly a more compelling language than PHP. Will it make things faster? I'm not sure what the advantages in that respect would be for Django vs.the PHP MVC frameworks. There are no magic bullets.
You do have to keep in mind that you're not just learning a new framework, but also a new language. There will be a bit of a learning curve if you've never used Python before. but I've found both Python and Django to be fairly easy to learn. The clean design of the language is fantastic and Django is veryt well designed, too. I do feel that it's boosting my productivity. I've found snippets for or articles about most everything I need to do in Django as I've been learning, so adapting has been pretty simple.