I'm attempting to write a simple HTTP/HTTPS proxy using Boost ASIO. HTTP is working fine, but I'm having some issues with HTTPS. For the record this is a local proxy. Anyway so here is an example of how a transaction works with my setup.
Browser asks for Google.com
I lie to the browser and tell it to go to 127.0.0.1:443
Browser socket connects to my local server on 443I attempt to read the headers so I can do a real host lookup and open a second upstream socket so I can simply forward out the requests.
This is where things fail immediately. When I try to print out the headers of the incoming socket, it appears that they are already encrypted by the browser making the request. I thought at first that perhaps the jumbled console output was just that the headers were compressed, but after some thorough testing this is not the case.
So I'm wondering if anyone can point me in the right direction, perhaps to some reading material where I can better understand what is happening here. Why are the headers immediately encrypted before the connection to the "server" (my proxy) even completes and has a chance to communicate with the client? Is it a temp key? Do I need to ignore the initial headers and send some command back telling the client what temporary key to use or not to compress/encrypt at all? Thanks so much in advance for any help, I've been stuck on this for a while.
HTTPS passes all HTTP traffic, headers and all, over a secure SSL connection. This is by design to prevent exactly what you're trying to do which is essentially a man-in-the-middle attack. In order to succeed, you'll have to come up with a way to defeat SSL security.
One way to do this is to provide an SSL certificate that the browser will accept. There are a couple common reasons the browser complains about a certificate: (1) the certificate is not signed by an authority that the browser trusts and (2) the certificate common name (CN) does not match the URL host.
As long as you control the browser environment then (1) is easily fixed by creating your own certificate authority (CA) and installing its certificate as trusted in your operating system and/or browser. Then in your proxy you supply a certificate signed by your CA. You're basically telling the browser that it's okay to trust certificates that your proxy provides.
(2) will be more difficult because you have to supply the certificate with the correct CN before you can read the HTTP headers to determine the host the browser was trying to reach. Furthermore, unless you already know the hosts that might be requested you will have to generate (and sign) a matching certificate dynamically. Perhaps you could use a pool of IP addresses for your proxy and coordinate with your spoofing DNS service so that you know which certificate should be presented on which connection.
Generally HTTPS proxies are not a good idea. I would discourage it because you'll really be working against the grain of browser security.
I liked this book as a SSL/TLS reference. You can use a tool like OpenSSL to create and sign your own certificates.
Related
I'm running a small web app on an EC2 instance and I want some friends to be able to use it. I also want to make it use HTTPS, just for basic security purposes (prevent packet snooping whenever possible). Of course I am using a self-signed certificate, because my budget for this project is $0. But Chrome throws up a warning page upon trying to visit it:
Your connection is not private
Attackers might be trying to steal your information from [...]
(for example, passwords, messages, or credit cards).
NET::ERR_CERT_AUTHORITY_INVALID
This server could not prove that it is [...]; its security certificate is not trusted by your computer's operating system. This may be caused by a misconfiguration or an attacker intercepting your connection.
Is is not true that "any encryption is better than no encryption"? On unenecrypted HTTP, I could be trying to steal information as well, and don't have to prove anything about my server identity, AND my communication can be read in plaintext by packet sniffing, but Chrome doesn't throw up any warning flags there...
What gives? Why does Chrome hate self-signed certificates so much? Why doesn't it just put a little red box over the padlock icon, instead of giving me a two-click warning page?
Edit Sep 2021 (this was applicable since 2016): Just suck it up and use one of the free key issuers. Let's Encrypt and AWS ACM will literally do it for free.
This question is not specific to chrome. Firefox and probably other browsers behave similar and in the last years the warnings even got stricter. Complaining about these warnings shows more a missing understanding of the role of certificates in HTTPS.
With HTTPS one expects encryption, i.e. private communication between the browser and the server with nobody sniffing or manipulating the transferred data. At the beginning of the encryption client and server exchange the encryption keys, so that one can encrypt the data and the other can decrypt the data. If some man-in-the-middle manages to manipulate the key exchange in a way that it gets control over the encryption keys, then the connection will still be encrypted but not private. Thus it is essential that the key exchange is protected and this is done with certificates. Only with proper checking of the certificates the client can verify that it talks to the server and not some man-in-the-middle and thus the critical key exchange can be protected.
Certificates are usually verified by
Checking the trust chain, i.e. if the certificate is directly or indirectly (via immediate certificates) issued by a certificate agency (CA) trusted by the browser or operating system.
Verifying that the certificate is issued for the expected hostname, i.e. the subject matches the hostname.
With self-signed certificates or certificates issued by a CA unknown to the browser/OS this check will fail. In this case it is unknown, if the original certificate was already not issued by a trusted CA or if there is some man-in-the-middle manipulating the connection. Being man-in-the-middle is not hard, especially in unprotected networks like public hotspots.
Because the browser can not verify the certificate in this cases it will throw a big fat warning to show the user that something is seriously wrong. If your friends know that you only have some self-signed certificate there they should also know that this is the expected behavior of the browser in this case. You also should provide them with the fingerprint of your certificate so that they can be sure that this is the expected certificate - because there is no other way to check the validity of this certificate. Note that this warning also comes once because the browser saves the fingerprint and from then on knows that your site is associated with this certificate. But if you change the certificate it will complain again.
If you don't like the trouble of teaching all of your friends how to properly verify your certificate then get yourself a certificate by a public CA. They don't need to be expensive and some also issue free certificates.
Is is not true that "any encryption is better than no encryption"?
While bad encryption might be better than no encryption, transferring sensitive data over en encrypted but man-in-the-middle connection is definitely worse then transferring non-sensitive data with no encryption. And contrary to plain HTTP you can actually detect a potential man-in-the-middle attack with HTTPS. What you can not do is find out if this a potential man-in-the-middle attack or if the non-verifiable certificate is actually the expected, because the browser has no previous knowledge what to expect. Thus a self-signed certificate is actually not that bad provided that the browser knows up-front that this site only provides a self-signed certificate. And it might also not bad if the transferred data are not sensitive. But how should the browser know what kind of data and what kind of certificate are to expect?
Because SSL/TLS are trying to solve two problems in a single stroke, but you are completely ignoring one of them.
SSL is meant to provide both encryption (between two endpoints) and authentication (where each endpoint is exactly who it says it is). This latter solution is generally meant to be solved via organizations known as Certificate Authorities (CAs), who are supposed to verify your identity before agreeing to give you a certificate. While there have been some spectacular failures of this level of trust in the past, we don't have anything better yet, and so browsers still expect to see SSL/TLS certificates to be issued by one of these Trusted authorities; if it's not, there's no way to know if you're actually talking to the party you intended to.
So, while it may be encrypted, having an encrypted conversation with someone who shouldn't be party to the conversation is actually WORSE than having a plaintext conversation with someone who SHOULD be party to the conversation.
There are a few free SSL providers out there such as Let's Encrypt that won't cause this warning and still fit in your $0 budget.
Put chrome://flags/#allow-insecure-localhost into the Chrome address bar, then select the enabled option.
The reason for the click through is to offer some protection from phishing attacks.
The $0 work-around is to create a certification authority verified by Justaskin_ (which is just a special file) and have your friends to install the public key derived from of it on their computers.
Instead, use the private key to sign your https certificate and their browsers will accept it. OpenSSL is one tool that can do this.
Im working on some JSON-based web service that is supposed to work with Android application.
I would like to encrypt data transport between client (android) and server (virtual server in datacenter).
I don't have to make sure that my server is my server, just data encryption.
I have no idea how to use HTTPS.
Do I just put my PHP files in private_html and use https://example.com url?
To use HTTPS, you don't have to do anything in the coding of your web service - it's all in your hosting. Here the are steps you can follow. The specific instructions differ in your hosting (IIS, Apache, AWS/Azure, etc), but you can google specifics on how to accomplish any of these steps for whatever host and application framework you decide.
Buy an SSL certificate (there are many different vendors, but expect between $75-$200 for the certificate) based on the vendor, reputation, and level of security you need.
Generate a certificate signing request (CSR) from the server you'll be hosting.
Upload the CSR to the SSL vendor who will validate and provide the certificate for your use.
Import the SSL certificate into your application server, and configure the site to use the certificate. For instance, if you're hosting Microsoft IIS, you'd import the SSL certificate and then add HTTPS bindings on 443 to the specific website hosting your web service.
Another point of security. Since you are deploying SSL, you don't have to do any application level encryption (assuming you are not putting sensitive information in query strings - use POST if you think you need to). You probably would want to implement some security to restrict access to your web service so only your app can access it. Best practice is some level of OAuth, but at a minimum some type of pre-shared key in the header of the request is a lot better than nothing.
Here are some additional sites for more information:
https://www.digicert.com/ssl-certificate-installation.htm
https://support.godaddy.com/help/category/742/ssl-certificates-installing-ssl-certificates?prog_id=GoDaddy
If you don't want to pay for a certificate, you can use certificate signet by your own CA and add the root certificates into your application using HTTPClient and keystores
Here there's some guides
http://datacenteroverlords.com/2012/03/01/creating-your-own-ssl-certificate-authority/
http://developer.android.com/reference/org/apache/http/client/HttpClient.html
KeyStore, HttpClient, and HTTPS: Can someone explain this code to me?
http://blog.antoine.li/2010/10/22/android-trusting-ssl-certificates/
You can limit users to use JUST and only HTTPS in apache, IIS or whatever do you use. If your client connects to your server, his communications will be likely to encrypted, because he is already using HTTPS. And for responsing in HTTPS you virtually cannot send HTTPS responses, as far as I know, unless that other side isn't also a website (for example, if you have your website, you could send such a response e.g. to Google). You should be okay to send data like http status codes (OK, NotModified, PageNotFound, ...), or if you want something more, or if it is a requirement, then there you still have JSON and you could encode it as well, with some encoding algorithms, or use binary JSON format.
Check if your hosting company provides a free public shared https address. Most of them do.
If you want to understand how to do it right, follow this thread
Warning: Don't stick with the solution below for production.
If you plan o use an https endpoint without a certificate you have to make sure to disable peer verification, check this answer
I have developed a Django application and now want to make sure the POST data transmitted through the page is safe.
I have couple of questions about this?
I see SSL certificates being displayed on many webpages. How do I get this certificate?
Do I need to change anything on my submitted form to encrypt the data or should I change any settings on my webserver?
I know its a general question but it would be great if someone provides a good answer.
First off, the POST data transmitted through the page is never safe from an application perspective. You don't have control over the user of the website. SSL and HTTPS helps prevent man in the middle attacks to ensure the request from the client (browser) to your server is encrypted. The underlying data that is sent can be malicious, so you should always validate inputs.
Secondly, if you want to use HTTPS and SSL, which I highly recommend, you'll need to obtain a certificate from one of the providers out there and install it with your webserver, which I presume is apache. Typically your domain provider can help you with obtaining an SSL certificate for your domain from one of the main certificate authorities. Regarding the installation and setup, there is tons of information about this online as it's a common task. I'm not familiar with Apache configuration to provide any specific recommendations. You'll also want to have rewrite rules so that your site can only be accessed via HTTPS and if someone tries to use HTTP, it simply redirects to HTTPS.
Lastly, you don't need to do anything in your Django application as your webserver should handle the basic interactions between your server and client to validate the HTTPS requests.
I understand, we use SSL to encrypt sensitive data like user name and password to transported to server without people in the network eavesdropping. So then server returns a secure token over HTTPS and its stored in cookie. We switch to HTTP after we have secure token, we attach cookie/secure token header to every HTTP request.
Now anybody can see my secure token and they can eavesdrop it and impersonate me. Is my understanding correct?
The cookies can be set per protocol, so that HTTPS cookies are not used for HTTP and vice versa. Also, the properly constructed secure token should include an IP address and have short expiration time.
But in general the best idea is of course to keep the authenticated session in secure channel - SSL is not that heavyweight these days (as computers became much faster than when SSL was first introduced) and also the heaviest part is handshake, which is performed only once if persistent HTTP connection is used (or when SSL session resuming is used).
I'm developing a client/server app that will communicate via rest. Some custom request data will be stored in the header of the request. Both the server sending the request and the receiving server have an SSL certificate - will the headers be encrypted, or just the content?
SSL encrypts the entire communications path from the client to the server and back, so yes - the headers will be encrypted.
By the way, if you develop networked applications and care about data security, the least you should do is read a book like Practical Cryptography, by Niels Ferguson and Bruce Schneier, and probably further reading that's more focused on web application security would be a good idea. If I may make an observation - and please, I don't mean that as a personal criticism - your question indicates a fundamental lack of understanding of very basic web security technologies, and that's never a good sign.
Also, it's never a bad idea to confirm that data which is assumed to be encrypted is indeed encrypted. You can use a network analyzer to monitor traffic on the wire and watch out for anything sensitive being sent in the clear. I've used Wireshark to do this before - the results can be surprising, sometimes.
As long as you're communicating in the SSL tunnel, everything sent between the server and the client will be encrypted. The encryption is done before any data is sent or received.
Both headers and content are encrypted.
You appear to think that REST is a distinct protocol.
REST is not a protocol. It is a design style for HTTP-based applications.
So, your a writing an HTTP application. Are the headers encrypted? Yes, if you are using the HTTPS (HTTP over SSL) protocol instead of plain HTTP.
Having certificates on both sides is not directly relevant to your question. SSL certificates are used for authentication. They help in detecting man-in-the-middle attacks such as are possible using DNS cache poisoning.
Having a certificate is not enough, you have to configure the web server to encrypt the connections (that is, to use the certificate) for that domain or virtual host. In addition, I think you would just need a single certificate, responses to requests will still be encrypted.
And yes, HTTP headers are encrypted as well as the data.
The other answers are correct that headers are indeed encrypted, along with the body, when using SSL. But keep in mind that the URL, which can include query parameters, is never encrypted. So be careful to never put any sensitive information in URL query parameters.
Update: as #blowdart pointed out below, this is wrong. See the comment below.
SSL..or rather HTTPS (HTTP over SSL) sends all HTTP content over SSL, and as HTTP content and headers are in fact the same thing, this means the headers are encrypted as well.
Seeing as GET and POST data is sent via HTTP headers, then it only makes sense then when sending data securely you wouldn't just want the response code or content to be encrypted.
Not everything is encrypted: the request query string is not encrypted. Believe me, I've seen requests like this:
https://mydomain.com/authenticate?user=username&password=MyStrongPasswordSentInTheClear
Please don't put sensitive data as parameters in the query string.