I'm programming some kind of browser in c++ using winsock and got struck in some kind of error. The program works just fine for some pages in the internet, but for some reason it doesn't work for all of them.
https://imageshack.com/a/v56q/1
As seen in these images the composed version of Fiddler works while my own program fails.
I thought that maybe the HTTPS connection might be the problem, but it doesn't even seem to need a handshake or something similar. The sending part of the program is certainly not the problem because it works with other pages (e.g. the ones in the comments).
Thanks in advance!
I thought that maybe the HTTPS connection might be the problem, but it doesn't even seem to need a handshake or something similar.
Yes, HTTPS requires that you perform a rather complicated handshake, which is why essentially nobody attempts to implement HTTPS directly on sockets, and instead uses WinHTTP or WinINET, the two HTTP stacks included in Windows.
While Fiddler is running, you can kinda "cheat" by sending your plaintext request (containing a HTTPS url) to Fiddler, which will then perform the proper secure handshake with the remote server on your behalf and then return the response to your client. (Fiddler is acting as a HTTP-to-HTTPS gateway in this scenario.
Related
For educational purposes I am trying to make a web api in c++. the web api needs to be able to listen for http requests(GET, POST etc.), when it receives a http request it needs to be able to send data back to the client. Because it is for educational purposes I would like to do it without unnecessary libraries. Now the first thing I need to do is make the api able to receive requests and respond on that, after some research on google I found out that winsock is probably the most basic way to setup sockets for windows but I could find very little on receiving http requests.
My question is: Is it possible with winsock to receive a http request from the browser, and send data back to the browser?.
My question is: Is it possible with winsock to receive a http request from the browser, and send data back to the browser?
Yes. ^^
It is, Because HTTP is a protocol that (usually) uses TCP as the underlying transportation protocol.
But trying to build a real HTTP layer on top of a simple win32 socket is a bit too much even for an experienced C++ developer.
Many un-experienced C++ developers would probably dismiss this task as a "well, you just need to read some data, parse the headers, assemble your own HTTP response and send it back".
but..
You will have to support
TLS, with all the nasty private keys/public keys implementation
Redirection
Chunked Transfer
G-Zip transfer
and the list goes on and on..
So practically speaking, if you just want to to accept a socket, read some data and send some basic HTTP response than yes. If you want a reliable, professional HTTP library - probably no.
You can check this page https://github.com/ReneNyffenegger/cpp-webserver to see simple winsock server implementation for HTTP. Web server implementation is not so difficult. Of course you should have time for it.
I'm attempting to write a simple HTTP/HTTPS proxy using Boost ASIO. HTTP is working fine, but I'm having some issues with HTTPS. For the record this is a local proxy. Anyway so here is an example of how a transaction works with my setup.
Browser asks for Google.com
I lie to the browser and tell it to go to 127.0.0.1:443
Browser socket connects to my local server on 443I attempt to read the headers so I can do a real host lookup and open a second upstream socket so I can simply forward out the requests.
This is where things fail immediately. When I try to print out the headers of the incoming socket, it appears that they are already encrypted by the browser making the request. I thought at first that perhaps the jumbled console output was just that the headers were compressed, but after some thorough testing this is not the case.
So I'm wondering if anyone can point me in the right direction, perhaps to some reading material where I can better understand what is happening here. Why are the headers immediately encrypted before the connection to the "server" (my proxy) even completes and has a chance to communicate with the client? Is it a temp key? Do I need to ignore the initial headers and send some command back telling the client what temporary key to use or not to compress/encrypt at all? Thanks so much in advance for any help, I've been stuck on this for a while.
HTTPS passes all HTTP traffic, headers and all, over a secure SSL connection. This is by design to prevent exactly what you're trying to do which is essentially a man-in-the-middle attack. In order to succeed, you'll have to come up with a way to defeat SSL security.
One way to do this is to provide an SSL certificate that the browser will accept. There are a couple common reasons the browser complains about a certificate: (1) the certificate is not signed by an authority that the browser trusts and (2) the certificate common name (CN) does not match the URL host.
As long as you control the browser environment then (1) is easily fixed by creating your own certificate authority (CA) and installing its certificate as trusted in your operating system and/or browser. Then in your proxy you supply a certificate signed by your CA. You're basically telling the browser that it's okay to trust certificates that your proxy provides.
(2) will be more difficult because you have to supply the certificate with the correct CN before you can read the HTTP headers to determine the host the browser was trying to reach. Furthermore, unless you already know the hosts that might be requested you will have to generate (and sign) a matching certificate dynamically. Perhaps you could use a pool of IP addresses for your proxy and coordinate with your spoofing DNS service so that you know which certificate should be presented on which connection.
Generally HTTPS proxies are not a good idea. I would discourage it because you'll really be working against the grain of browser security.
I liked this book as a SSL/TLS reference. You can use a tool like OpenSSL to create and sign your own certificates.
I want to upload files from a client location to a server. At present, I have a client-server socket program by which I could send/receive files across, but I would like to improvise it.
My idea would be to transfer the file using HTTP PUT/POST from client (most of the coding on client side) to the server. Since I have no idea about HTTP programming, so I need some guidance on how to achieve that. I want to use C++ with BSD sockets in doing that, and no other libraries. My aim is to send the server a form, like as given below with a HTTP POST/PUT request, and get the file "main.cpp" uploaded to the server.
PUT http://localhost/ HTTP/1.0
Host: localhost
Content-type: form-data
Content-length: 90
FileUpload: /Users/SG/files/main.cpp
I was able to write a dummy program that does some PUT from a client, and the web server running Apache returns a HTTP 200. What I am failing to understand currently would be the following two things, which I guess are somewhat connected:
How one could specify a file to be uploaded from the client in the form above?
If I understand correctly, the file would be read at client site and then the file content would be sent to the server, where a server side script would read the bytes from client and create a copy of the file at the server. Is it correct?
Any help is appreciated.
Thanks,
Sayan
As stated it's not possible. C++ alone has no sockets API. (Edit: with the addition of a BSD sockets API, it's now possible).
Your implementation might provide an OS-specific sockets API, in which case your question boils down to, "how do I write an HTTP client?". To which the answer is (a) don't, or (b) read the HTTP specification very carefully and do as it tells you. The basic steps are:
(Possibly) parse a URL and use gethostbyname to get an IP address.
Open a TCP socket, using connect.
Write the request (you already have an idea what that looks like) using send.
Read the response using read.
Close the connection.
The difficult part is parsing the response robustly, because there are a lot of features to worry about. If your client is tied to a particular server, then its responses will be quite predictable, and you can ignore quite a lot of the HTTP spec (at least until you change the configuration of the server, update its software, etc).
If you're ready to give up before you finish, there are perfectly good HTTP libraries available, such as libcURL.
I want to use C++ for doing this,
without the help of any libraries or
system() calls.
Socket programming always requires system calls. C++ is an extremely general language. The ISO language definition does not specify anything about sockets or network programming, so in order to do this you need to rely on another system-specific library/standard, such as the BSD socket API on UNIX/Linux. Usually, in C++ you would use a higher-level third-party library that wraps these lower level system calls, such as Boost.ASIO. However, even that takes a bit of learning, to say nothing of correctly implementing the HTTP standard itself on top of your sockets.
I was able to attain what I wanted. My http post request string looks like below:
"POST %s HTTP/1.0\r\n"
"Host: %s\r\n"
"Content-type: multipart/form-data\r\n"
"Content-length: %d\r\n\r\n"
"Content-Disposition: %s; filename: %s\n"
I am sending the above string from a cpp program in client to the server, running Apache, which handles the data through a cgi script, also written in cpp. The data is placed with the Content Disposition attribute. Can anybody please point it out to me as to how would I transform the above to a HTTP PUT?
I hate CURL it is too bulky with too many dependencies when all I need to do is quickly open a URL. I don't even need to retrieve the contents of the web page, I just need to make the GET HTTP request to the server.
What's the most minimal way I can do this and don't say CURL !##$
There are lots of choices! Try libwww -- but be warned, most people strongly prefer libcurl. It's much easier to use.
There's a very light way and I've done this myself when implementing a high-scale back end service for a large media provider in the UK.
This method is extremely operating-system specific.
open a TCP socket to the HTTP server
send a "GET /path/to/url HTTP/1.1\r\nHost: www.host.com\r\n\r\n" (the Host header is required for HTTP/1.1 and most virtual servers, don't forget the two blank lines, and a newline requires a carriage return as well for HTTP headers)
wait for the response
close the socket
If you are going to close the socket at the end of the connection you may also want to send a Connection: close\r\n as part of your headers to inform the web server that you will terminate the connection after retrieving the web page.
You may run into trouble if you're fetching an encoded or generated web page in which case you'll have to add logic to interpret the fetched data.
On Windows (Windows XP, Windows 2000 Professional with SP3 and above) you could use WinHttpReadData API. There's also an example at the bottom of that page.
More info on Windows HTTP Services on MSDN
I have used Winsock when I need as few dependencies as possible and it has worked well. You need to write more code than using a separate library or the Microsoft WinHTTP library.
The functions you need are WSAStartup, socket, connect, send, recv, closesocket and WSACleanup.
See sample code for the send function.
system("wget -q -O file.htm http://url.com");
I'm developing a client/server app that will communicate via rest. Some custom request data will be stored in the header of the request. Both the server sending the request and the receiving server have an SSL certificate - will the headers be encrypted, or just the content?
SSL encrypts the entire communications path from the client to the server and back, so yes - the headers will be encrypted.
By the way, if you develop networked applications and care about data security, the least you should do is read a book like Practical Cryptography, by Niels Ferguson and Bruce Schneier, and probably further reading that's more focused on web application security would be a good idea. If I may make an observation - and please, I don't mean that as a personal criticism - your question indicates a fundamental lack of understanding of very basic web security technologies, and that's never a good sign.
Also, it's never a bad idea to confirm that data which is assumed to be encrypted is indeed encrypted. You can use a network analyzer to monitor traffic on the wire and watch out for anything sensitive being sent in the clear. I've used Wireshark to do this before - the results can be surprising, sometimes.
As long as you're communicating in the SSL tunnel, everything sent between the server and the client will be encrypted. The encryption is done before any data is sent or received.
Both headers and content are encrypted.
You appear to think that REST is a distinct protocol.
REST is not a protocol. It is a design style for HTTP-based applications.
So, your a writing an HTTP application. Are the headers encrypted? Yes, if you are using the HTTPS (HTTP over SSL) protocol instead of plain HTTP.
Having certificates on both sides is not directly relevant to your question. SSL certificates are used for authentication. They help in detecting man-in-the-middle attacks such as are possible using DNS cache poisoning.
Having a certificate is not enough, you have to configure the web server to encrypt the connections (that is, to use the certificate) for that domain or virtual host. In addition, I think you would just need a single certificate, responses to requests will still be encrypted.
And yes, HTTP headers are encrypted as well as the data.
The other answers are correct that headers are indeed encrypted, along with the body, when using SSL. But keep in mind that the URL, which can include query parameters, is never encrypted. So be careful to never put any sensitive information in URL query parameters.
Update: as #blowdart pointed out below, this is wrong. See the comment below.
SSL..or rather HTTPS (HTTP over SSL) sends all HTTP content over SSL, and as HTTP content and headers are in fact the same thing, this means the headers are encrypted as well.
Seeing as GET and POST data is sent via HTTP headers, then it only makes sense then when sending data securely you wouldn't just want the response code or content to be encrypted.
Not everything is encrypted: the request query string is not encrypted. Believe me, I've seen requests like this:
https://mydomain.com/authenticate?user=username&password=MyStrongPasswordSentInTheClear
Please don't put sensitive data as parameters in the query string.