I am trying to load CA certificate from memory instead of file. But I keep getting handshake error while connecting. The file loading works perfectly, memory loading fails. What am I missing?
std::ifstream file("message_server_ca.crt");
std::vector<char> fileContents((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());
boost::asio::const_buffer buffer(&fileContents.at(0),fileContents.size());
bool useFile = false; // switch between file and memory loading.
boost::asio::ssl::context ctx(io_service, boost::asio::ssl::context::sslv23);
ctx.set_verify_mode(boost::asio::ssl::context::verify_peer);
if(useFile)
{
// This works perfectly!
ctx.load_verify_file("message_server_ca.crt");
}
else
{
// This fails the handshake (asio.ssl:336134278)
ctx.use_certificate(buffer,boost::asio::ssl::context_base::pem);
}
client c(io_service, ctx, iterator);
io_service.run();
It appears that you want add_certificate_authority():
This function is used to add one trusted certification authority from
a memory buffer.
use_certificate() and use_certificate_file() are for the server or client certificate presented in the handshake, i.e. not the CA used to test those certificates.
These functions (load_verify_file() and add_certificate_authority()) are not consistently named. I guess it is because the memory buffer versions were added relatively recently.
Related
When attempting to securely connect to a remote IMAP server using Boost ASIO, the server handshake fails on every connection. The exception message reads:
handshake: unregistered scheme (STORE routines) [asio.ssl:369098857]
My code is below (url is a std::string_view containing the host URL):
using boost::asio::ip::tcp;
namespace ssl = boost::asio::ssl;
using SSLSocket = ssl::stream<tcp::socket>;
boost::asio::io_context context;
ssl::context ssl_context(ssl::context::tls);
SSLSocket socket(context, ssl_context);
ssl_context.set_default_verify_paths();
tcp::resolver resolver(context);
auto endpoints = resolver.resolve(url, "993");
boost::asio::connect(socket.next_layer(), endpoints);
socket.set_verify_mode(ssl::verify_peer);
socket.set_verify_callback(ssl::host_name_verification(url.data()));
socket.handshake(SSLSocket::client);
The code immediately throws an exception on the final line, which is a blocking synchronous handshake.
The prior two lines set up host name verification, similar to how it's done in the official ASIO tutorial. These checks seem to be causing an issue, however, because when they are removed the handshake succeeds. Obviously, this is not a good solution.
After stepping through some of ASIO's internals, I found that the last three lines of the above snippet could be replaced by:
SSL_set_verify(socket.native_handle(), SSL_VERIFY_PEER, nullptr);
socket.handshake(SSLSocket::client);
and the same error occurs. SSL_set_verify is an OpenSSL function, and the fact that setting a null callback directly causes the same issue makes me think that the issue is with my system's OpenSSL environment and not ASIO or the host name verification callback. However, I have not been able to determine what exactly the error means and what could be causing it.
Here is a list of things I have tried while troubleshooting:
Load the system's certificate (.pem) file explicitly Thinking maybe ASIO and/or OpenSSL's were not able to load the right certificates to do the validation, I found my system's (a Mac) certificate file at /private/etc/ssl/cert.pem. I then inserted the following line:
ssl_context.load_verify_file("/private/etc/ssl/cert.pem");
directly after set_default_verify_paths() is called. My program loads this certificate file without complaining, but it doesn't fix the handshake error.
Use a different version of OpenSSL At first I was using Apple's system version of OpenSSL (which is really LibreSSL 2.8.3). I then rebuilt my code using the Homebrew package manager's version of OpenSSL (OpenSSL 3.0.4). This also did not fix the issue, even when I tried calling load_verify_file as in point 1.
Sanity check using the OpenSSL command-line tool To make sure my network connection and URL/port number were correct, I tried connecting to the IMAP server over SSL using the following command:
openssl s_client -connect my.url.com:993 -crlf -verify 1
and it worked fine, connecting to the IMAP server and enabling me to send/receive IMAP responses.
Has anyone encountered similar issues when using OpenSSL and ASIO? I'm not very familiar with setting up an SSL/TLS connection, but I don't see what could be causing the problem.
Thanks for your help!
Given that openssl s_client -connect my.url.com:993 -crlf -verify 1 succeeds there is not a lot that seems wrong. One thing catches my eye: I'd configure the context before constructing an SSL stream from it:
ssl::context ssl_context(ssl::context::tls);
ssl_context.set_default_verify_paths();
SSLSocket socket(context, ssl_context);
Also, openssl likely uses SNI extensions:
// Set SNI Hostname (many hosts need this to handshake successfully)
if(! SSL_set_tlsext_host_name(socket.native_handle(), hostname.c_str()))
{
throw boost::system::system_error(
::ERR_get_error(), boost::asio::error::get_ssl_category());
}
Finally, make sure the url string view contains correct data, notably that it's a valid hostname and null-terminated string. In this case I'd prefer to use a string representation that guarantees null-termination:
Summary
#include <boost/asio.hpp>
#include <boost/asio/ssl.hpp>
using boost::asio::ip::tcp;
namespace ssl = boost::asio::ssl;
using SSLSocket = ssl::stream<tcp::socket>;
int main() {
boost::asio::io_context context;
ssl::context ssl_context(ssl::context::tls);
ssl_context.set_default_verify_paths();
SSLSocket socket(context, ssl_context);
tcp::resolver r(context);
std::string hostname = "www.example.com";
auto endpoints = r.resolve(hostname, "443");
boost::asio::connect(socket.next_layer(), endpoints);
socket.set_verify_mode(ssl::verify_peer);
socket.set_verify_callback(ssl::host_name_verification(hostname));
// Set SNI Hostname (many hosts need this to handshake successfully)
if(! SSL_set_tlsext_host_name(socket.native_handle(), hostname.c_str()))
{
throw boost::system::system_error(
::ERR_get_error(), boost::asio::error::get_ssl_category());
}
socket.handshake(SSLSocket::client);
}
In my case I manged to make it work with this:
ssl_context.load_verify_file("/etc/ssl/certs/ca-bundle.crt");
so pointing to the ca-bundle file instead of the cert file.
I'm currently running a FileZilla FTP server on a network. My issue is that on seemingly random machines, when the user navigates to a directory (which they are able to do) and attempts to ls (i.e. data transfer) their end hangs waiting for a response, while the server reports this 425: Can't open data connection for transfer mentioned above. This result varies depending on the client machine used, where some (either local or remote) are able to proceed and others stuck here. I understand that this is because simple FTP commands like CWDing operate on the 20/21 ports, whereas FTP data transfer operate on some other port number, which in turn may be blocked by a firewall somewhere along the chain. My question is, how do I account for these varying ports (if this truly is the issue), as as best I know they could be anything above 1024?
My end goal with this project is to implement a very simple FTP solution, ideally using WinINet, however, so far I've run into the same problem:
BOOL CWebFileFinder::FindFile(const CString& URL)
{
CString ServerName;
CString strObject;
INTERNET_PORT nPort;
DWORD dwServiceType = AFX_INET_SERVICE_FTP;
if (AfxParseURL(URL, dwServiceType, ServerName, strObject, nPort))
{
m_Connection = m_Session.GetFtpConnection(ServerName, m_Username, m_Password, nPort/*, true*/); // results in findfile still failing
if (m_Connection)
{
m_Connection->SetCurrentDirectory("sms"); // CDs into this dir
m_Finder = new CFtpFileFind(m_Connection);
if (m_Finder)
{
More = m_Finder->FindFile(_T("*.*")); // hangs here
}
}
}
catch (CException* pEx)
{
CString str;
LPTSTR error = str.GetBuffer(255);
pEx->GetErrorMessage(error, 255);
pEx->Delete();
str.ReleaseBuffer();
}
return More;
}
As far as I can see, either I need to call to open this data port prior to the LIST, or find the firewalls blocking these ports and create a rule to prevent that (What ports does Wininet listen on for Active FTP data connection?). Of course I could also be just completely off-base – Any insights at all would be greatly appreciated!
Your FTP server seems to require an encrypted connection (TLS/SSL).
WinInet does not support encrypted FTP.
See C++/Win32 The basics of FTP security and using SSL.
I'm working on Boost Asio and Boost Beast for simple RESTful server. For normal HTTP and TCP socket, it works perfectly. I put it under load test with JMeter, everything works fine.
I tried to add the SSL socket. I set the 'ssl::context' and also called the 'async_handshake()' - additional steps for SSL compared to normal socket. It works for the first time only. Client can connected with me (server) and I also able to receive the data via 'boost::beast::http::async_read()'.
Because this is RESTful, so the connection will drop after the request & respond. I call 'SSL_Socket.shutdown()' and follow by 'SSL_Socket.lowest_layer().close()' to close the SSL socket.
When the next incoming request, the client able to connect with me (server). I called 'SSL_Socket.async_handshake()' and then follow by 'boost::beast::http::async_read()'. But this time I not able to receive any data. But the connection is successfully established.
Anyone has any clue what i missed?
Thank you very much!
If you want to reuse the stream instance, you need to manipulate SSL_Socket.native_handle() with openssl lib function. After ssl shutdown, use SSL_clear() before start a new ssl handshake.
please read(pay attention to warnings) link for detail
SSL_clear() resets the SSL object to allow for another connection. The reset operation however keeps several settings of the last sessions (some of these settings were made automatically during the last handshake)
.........
WARNINGS
SSL_clear() resets the SSL object to allow for another connection. The reset operation however keeps several settings of the last sessions (some of these settings were made automatically during the last handshake). It only makes sense for a new connection with the exact same peer that shares these settings, and may fail if that peer changes its settings between connections. Use the sequence SSL_get_session(3); SSL_new(3); SSL_set_session(3); SSL_free(3) instead to avoid such failures (or simply SSL_free(3); SSL_new(3) if session reuse is not desired).
In regard to ssl shutdown issue, link explain how boost asio ssl shutdown work.
In Boost.Asio, the shutdown() operation is considered complete upon error or if the party has sent and received a close_notify message.
If you look at boost.asio (1.68) source code boost\asio\ssl\detail\impl\engine.ipp, it shows how boost.asio do ssl shutdown and stream_truncated happens when there is data to be read or ssl shutdown expected from peer not received.
int engine::do_shutdown(void*, std::size_t)
{
int result = ::SSL_shutdown(ssl_);
if (result == 0)
result = ::SSL_shutdown(ssl_);
return result;
}
const boost::system::error_code& engine::map_error_code(
boost::system::error_code& ec) const
......
// If there's data yet to be read, it's an error.
if (BIO_wpending(ext_bio_))
{
ec = boost::asio::ssl::error::stream_truncated;
return ec;
}
......
// Otherwise, the peer should have negotiated a proper shutdown.
if ((::SSL_get_shutdown(ssl_) & SSL_RECEIVED_SHUTDOWN) == 0)
{
ec = boost::asio::ssl::error::stream_truncated;
}
}
Also you can see boost.asio ssl shutdown routine may call openssl SSL_shutdown() twice if first return 0, openssl document allows it but advice call SSL_read() to do a bidirectional shutdown if first SSL_shutdown() returns 0.
Read link for details.
I had a similar issue, the 2nd time onward my asynchonous accept always failed with session id uninitialized.
I solved this problem calling SSL_CTX_set_session_id_context on context or
setting context cache mode with SSL_SESS_CACHE_OFF and SSL_OP_NO_TICKET on context options.
This is my cents to someone else's problem.
I managed to resolve the problem by switching 'ssl::stream' socket to 'boost::optional' and then added 'SSL_Socket.emplace(io_context, oSSLContext)' each time the socket is shutdown and closed.
Big credit to sehe at 'Can't implement boost::asio::ssl::stream<boost::asio::ip::tcp::socket> reconnect to server'. His statement "the purest solution would be to not reuse the stream/socket objects" rocks! Save my time.
Thanks.
I'm trying to implement OpenSSL into my application which uses raw C sockets and the only issue I'm having is the SSL_accept / SSL_connect part of the code which starts the KeyExchange phase but does not seem to complete it on the serverside.
I've had a look at countless websites and Q&A's here on StackOverflow to get myself through the OpenSSL API since this is basically the first time I'm attempting to implement SSL into an application but the only thing I could not find yet was how to properly manage failed handshakes.
Basically, running process A which serves as a server will listen for incoming connections. Once I run process B, which acts as a client, it will successfully connect to process A but SSL_accept (on the server) fails with error code -2 SSL_ERROR_WANT_READ.
According to openssl handshake failed, the problem is "easily" worked around by calling SSL_accept within a loop until it finally returns 1 (It successfully connects and completes the handshake). However, I do not believe that this is the proper way of doing things as it looks like a dirty trick. The reason for why I believe it is a dirty trick is because I tried to run a small application I found on https://www.cs.utah.edu/~swalton/listings/articles/ (ssl_client and ssl_server) and magically, everything works just fine. There are no multiple calls to SSL_accept and the handshake is completed right away.
Here's some code where I'm accepting the SSL connection on the server:
if (SSL_accept(conn.ssl) == -1)
{
fprintf(stderr, "Connection failed.\n");
fprintf(stderr, "SSL State: %s [%d]\n", SSL_state_string_long(conn.ssl), SSL_state(conn.ssl));
ERR_print_errors_fp(stderr);
PrintSSLError(conn.ssl, -1, "SSL_accept");
return -1;
}
else
{
fprintf(stderr, "Connection accepted.\n");
fprintf(stderr, "Server -> Client handshake completed");
}
This is the output of PrintSSLError:
SSL State: SSLv3 read client hello B [8465]
[DEBUG] SSL_accept : Failed with return -1
[DEBUG] SSL_get_error() returned : 2
[DEBUG] Error string : error:00000002:lib(0):func(0):system lib
[DEBUG] ERR_get_error() returned : 0
[DEBUG] errno returned : Resource temporarily unavailable
And here's the client side snippet which connects to the server:
if (SSL_connect(conn.ssl) == -1)
{
fprintf(stderr, "Connection failed.\n");
ERR_print_errors_fp(stderr);
PrintSSLError(conn.ssl, -1, "SSL_connect");
return -1;
}
else
{
fprintf(stderr, "Connection established.\n");
fprintf(stderr, "Client -> Server handshake completed");
PrintSSLInfo(conn.ssl);
}
The connection is successfully enstablished client-side (SSL_connect does not return -1) and PrintSSLInfo outputs:
Connection established.
Cipher: DHE-RSA-AES256-GCM-SHA384
SSL State: SSL negotiation finished successfully [3]
And this is how I wrap the C Socket into SSL:
SSLConnection conn;
conn.fd = fd;
conn.ctx = sslContext;
conn.ssl = SSL_new(conn.ctx);
SSL_set_fd(conn.ssl, conn.fd);
The code snippet here resides within a function that takes a file-descriptor of the accepted incoming connection on the raw socket and the SSL Context to use.
To initialize the SSL Contexts I use TLSv1_2_server_method() and TLSv1_2_client_method(). Yes, I know that this will prevent clients from connecting if they do not support TLS 1.2 but this is exactly what I want. Whoever connects to my application will have to do it through my client anyway.
Either way, what am I doing wrong? I'd like to avoid loops in the authentication phase to avoid possible hang ups/slow downs of the application due to unexpected infinite loops since OpenSSL does not specify how many attempts it might take.
The workaround that worked, but that I'd like to avoid, is this:
while ((accept = SSL_accept(conn.ssl)) != 1)
And inside the while loop I check for the return code stored inside accept.
Things I've tried to workaround the SSL_ERROR_WANT_READ error:
Added usleep(50) inside the while loop (still takes several cycles to complete)
Added SSL_do_handshake(conn.ssl) after SSL_connect and SSL_accept (didn't change anything on the end-result)
Had a look at the code shown on roxlu.com (search on Google for "Using OpenSSL with memory BIOs - Roxlu") to guide me through the handshaking phase but since I'm new to this, and I don't directly use BIOs in my code but simply wrap my native C sockets into SSL, it was kind of confusing. I'm also unable to re-write the Networking part of the application as it'd would be too much work for me right now.
I've done some tests with the openssl command-line as well to troubleshoot the issue but it gives no error. The handshake appears to be successful as no errors such as:
24069864:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:656
appear. Here's the whole output of the command
openssl s_client -connect IP:Port -tls1_2 -prexit -msg
http://pastebin.com/9u1bfuf4
Things to note:
1. I'm using the latest OpenSSL version 1.0.2h
2. Application runs on a Unix system
3. Using self-signed certificates to encrypt the network traffic
Thanks everyone who's going to help me out.
Edit:
I forgot to mention that the sockets are in non-blocking mode since the application serves multiple clients in one-go. Though, client-side they are in blocking mode.
Edit2:
Leaving this here for future reference: jmarshall.com/stuff/handling-nbio-errors-in-openssl.html
You have clarified that the socket question is non-blocking.
Well, that's your answer. Obviously, when the socket is in a non-blocking mode, the handshake cannot be immediately completed. The handshake involves an exchange of protocol packets between the client and the server, with each one having to wait to receive the response from its peer. This works fine when the socket is in its default blocking mode. The library simply read()s and write()s, which blocks and waits until the message gets succesfully read or written. This obviously can't happen when the socket is in the non-blocking mode. Either the read() or write() immediately succeeds, or fails, if there's nothing to read or if the socket's output buffer is full.
The manual pages for SSL_accept() and SSL-connect() explain the procedure you must implement to execute the SSL handshake when the underlying socket is in a non-blocking mode. Rather than repeating the whole thing here, you should read the manual pages yourself. The capsule summary is to use SSL_get_error() to determine if the handshake actually failed, or if the library wants to read or write to/from the socket; and in that eventuality call poll() or select(), accordingly, then call SSL_accept() and SSL_connect() again.
Any other approach, like sprinkling silly sleep() calls, here and there, will result in an unreliable house of cards, that will fail randomly.
I have some legacy code that uses OpenSSL for communication. Just like any other session it does a handshake using the SSL functions and then encrypted communication over TCP. We recently changed our code to use IO completion ports. The way it works is contrary to that of OpenSSL. Basically, I'm having a hard time migrating our secure communication code from full OpenSSL usage to IOCP sockets and OpenSSL encryption.
Does anyone have/anyone know of any references that might help me with such a task?
How would TLS handshaking work over IOCP?
In order to use OpenSSL for encryption, but do your own socket IO, what you basically do is create a memory BIO, that you read and write socket data into as that becomes available, and attach that to the SSL context.
Each time you do a SSL_write call, you follow up with a call to the memory BIO to see if it has data in its read buffer, read that out and send it.
Conversely, when data arrives on the socket via your io completion port mechanism, you write it to the BIO and call SSL_read to read the data out. SSL_read might return an error code indicating its in a handshake, which usually means its generated more data to write - which you handle by reading the memory BIO again.
To create my SSL session, I do this:
// This creates a SSL session, and an in, and an out, memory bio and
// attaches them to the ssl session.
SSL* conn = SSL_new(ctx);
BIO* bioIn = BIO_new(BIO_s_mem());
BIO* bioOut = BIO_new(BIO_s_mem());
SSL_set_bio(conn,bioIn,bioOut);
// This tells the ssl session to start the negotiation.
SSL_set_connect_state(conn);
As I receive data from the network layer:
// buf contains len bytes read from the socket.
BIO_write(bioIn,buf,len);
SendPendingHandshakeData();
TryResendBufferedData(); // see below
int cbPlainText;
while( cbPlainText = SSL_read(ssl,&plaintext,sizeof(plaintext)) >0)
{
// Send the decoded data to the application
ProcessPlaintext(plaintext,cbPlaintext);
}
As I receive data from the application to send - you need to be prepared for SSL_write to fail because a handshake is in progress, in which case you buffer the data, and try and send it again in the future after receiving some data.
if( SSL_write(conn,buf,len) < 0)
{
StoreDataForSendingLater(buf,len);
}
SendPendingHandshakeData();
And SendPendingHandshakeData sends any data (handshake or ciphertext) that SSL needs to send.
while(cbPending = BIO_ctrl_pending(bioOut))
{
int len = BIO_read(bioOut,buf,sizeof(buf));
SendDataViaSocket(buf,len); // you fill this in here.
}
Thats the process in a nutshell. The code samples arn't complete as I had to extract them from a much larger library, but I believe they are sufficient to get one started with this use of SSL. In real code, when SSL_read/write / BIO_read/write fail, its probably better to call SSL_get_error and decide what to do based on the result: SSL_ERROR_WANT_READ is the important one and means that you could not SSL_write any more data, as it needs you to read and send the pending data in the bioOut BIO first.
You should look into Boost.Asio