We want to replicate our on-site MySQL database to AWS. Needless to say we need a secure connection, and setting that up has proven to be surprisingly difficult. The AWS Data Migration Service looked like the right thing, and it has the option to import a certificate.
After trial and error I discovered that AWS needed the .pem extension to even try to upload the file. According to the documentation, it expects a chained certificate file, which I have. But I get a validation error every time, with absolutely no helpful details as to what is wrong. I ensured that my private key is only 2048 bits long, as AWS seems to have trouble with larger ones. I have run the certificate through various online tools and they say it is OK. It seems that the first part of the certificate is the problem, Data Migration is happy to import the second part all by itself. Also worth noting, AWS Certificate Manager was willing to take the entire thing (though there it forces you to split the file). The certificate was generated by Let's Encrypt/Zero SSL with my CSR. I have confirmed there are no extra whitespaces in uploading the file.
-----BEGIN CERTIFICATE-----
MIIFBTCCA+2gAwIBAgISA9w/KQIPfQ7G1qbtWy6eruVwMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzEwMTUxMjU2MzZaFw0x
ODAxMTMxMjU2MzZaMBwxGjAYBgNVBAMTEWVxLnNlbnRyeWxpbmsuY29tMIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4GXdQB1u967ZpUuSpaBpzVWFsIXk
YYvDVbX+1DoygaIhqAAoL+s8RgZf8jz49tbFlBc06eXhDH9qL47ZcdLUahY3TY0G
Aksl1uUpxivt7Am3WvzoeTuCO8vhObVNpVLLcyKQ7H543jswLehhSgcPKiSF3ffw
qHJMst4bw4bmzHeTp6ZX83xek8YbXE48PUktpE4sGxwHbQVTuWLDCmMJZr/Pwz6i
fpbkxoUhv4jzlwsAtyPmRIa/XTYhGhnRuPD5m1ZX2LkAuKCH4crYuXPp+F+lMc1R
K0DGoQYk0QjP2nuLmqmJPByHRaTBMb+UwvJn1Ady7qyyS+3nIzG87fzvmwIDAQAB
o4ICETCCAg0wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggr
BgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQFXP07WkX+3XSghpNPYr0W
3Q6LBjAfBgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggrBgEFBQcB
AQRjMGEwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRzZW5jcnlw
dC5vcmcwLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5jcnlw
dC5vcmcvMBwGA1UdEQQVMBOCEWVxLnNlbnRyeWxpbmsuY29tMIH+BgNVHSAEgfYw
gfMwCAYGZ4EMAQIBMIHmBgsrBgEEAYLfEwEBATCB1jAmBggrBgEFBQcCARYaaHR0
cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwgasGCCsGAQUFBwICMIGeDIGbVGhpcyBD
ZXJ0aWZpY2F0ZSBtYXkgb25seSBiZSByZWxpZWQgdXBvbiBieSBSZWx5aW5nIFBh
cnRpZXMgYW5kIG9ubHkgaW4gYWNjb3JkYW5jZSB3aXRoIHRoZSBDZXJ0aWZpY2F0
ZSBQb2xpY3kgZm91bmQgYXQgaHR0cHM6Ly9sZXRzZW5jcnlwdC5vcmcvcmVwb3Np
dG9yeS8wDQYJKoZIhvcNAQELBQADggEBAFxM5i3VlCpyrK0/Dnw5vtkV5TG+o/fJ
TG8elTQD8NDEuuZUee0u0jdcvP3CSGkRJo7tF1lBih8ns7Dhu2wxouM9r+3nP+7F
CYRF1BRAzj6gsTPpHX1XOv98Nq+s8NXQNFe+WxPlYtUQ4ZoJ+gVcNpm8zQY1GaMA
vb6osuru0WoOA3YeNiuRUSvMFnUMt0hO9DuknUdbbr/i9OphOz6xiWCLFPUtNNos
79yoanGZs9Kt40Ou4yhW1gZLHJfp461r0bzoh848f3+R2fwVaBUGBEYLxPNBCu7U
CK0Iualw5hRhh6620f79Lv2Z2FNrPq5kMIySaLpDWgaj5pQjXAjAVXM=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
UPDATE: The answer, which was to focus on the intermediate certificate, is correct. Using just the intermediate was enough to get a data load completed. However, to get data load with continuing replication, you will indeed have to add in the root certificate from your certificate authority as well. Convert it to PEM format and add that to the end of the intermediate bundle, and give the resulting file to AWS.
Data Migration is happy to import the second part all by itself.
The second part is what DMS needs, not the first.
To assign a certificate to an endpoint, you provide the root certificate or the chain of intermediate CA certificates leading up to the root (as a certificate bundle), that was used to sign the server SSL certificate that is deployed on your endpoint.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Security.SSL.html#CHAP_Security.SSL.Limitations
DMS wants to be able to validate the cert on your endpoint, so it only wants the cert of the authority that signed it -- and apparently not the actual endpoint cert itself.
The "validation failed" is probably related to the fact that your first cert is your endpoint cert, which is not a CA cert.
If you check your certs with openssl x509 you note that the first file isn't a CA file...
X509v3 Basic Constraints: critical
CA:FALSE
...but the second one is.
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
I suspect that is all you need.
However, what you have here is an intermediate. If you want to build the full chain, then you need to append the root. According to Let's Encrypt, their Intermediate CA that signed your cert was in turned signed by IdenTrust DST Root CA X3.
This checks -- confirmed out by comparing the Authority Key Identifier of Let's Encrypt Authority X3 to the Subject Key Identifier of DST Root CA X3 (again, using openssl x509).
So, remove your first cert, and upload only the second.
If this isn't sufficient, add the body of the IdentTrust DST Root CA X3 certificate to the end of your file. It's at the URL linked above, and also pasted below:
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
Related
A little new to windows programming/C++. I'm trying to install a .p7b root certificate file to the Trusted Root Certificate Store. I want to use the Windows Wincrypt library. Specifically, these are the suggested steps that I got from an old forum:
Call CertCreateCertificateContext using your certificate content bytes
in order to obtain a PCCERT_CONTEXT
Call CertOpenSystemStore with szSubsystemProtocol set to "ROOT" in
order to obtain a HCERTSTORE
Call CertAddCertificateContextToStore using the above HCERTSTORE and
PCCERT_CONTEXT.
[Here's] the api documentation for CertCreateCertificateContext. Not sure how to just point pbCertEncoded to my actual cert file. Should I just point it to the path? Do I have to load the cert in? What should the type be?
From Simon Rozman's answer in this post: We have to use CertOpenStore() instead of
CertCreateCertificateContext(), which supports one certificate only, whereas PKCS #7 file can contain many.
After the certificate store is open, you can use CertEnumCertificatesInStore() to retrieve certificate context of individual certificates from store.
So from my original steps to successfully install a p7b into the root store:
Call CertOpenStore() for the root store and for the actual certificate itself. This will give you two HCERTSTORE handles.
Have a while loop that will add the certificate contexts to the opened root store (using CertAddCertificateContextToStore()) as long as the certificate context exists (check using CertEnumCertificatesInStore() on the opened certificate store).
I have a leaf certificate installed on my machine, which was issued by a Certificate Authority (CA); this CA is not present on the system.
If I am given the thumbprint (i.e. the SHA-1 hash) of the CA, can I look up and thereby retrieve the installed leaf signers issued by this CA? If I am able to, then what are the required Windows function calls for accomplishing this?
I have been examining a leaf certificate and I only see the standard string representation name of the CA and not a thumbprint. This string name is not unique, hence why I am utilizing the thumbprint (SHA-1’s poor security here is not a problem as it is only used for looking up a proper signer). Microsoft has the CertGetIssuerCertificateFromStore function, but this requires having the CA in memory.
No, it is not possible to look up an issued leaf signer using only the issuer's CA's thumbprint. You first have to install the CA on the machine and then use the entirety of the CA to find its issued leaf signers. The CA's thumbprint is not stored in the leaf signers it issues from what it appears.
All in all, the first operation that needs to be done is installing the CA on the machine you want to find its issued leaf signers on.
I want to set up my local server to communicate with my client. They build TLS connection using Openssl. I try to implement double side authentication, like server would verify client and client also needs to verify server.
When I use certificates generated by my self, everything works fine. The code is as following. It's C++ code in client. I set up client cert, private key and intermediate cert. In server side I saved a CA cert.
The relationship is: CA signs intermediate cert, intermediate cert signs client cert.
As we know, the reason that we need to provide client private key is the client will signature a "challenge" then send to server. Server could get client public key by certificate chain and use it to decode the encrypt "challenge" to see if they matched. You could see this link for detailed process:
https://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_handshake
However in my scenario, I have no permission to get the private key. I only have an API to call, which takes the digest or anything we want to encode as input and return a string encoded by client private key.
Therefore I'm not able to pass any "ClientPrivateKeyFileTest" to TLS.
I searched openssl source code but all handshakes were done in this function: SSL_do_handshake() and I'm not allowed to modify this function.
// load client-side cert and key
SSL_CTX_use_certificate_file(m_ctx, ClientCertificateFileTest, SSL_FILETYPE_PEM);
SSL_CTX_use_PrivateKey_file(m_ctx, ClientPrivateKeyFileTest, SSL_FILETYPE_PEM);
// load intermediate cert
X509* chaincert = X509_new();
BIO* bio_cert = BIO_new_file(SignerCertificateFileTest, "rb");
PEM_read_bio_X509(bio_cert, &chaincert, NULL, NULL);
SSL_CTX_add1_chain_cert(m_ctx, chaincert)
m_ssl = SSL_new(m_ctx);
// get_seocket is my own API
m_sock = get_socket();
SSL_set_fd(m_ssl, m_sock)
// doing handshake and build connection
auto r = SSL_connect(m_ssl);
I think all handshake processes would be done after I call SSL_connect(). So I wonder is there other way I can do to complete the client-authentication?
For example, I could skip adding private key step but set up a callback function somewhere which can handle all cases when SSL needs to use private key to calculate something.
PS: The API is a black box in the client machine.
One more thing, these days I found that openssl engine may help this problem. But does anybody know what kind of engine is useful for this problem? The EC sign, verification or others?
Final update: I implemented a OpenSSL engine to reload EC_KEY_METHOD so that I'm able to use my own sign function.
Thanks a lot!
I am new to python and still learning it so my question can be little naive. Please bear with it ;)
The problem is client will be sending CSR and I want to sign it with my CA root certificate and return the signed certificate back to client.
I have been using this command to do it using command line
openssl x509 -req -in device.csr -CA root.pem -CAkey root.key -CAcreateserial -out device.crt -days 500
same thing I want achieve using python. I have come across python library for openssl pyopenssl
is it possible using this library ? How ? or shoudl I go for M2Crypto ?
You can indeed go with pyOpenSSL. As you are saying you already have CA root certificate and a private key, and CSR will be sent by a client then you can use functions of crypto to read all those ( CA cert, private key and Device CSR ) from file or manage to have them in buffer.
Use below functions to start with. Check dir(crypto) and crypto.function_name.__doc__on python interpreter for more info :) You need to import crypto from pyOpenSSL
crypto.load_certificate_request() - to get device CSR obj
crypto.load_privatekey() - to get private key obj for CA private key
crypto.load_certificate() - to get CA root certificate
then you can write simple funcation to return certificate
def create_cert():
cert = crypto.X509()
cert.set_serial_number(serial_no)
cert.gmtime_adj_notBefore(notBeforeVal)
cert.gmtime_adj_notAfter(notAfterVal)
cert.set_issuer(caCert.get_subject())
cert.set_subject(deviceCsr.get_subject())
cert.set_pubkey(deviceCsr.get_pubkey())
cert.sign(CAprivatekey, digest)
return cert
where caCert , deviceCsr and CAprivatekey are values from above three funcations.
Now that you have certificate with you, you can write this to a file using crypto.dump_certificate(crypto.FILETYPE_PEM, cert) with file name of your choice.
You can modify this function as per your requirement. After this you can verify generated device certificate with CA root certificate with openssl command e.g. openssl verify -CApath <CA cert path> <name of device cert file>
You can also go through few examples from github.
M2Crypto Example , pyOpenSSL example
Hope this gives you idea about the implementation
The maintainer of pyOpenSSL recommends to use cryptography module for X509 manipulation (see note on top of the documentation page: https://www.pyopenssl.org/en/stable/api/crypto.html).
Here is the code to create a certificate from a CSR signed by a CA:
def sign_certificate_request(csr_cert, ca_cert, private_ca_key):
cert = x509.CertificateBuilder().subject_name(
csr_cert.subject
).issuer_name(
ca_cert.subject
).public_key(
csr_cert.public_key()
).serial_number(
x509.random_serial_number()
).not_valid_before(
datetime.utcnow()
).not_valid_after(
# Our certificate will be valid for 10 days
datetime.utcnow() + timedelta(days=10)
# Sign our certificate with our private key
).sign(private_ca_key, hashes.SHA256())
# return DER certificate
return cert.public_bytes(serialization.Encoding.DER)
csr_cert is the cryptography CSR certificate object - can be loaded from a file with x509.load_der_x509_csr()
ca_cert is the cryptography certificate object - can be loaded from a file with x509.load_pem_x509_certificate()
private_ca_key is the cryptography private key object - can be loaded from a file with serialization.load_pem_private_key()
I'm trying to use SSLSniff's tool, and I have some technical issues... I've been looking for any similar problems, but the only results are from Twitter feeds, with no public useful answer. So, here it is:
(My version of SSLSniff is 0.8) I'm launching sslsniff with args:
sslsniff -a -c cert_and_key.pem -s 12345 -w out.log
where: cert_and_key.pem file is my authority's certificate concatenate with my unencrypted private key (in PEM format of course), and 12345 is the port where I redirect traffic with my iptables rule.
So sslsniff is correctly running:
INFO sslsniff : Certificate ready: [...]
[And anytime I connect with a client, there are these 2 following lines:]
DEBUG sslsniff : SSL Accept Failed!
DEBUG sslsniff : Got exception: Error with SSL connection.
On my client' side, I've register my AC as a trusted CA (with FF). Then when I connect through SSL I'm having the error:
Secure Connection Failed.
Error code: ssl_error_bad_cert_domain
What is super strange (moreover the fact that the certificate is not automatically accepted since it should be signed by my trusted CA) is that I cannot accept the forged certificate by clicking on "Add exception..." : I am always returning to the error page asking me to add an(other) exception...
Moreover, when I try to connect to for example: https://www.google.com, SSLSniff's log is completed with a new line :
DEBUG sslsniff : Encoded Length: 7064 too big for session cache, skipping...
Does anyone know what I'm doing wrong?
-- Edit to summer up the different answers --
The problem is that SSLSniff is not taking care of alternive names when it forges certificates. Apparently, Firefox refuses any certificate as soon as the Common Name doesn't match exactly the domain name.
For example, for Google.com : CN = www.google.com and there is no alternative name. So when you connect to https://www.google.com, it's working fine.
But for Google.fr : CN = *.google.fr, with these alternative names: *.google.fr and google.fr. So when you connect to https://www.google.fr, FF is looking for alternative names and, since it obviously doesn't find any, refuses the malformed certificate.
... So a solution would be to patch/commit... I don't know if Moxie Marlinspike has intentionally forgot this functionnality because it was too complicated, or if he was just not aware of this issue. Anyway, I'll try to have a look at the code.
The session encoded length error message: When caching the SSL session fails, it means that SSL session resumption on subsequent connections will fail, resulting in degraded performance, because a full SSL handshake needs to be done on every request. However, despite using the CPU more heavily, sslsniff will still work fine. The caching fails because the serialized representation of the OpenSSL session object (SSL_SESSION) was larger than the maximum size supported by sslsniff's session cache.
As for your real problem, note that sslsniff does not support X.509v3 subjectAltNames, so if you are connecting to a site whose hostname does not match the subject common name of the certificate, but instead matches only a subjectAltName, then sslsniff will generate a forged certificate without subjectAltNames, which will cause a hostname verification mismatch on the connecting client.
If your problem happens only for some specific sites, let us know the site so we can examine the server certificate using e.g. openssl s_client -connect host:port -showcerts and openssl x509 -in servercert.pem -text. If it happens for all sites, then the above is not the explanation.
Try a straight MITM with a cert you fully control , and make sure you don't have some OCSP/Perspectives/Convergance stuff meddling with things. Other than that, maybe add the cert to the OS trusted roots. I think FF on windows uses the windows cert store (start->run->certmgr.msc). It may also be worth trying with something like Burp to see if the error is localized to SSLSniff or all MITM attempts.