Situation :
The target site (a pre-prod URL, say https://my-pre-prod-site.com/login, for example) is using a self-signed certificate.
From the browser, the site is accessible over https without any issues (the self-signed certificate warning is suppressed by adding the certificate to the trust store in the browser)
Problem Statement :
A simple python script that makes a get call to the target site using requests fails with either of the below errors in different situations :
requests.exceptions.SSLError: [Errno 0] _ssl.c:344: error:00000000:lib(0):func(0):reason(0)
or
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
The simple script used (on the python prompt) is :
import requests
res = requests.get('https://my-pre-prod-site.com/login')
**Things already tried **
I do NOT want to skip the ssl verification. Hence, verify = false is not an option for me.
I have already used the below with the same error
res = requests.get('https://my-pre-prod-site.com/login', verify = os.path.join(os.getcwd(),'test.pem') where test.pem is a pem file created by concatenating the output of the below commands in that order :
openssl rsa -in ~/Desktop/CertPath/private.key -check
and
openssl x509 -pubkey -noout -in ~/Desktop/CertPath/certificate.pem
The script is run from ~/Desktop/CertPath so getcwd() gives the right path to the certificate.
I tried another test.pem file as well where the order of concatenation was reversed. It still throws the same error.
Have tried passing the .pem file holding the public key and the .key file holding the private key, separately (individually) as well, with the same error as the outcome.
Environment details if it helps
OS - ElCapitan Mac
Requests - 2.9.0
Python - 2.7.10
OpenSSL being used by Python - 'OpenSSL 0.9.8zg 14 July 2015'
Note - The openssl version does not seem to be an issue. Because even with an updated version of openssl, the errors are the same - tested on Ubuntu
with Python 2.6 that uses the Openssl 1.x
This question is old but In case someone wonders off here.
You are putting the private key and public key in you test.pem. This is wrong. What verify param requires is certs which it can trust.
res = requests.get('https://my-pre-prod-site.com/login', verify = os.path.join(os.getcwd(),'test.pem')
The test.pem is supposed to contain the list of all the Trusted Certificates. But what you're providing in your test.pem is your public and private key. You're ~/Desktop/CertPath/certificate.pem file itself should go into it.
Try this:
res = requests.get('https://my-pre-prod-site.com/login', verify = '~/Desktop/CertPath/certificate.pem')
In order to specify certificate for SSL verification you can use :
requests.get('https://my-pre-prod-site.com/login', cert=os.path.join(os.getcwd(),'test.pem'))
Related
I'm attempting to build a simple client/server application in C++ using the IXWebsocket library, using the example code as an example, as shown on this page - https://machinezone.github.io/IXWebSocket/usage/
The code works fine when using an unsecured connection (as denoted by a ws:// url), but I can't get it working at all when using a secured connection (as denoted by a wss:// url).
The website states under the "TLS Support and configuration" section that
Then, secure sockets are automatically used when connecting to a wss://* url.
Additional TLS options can be configured by passing a ix::SocketTLSOptions instance to the setTLSOptions on ix::WebSocket (or ix::WebSocketServer or ix::HttpServer)
This implies to me that simply changing the ws:// url to a wss:// url is enough to instruct the application to secure the connection, however this does not work.
When I attempt to connect using a wss:// url, the server returns the following
WebSocketServer::handleConnection() HTTP status: 400 error: Error reading HTTP request line
The website goes on to say that
Additional TLS options can be configured by passing a ix::SocketTLSOptions instance to the setTLSOptions on ix::WebSocket (or ix::WebSocketServer or ix::HttpServer)
and...
Specifying certFile and keyFile configures the certificate that will be used to communicate with TLS peers. On a client, this is only necessary for connecting to servers that require a client certificate. On a server, this is necessary for TLS support.
This implies to me that for the server to support TLS, I must provide a cert file, and a key file.
The github repo includes the script generate_certs.sh which produces a series of certificates in pem format, which should be enough to get things working. Included among them are selfsigned-client-crt.pem and selfsigned-client-key.pem, which seem like obvious candidates, however they specifically state client in the names, which suggests that they should not be used in the server application, rather they belong in the client.
The website also includes the example snippet:
webSocket.setTLSOptions({
.certFile = "path/to/cert/file.pem",
.keyFile = "path/to/key/file.pem",
.caFile = "path/to/trust/bundle/file.pem", // as a file, or in memory buffer in PEM format
.tls = true // required in server mode
});
I have attempted to populate the certFile and keyFile properties, and specified "NONE" for the caFile property as explained in the example, however this results in the server application printing SocketServer::run() tls accept failed: error in handshake : SSL - The connection indicated an EOF to the console.
What's more, the example snippet listed above states "path/to/cert/file.pem" and "path/to/key/file.pem" but doesn't explicitly state whether those should be client, or server usage.
The example doesn't come with a complete runnable implementation, and doesn't explain clearly what is needed to make TLS work in this particular form, and I'm at a bit of a loss now.
There is an example application in the github repo, however it includes a number of different variations, all of which are far more complicated than this trivial example, and it is this trivial example that I need to get working so I can understand how to implement this further.
In my server application, I have implemented the following for the TLS options:
int port = 8443;
ix::WebSocketServer server(port);
ix::SocketTLSOptions tlsOptions;
tlsOptions.certFile = "certs/selfsigned-client-crt.pem";
tlsOptions.keyFile = "certs/selfsigned-client-key.pem";
tlsOptions.caFile = "NONE";
tlsOptions.tls = true; //Required for TLS
server.setTLSOptions(tlsOptions);
I am pretty sure that the issue in in how I've set up the key and cert files. I have used the client files here, but I also tried generating and signing a server cert and key, which also did not work.
I have even tried using the trusted key and cert for both the client and server applications, and still did not get a working TLS connection (the following files were generated by the generate_cert.sh script -
selfsigned-client-crt.pem, selfsigned-client-key.pem, trusted-ca-crt.pem, trusted-ca-key.pem, trusted-client-crt.pem, trusted-client-key.pem, trusted-server-crt.pem, trusted-server-key.pem, untrusted-ca-crt.pem, untrusted-ca-key.pem, untrusted-client-crt.pem, untrusted-client-key.pem
... none of which is a self signed server cert.
What I can gather from the example page is that I need to do the following to get this working.
Generate a server cert and key
Self sign the cert
Specify the cert and key file in the tlsOptions on the server
Set the tls property in tlsOptions to true on the server
Set the caFile property in tlsOptions on the server to "NONE"
Set the url in the client to a wss:// url
But this did not work when I tried it, so there's clearly something I've missed.
All I'm aiming to do for the moment is to use self signed certs so that I can test my client and server, both running on localhost.
If anybody can steer me in the right direction, I'd be immensely grateful. I've been on this for 4 days now and I'm really lost.
Many thanks
Check this file https://github.com/machinezone/IXWebSocket/blob/master/ws/test_ws.sh / it does a full client + server encrypted exchange.
Note that on macOS there are limitations, but on windows or linux, using mbedtls and openssl everything should work fine.
ps: You will need to supply the same set of certs on the client and on the server.
https://machinezone.github.io/IXWebSocket/build/
-DUSE_TLS=1 will enable TLS support
so I do the following :
mkdir build
cd build
cmake -DUSE_TLS=1 -DUSE_WS=1 ..
works for me
I just started using Poco C++ libraries and i just compiled the NetSSL-OpenSSL download example.
When using it with an http site, everything works, but when i use it with an https site, following error occurs:
>download.exe https://api.github.com
WARNING: Certificate verification failed
----------------------------------------
Issuer Name: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV Root CA
Subject Name: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 High Assurance Server CA
The certificate yielded the error: unable to get local issuer certificate
The error occurred in the certificate chain at position 1
Accept the certificate (y,n)?
weirdly the content is still loaded fine after saying yes, but i would really like my program to be clean and also safe regarding internet security.
Here is my code in its current state, but i really don't know what i'm doing, so please point me in the right direction:
// POCO C++ Libraries used
SharedPtr<InvalidCertificateHandler> ptrCert = new ConsoleCertificateHandler(false);
Context::Ptr ptrContext = new Context(Context::CLIENT_USE, "", "cert.pem", "", Context::VerificationMode::VERIFY_NONE);
SSLManager::instance().initializeClient(0, ptrCert, ptrContext);
URI uri(argv[1]);
std::unique_ptr<std::istream> pStr(URIStreamOpener::defaultOpener().open(uri));
StreamCopier::copyStream(*pStr.get(), std::cout);
I generated the cert.pem file with openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes, hoping this would generate a certificate, but i don't know what certificates i need and where to specify them in my c++ program.
My aim is to connect to a Github API page and read a GET Request, any help appreciated...
I just managed to do it, for anyone interested:
Since i only need to access the Github API, i only need to verify the certificate used by github, which is DigiCert.
I visited this page https://www.digicert.com/kb/digicert-root-certificates.htm and downloaded the file DigiCertHighAssuranceEVRootCA.crt.pem, copied it next to my exe and then specified it in my c++ program as such:
SharedPtr<InvalidCertificateHandler> ptrCert = new ConsoleCertificateHandler(false);
Context::Ptr ptrContext = new Context(Context::CLIENT_USE, "", "", "DigiCertHighAssuranceEVRootCA.crt.pem", Context::VerificationMode::VERIFY_RELAXED);
SSLManager::instance().initializeClient(0, ptrCert, ptrContext);
Now i get the answer immediately, without any errors or warnings
I'm looking to scrape data off a website, other https sites work and this was working last week but now fails
<cfhttp url="https://www.cliftoncameras.co.uk/all-brands-and-types-of-used-cameras/"></cfhttp>
If I run a dump of cfhttp
Exception: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target
I have tried running with the latest JRE version 12 - no change
https://helpx.adobe.com/coldfusion/kb/import-certificates-certificate-stores-coldfusion.html
Reverted back to CF original JRE, downloaded the target SSL certificate and installed it using the keytool - no change
c:\ColdFusion2018\jre\bin\keytool -import -keystore cacerts -alias
cliftoncameras -file
c:\ColdFusion2018\jre\lib\security\cliftoncameras.cer
I changed the websocket in the CFAdmin to proxy - no change
I did restart the CF Application Server each time.
What else can I do?
I have also seen this java.security.cert.CertPathBuilderException error before from Java and Coldfusion on sites that load ok in a regular browser, but which still error from cfhttp even after adding the certificate to the CF keystore and restarting.
This happens when the target site server certificate configuration has a trust chain issue - when one or more trust chain paths requires the browser to perform an "extra download" of a certificate. This can be because of a missing intermediate certificate in a single trust chain path, or because there are multiple branches in the trust chain with different fingerprints and one or more certificates from one or more of those branches is not being served.
If you run the target site through an SSL Analyzer like ssllabs.com - eg
https://globalsign.ssllabs.com/analyze.html?d=www.cliftoncameras.co.uk&hideResults=on - you'll see that their intermediate certificate Starfield Secure Certificate Authority - G2 is not being served by their server, which forces the client to do an "extra download" - which won't be a problem for most proper browsers, but the Java client used by cfhttp needs the server to provide pretty much every intermediate and root cert directly. It used to be the same for most mobile OSs up until a few years ago.
So the ideal solution is to contact cliftoncameras and have their server admin install the correct Starfield Intermediate certificate so that it is served correctly.
A possible workaround on your side is to install the Starfield Secure Certificate Authority - G2 intermediate certificate in your CF keystore.
On my development platform I added
-Dcom.sun.security.enableAIAcaIssuers=true
To the java.args in the file in ColdFusion2018\cfusion\bin\jvm.config
Then restarted the CF Application Server, and now my CFHTTP call is successful.
Thanks to #agreax for this solution
Thanks to #sevroberts who's answer was probably the correct one, even though I couldn't get it to work. The production host installed the SSL certificate to the keystore and successfully resolved it this way. They said:
If you use FireFox browser and click on the lock icon when browsing the URL you are wanting to have the cfhttp request access you can then get the more info and click the View Certificate option.
You will need to download the PEM (cert) not the Chain. Once downloaded, you need to run the keytool in order to import it to the keystore.
If you are using the default JRE within your JVM for ColdFusion you will need to install a JDK to your development machine.
You can see the details and steps we have listed on our wiki regarding the commands from the command prompt to import the SSL into the store.
https://wiki.hostek.com/ColdFusion_Tips_%26_Tricks#Fixing_cfhttp_Connection_Failures_with_Keytool
Thanks to #alexbaban his workaround, whilst it worked, it was a solution I could not implement due to requiring the use of the tag cfexecute.
If you can not get the keystore thing working maybe you'll want to try this.
Create a dedicated command line executable (.exe) which will read the web page and save the source to a text file. You can then use ColdFusion to read the file and work with the data.
Here is the ColdFusion code:
<cfscript>
_execPath = "c:/bin/clifton.exe";
_filePath = "c:/bin/clifton.txt";
// run your command-line app (clifton.exe)
cfexecute(name="#_execPath#");
// wait for the file
do {
sleep(100);
} while ( not fileExists(_filePath) )
// wait for write to finish
do {
sleep(100);
_fileInfo = getFileInfo(_filePath);
writeOutput(" ## ");
cfflush();
} while ( _fileInfo.size eq 0 || dateDiff("s", _fileInfo.lastmodified, now()) lte 3 )
writeOutput("<hr />")
_result = fileRead(_filePath);
writeDump(_result);
</cfscript>
As you can see it depends on clifton.exe and reads clifton.txt (clifton.txt is the result of executing clifton.exe).
How to make clifton.exe
You will use the Dart SDK and the dart2native tool to create the executable on your development computer. You can deploy the executable on your production server as a standalone (You don't need the Dart SDK installed on production).
Create a bin folder on your C drive.
From https://ssl-ccp.secureserver.net/repository/?origin=CALLISTO download the certificate sfig2.crt.pem (PEM) and save it inside c:\bin.
Inside c:\bin create a text file clifton.dart with the following code:
// clifton.dart
import 'dart:convert';
import 'dart:io';
main() {
//
const String _certFilePath = 'c:/bin/sfig2.crt.pem';
const String _responseFilePath = 'c:/bin/clifton.txt';
const String _uri =
'https://www.cliftoncameras.co.uk/all-brands-and-types-of-used-cameras/';
final File _file = new File(_responseFilePath);
final IOSink _sink = _file.openWrite();
final SecurityContext _context = new SecurityContext();
_context.setTrustedCertificates(_certFilePath);
final HttpClient _client = new HttpClient(context: _context);
saveSourceToFile(_client, _uri, _sink);
_client.close();
//
}
// get web page source then write it to file
void saveSourceToFile(HttpClient _client, String _uri, IOSink _sink) {
//
_client
.getUrl(Uri.parse(_uri))
.then((req) => req.close())
.then((res) => res.transform(Utf8Decoder()).listen((data) {
// as data is received write to file
_sink.write(data);
}, onDone: () {
_sink.close();
}));
//
}
Download and install the Dart SDK from https://dart.dev/
Open a terminal window and test the installation of Dart with dart --version (you should be able to run dart from any folder, if needed add dart to your PATH)
In a terminal window, change directory to c:\bin with cd c:\bin
Next, run dart2native clifton.dart -o clifton.exe
If compilation goes well you should have inside c:\bin the three files: clifton.dart, clifton.exe and the certificate sfig2.crt.pem.
If you wish you can test run clifton.exe in the terminal window, which should create the clifton.txt file.
Test the ColdFusion page which calls clifton.exe, waits for clifton.txt then outputs the content.
If you deploy in production you need both files clifton.exe and sfig2.crt.pem (the certificate).
Good luck!
I am trying to update a wildcard certificate for EC2 instances on AWS. The service these servers belong to consists of a single server and a set of servers behind AWS ELB.
The certificate has been successfully updated and verified on the single server.
The same is true for an instance pulled up from the image the ELB uses for AutoScaling.
However, when trying to add a new certificate to the load-balancer, I get the above error. I'm sure the certificate is correct and is in PEM format. I first tried via the web console, then using the aws aim command line tools with the same result.
Anyone came across similar issue recently?
The AWS CLI requires file:// prefix for local files. For example file://private.key, file://cert_file, etc.
Just ran into the same exact issue: web console and AWS CLI reporting the same error in not being able to parse the certificate.
The error's root cause turned out to be in the private key -- converting my private key to a "RSA PRIVATE KEY" fixed the issue:
openssl rsa -in server.key -out server.key.rsa
Then, use the server.key.rsa in the private key field and leave the public cert as is.
I just spent the last hour struggling with this issue, on the web console. For sake of documenting, I would like to share what fixed the problem for me:
Ensure all keys and certs to be in RSA (as is demonstrated in Vikram's answer)
Ensure the ---- TEXT HERE ---- start and end labels are included in what you are uploading/copy-pasting into the web-console
My issue was: The RapidSSL certificates I'd purchased on Name.com, when copy-pasted into an Evernote, resulted in the conversion of newlines into spaces. I only realized this when I inputted the text into Vim, and the monospaced text wasn't aligned properly. After a series of f, [space], a, [return], I'd fixed the file in Vi and it now seems to be working. AWS' interface should be smart enough to recover from common errors such as this - nevertheless, this fixed the issue for me.
Hope this helps save an hour for somebody else :)
According to installation steps of ZeroSSL you have to copy all the content of the private.key after opening it in notepad.
The private key must start with -----BEGIN RSA PRIVATE KEY----- and end with -----END RSA PRIVATE KEY-----
I did that also with certificate.crt and the problem solved and saved successfully
You might get this or a similar error if you get ahead of yourself and paste the CSR (Certificate Signing Request) .pem file into the AWS certificate console where you should be pasting a certificate .pem file.
The CSR file typically includes REQUEST in the first line:
-----BEGIN CERTIFICATE REQUEST-----
The certificate file does not:
-----BEGIN CERTIFICATE-----
I am using openssl in c to verify a certificate. Is there any way i can skip the self signed certificate error? I am getting that error for all the sites that has invalid/expired/mismatched url certificates and i am unable to detect any of the other errors.
And I use the function
SSL_CTX_load_verify_locations(ctx,0,CA_LIST)) to load CA_LIST. What does it exactly do?
The error self-signed certificate in certificate chain comes, when the root or self-signed certificate is present in the certificate list sent by the peer, but, the same is not loaded in your Trust Store.
The SSL_CTX_load_verify_locations(ctx,0,CA_LIST)) will try to load the CAs present in the path mentioned in CA_LIST.
The function prototype is int SSL_CTX_load_verify_locations(SSL_CTX *ctx, const char *CAfile, const char *CApath);
The OpenSSL Help Page Says:
"If CApath is not NULL, it points to a directory containing CA certificates in PEM format. The files each contain one CA certificate. The files are looked up by the CA subject name hash value, which must hence be available. If more than one CA certificate with the same name hash value exist, the extension must be different (e.g. 9d66eef0.0, 9d66eef0.1 etc). The search is performed in the ordering of the extension number, regardless of other properties of the certificates. Use the c_rehash utility to create the necessary links.
The certificates in CApath are only looked up when required, e.g. when building the certificate chain or when actually performing the verification of a peer certificate."
You can get more information from the OpenSSL Page here.