CFHTTP unable to find valid certification path to requested target - coldfusion

Question

I'm looking to scrape data off a website, other https sites work and this was working last week but now fails
<cfhttp url="https://www.cliftoncameras.co.uk/all-brands-and-types-of-used-cameras/"></cfhttp>
If I run a dump of cfhttp
Exception: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target
I have tried running with the latest JRE version 12 - no change
https://helpx.adobe.com/coldfusion/kb/import-certificates-certificate-stores-coldfusion.html
Reverted back to CF original JRE, downloaded the target SSL certificate and installed it using the keytool - no change
c:\ColdFusion2018\jre\bin\keytool -import -keystore cacerts -alias
cliftoncameras -file
c:\ColdFusion2018\jre\lib\security\cliftoncameras.cer
I changed the websocket in the CFAdmin to proxy - no change
I did restart the CF Application Server each time.
What else can I do?

Answer 1

I have also seen this java.security.cert.CertPathBuilderException error before from Java and Coldfusion on sites that load ok in a regular browser, but which still error from cfhttp even after adding the certificate to the CF keystore and restarting.
This happens when the target site server certificate configuration has a trust chain issue - when one or more trust chain paths requires the browser to perform an "extra download" of a certificate. This can be because of a missing intermediate certificate in a single trust chain path, or because there are multiple branches in the trust chain with different fingerprints and one or more certificates from one or more of those branches is not being served.
If you run the target site through an SSL Analyzer like ssllabs.com - eg
https://globalsign.ssllabs.com/analyze.html?d=www.cliftoncameras.co.uk&hideResults=on - you'll see that their intermediate certificate Starfield Secure Certificate Authority - G2 is not being served by their server, which forces the client to do an "extra download" - which won't be a problem for most proper browsers, but the Java client used by cfhttp needs the server to provide pretty much every intermediate and root cert directly. It used to be the same for most mobile OSs up until a few years ago.
So the ideal solution is to contact cliftoncameras and have their server admin install the correct Starfield Intermediate certificate so that it is served correctly.
A possible workaround on your side is to install the Starfield Secure Certificate Authority - G2 intermediate certificate in your CF keystore.

Answer 2

On my development platform I added
-Dcom.sun.security.enableAIAcaIssuers=true
To the java.args in the file in ColdFusion2018\cfusion\bin\jvm.config
Then restarted the CF Application Server, and now my CFHTTP call is successful.
Thanks to #agreax for this solution
Thanks to #sevroberts who's answer was probably the correct one, even though I couldn't get it to work. The production host installed the SSL certificate to the keystore and successfully resolved it this way. They said:
If you use FireFox browser and click on the lock icon when browsing the URL you are wanting to have the cfhttp request access you can then get the more info and click the View Certificate option.
You will need to download the PEM (cert) not the Chain. Once downloaded, you need to run the keytool in order to import it to the keystore.
If you are using the default JRE within your JVM for ColdFusion you will need to install a JDK to your development machine.
You can see the details and steps we have listed on our wiki regarding the commands from the command prompt to import the SSL into the store.
https://wiki.hostek.com/ColdFusion_Tips_%26_Tricks#Fixing_cfhttp_Connection_Failures_with_Keytool
Thanks to #alexbaban his workaround, whilst it worked, it was a solution I could not implement due to requiring the use of the tag cfexecute.

Answer 3

If you can not get the keystore thing working maybe you'll want to try this.
Create a dedicated command line executable (.exe) which will read the web page and save the source to a text file. You can then use ColdFusion to read the file and work with the data.
Here is the ColdFusion code:
<cfscript>
_execPath = "c:/bin/clifton.exe";
_filePath = "c:/bin/clifton.txt";
// run your command-line app (clifton.exe)
cfexecute(name="#_execPath#");
// wait for the file
do {
sleep(100);
} while ( not fileExists(_filePath) )
// wait for write to finish
do {
sleep(100);
_fileInfo = getFileInfo(_filePath);
writeOutput(" ## ");
cfflush();
} while ( _fileInfo.size eq 0 || dateDiff("s", _fileInfo.lastmodified, now()) lte 3 )
writeOutput("<hr />")
_result = fileRead(_filePath);
writeDump(_result);
</cfscript>
As you can see it depends on clifton.exe and reads clifton.txt (clifton.txt is the result of executing clifton.exe).
How to make clifton.exe
You will use the Dart SDK and the dart2native tool to create the executable on your development computer. You can deploy the executable on your production server as a standalone (You don't need the Dart SDK installed on production).
Create a bin folder on your C drive.
From https://ssl-ccp.secureserver.net/repository/?origin=CALLISTO download the certificate sfig2.crt.pem (PEM) and save it inside c:\bin.
Inside c:\bin create a text file clifton.dart with the following code:
// clifton.dart
import 'dart:convert';
import 'dart:io';
main() {
//
const String _certFilePath = 'c:/bin/sfig2.crt.pem';
const String _responseFilePath = 'c:/bin/clifton.txt';
const String _uri =
'https://www.cliftoncameras.co.uk/all-brands-and-types-of-used-cameras/';
final File _file = new File(_responseFilePath);
final IOSink _sink = _file.openWrite();
final SecurityContext _context = new SecurityContext();
_context.setTrustedCertificates(_certFilePath);
final HttpClient _client = new HttpClient(context: _context);
saveSourceToFile(_client, _uri, _sink);
_client.close();
//
}
// get web page source then write it to file
void saveSourceToFile(HttpClient _client, String _uri, IOSink _sink) {
//
_client
.getUrl(Uri.parse(_uri))
.then((req) => req.close())
.then((res) => res.transform(Utf8Decoder()).listen((data) {
// as data is received write to file
_sink.write(data);
}, onDone: () {
_sink.close();
}));
//
}
Download and install the Dart SDK from https://dart.dev/
Open a terminal window and test the installation of Dart with dart --version (you should be able to run dart from any folder, if needed add dart to your PATH)
In a terminal window, change directory to c:\bin with cd c:\bin
Next, run dart2native clifton.dart -o clifton.exe
If compilation goes well you should have inside c:\bin the three files: clifton.dart, clifton.exe and the certificate sfig2.crt.pem.
If you wish you can test run clifton.exe in the terminal window, which should create the clifton.txt file.
Test the ColdFusion page which calls clifton.exe, waits for clifton.txt then outputs the content.
If you deploy in production you need both files clifton.exe and sfig2.crt.pem (the certificate).
Good luck!