Haproxy to authorize traffic from AWS API Gateway - amazon-web-services

Question

I have set up a basic API using AWS API Gateway and I would like to link my endpoints to a service I have running on an EC2 instance (using the "HTTP Proxy" integration type). I have read that in order to lock down my EC2 server from only accepting traffic from the API Gateway, I basically have one of two options:
Stick the EC2 instance behind VPC and use Lambda functions (instead of HTTP proxy) that have VPC permissions to act as a "pass through" for the API requests
Create a Client Certificate within API Gateway, make my backend requests using that cert, and verify the cert on the EC2 instance.
I would like to employ a variation of #2 and instead of verifying the cert on the EC2 service instance itself, I would like to instead do that verification on another instance running Haproxy. I have set up a second EC2 instance with Haproxy and have that pointed at my other instance as the backend. I have locked down my service instance so it will only take requests from the Haproxy instance. That is all working. What I have been struggling to figure out is how to verify the AWS Gateway Client Certificate (that I have generated) on the Haproxy machine. I have done tons of googling and there is surprisingly zero information on how to do this exact thing. A couple questions:
Everything I have read seems to suggest that I need to generate SSL server certs on my Haproxy machine and use those in the config. Do I have to do this, or can I verify the AWS client cert without generating any additional certs?
The reading I have done suggests I would need to generate a CA and then use that CA to generate both the server and client certs. If I do in fact need to generate server certs (on the Haproxy machine), how can I generate them if I don't have access to the CA that amazon used to create the gateway client cert? I only have access to the client cert itself, from what I can tell.
Any help here?
SOLUTION UPDATE
First, I had to upgrade my version of HAproxy to v1.5.14 so I could get the SSL capabilities
I originally attempted to generate an official cert with letsencrypt. While I was able to get the API gateway working with this cert, I was not able to generate a letsencrypt cert on the HAproxy machine that the API gateway would accept. The issue surfaced as an "Internal server error" response from the API gateway and as "General SSLEngine problem" in the detailed CloudWatch logs.
I then purchased a wildcard certificate from Gandi, and tried this on the HAproxy machine, but initially ran into the exact same problem. However, I was able to determine that the structure of my SSL cert was not what the API gateway wanted. I googled and found the Gandi chain here:
https://www.gandi.net/static/CAs/GandiStandardSSLCA2.pem
Then I structured my SSL file as follows:
-----BEGIN PRIVATE KEY-----
# private key I generated locally...
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
# cert from gandi...
-----END CERTIFICATE-----
# two certs from file in the above link
I saved out this new PEM file (as haproxy.pem) and used it in my HAproxy frontend bind statement, like so:
bind :443 ssl crt haproxy.pem verify required ca-file api-gw-cert.pem
The api-gw-cert.pem in the above bind statement is a file that contains a client cert that I generated in the API gateway console. Now, the HAproxy machine properly blocks any traffic coming from anywhere but the gateway.

Answer 1

The reading I have done suggests I would need to generate a CA and then use that CA to generate both the server and client certs.
That's one way to do it, but it is not applicable in this case.
Your HAProxy needs to be configured with a working SSL certificate signed by a trusted CA -- not the one that signed the client certificate, and not one you create. It needs to be a certificate signed by a public, trusted CA whose root certificates are in the trust store of the back-end systems at API Gateway... which should be essentially the same as what your web browser trusts, but may be a subset.
Just as your web browser will not speak SSL to a server sporting a self-signed certificate without throwing a warning that you have to bypass, the back-end of API Gateway won't negotiate with an untrusted certificate (and there's no bypass).
Suffice it to say, you need to get API Gateway talking to your HAProxy over TLS before trying to get it to use a client cert, because otherwise you are introducing too many unknowns. Note also that you can't use an Amazon Certificate Manager cert for this, because those certs only work with CloudFront and ELB, neither of which will support client certs directly.
Once the HAProxy is working with API Gateway, you need then to configure it to authenticate the client.
You need ssl and verify required in your bind statement, but you can't verify an SSL client cert without something to verify it against.
I only have access to the client cert itself, from what I can tell.
And that's all you need.
bind ... ssl ... verify required ca-file /etc/haproxy/api-gw-cert.pem.
SSL certs are essentially a trust hierarchy. The trust at the top of the tree is explicit. Normally, the CA is explicitly trusted and anything it has signed is implicitly trusted. The CA "vouches for" the certificates it signs... and for certificates it signs with the CA attribute set, which can also sign certificates under them, extending that implicit trust.
In this case, though, you simply put the client certificate in as the CA file, and then the client certificate "vouches for"... itself. A client presenting the identical certificate is trusted, and anybody else is disconnected. Having just the certificate is not enough for a client to talk to your proxy, of course -- the client also needs the matching private key, which API Gateway has.
So, consider this two separate requirements. Get API Gateway talking to your proxy over TLS first... and after that, authenticating against the client certificate is actually the easier part.

Answer 2

I think you are mixing up server certs and client certs. In this instance API Gateway is the client, and HAProxy is the server. You want HAProxy to verify the client cert sent by API Gateway. API Gateway will generate the certificate for you, you just need to configure HAProxy to verify that certificate is present in every request it processes.
I'm guessing you might be looking at this tutorial where they are telling you to generate the client cert, and then configure HAProxy to verify that cert. The "generate the cert" part of that tutorial can be skipped since API Gateway is generating the cert for you.
You just need to click the "Generate" button in API Gateway, then copy/paste the contents of the certificate it presents you and save that as a .pem file on the HAProxy server. Now I'm not a big HAProxy user, but I think taking the example from that tutorial your HAProxy config would look something like:
bind 192.168.10.1:443 ssl crt ./server.pem verify required