Allow Stripe webhook to access AWS EC2 instance - amazon-web-services

I do have a Stripe webhook which is successfully caught and processd in Stripe's TEST MODE, on http local host server.
However, when switching to Stripe's LIVE MODE DATA, the webhook returns status code 500, while the EC2 instance is untouched, no logs being generated.
There is no issue with Signing secrets or Stripe keys, the event never reaches the HTTPS endpoint of the EC2 created using a Load Balancer.
Stripe's support cannot pronounce to this so any suggestions of why this could happen or how to handle it is very welcome.
The error displayed on Stripe is:
HTTP status code 500 (Internal Server Error)
Response Failed to connect to remote host
I have added a whitelist middleware to the express server running on EC2:
app.use((req, res, next) => {
console.log('Always inside ', req.originalUrl);
next();
});
before handling the stripe webhook URL
app.use('/afterpayment', bodyParser.raw({ type: 'application/json' }), afterPaymentRoutes);
in order to see if Stripe event reaches the server, which is not happening.
However, if i manually enter into browser the Stripe Webhook URL, domain/afterpayment, the result is as expected: whitelist middleware prints the message and webhook handler takes over.

I was having a similar problem, and watching this thread. In my case, the issues were a few different things. I'm forcing https to my site (elb is redirecting any traffic from 80 to 443). The app on my ec2 was accepting connections over port 80. Access to the site was working. I thought maybe stripe sending the webhook data to the elb was breaking because of the redirect. This wasn't the case. However, I had a security group that was only allowing access from my IP address (for testing). Changing this to 0.0.0.0/0 from the internet (actual production access) didn't completely fix the problem but I wanted to get things set up to as close as real-world as possible. In the stripe dashboard I created a new webhook pointing to the app endpoint I exposed for testing. From the Stripe dashboard I hit the "Send a test webhook" button. This time instead of getting a timeout the error was invalid signature. So, I knew that exposing the site to the internet was part of the problem., (Yes, I could have created a security group that only allowed access from the IP addresses where the webhook data originates from, but again - I wanted to keep this as close to production as possible thanks #justin-michael for the nudge in the right direction). My app was still using the test webhook I set up for development. When I created the new webhook it also created a new signing secret. I pulled this new webhook signing secret into my app then ran the "send test webhook" again and it was successful. So, allowing the correct access from Stripe and making sure the signing secret was correct fixed the problem for me.

The problem was that the domain was not properly exposed on the internet.
So I have Elastic Beanstalk environment running a node.js server app on which I set a Load Balancer and exposed the server over HTTPS.
While trying to catch a webhook sent by a 3rd party app, like Stripe, nothing arrived on the server, even though I could successfully simulate POST request to the domain endpoint. The domain was also accessible through browser (or so it seemed).
The issue was that the domain name linked to load balancer was not resolvable publicly on the internet. Here are 2 useful links:
https://www.ssllabs.com/ssltest/index.html
https://dns.google.com/query?name=&rr_type=ALL&ecs=
Running tests on them unveiled problems related to DNSSEC configuration of my domain, which was not enabled on my domain.
While following this instructions i did:
On Hosted Zones, under DNSSEC signing -> Enable DNSSEC signing.
Created KSK and Customer managed CMK
Under DNSSEC signing, copied the information from View information to create DS record
On Route 53, on Registered Domains -> on the domain -> DNSSEC status, created a new key with info from previous step
After this, all tests passed and the webhook was successfully handled.

Related

AWS Internal ALB is unable to re-direct to private MWAA webserver

I am attempting to setup MWAA in AWS and the UI web server needs to be inside a private subnet. Based on documentation the way to setup access to the web server VPC endpoints requires using a VPN/Bastion/Load Balancer and I would ideally like to use the load balancer to grant users access.
I am able to see the VPC endpoint created and it is associated to an IP in each subnet (two subnets total) that were chosen during the initial environment setup.
Target groups were setup to these IP addresses with HTTPS:443.
An internal ALB was created with the target groups above.
The UI is presented in the MWAA console as a pop-out link. When accessing that I am sent sent to a page that says The site can't be reached and the URL has a syntax similar to
https://####-vpce.c71.us-east-1.airflow.amazonaws.com/aws_mwaa/aws-console-sso?login=true<MWAA_WEB_TOKEN>
If I replace the beginning of the URL with below I am able to get to the proper MWAA webpage but there are some HTTPS certificate issues which I can figure out later but this seems to be the proper landing page I need to reach.
https://<INTERNAL_ALB_A_RECORD>/aws_mwaa/aws-console-sso?login=true<MWAA_WEB_TOKEN>
If I access just the internal ALB A record in my browser
https://<INTERNAL_ALB_A_RECORD>
I get redirected to a login page for MWAA, click the login button, then I get re-directed to the below which has the This site can't be reached page.
https://####-vpce.c71.us-east-1.airflow.amazonaws.com/aws_mwaa/aws-console-sso?login=true<MWAA_WEB_TOKEN>
I am not sure exactly where the issue is but it seems to be that I am not being re-directed to where I need to go.
Should I try a NLB pointing to the ALB as a target group? Additionally when accessing an internal ALB I read that you need access to the VPC. What does this mean exactly?
Still unsure of what the root cause is for the re-direction not taking place.
Our networking does not need an ALB or NLB in this scenario since we have a DirectConnect setup established which allows us to resolve VPC endpoints if we are on our corporate VPN. I still do end up at a page with an error message regarding This site can't be reached and to get to the proper landing page I just have to hit Enter again in my browser's URL search bar.
If anyone else comes across this make sure that you have the login token appended to the end of your URL entry before hitting enter again.
I have a case open with AWS on this so I will update if they are able to figure out the root cause.

AWS lightsail container service private domain - "net::ERR_NAME_NOT_RESOLVED"

How can I send https request from one deployment to another deployment using AWS lightsail's private domain?
I've created two AWS Lightsail Container deployments using two docker images. I'd like to send https request from one image deployment ("sender") to another image deployment ("receiver"). This works fine when the receiver's public endpoint is enabled. However, I don't want to expose this service to the public but instead route traffic using AWS Lightsail's private domain.
My problem is when I try and send https request from "sender" to the "receiver"'s private domain (.service.local:) I get https://<service_name>.service.local:52020/tester/status net::ERR_NAME_NOT_RESOLVED on the "sender"'s html page. According to the Lightsail docs (section "Private domain") this should be accessible to my "Lightsail resources in the same AWS Region as your service".
I've found a similar Question & Answer in stackoverflow. I tried this answer using my region but failed because Lightsail container required https while .service.local required http. After creating a Amazon Linux instance, I succeeded making http request but failed to make https request. (screenshot below). In the meantime, Lightsail strictly asks you to use https.
If I force to send http request from https webpage, chrome generates Mixed content: The page at ... was loaded over HTTPS but requested an insecure ... error. I can go around the https problem by using next.js api routes, but this doesn't feel secure because next.js api routes are publicly accessible.
Is there anything that I may be missing here?
Things I've verified:
The image is up and running and works fine when connecting to it using the public domain
I'm running both instance and container service in the same region
Thank you in advance.
Some screenshots
Running dig inside docker's entrypoint script
Error message when sender sends http request to receiver
I made my two AWS Lightsail Containers, Frontend Container with next.js and Backend Container with flask, talk to each other using the following steps:
Launch a Lightsail "instance" using "Amazon Linux" in the region I want to deploy my Container. Copy /etc/resolv.conf from this "Amazon Linux" instance. Update Dockerfile to overwrite /etc/resolv.conf file in my docker.
To make API request using http instead of https and go around the Mixed content: The page at ... was loaded over HTTPS but requested an insecure ... error, I used next.js' API route to relay the API request. So, for instance, a page on Frontend Container will make API request to /api on the same Container and the /api route will make http request to Backend Container.
API route was properly coded with security measures so that users cannot use API route to access random endpoint in Backend Container.
https "pages" are often mixed content where resources such as pictures are drawn from the http folders not the "https" site folder, hence the request to get such a resource is http because of its configuration by location, so it will be called by http to obtain and then not be crypted (see server configuration for https folder location that requires access to it by that protocol).
Of protocols, the message from another post implies and may be that to communicate "privately" is NOT a web service for public so such communications require using ssl:// secure protocol (alike using ssh://) NOT https:// secure public web server protocol of both require certificate. (hazard a guess)
ssl may be what is used privately across local.
The following AWS links recommend having differnet accounts for developing and the service.
https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/
https://aws.amazon.com/cli/

Kibana redirects back to login after successful auth on AWS Elasticsearch Service when accessing remotely via an NGINX reverse proxy

I have Elasticsearch (AWS Elasticsearch Service 7.7) running in my AWS VPC.
I'm trying to access Kibana, from a web browser, on my laptop at home.
I followed these instructions, to setup an nginx reverse proxy:
https://medium.com/#k.ashu403/aws-elasticsearch-nginx-reverse-proxy-for-accessing-kibana-86292edc6f14
My config file based on this one (authored by the blog post author):
https://github.com/kin-kins/AWS-services/blob/34c94abeaac5e8e7f5371f5d0df3f49c0417ec56/nginx_reverse_proxy.conf
Everywhere that file shows 3.226.189.187, I have replaced it with the external IP of my reverse proxy.
Everywhere that file shows vpc-ngelasti-qmazoh6hzvpxiludpnzasoi2nu.us-east-1.es.amazonaws.com, I have replaced it with the fqdn of the Kibana instance running in my VPC.
(And nginx has been restarted).
If I then access, my equivalent of: http://3.226.189.187/_plugin/kibana/, I get properly requested for my username & password, and I pass basic auth.
It then redirects to the equivalent of http://3.226.189.187/_plugin/kibana/login?nextUrl=%2F_plugin%2Fkibana%2F#/
If I put in an incorrect username or password, it tells me that it's wrong.
If I put in the correct username & password, it sends me right back to http://3.226.189.187/_plugin/kibana/login?nextUrl=%2F_plugin%2Fkibana%2F#/
In short, I keep getting prompted for the username/password (i.e. redirected to the login page). I assume something is wrong in the nginx conf, but I'm not sure, and have spend a lot of time w/ trial and error, and haven't made much progress. (AWS Elasticsearch Service is configured for username/password auth, which works within the VPC, for Elasticsearch)
What worked for me:
add https (443) on the inbound security group with your ip/office ip range/all traffic
generate a certificate (be aware chrome doesn't like self signed, might have to use firefox)
change your nginx config to use https (port 443) instead of http (port 80) and add the path to the certificate in the config as well.
Example config, namely the listen 443 line as well all the lines starting with ssl
Explanation:
When logging in a security cookie is supposed to be set with a token but it won’t be set when using http for security reasons, changing to https will allow this cookie to be set.
This can be seen by looking at the network request to _plugin/kibana/api/ui_metric/report and looking at the headers and response. It shows the redirect back to log in as well the session expired message.

CNAME to AWS Service - Browser Not Accepting Certificate

I am trying to access an AWS service directly from the browser- specifically the SNS service. I want to be able to post a message directly to an sns topic, but using a CNAME record so I can control which region the browser ultimately goes to (sns.mydomain.com -> sns.us-east-1.amazonaws.com | sns.eu-west-1.amazonaws.com depending on requesters region).
My issue is that if I make an HTTPS request to my aliased endpoint, the returned certificate will not be signed to my endpoint and the browser will refuse to work with it. And while I can get around this by making only HTTP requests, the browser will refuse to make an HTTP request from a secure origin (a site served on HTTPS).
Is it possible to have a CNAME point to an AWS service in the way that I'm trying to do it?
Ultimately, i'm trying to avoid locking the client application in the browser into an aws region.
Is it possible to have a CNAME point to an AWS service in the way that I'm trying to do it?
No. You're hitting up against a central feature of https verification, namely the Common Name of the cert or a SAN ( Subject Alternative Name) must match the certificate. If it weren't so, HTTPS would not be validating that the server is who they claim to be.
Ultimately, i'm trying to avoid locking the client application in the browser into an aws region.
That's a fine goal. Instead of doing so at the DNS layer, why not create an endpoint or configuration setting that supplies region or regions to use? A smart client could even iterate through regions in the case of some failures that appeared to be regional outages, which is somewhat better than a CNAME that you still have to fix when a region goes down.

AWS ALB Listener Rules - OIDC - Google Oauth

I am trying to set Listener rules on an ALB. I want to add Google Oauth support to one of my servers.
Here are the Google endpoints I am using
I see google auth page alright, but on the callback url I'm seeing 500 Internal Server Error. I've also set the callback URL. Am at a loss as to what's wrong here. Any help is most appreciated!
After authentication, I'm not redirecting to my application, instead I've set ALP to show a text based simple response.
I struggled with the same problem for hours, and in the end it turned out to be the user info endpoint that was wrong. I was using the same one as you, but it should be https://openidconnect.googleapis.com/v1/userinfo.
I haven’t found any Google documentation saying what the value should be, but found this excellent blog post that contained a working example: https://cloudonaut.io/how-to-secure-your-devops-tools-with-alb-authentication/ (the first example uses Cognito, but the second uses OIDC and Google directly).
From AWS documentation
HTTP 500: Internal Server Error
Possible causes:
You configured an AWS WAF web access control list (web ACL) and there was an error executing the web ACL rules.
You configured a listener rule to authenticate users, but one of the following is true:
The load balancer is unable to communicate with the IdP token endpoint or the IdP user info endpoint. Verify that the security groups for your load balancer and the network ACLs for your VPC allow outbound access to these endpoints. Verify that your VPC has internet access. If you have an internal-facing load balancer, use a NAT gateway to enable internet access.
The size of the claims returned by the IdP exceeded the maximum size supported by the load balancer.
A client submitted an HTTP/1.0 request without a host header, and the load balancer was unable to generate a redirect URL.
A client submitted a request without an HTTP protocol, and the load balancer was unable to generate a redirect URL.
The requested scope doesn't return an ID token.