kind of an unusual setting here:
We have an SMTP service running on Tomcat / Elastic Beanstalk on AWS in an auto-scaling group behind an ELB load-balancer.
Now, for one of our clients we need to have a static IP for the SMTP service. Since this is not possible with the out-of-the-box load-balancer on AWS, we have a separate HAProxy instance transparently routing the :25 traffic trough the AWS load-balancer.
For some reason, the HAProxy chokes after exactly 3 SMTP calls. After that connections either time out or take minutes to go through.
The interesting part is that the following configurations work perfectly fine:
Calling the SMTP service on the AWS load-balancer directly
Load-balancing the Elastic Beanstalk's nodes through HAProxy directly.
Target setting with HTTP calls on port 80, instead SMTP on port 25
Help is really appreciated
That sounds like EC2 rate limiting what appears -- to the system -- to be "outbound" SMTP from your HAProxy instance.
You're accessing the ELB from the HAProxy by one of this outside addresses, and this is causing your traffic to be treated as Internet-bound.
In order to maintain the quality of Amazon EC2 addresses for sending email, we enforce default limits on the amount of email that can be sent from EC2 accounts. If you wish to send larger amounts of email from EC2, you can apply to have these limits removed from your account by filling out this form.
https://portal.aws.amazon.com/gp/aws/html-forms-controller/contactus/ec2-email-limit-rdns-request
One solution is to had those limits removed, but consider your next step carefully -- you'd be better served by load-balancing the EB nodes through the HAProxy directly, using the nodes' private IP addresses -- because there is a charge for traffic to your ELB from within EC2 on the public IP.
Data Transfer OUT From Amazon EC2 To ... Amazon Elastic Load Balancing ... in the same Availability Zone ... Using a public or Elastic IP address ... $0.01/GB.
http://aws.amazon.com/ec2/pricing/
Not a massive charge, perhaps, but it should be an avoidable charge nonetheless.
Additionally, there's no way to configure HAProxy to look up the IP address behind the hostname you've configured for the ELB with each request. HAProxy resolves hostnames on startup and if the ELB's IP address changes, HAProxy will not detect this change.
On the flip side, you can't reliability configure HAProxy to directly connect to the EB instances, since they're dynamically-addressed as well.
The simplest way to prove that my diagnosis is correct is to set the ELB's TCP listener on another port, such as 587 (or 2025, or whatever), mapped to port 25 on the EB instances. Then have the HAProxy target the traffic to port 587. That should eliminate the EC2 rate limiting on SMTP, although you do still have an issue to deal with if the ELB's external IP changes.
Related
so I have a system that serves 100 EC2 instances. they are all behind the load balancer and they have a random pull range of IPs.
I want to use each app server to communicate with a local server however, that local server can only whitelist a set of IP or domains (not wild card).
I have 200 EC2 with 200 IPs that regularly change and public DNS is ec2 XX.XX.XX.XX.compute-1.amazonaws.com which cannot be whitelisted because the IP varies.
the question is what would be the easiest way to make all outgoing request go out in a single IP/domain>?
I read something about reverse proxy but I do not know if that is the best way to go. any help would be appreciated
You could use something like a NAT Gateway that translates the original requesters’ IP addresses into its own fixed address before forwarding the requests.
AWS NAT Gateway documentation: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html
I have an instance running on premise and its IP address is changed regularly. My other services are running on AWS and they are using IP to connect to the premise's services. I have to update the IP address saved on AWS services whenever the IP is changed on premise network. I have a thought about using DNS but it is still a need to update A record.
I am looking for a way to do some auto-detect instead of manual updating. I wonder whether I can use load balancer to do the check. I know there will be a range of IP addresses on premise network. Can load balancer do a health check on these IP within the range? So my AWS service can send request to the load balancer. Is there any side-effect on this approach?
You need to use hostname instead of IP address as you mentoned the IP addresses keeps changing. AWS VPC can use a DNS forwarder like Unbound, which can forward the requests to your on premise DNS server when VPC resolution is unable to resolve the hostnames. This appraoch is quite effective as you send only those DN resolution to on-premise DNS that are missed by AWS VPC DNS.
Unbound allows resolution of requests originating from AWS by
forwarding them to your on-premises environment—and vice versa. For
the purposes of this post, I will focus on a basic installation of
Amazon Linux with the configuration necessary to direct traffic to
on-premises environments or to the Amazon VPC–provided DNS, as
appropriate. Review the Unbound documentation for details and other
configuration options.
Further reading : How to setup DNS resolution from AWS to on premise servers
I have configured and passed the health check for my AWS ELB(load balancer), but I was trying to do a ping or send a packet to the tcp port 9300 there is no ip address for the ELB.
I have an EC2 instance at the end of the ELB which has Elasticsearch running on it.
The ELB that I configured is an internal ELB so it doesn't have a public IP address for it.
I was wondering if there is a way I can ssh? or do something to ping the ELB?
I am pretty new to AWS and read all the trouble shooting from AWS official website, but couldn't find a solution.
The goal that I am trying to achieve is to test whether my internal Amazon EC2 load Balancer is working properly.
I got the internal ELB ip address with the ping command, however, I am not able to ping or crul to that IP address.
I what to know what I am doing wrong.
Is it the way that I want to access a private network is in correct?
An Elastic Load Balancer is presented as a single service, but actually consists of several Load Balancing servers spread across the subnets and Availability Zones you nominate.
When connecting to an Elastic Load Balancer, you should always use the DNS Name of the Elastic Load Balancer. This will then resolve into one of the several servers that are providing the load balancing service.
Load Balancers are designed to pass requests and return responses. The next time a user sends a request, it might be sent to a different back-end service. Thus, it is good for web-type traffic but not suitable for situations requiring a permanent connection, such as SSH. You can configure sticky sessions for HTTP connections that will use cookies to send the user to the same back-end server if required.
The classic Elastic Load Balancer also supports TCP protocol, but these requests are distributed in a round-robin fashion to the back-end servers so they are also not suitable for long-lasting sessions.
Bottom line: They are great for request/response traffic that needs to be distributed across multiple back-end servers. They are not suitable for SSH.
Site-note: Using PING to test services often isn't a good idea. Ping is turned off in Security Groups by default since it can expose services and isn't good from a security perspective. You should test connectivity by connecting via the expected protocols (eg HTTP requests) rather than using Ping. This applies to testing EC2 connectivity, too.
First things first, I don't know how AWS components work or are configured. I'm just designing an architecture from a top perspective, and then some sysadmin is going to implement it.
If I have an ELB and EC2 instances either running a HTTPS server or accepting WSS connections using a wildcard certificate (server1.domain.com, server2.domain.com...), should the ELB listen to different ports (like ELB:443 -> server1, ELB:444 server2...) or can it listen just to 443 and 8080 (for WSS) and then redirect to the specific server?
ELB does not work like that. ELB will take requests and forward them to any EC2 instance that is listening behind it. You cannot direct requests to specific EC2 instances in any way (port, domain, etc.).
If you want certain domains to go to specific EC2 instances, then you would either map your domain to the EC2 instances directly via your DNS server (and use Elastic IP addresses), or you would have individual ELBs infront of the different EC2 instances.
With a webserver (apache or nginx) I am able to find the x-forwarded-for header and find the client IP rather than the ELB's IP.
Can I do the same thing using IPTables so that I am able to block certain IP addresses?
I can do this at the webserver level. However, I think this is a bit inefficient and I am hoping I can achieve this with IP Tables or something similar?
Since July 30th ELB supports Proxy Protocol. As stated in the end of this thread in the AWS forum:
Elastic Load Balancing (ELB) now supports Proxy Protocol version 1.
This feature allows you to identify the client’s connection
information when using TCP load balancing, providing additional
insight into visitors to your applications. Having this information
can be useful for analyzing traffic logs, gathering connection
statistics, troubleshooting, or managing whitelists of IP addresses.
You must enable Proxy Protocol in the ELB.
The developers guide has more information about Proxy Protocol.
You can't do this with iptables, because iptables will only ever see the IP address of the elastic load balancer, since the ELB is what is establishing the connections to your instance.
Using the web server to block certain x-forwarded-for values isn't particularly inefficient, but if you want to control who can access your ELB by IP address, that can also be accomplished with the Security Group attached to the ELB.
Update: Your comment is partially correct, because, at least as of now, ELB on EC2 "classic" does not support an inbound security group or network access control list, but ELB on VPC does.
Q: Can I configure a security group for the front-end of the Elastic Load Balancer?
If you are using Amazon Virtual Private Cloud, you can configure security groups for the front-end of your Elastic Load Balancer. — http://aws.amazon.com/ec2/faqs/#ELB6
Security groups are easiest to use when you need to allow a relatively small set of specific IP address ranges. If you want to allow most but block a few, then a VPC Network Access Control List is the easier approach.