I am trying to setup couchbase cluster on AWS. I want my nodes to be distributed across multiple availability zones.
Ec2 instances with in an availability zone are able to access each other using the ip (Private DNS) which is assigned to them during creation and which does not change even if we restart the machine.
I am not able to access an Ec2 instance from other AZ using this (Private DNS). One way this can be done is by using Elastic Ip which has a limit per region.
Question here is How to access one Ec2 instance from other EC2 instance in another AZ without elastic ip?
You do not want to use Elastic IP for this. Your statement that Elastic IP is a solution to your issue is not correct. You want to use the Private IP assigned to the instance when you created it.
The private IP will not change as long as the instances are deployed inside a VPC.
You have to use the private IP in order to keep all network traffic inside the VPC. Then you just need to make sure your Security Groups are configured correctly to allow traffic between the instances.
Amazon Web Services Operates Split-horizon DNS (AKA Split-Brain DNS). The best practice when deploying couchbase onto EC2 is to use hostnames not IP addresses, see http://developer.couchbase.com/documentation/server/current/install/cloud-deployment.html . Amazon will automatically give a different IP when resolving the hostname depending if the source of the request is internal or external.
Related
I have 3 AWS Elastic Beanstalk instances which are running Spring microservices. All microservices are making POST requests to each other and using RDS service for database.
Should I isolate database traffic and microservices traffic into separate subnets?
In case it's a good practice is it possible to assign 2 private network IP's for each subnet for every AWS Elastic Beanstalk instance?
I think you cannot do it using EBS as the instances will auto create and terminate. So you should try to create instances separately and add autoscaling policy on it.
What I usually do is create my EC2 instances in Public subnet and RDS in private subnet and use RDS Security Group and add EC2 instance's Elastic IP, so that all the traffic is going through the EC2 instance and all the traffic coming to EC2 instance is HTTPS coming from ELB.
Adding the below steps as requested:
Ok, So I am assuming you already know a bit about how to create the servers and RDS etc.
Create an EC2 instance for each of your microservices.
Attach an EIP to each of these instances.
Add an Auto-Scaling policy to increase or decrease the instances based on the traffic/CPU Utilization. Make sure you terminate the newest created instance.
Add an ELB for this instance and add HTTPS/SSL certificate to secure your traffic.
Create RDS in a Private subnet and add instance EIP in RDS SG for 3306 port.
I think you should be able to do this then.
It's not a good practice to directly communicate between instances in EB. The reason is that that EB instances run in autoscalling group. So they can be terminated and replaced at any time by AWS leading to change in their private Ip addresses.
The change in IP will break your application sooner or later. Instances in EB should be accessed using Load Balancer or private IP.
So if you have some instances that are meant for private access only you could separate them to internal EB environment.
After restarting AWS EC2, hostname & public IP gets changed.
Remote docker clients get affected as they rely(export DOCKER_HOST) on these public names.
How to resolve this dynamic IP(public) problem of EC2?
By default, AWS assigned public IP addresses as well as hostnames are ephemeral, meaning they will be released back to the pool if you restart the instance. If you really need a persistent IP address, you can use Elastic IPs, but bear in mind there’s a limit per region.
Note: I’d still recommend evaluating the need for using a public IP from the IPv4 pool, as they are a rare resource. Most of the times, one can get by well by using the correct combination of security groups and private IPs, along with Route53 hosted zones for friendly naming, assuming instances are in the same VPC or can communicate via VPC peering.
I currently have an elastic beanstalk setup in AWS currently whenever i make API calls it is coming from the EC2 instance's external IP. is there a way to have all servers in that group use the same IP?
Put your EC2 instances in private subnets and direct all outbound traffic through a NAT. This way, all outbound connections appear to come from the NAT's IP address.
See the following for more information. It's a different problem, but the NAT solution is the same.
How do you allocate STATIC addresses to an EBS (beanstalk) within a VPC?
Note, for security, you should follow this architecture anyways. When using ELB, don't have your EC2 instances in a public subnet.
As far as high availability goes, what is the difference between using an Elastic IP or an Elastic Network Interface to mask instance failure? Is the only difference because ENIs can be used for private instances and Elastic IPs can't?
I'm trying to explain the advantages of both, so if someone can help me with this, I would appreciate it!
To achieve High Availability, you need the ability to redirect traffic in the case of instance failure. There are several options:
1. Use an Elastic Load Balancer
This is the preferred way to provide High Availability.
Run multiple Amazon EC2 instances, preferably in different Availability Zones (AZs). Users connect to the ELB (via the supplied DNS name), which redirects traffic to the EC2 instances. If an instance fails, ELB notices this via regular Health Checks, and will only direct traffic to the healthy instances.
Auto Scaling can be used to create these multiple instances across multiple Availability Zones, and it can also update the Load Balancing service when it adds/removes instances.
2. Redirect an Elastic IP address
Run multiple instances (preferably across multiple Availability Zones). Point an Elastic IP address to the instance you desire. Users connect via the Elastic IP address and are directed to the instance. If the instance fails, reassociate the Elastic IP address to a different instance, which will then start receiving the traffic immediately.
This method is not recommended because only one instance is receiving all the traffic while the other instance(s) are sitting idle. It also requires a mechanism to detect failure and reassociate the Elastic IP (which you must do yourself).
3. Reassign an Elastic Network Interface (ENI)
All EC2 instances have a primary ENI. They can optionally have additional ENIs.
It is possible to direct traffic to a secondary ENI and then move that secondary ENI to another instance. This is similar to reassigning an Elastic IP address.
This method is not recommended for the same reason as reassociating an Elastic IP address (above), but also because an ENI can only be reassigned within the same AZ. It cannot be used to direct traffic to an EC2 instance in a different AZ.
Bottom line: Use an Elastic Load Balancer. It provides true High Availability and can do it automatically.
See documentation: What Is Elastic Load Balancing?
Is it possible to use S3 APIs from an EC2 instance that does not have a elastic / public IP? Alternatively, is it possible to whitelist the connection with S3?
We intend to use the APIs to read/write objects from a few buckets on S3.
Based on this:
Q: Do I need one Elastic IP address for every instance that I have running?
No. You do not need an Elastic IP address for all your instances. By default, every instance comes with a private IP address and an internet routable public IP address. The private address is associated exclusively with the instance and is only returned to Amazon EC2 when the instance is stopped or terminated. The public address is associated exclusively with the instance until it is stopped, terminated or replaced with an Elastic IP address. These IP addresses should be adequate for many applications where you do not need a long lived internet routable end point. Compute clusters, web crawling, and backend services are all examples of applications that typically do not require Elastic IP addresses.
It looks like you have a public IP even if you don't have an elastic IP, it'll just go away when you stop your instance (and can't be given to another box like elastic IPs can).
So yeah, you should be able to access S3 from any running EC2 instance.
As for whitelisting, I think it depends on how you're doing it? If you're using security groups, all outbound traffic is allowed by default and I don't think you can change that unless you're using VPC. If you're using an internal firewall you'll have to find the S3 IP ranges somewhere on the AWS forums.
You can whitelist and blacklist users and instances using S3 Bucket Policies or IAM Policies or both.
S3 Bucket Policies
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucketPolicies.html
IAM Policies
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingIAMPolicies.html