I want to know if someone have been implemented and automatic failover solution in AWS. I have an environment where I'm replicating an EC2 Instance behind an ALB in US-EAST-1 and the target Region is US-EAST-2 and I use Route 53 with a failover routing policy to redirect the traffic from Virginia's ALB to Ohio's ALB in case of a disaster.
The problem is not this, the replication is working good but in case of a disaster in Virginia happen, I should need to manually go to the AWS Console and perform an Failover manually, so, my question is if there is a way to perform this automatically?
I thought about an Lambda Function that performs the failover process when a specific health check failed or something like that, but I want to know if there are another options. Have do you implemented something like this?
I expect that someone give an option about how they solved this problems
Related
To create a Glue Connection Resource we can just pick one AZ/Subnet form the account VPC. Does it mean if that AZ goes down the Glue jobs that using that connection will fail? If yes how we can make it multi AZs to avoid a single point of failure?
There is no option to switch a subnet at runtime. The Elastic network interfaces with a glue prefix as seen in CloudTrail, show that they are attached to a specific subnet, VPC and security group. These match the connection metadata that you would have added in Glue. If a job fails due to the unlikely event of a complete AZ going down, then you can edit the connection and switch to another subnet. Maybe the folks at AWS support could take a feature request to enhance the product in future.
I able to find that in my research there are some of the ways to find all our aws resources but will you suggest any solution to find public resources in aws our account.
Unfortunately, there is no easy way that you can find all resources that are allowed to communicate with outside world. However, the only way for most of the resources in AWS can connect to the internet through a VPC that is configured to allow access to 0.0.0.0/0 network with Security Group and ACL. So what you can do is to search for VPC ID on resources categories and if any results you get can connect to the internet.
This can be a very tedious task and this is NOT 100% guaranteed to find all resources as some resources has self-sustained VPC per resource such as S3. So my suggestion is you get used to using tags for resources.
Also maybe think about using infrastructure as code approach using something like Terraform.
If you have all of your instances in a VPC you could enable VPC Flow Logs on your VPC and then monitor all VPC traffic that way.
For the instances that are not in VPC's you could have CloudWatch logs enabled for that instance and then monitor the traffic that way.
Your instances should have security groups attached to them so that you could monitor the rules, and see if you have an OutBound Set to 0.0.0.0/0
You can give Netflix's Security Monkey a try. The tool will make a complete report of all secure and insecure AWS assets including Security Groups, IAM Policies and S3 Buckets. In fact, it also provides a ticketing like system to collab on such problems.
You can use IAM Access Analyzer console
I have several pre-prod VPC's for dev, qa, etc. So far everything is managed via CloudFormation + Lambda and is very clean.
Since none of the VPC's talk to each other, I can launch and tear down environments, all from one single set of templates and it is very idempotent.
Now I have a requirement to add a management VPC for some common things between the environments. I have used VPC peering through the UI before and it's easy to manage.
However I like the flexibility of tearing down and launching new environments, especially since prod will end up being blue/green.
So to avoid having to use the UI I was thinking of adding two Lambda functions:
1. on the mngmt VPC, list all VPC's and if there's not an existing VPC peering connection with one, request it
2. on pre-prod VPC's poll for VPC connection peering requests and accept if from a trusted owner
This way I can tear down the stack and re-launch them without any manual intervention.
Is this a wasteful design? Are there any problems that I am not foreseeing? What is a better way to accomplish this?
The AWS documentation states that only the owner of an account can accept a VPC peering connection. You could create a Lambda for each account and then use DescribeVpcPeeringConnections to detect pending-acceptance state. Then call AcceptVpcPeeringConnection. To create a VPC peering connection call AcceptVpCPeeringConnection.
Notes:
I do not like polling designs. In your case you would need to constantly poll to detect a new peering connection request. A better solution would be to invoke (or trigger via SNS) the Lambda functions via a script (program) after creating the VPCs.
This link is to the API Reference. From here you can select the language that you will be using at the bottom of the page.
AcceptVpcPeeringConnection
I have two VPCs:
VPC1 which holds our RDS instance.
VPC2 which holds our cluster of EC2 instances.
We have successfully setup a VPC peering connection, routes and security groups to allow appropriate communication.
In order to resolve the RDS instance AZ-appropriate local IP address from it's hostname, we need to follow these instructions and set --requester-peering-connection-options AllowDnsResolutionFromRemoteVpc=true.
If I do this manually through the AWS Console or the AWS CLI it all works fine, however I'm creating the cluster of EC2 instances through CloudFormation and the option is missing from the CloudFormation documentation.
The effect of this is that my stack starts up and fails because the services themselves cannot connect to the database.
Am I doing something obvious wrong, or is this just Amazon being incomplete?
Thanks!
Due to the frequency of updates, there are many times where an AWS feature isn't available in CloudFormation (ALB targeting Lambda used to be) - you end up having to create a custom resource to manage it. It's not too bad, just make sure that your lambda responds with success or failure in all scenarios, including exceptions, otherwise your stack will be 'in progress' for hours.
I have one VPC with an RDS instance in it. They are both located in the same region.
I want to use the RDS instance in another VPC, that is in another region on another AWS account (we have multiple AWS accounts). If that's not complicated enough the 2nd VPC comes up via CloudFormation (i.e. dynamic). Whenever I am bringing up a CloudFormation stack I want to attach the RDS instance automatically.
I have looked at:
exposing the RDS instance on the public internet :(
an ELB w/ TCP transport to put the database instance behind
VPC peering but the different regions and the approval workflow in the AWS console make little sense in the case we are using CloudFormation
All of these seem suboptimal to me and was wondering if somebody already did this before. If yes, please share what you did and what the though process behind it was.
Use a VPN tunnel from one VPC to the other. You could build your own or look at Vyatta. Ideally the two VPCs do not have overlapping CIDRs. Note that you cannot use VPC peering inter-region.
For anyone who stumbles around here, it looks like AWS VPC Peering can now be done cross region: https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html