I am using EC2 with Elastic Beanstalk to deploy a Spring Boot application. This deployment connects to an RDS MySQL instance and an assigned default security group allows the communication.
For a 3rd time, I have found the security group has been dropped from the EC2s list of groups, resulting in degraded Spring Boot, in which Boot is stuck in a startup loop (I am not sure why brought it down)
A separate Boot/EBS deployment uses this same group for RDS connectivity, and has never experienced this.
Has anyone else experienced this? Logs reveal nothing other than connection timeout to RDS.
To troubleshoot this issue, you can use AWS CloudTrail. Using AWS CloudTrail, you can trace who is detaching security group from the related AWS EC2 Instance. This kind of event is logged as ModifyNetworkInterfaceAttribute with event source as ec2.amazonaws.com.
Here you can find AWS CloudTrail user guide.
Note:Typically, CloudTrail delivers an event within 15 minutes of the API call/event.
I believe your problem is you are attaching the security groups to the instance using EC2 console instead of using EB environment's configuration.
Go to EB console, chose your environment, click on configuration.
Click Edit on the Instances section, add security groups from this location. Doing so will ensure that all your security groups are applied when EB is creating instances as an example when it scale-out.
Related
This might be a duplicate of this post, but I wanted a separate question for my specific situation. Because none of the found answers on Stackoverflow have helped me so far.
I created an Elastic Beanstalk instance a while ago, complete with RDS instance, by following along with the Setup Wizard. But when I now try to upload new code to it I get the following error:
Environment update is starting.
Environment health has transitioned from Ok to Info. Application
update in progress (running for 2 seconds).
Service:AmazonCloudFormation, Message:Stack named
'awseb-e-gmz62hycax-stack' aborted operation. Current state:
'UPDATE_ROLLBACK_IN_PROGRESS' Reason: null
Updating RDS database named: xxxxxxxxxxxxxx failed Reason: DB Security
Groups can no longer be associated with this DB Instance. Use VPC
Security Groups instead.
In the Configuration section of the Elastic Beanstalk, under Security or Database I can't find any setting for DB Security Groups.
If I go to the RDS instance that this Elastic Beanstalk is using, under the Security group rules I see two Security Group - Inbound rules.
One with a name like:
awseb-e-abcde12345-stack-AWSEBSecurityGroup-123456ABCDE (sg-1234abcd)
And the other with a name like:
rds-awseb-e-abcde12345-stack-awsebrdsdbsecuritygroup-1234567abcdef-abcd (sg-4321dcba)
And the rest is all: CIDR/IP - Inbound.
Am I to remove/replace these Security Group - Inbound rules with VPC rules? Do I need to completely build a new Elastic Beanstalk, with other rules? And especially how to do this.
Also, all the security groups associated with this elastic beanstalk and RDS appear in the Security Groups section of the VPC Dashboard: https://eu-west-1.console.aws.amazon.com/vpc/home?region=eu-west-1#securityGroups:.
I am at a loss here. It feels like this should be very straightforward, since I created this with the Setup Wizard from AWS, but I can't figure it out. Anyone with some idea, please comment or answer, thanks!
The error clearly says, DB Security Groups can no longer be associated with this DB Instance.
Earlier in RDS, they were using a separate security group called DB Security group. Now everything moved to VPC security group. If you are using older AWS account, still you can see the DB security groups, but its not working now. So use VPC security group instead.
Best practice:
Launching RDS via Elastic Beanstalk or Cloudformation is not a good practice (particularly for production), because when you delete the CF template it'll remove all of your resources including the DB. And Beanstalk also will do the same on the new deployment.
I have two VPCs:
VPC1 which holds our RDS instance.
VPC2 which holds our cluster of EC2 instances.
We have successfully setup a VPC peering connection, routes and security groups to allow appropriate communication.
In order to resolve the RDS instance AZ-appropriate local IP address from it's hostname, we need to follow these instructions and set --requester-peering-connection-options AllowDnsResolutionFromRemoteVpc=true.
If I do this manually through the AWS Console or the AWS CLI it all works fine, however I'm creating the cluster of EC2 instances through CloudFormation and the option is missing from the CloudFormation documentation.
The effect of this is that my stack starts up and fails because the services themselves cannot connect to the database.
Am I doing something obvious wrong, or is this just Amazon being incomplete?
Thanks!
Due to the frequency of updates, there are many times where an AWS feature isn't available in CloudFormation (ALB targeting Lambda used to be) - you end up having to create a custom resource to manage it. It's not too bad, just make sure that your lambda responds with success or failure in all scenarios, including exceptions, otherwise your stack will be 'in progress' for hours.
I have an RDS database running (postgres) that I'm now trying to connect to an Elastic Beanstalk application. The connection times out.
I am guessing this has something to do with the VPC and security groups, but AWS continues to confuse me here and I'm kind of new to this. I'm trying follow this guide: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.RDS.html?icmpid=docs_elasticbeanstalk_console
When I try to add the rds_launch_wizard security group to the EB instance, it tells me that the security group is not found. rds_launch_wizard is in a VPC, whereas the EB application is not.
I have tried going into the EB configuration, choosing Instance and trying to add a security group:
By name, this pops up an error saying that "I cannot add groups by name".
By group ID, this tells me group sg-xxxxx not found.
Similarly, I can't add the EB security group to RDS, as this is also not found
How do I get the two talking to each other?
You have two options:
Move the EB environment into the VPC
Enable ClassicLink so that your EC2 Classic instances inside your EB environment can talk to the VPC.
ClassicLink is the obvious quick fix, but I think you will eventually want to move everything into a VPC. The VPC is more secure and there are quite a few features and services now on AWS that require a VPC.
Trying to create an Amazon Cloudwatch alert to monitor an Elastic Beanstalk deployment of a public facing website. The alert options for Elastic Beanstalk don't seem to allow for monitoring specific instances that fail Beanstalk's health check URL query. I need to identify the specific unhealthy INSTANCE and terminate it. From there, my autoscaling policy will automatically replace the terminated instance.
Some background
Setup: Elastic Beanstalk deployment running LAMP for a public facing site.
Purpose: For additional failsafe security, I've added a daemon to monitor the state of the file system at /var/www. If the timestamp or size of the filesystem changes (i.e., unwanted file introduction or change), the monitor fires a script that deletes the php file located at elasticbeanstlak's health check URL (random url in the /var/www dir) and forces an "unhealthy" state at the ELB monitoring level.
All is working fine except I can't seem to find a way to get Amazon to identify the specific instance which has caused the health check to fail and let me terminate only that instance.
The AWS docs for creating alarms to handle this specific functionality and initiating instance termination is unclear. I've tried setting up health monitoring at the Beanstalk level, which works to identify an unhealthy state, but not the specific instance. Not new to AWS, but relatively new to Cloudwatch metrics.
Thanks for suggestions.
So it looks like your base use case is this:
Something is wrong in the /var/www dir and your script deleted the health check script.
Instance fails the health check
The instance gets terminated then replaced by Autoscaling
One option would be to use Elastic Beanstalk's Scaling Triggers setting to configure your Autoscaling Group to immediately replace hosts on the UnhealthyHostCount trigger measurement. If you are using the API you can set the triggers with these option settings.
When creating an application in Amazon Elastic Beanstalk, you have the option of creating a new Amazon RDS database instance. Is it possible to associate an existing RDS database instance with an Elastic Beanstalk application?
If you want to share an RDS instance between multiple applications, the best thing to do is set that up independent of your beanstalk application.
Set up privs for each application and configure each application to use the RDS instance.
You will probably have to manually configure your application security group to access the database instance.
I was facing same issue and fixed using following steps :
1) Go to EC2 instance and note your security group example "sg-121212121212"
2) Go to RDS Security Group ad=nd inbound traffic
3) Edit rule select all traffic and add your new ebs security group "sg-121212121212"
Hope it will helpful
A single database can only be used with one Elastic Beanstalk environment. Although, it can be moved between environments.
Also important to remember that a clone operation doesn't clone the database.