I have a Django app that fetches DB secret from AWS Secret Manager. It contains all the DB parameters like username, password, host, port, etc. When I start the Django application on EC2, it successfully retrieves the secret from the Secret Manager and establishes a DB connection.
Now the problem is that I have a password rotation policy set for 30 days. To test the flow, at present, I have set it to 1 day. Every time the password rotates, my Django app loses DB connectivity. So, I have to manually restart the application to allow the app to fetch the new DB credentials from the Secret Manager.
Is there a way that secret fetching can happen automatically and without a manual restart of the server.? Once way possibly is to trigger an AWS CodeDeploy or similar service that will restart the server automatically. However, there will be some downtime if I take this approach.
Any other approach that can seamlessly work without any downtime.
If the old DB credentials are invalidated immediately during the rotation, then it will probably be pretty difficult to do this without some downtime. One option would be to have your app catch the credential error and (try to) fetch the new secret at that point from Secrets Manager, creating a new DB connection. Another other option is to have two valid user/password pairs, leaving the old valid while creating the new. I'm not sure if automatic rotation gives you this option. Then you can restart your app as you like. To do even that without a brief outage probably requires a load balancer and multiple instances of your application running, so that you can up one with new creds before you terminate the old one.
This was previously answered in how to use new secret created by key rotation.
If you are using multi user rotation (the "Use a secret that I have previously stored in AWS Secrets Manager" option in the console) you can use the Secrets Manager python caching library to cache and periodically refresh the secret.
If you use the single user rotation option you will need to write a connection wrapper (similar to the JDBC wrapper) that refresh the credentials when you get an error establishing a new connection.
Related
we have a Windows EC2 instance created. We have a need to remote desktop into the instance occasionally to execute a PowerShell script that updates Microsoft's Power BI Gateway. This is all implemented in a sandbox environment. We're looking at steps/actions to implement in a production environment. From research, I have found that we need to have a different key pair for each user. I think this makes sense. But, the logistics of implementing this is beyond my knowledge. I have some basic questions that perhaps can help steer me in the right direction:
How do I generate a key pair for each user? Where would these be maintained? How do I assign the key pair by user? Do I need to create a new user in the IAM console?
Is there a way to rotate the key pairs?
An instance can only have a single key pair associated with it at a time. Does this mean that after each use, I should detach my key pair and the next user needs to attach their key pair? This all seems very manual.
This is the first I've had to deal with virtual servers and security management of said server. But, it seems that AWS would offer a service that automates this process?
Thank you.
Use AWS Systems Manager Run Command to automate common admin tasks across your EC2 instances.
When a Windows Amazon EC2 instance is launched, a random Administrator password is generated.
This password is then encrypted using the keypair selected when the instance is launched. You can access it via "Get Windows Password", and supplying the keypair.
Once you login, you are welcome to change the Administrator password or connect the instance to Active Directory.
You can also create additional Windows logins for your other users.
So, your choice is to either "share" a login for whoever needs to run the script, or give them each their own Windows login and their own password.
I also like #jarmod's idea of using the Systems Manager Run Command to execute a script without logging in!
I have a Java application using MongoDB (or it could be any service like that). On start up, the app creates a singleton connection to database. To connect, I get the MongoDB from AWS Secrets Manager... and thus the application runs merrily ever after communicating with MongoDB.
My question is: What happens when AWS Secrets Manager rotates keys?
How does my app come to 'know' that secret has been rotated.
Do I have to synchronize the timing between Secrets Manager and my app?
e.g. rotation is set to 7 days. So I code in my app to refresh every 7 days... not good, as very hard to time precisely.
Another way could be, if my app faces authentication exception, just refresh password and make a new connection and retry app logic.
What is the industry standard?
How does my app come to 'know' that secret has been rotated?
-AWS Secrets Manager publishes the CloudTrail event - 'RotationSucceeded' when rotation succeeds and the cloudtrail event 'RotationFailed' when rotation fails. You can setup a cloudwatch rule on this cloudtrail event - https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-CloudTrail-Rule.html
and have a SNS or a Lambda setup as the target for the rule and execute any logic you want after rotation succeeds
This is generally dealt with using one of two strategies, or in Secrets Manager speak, by using single user rotation or multi user rotation. Secrets Manager provides lambda implementations for both single and multi user rotations of MongoDB.
In single user rotation there is one DB user/password pair. During rotation the password is updated either using the original user/password or by fetching the master user creds and using those to update the password. In this case any connections established using the old creds would fail after rotation. To deal with this the application would use a connection manager that detected an authentication error (or all errors if necessary) and refreshes the secret before retrying. This is the strategy used by the Secrets Manager provided JDBC wrapper.
The other alternative (multi user rotation) is to read the user name from the original secret, and then, on the first rotation, create a clone of that user with a new password using the master user secret. After that rotation consists of alternating the secret user/password pair between the original and clone and updating the password. In this case the application only needs to refresh the secret once in the rotation interval. If it is using the old user/password pair, it will remain valid for two rotation intervals.
If you are using MongoDB on AWS (as apposed to DocumentDB with Mongo compatability), the easiest thing to do is spin up a temporary DocumentDB and use the Secrets Manager console to setup rotation on that. Then copy the Lambdas, roles and policies, and secrets used there for your Mongo application before tearing down the DocumentDB instance. If you are already using DocumentDB then as mentioned just use the SecretsManager console to set it up.
I was successfully able to connect to RDS like any other database connection.
I use spring jpa data ( repository ) to do CRUD operation on postgres db.
currently I provide the db url and the credential in the properties file
spring:
datasource:
url: jdbc:postgresql://<rds-endpoint>:5432/<dbschema>
username: <dbuser>
password: <dbpassword>
However this is not an option while connecting to production or preproduction.
what is the best practise here.
Does AWS provide any inbuild mechanism to read these details from an endpoint like in the case of accessing S3 ?
My intention is not expose the password.
Several options are available to you:
Use the recently announced IAM access to Postgres RDS
Use Systems Manager Parameter Store to store the password
Use Secrets Manager to store the password and automatically rotate credentials
For 2 and 3, look up the password on application start in Spring using a PropertyPlaceholderConfiguration and the AWSSimpleSystemsManagement client (GetParameter request). SystemsManager can proxy requests to SecretsManager to keep a single interface in your code to access parameters.
IAM credentials is more secure in that:
If using EC2 instance profiles, access to the database uses short lived temporary credentials.
If not on EC2 you can generate short lived authentication tokens.
The password is not stored in your configuration.
If you have a separate database team they can manage access independent of the application user.
Removing access can be done via IAM
another generic option I found was to use AWS Secret Manager
(doc link)
RDS specific solution is to connect to DB Instance Using the AWS SDK using IAMDBAuth
We are a small startup currently in prototype phase. We are still in development phase, and are using AWS to host our application and (test) domain. We have hosted our domain on Route 53, and registered that with SES for email services.
I am new to AWS, and have used domination to understand how to set these things up. Now it appears that our account(s) have been compromised/hacked and someone is missing it to send malicious emails. I am unsure what is the extend of hack, and if the users is only managed to get access to SES and Database credentials. I received an email from SES team, which shows emails have been send through my domain (not by me), but I never created that email on my domain.
Additionally, I have noticed that someone is trying to access my database (from China) and database is always at 100%. Database log says it has blocked IP (which is based in China).
We are using GitHub to store code, and in our code we had credentials for AWS and SMTB servers so I think its possible that someone stoke keys from there (we have taken credential out of GitHub now).
Can someone help me understand what steps do I need to take. I am thinking to shut down this environment and create a new one, but I am unsure how to get hold of my domain and shut down all emails created by spammer on my domain. I am also unclear what is the extend of hack, and if this will come back.
Cam someone please help.
You should never store your credentials in github.
In fact, you should use roles instead of credentials stored directly in the code.
So, step by step you should:
Remove the credentials from github and from your code (done)
Reset your credentials and do not store them
Create a role with the policy according to your needs
Assign that role to your resources.
Here you can found more info
I set up IAM authentication on an RDS instance, and I'm able to use IAM to get database passwords that work for 15-minutes. This is fine to access the database for backups, but this database backs an web application so currently after 15 minutes the password used by the app to connect to the DB becomes invalid and the app crashes as it can no longer access the DB.
However, in the RDS IAM docs there's this line:
For applications running on Amazon EC2, you can use EC2 instance profile credentials to access the database, so you don't need to use database passwords on your EC2 instance.
This implies that on EC2 there's no need to use the IAM temporary DB password, which would mean that my app should be able to connect to the DB as long as it's running on EC2 and I set up the role permissions (which I think I did correctly). However, I can't get my app running on EC2 to be able to connect to the RDS DB except by using the 15-minute temporary password. If I try connecting with a normal MySQL connection with no password I get permission denied. Is there something special that needs to be done to connect to RDS using the EC2 instance profile, or is it not possible without using 15-minute temporary passwords?
According to the documentation you linked (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html), you need to perform the following steps (See under "Authenticating to a DB Instance or DB Cluster Using IAM Database Authentication"):
Use the AWS SDK for Java or AWS CLI to get an authentication token you can use to identify the IAM user or role. To learn how to get an authentication token, see Getting an Authentication Token.
Connect to the database using an SSL connection, specifying the IAM user or role as the database user account and the authentication token as the password. For more information, see Connecting to a DB Instance or DB Cluster Using IAM Database Authentication.
That means for every connection you intend to open, you need to get a valid Token using the AWS SDK. This is where using the correct instance profile with the RDS permission is needed. See also the code examples further down the AWS documentation page.
I think however this requires quite a bit of effort on your side, to always get a valid token before opening a connection. It makes using an off-the-shelf connection pool difficult. Probably once open, the connection will remain open even after the token expires, but you still need to handle the case where more connections need to be opened at a later time.
I would stick with a normal user/password access for the application, using IAM for this case seems to be too much effort.
For applications running on Amazon EC2, you can use EC2 instance profile credentials to access the database, so you don't need to use database passwords on your EC2 instance.
You're misinterpreting what this means. It means you don't have to use static passwords or store them on the instance.
The idea is that you generate a new authentication token each time you establish a connection to the database. The token is generated on your instance, using the instance role credentials. It can only be used to authenticate for 15 minutes, but once connected, you don't lose your database connection after 15 minutes. You remain connected.
If your application doesn't reuse database connections, then you likely have a design flaw there.