We all know about what happened to Cold Spaces getting hacked and their AWS account essentially erased. I'm trying to put together recommendation on set of tools, best practices on archiving my entire production AWS account into a backup only where only I would have access to. The backup account will be purely for DR purposes storing EBS snapshots, AMI's, RDS etc.
Thoughts?
Separating the production account from the backup account for DR purposes is an excellent idea.
Setting up a "cross-account" backup solution can be based on the EBS snapshot sharing feature that is currently not available for RDS.
If you want to implement such a solution, please consider the following:
Will the snapshots be stored in both the source and DR accounts? If they are, it will cost you twice.
How do you protect the credentials of the DR account? You should make sure the credentials used to copy snapshots across accounts are not permitted to delete snapshots.
Consider the way older snapshots get deleted at some point. You may want to deal with snapshot deletion separately using different credentials.
Make sure your snapshots can be easily recovered back from the DR account to the original account
Think of ways to automate this cross-account process and make it simple and error free
The company I work for recently released a product called “Cloud Protection Manager (CPM) v1.8.0” in the AWS Marketplace which supports cross-account backup and recovery in AWS and a process where a special account is used for DR only.
I think you would be able to setup A VPC and then use VPC peering to see the other account and access S3 in that account.
To prevent something like coldspaces, make sure you use MFA authentication (no excuse for not using it, the google authentication app for your phone is free and safer than just having a single password as protection.
Also dont use the account owner but setup a separate IAM role with just the permissions you need (and enable MFA on this account as well).
Only issues is that VPC peering doesnt work across regions which would be nicer than having the DR in a different AZ in the same region.
Related
We are running AWS RDS PostgreSQL, with daily automatic snapshots, encrypted by AWS managed KMS key. My objective is to minimize risks and data loss, in case when main AWS account (running RDS) got compromised or RDS deleted/damaged in some way.
What we've implemented so far: RDS snapshots are shared with different (backup) account, periodically copied to backup account and re-encrypted with the KMS key from the backup account, to make copies local, and independent from the main AWS account.
I'm wondering if there are better ways to minimize recovery time objective and recovery point objective in case of a disaster event?
This AWS blog post seems to weigh the options well.
Automated backups are limited to a single AWS Region while manual snapshots and Read Replicas are supported across multiple Regions.
Having cross region Read replica would give you the best RPO and RTO as you can promote replica to be an independent instance which should improve your RPO / RTO
Alternatively, if you choose to use Amazon Aurora Backtrack it seems to offer a similar option to having a read replica but I do not have a personal experience with this feature so can't say how effective it is in improving RTO and RPO.
I wrote two scripts implementing flow at the diagram drawn above ^^^, the idea is to run them daily:
src_acc_take_share_rds_snapshot.py in src account:
list available RDS snapshots according to provided regexp
recrypt them with KMS key, shared from dst account
share recrypted RDS snapshots with the dst account
remove old decrypted snapshots
dst_acc_copy_shared_rds_snapshot_to_local.py in dst account
list RDS snapshots, shared in src account with dst account
copy RDS snapshots from src account to dst account
remove old decrypted snapshots
fire an SNS message if desired snapshot count != actual
and put them at GitHub https://github.com/mvasilenko/dr-rds-share-snapshot
I am using a shared AWS account (Everyone in the account has root access) to deploy servers on an EC2 instance that I created. Is there a way to prevent anyone else that have access to the same AWS account to access the content that I put on the EC2 instance?
As far as I know, creating a separate key pair won't work because someone else can snapshot the instance and launch it with another key pair that they own.
I am using a shared AWS account (Everyone in the account has root
access) to deploy servers on an EC2 instance that I created. Is there
a way to prevent anyone else that have access to the same AWS account
to access the content that I put on the EC2 instance?
No, you cannot achieve your objective.
When it comes to access security, you either have 100% security or none. If other users have root access, you cannot prevent them from doing what they want to your instance. They can delete, modify, clone, and access. There are methods to make this more difficult, but anyone with solid experience will bypass these methods.
My recommendation is to create a separate account. This is not always possible, as in your case, but is a standard best practice (separation of access/responsibility). This would isolate your instance from others.
There are third-party tools that support the encryption of data. You will not be able to store the keys/passphrase on the instance. You will need to enter the keys/passphrase each time you encrypt/decrypt your data.
As far as I know, creating a separate key pair won't work because
someone else can snapshot the instance and launch it with another key
pair that they own.
With root access, there are many ways to access the data stored on your instance's disk. Clone the disk and just mount it on another instance is one example.
By default, IAM Users do not have access to any AWS services. They can't launch any Amazon EC2 instances, access Amazon S3 data or snapshot an instance.
However, for them to do their work, it is necessary to assign permissions to IAM Users. It is generally recommended not to grant Admin access to everyone. Rather, people should be assigned sufficient permissions to do their assigned job.
Some companies separate Dev/Test/Prod resources, giving lots of people permission in Dev environments, but locking-down access to Production. This is done to ensure continuity, recoverability and privacy.
Your requirement is to prevent people from accessing information on a specific Amazon EC2 instance. This can be done by using a keypair that only you know. Thus, nobody can login to the instance.
However, as you point out, there can be ways around this such as copying the disk (EBS Snapshot) and mounting it on another computer, thereby gaining access to the data. This is analogous to security in a traditional data center — if somebody has physical access to a computer, they can extract the disk, attach it to another computer and access the data. This is why traditional data centers have significant physical security to prevent unauthorized access. The AWS equivalent to this physical security are IAM permissions that grant specific users permission to perform certain actions (such as creating a disk snapshot).
If there are people who have Admin/root access on the AWS account, then they can do whatever they wish. This is by design. If you do not wish people to have such access, then do not assign them these permissions.
Sometimes companies face a trade-off: They want Admins to be able to do anything necessary to provide IT services, but they also want to protect sensitive data. An example of this is an HR system that contains sensitive information that they don't want accessible to general staff. A typical way this is handled is to put the HR system in a separate AWS Account that does not provide general access to IT staff, and probably has additional safeguards such as MFA and additional audit logging.
Bottom line: If people have physical access or Admin-like permissions, they can do whatever they like. You should either restrict the granting of permissions, or use a separate AWS Account.
I have an AWS account where multiple EC2 instances, load balancers, target groups, security groups etc are setup by multiple owners.
We use terraform to set this up but sometimes due to corruption, the state becomes inconsistent. Current mechanism to recover is to manually destroy all resources in that account owned by a particular owner.
Is there an easy way to nuke all resources in an AWS account belonging to a particular owner?
There is no way to delete all resources in an account owned by a particular user but there is a way to delete all resources in an account.
You can use aws-nuke which was created somewhat out of the same use case you described.
At first, you need to set an account alias for your account.
You must create a config file.
Then you can list down all resources that will be deleted using the following command:
aws-nuke -c config/nuke-config.yml --profile aws-nuke-example
Add --no-dry-run option to permanently delete all resources in the same command.
There are also multiple filter options available such as target, resource type, exclude, etc. that you can leverage to suit your needs.
Agree with the other answer that there is no easy way delete orphan resources.
But I see the original issue is that the terraform state is corrupted.
You can checkout the terraform import feature which lets you generate state file from aws resources. In that way you can connect your config to resources again.
Short answer: no.
Longer answer: actually, that's also no. There's no built-in capabillity for this.
The case you're describing is not within the bounds of typical AWS usage... destroying everything in an account -- usually -- should not be easy.
Of course, you could script it, fairly trivially, by wrapping calls to aws-cli to custom code to iterate through the resources and generate additional requests to destroy them... but if you do, lock that code away, since such capability is inherently dangerous.
You can delete all your resources you created, you'll need to automate, see a sample here:
Creation
https://github.com/jouellnyc/AWS/tree/master/create_aws_vpc2
Deletion
https://github.com/jouellnyc/AWS/blob/master/create_aws_vpc2/delete_lb_and_vpc.sh
Other
I've had some success with cloud nuke (played around for a few min; not in depth):
https://github.com/gruntwork-io/cloud-nuke
I dont think there is any state forward way to do it but to check if you have any active resources in your account, do the following:
Open the Billing and Cost Management console.
Choose Bills in the navigation pane.
You can see the charges incurred by different services in the Bill details by service section.
You can see the charges incurred in different AWS Regions in the Bill details by account section.
For each service, identify the Regions where the services have incurred charges.
To terminate the identified active resources under different services, do the following:
Open the AWS Management Console.
For Find services, enter the service name.
After opening the service console, terminate all your active resources. Be sure to check each Region where you have allocated resources.
if the issue is a corrupted terraform state, perhaps storing the state in a versioned S3 bucket would help reduce the impact of that.
Use Terraformer to import all resources into terraform configuration then do whatever you want:
terraformer import aws --resources="*"
https://github.com/GoogleCloudPlatform/terraformer
Take care of your state file lock f.e. by using dynamodb & enable s3 versioning.
Is there an easy way to nuke all resources in an AWS account
belonging to a particular owner?
Since you are using Terraform, you can use Blank Apply.
It will destroy all the Resources in a state file.
We use terraform to set this up but sometimes due to corruption, the
state becomes inconsistent.
It's better to use version control system to avoid drifts and inconsistencies in your state file and use remote states in order to make sure everyone is on the same page.
I was thinking what if My AWS Account get deleted/inaccessible one fine day? (may sound weird). Have anyone implemented any solution for this? Can we have back from one AWS account to another AWS account?
There are several things you can do, one is to make sure you have at least two administrator accounts, one that you use, and one that you store away in a safe place and only use for emergencies.
The second is to setup a completely seperate AWS account as a 'backup', with its own set of credentials. You can grant cross-account access from your primary account, to your backup account but only allow the primary account to 'put' or backup objects to the backup account, so that even if your primary account is compromised, the attacker can't do harm to the second account from the primary account.
The actual process to backup your services on one account to another is going to vary depending on which services you are using, but the concept is the same - backup the data to s3 and then copy the data from s3 in your primary account to s3 in the backup account - and make sure the primary account only has enough access to the second account to 'put' things, not delete. Nobody in your company should have access to both of those sets of credentials (assuming your company is not tiny).
You don't want to be this company that was put out of business when there account was compromised:
https://threatpost.com/hacker-puts-hosting-service-code-spaces-out-of-business/106761/
Also, this video, from AWS Reinvent 2015 (starting around 50 minutes in), listen to how AirBNB protects against these issues in just this way:
https://www.youtube.com/watch?v=eHg8LD5KNC0
Finally After lot of reasearch and help from Skeddly, I found a solution for Linux machines.
Via Skeddly we can copy EBS snapshots to another AWS account hassle free.It works brilliant.Now even if my Primary AWS account is compromised, I have all EBS snapshots to start with by BCP AWS account for linux machines ;)
Now am hunting for windows machine solution for which I already got an idea... ;)
What the title says.
Within the master AWS account, I have several personal accounts, i.e. AWS Identity and Access Management (IAM) users. I would like to assign certain IAM users to groups and prevent them from terminating certain Amazon EC2 instances, de-registering certain Amazon Machine Images (AMIs), etc.
I don't mind if they're playing with their own stuff, but I don't want them to touch my stuff.
Is that possible?
Update
AWS has just announced Resource-Level Permissions for Amazon EC2 and Amazon RDS to address this long standing shortcoming of IAM support within EC2 and RDS (in comparison to other AWS services, see my original answer below for details/background):
Today we are making IAM even more powerful with the introduction of resource-level permissions for Amazon EC2 and Amazon RDS. [...]
On the EC2 side, you can now construct and use IAM policies to control
access to EC2 instances, EBS volumes, images, and Elastic IP
addresses. [...]
Here are just a few of things that you can do:
Allow users to act on a limited set of resources within a larger, multi-user EC2 environment.
Set different permissions for "development" and "test" resources.
Control which users can terminate which instances.
Require additional security measures, such as MFA authentication, when acting on certain resources.
This solves a plethora of security complications and enables quite a few new use cases as well.
Furthermore, EC2 policy statements can include reference to tags on EC2 resources, which allows to use the same tagging model and schema for permissions and for billing reports. Finally, there's an expanded set of condition tags [...] including ec2:Region, ec2:Owner, and ec2:InstanceType, see Condition Keys for Amazon EC2 for details.
Solution
Here's a variation of Example 3: Allow users to stop and start only particular instances for the use case at hand, which allows users to start and stop [and terminate] only the instances that have the tag "department=dev":
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"ec2:StopInstances",
"ec2:StartInstances",
"ec2:TeminateInstances"
],
"Resource": "arn:aws:ec2:us-east-1:123456789012:instance/*",
"Condition": {
"StringEquals": {
"ec2:ResourceTag/department": "dev"
}
}
}
]
}
Caveat
Support for resource-level permissions is restricted to the following set of actions on the indicated resources still, which excludes parts of the use case (e.g. de-registering AMIs) - the confidence in the foundation for this complex and far-reaching feature is apparently high enough though to announce that they plan to add support for additional APIs throughout the rest of 2013 (AWS doesn't usually publish any roadmaps):
Instances - Reboot, Start, Stop, Terminate.
EBS Volumes - Attach, Delete, Detach.
Original Answer
I'm afraid this isn't possible the way you'd like to do it (and many others for that matter, including myself).
Problem
You want to restrict access to a particular service's resources rather than its actions - while AWS Identity and Access Management (IAM) supports both in principle, not every AWS product/service offers restrictions based on resources yet; unfortunately Amazon EC2 is one of these and even featured as an example for this very difference, see Integrating with Other AWS Products:
The following table summarizes whether you can grant IAM permissions
that control access to a service's actions, resources, or both. For
example, you can use IAM to control which Amazon EC2 actions users
have access to, but you can't use IAM to control users' access to
AMIs, volumes, instances, etc. [emphasis mine]
(Partial) Workaround
Depending on the needs of the other accounts, you might still be able to at least limit their ability to perform those actions considered destructive - you can explore the available actions via the AWS Policy Generator, for example:
ec2:DeregisterImage - obvious effect, when denied for a user/group
ec2:ModifyInstanceAttribute - this would help via Enabling Termination Protection for an Instance, when denied for a user/group:
By default, you can terminate any instances you launch. If you want to
prevent accidental termination of the instance, you can enable
termination protection for the instance.
That is, once you've enabled termination protection, anyone without permission to use ec2:ModifyInstanceAttribute cannot terminate these instances at all.
Obviously the respectively restricted accounts won't be able to facilitate those calls for their own resources anymore.
Furthermore this won't prevent them from running a fancy Cluster Compute Eight Extra Large Instance or so, incurring respective costs in turn ;)
Alternative Approach
Depending on your setup/environment you might want to look into Consolidated Billing instead, which essentially provides a way to gather one or many AWS accounts under another one, which is paying for the resources used by the others.
While this is primarily an accounting feature, it can be used to separate areas of concern as well - for example, it's quite common to facilitate separate development and production accounts to achieve respectively independent operation, not the least regarding IAM rights and the like.
The introductory blog post New AWS Feature: Consolidated Billing provides a good overview, and here is the relevant topic from the AWS Consolidated Billing Guide regarding your apparent use case:
The paying account is billed for all costs of the linked accounts.
However, each linked account is completely independent in every other
way (signing up for services, accessing resources, using AWS Premium
Support, etc.). The paying account owner cannot access data belonging
to the linked account owners (e.g., their files in Amazon S3). Each
account owner uses their own AWS credentials to access their resources
(e.g., their own AWS Secret Access Key). [emphasis mine]
Obviously this functionality is targeted at larger customers, but depending on your scenario you might be able to come up with a solution to separate your AWS accounts and resources as needed still.