Is there a way how to switch from AWS managed key to Customer managed key for already existing EFS?
The EFS was created by the key provided by AWS (aws/elasticfilesystem), but because of Security audit we have to use CMK.
Unfortunately, you can not change the key for an existing EFS. Disabling or deleting the AWS managed Key will lead to the lost of your filesystem.
But you have several options to workaround this. The first one I see, is to create a new EFS with a CMK, mount it on a host which has also the old EFS and backup all your file using rsync or a similar tool. Then switch when synchronisation is finish. I don't know how much data you have this can take a while and cost money.
I also found a similar procedure that use datapipeline that seems to do the same things but all package by AWS.
To be honest, I never use this tool. You can find information here https://docs.aws.amazon.com/efs/latest/ug/alternative-efs-backup.html
The second option, is to use AWS backup. Create an "on demand backup" of your EFS. When backup is done, create a restore job using a new filesystem which will use your CMK. What I don't like with this method, is that AWS backup will create a directory inside the root filesystem. I thinks this is kind of dirty.
root#ip-172-31-16-39:/data1# df -h .
Filesystem Size Used Avail Use% Mounted on
fs-fc09d4c8.efs.eu-west-1.amazonaws.com:/ 8.0E 0 8.0E 0% /data1
root#ip-172-31-16-39:/data1#
root#ip-172-31-16-39:/data1# ls -l
total 4
drwxr-xr-x 3 root root 6144 May 14 17:55 aws-backup-restore_2021-05-14T19-03-08-145Z
[1]. https://docs.aws.amazon.com/efs/latest/ug/troubleshooting-efs-encryption.html
Related
I am backing up my EFS using AWS Backup service. Wanted to know where the Backup is actually stored? Is this getting stored in the EFS itself or not? I noticed a huge hike in the data stored in the EFS so wanted to know if the automated backup can be causing this hike or not, I tried deleting few backup recovery points from the backup vault but is not making much of a difference.
Thanks in advance for your help.
Your backups are not stored on your EFS itself. They are stored in Recovery points in a backup vault. Term Recovery point is interchangeable with backup according to aws docs.
you need to check you application etc which stores data on efs or check efs itself what has been added which has caused this hike.
EFS Automatic backups and Regular EFS Backups are stored in different Backup Vaults.
The EFS Automatic backup vaults are stored in aws/efs/automatic-backup-vault vault.
By default you won't be able to delete the EFS Automatic Backup Recovery Points from aws/efs/automatic-backup-vault vault since the Access policy is set to deny deletions.
If you want to disable EFS Automatic Backups and delete those recovery points you can follow this document:
https://aws.amazon.com/premiumsupport/knowledge-center/efs-disable-automatic-backups/
I have an EC2 instance running in us-east-1 that needs to be able to access/manipulate data available in the KITTI Vision Benchmark public dataset. I'd like to make this data available to the instance, but would also like to be able to reuse it with other instances in the future (more like a mounted S3 approach).
Other similar questions get at downloading from the bucket directly... and I understand that I can view the bucket and recursively download the data to a local folder using AWS cli from within the instance:
aws s3 ls --no-sign-request s3://avg-kitti/
aws s3 sync s3://avg-kitti/ or aws s3 cp s3://avg-kitti/ . --recursive
However, this feels like a brute force approach and would likely require me to increase my EBS volume size... and would limit my reuse of this data elsewhere (unless I was to snapshot and reuse the entire EBS volume). I did find some stackoverflow solutions that mentioned some of the open data sets being available as a snapshot you could copy over and attach as a volume. But the KITTI Vision Benchmark public dataset appears to be on S3 so I don't think it would have a snapshot like it would on EBS datasets...
That being said, is there an easier way to copy public data over to an existing S3 bucket? and then mount my instance to that? I have played around with S3FS and feel like that might be my best bet, but I am worried about 1) the cost of copying / downloading all data from public bucket to my own 2) best approach for reusing this data on other instances 3) simply not knowing if there's a better/cheaper way to make this data available without downloading or needing to download again in the future.
I have mounted a Filestore instance in my Kubernetes cluster. Can I upload some files from my local computer to the Filestore instance? It would be like uploading files to google drive, but in Filestore.
Connect to Filestore outside the VPC network is not possible.
But it is possible to upload/dowload files from a machine outside the VPC network to Google Storage.
So the solution is upload the files to Google storage and then copy them inside the cluster using a pod that executes gsutil rsync gs://bucket mount-directory.
The pod can be replaced by a VM machine connected to the VPC network.
Because the files you would like to upload are configuration and credentials files for your app, using Filestore doesn't look like the best choice. Filestore could be justified if you need some data mounted as ReadWriteMany, because minimum volume size is 1Tb, and it's too much for configs and credentials.
Instead, a better way to achieve your goal would be using ConfigMaps to store configuration files and Secrets to store credentials. This should be also a cheaper option:
You are charged based on the provisioned capacity, not based on the
capacity used. For example, if you create a 1 TiB instance and store
100 GiB of data on it, you incur charges for the entire 1 TiB.
What is the best way to move a system partition, ie. /var, /opt to a larger volume that is attached during the EC2 instance creation? For example, I want to create a generic AMI but if I use this AMI for MySQL server then I need to attach a larger volume for /var. Is there a way during the creation process to tell it to provision the instance with /var on the additional volume?
The concern you mentioned doesn't actually exist in real life. You don't need to specify the volume or size of /var or /opt directories. Linux directory structure is mounted to the root storage device. Any sub-directories(e.g. /var, /etc) can expand unless there is available free space in root storage. You don't need extra parrtition for that.
In your case the solution is: while launching new instance from your generic AMI in Step 4: Add Storage choose your desired storage size!
We need an option to mount /var/www/html/ folder on multiple servers to EFS or some other services for setting up autoscaling environment.
Firstly we have used EFS and faced an issue such that, at high throughput, and if the file size is not increasing, then it will burst.
So could you please suggest an alternative for high throughput and the file size not increasing rapidly.
I have tried using EFS with auto-scaling and I have faced the same issue as yours then I have done the below solution.
I have mounted a drive at any mount point at "/home/code" which is using XFS filesystem and deploy the code in this particular directory. I am taking up snapshot after every deployment using script (Link).
I am using the ec2 bootstrap script to create a volume from this snapshot with high IOPS which automatically attaches to the new instance which are coming in.
Alternatively, you can tar your code at every time you deploy and copy the code to the new servers which are coming in through bootstrap script itself.