I am transferring an existing PHP application to the Elastic Beanstalk and have a newbie question. My application has a data folder that grows and changes over time and can grow quite large currently the folder is a subfolder in the root directory of the application. In the traditional development model I just upload the changed PHP files are carry on using the same data folder, how can I do this in the Elastic Beanstalk?
I dont want to have to download and upload the data folder everytime I deploy a new version of the application. What is the best practice to do this in the AWS Beanstalk?
TIA
Peter
This is a question of continious deployment.
Elastic BeanStalk supports CD from AWS CodePipeline: https://aws.amazon.com/getting-started/tutorials/continuous-deployment-pipeline/
To address "grows and changes over time and can grow quite large currently the folder is a subfolder in the root directory of the application", you can use CodeCommit to version your code using Git. If you version the data folder with your application, the deployment will include it.
If the data is something you can offload to an object store (S3) or a database (RDS/DynamoDB/etc), it would be a better practice to do so.
As per the AWS documentation here, Elastic Beanstalk applications run on EC2 instances that have no persistent local storage, as a result your EB application should be as stateless as possible, and should use persistent storage from one of the storage offerings offered by AWS.
A common strategy for persistent storage is to use Amazon EFS, or Elastic File Service. As noted in the documentation for using EFS with Elastic Beanstalk here:
Your application can treat a mounted Amazon EFS volume like local storage, so you don't have to change your application code to scale up to multiple instances.
Your EFS drive is essentially a mounted network drive. Files stored in EFS will be accessible across any instances that have the file system mounted, and will persist beyond instance termination and/or scaling events.
You can learn more about EFS here, and using EFS with Elastic Beanstalk here.
Related
I'm new to AWS and a little perplexed as to the situation with var/www/html folder in an EC2 instance in which Apache has been installed.
After setting up an Elastic Beanstalk service and uploading the files, I see that these are stored in the regular var/www/html folder of the instance.
From reading AWS documents, it seems that instances may be deleted and re-provisioned, which is why use of an S3 bucket, EFS or EBS is recommended.
Why, then, is source code stored in the EC2 instance when using an apache server? Won't these files potentially be deleted with the instance?
If you manually uploaded some files and data to /var/www/html then off course they will be wiped out when AWS is going to replace/delete the instance, e.g. due to autoscalling.
All files that you use on EB should be part of your deployment package, and all files that, e.g. your users upload, should be stored outside of Eb, e.g. on S3.
Even if the instance is terminated for some reason, since the source code is part of your deployment package on Beanstalk it can provision another instance to replace with the exact same application and configurations. Basically you are losing this data, but it doesn't matter.
The data loss concern is for anything you do that is not part of the automated provisioning/deployment, ie any manual configuration changes or any data your application may write to ephemeral storage. This is what you would need a more persistent storage option for.
Seems that, when the app is first deployed, all files are uploaded to an S3 bucket, from where these are copied into the relevant directory of each new instance. In the event new instances have to be created (such as for auto-scaling) or replaced, these instances also pull the code from the S3 bucket. This is also how the app is re-deployed - the bucket is updated and each instance makes a new copy of the code.
Apologies if this is stating the obvious to some people, but I had seen a few similar queries.
I am trying to create an Centralized file based repository where I can upload all the configuration files needed for an application to run which is deployed as a pod inside the Kubernetes. Any suggestion on achieving this functionality ? Can the file based repository version the files uploaded ?
I see that s3fs-fuse can be used to achieve this, but i lack to see that, it wont support versioning the added config files in the S3 bucket.
https://github.com/s3fs-fuse/s3fs-fuse
Any other suggestion ?
You could use elastic file system which is supported by EKS:
Applications running in Kubernetes can use EFS file systems to share data between pods in a scale-out group, or with other applications running within or outside of Kubernetes. EFS can also help Kubernetes applications be highly available because all data written to EFS is written to multiple AWS Availability zones. If a Kubernetes pod is terminated and relaunched, the CSI driver will reconnect the EFS file system, even if the pod is relaunched in a different AWS Availability Zone.
But its not S3 and it does not have versioning of files such as S3 has. You would have to add such functionality yourself, e.g. by keeping everything in a git repository on the EFS file system.
Why not use git?
The following article contains an example which runs a git clone within an initContainer:
https://blog.frankel.ch/versatility-kubernetes-initcontainer/
I am new to AWS infrastructure, I would like to know what storage mechanism should I use which can be accessible by my war (hosted in AWS elasticbeanstalk) and one windows service hosted on one of my AWS machine. I have little knowledge about S3, EBS and EFS.
Use Case:
my webapp in elasticbeanstalk would like to create objects in some storage system.
my executable on one of aws machine produces files that should be accessible by my webapp deployed in elasticbeanstalk
Questions:
Is it possible to share some storage to both my webapp and my executable.
If answer of 1 is yes, What storage mechanism should I use ?
Please advise.
S3 is Simple Storage Service that is reachable through a web interface. I reckon this is what you are looking for. It is reachable through an URL and is used for storing objects such as files, images, and so on.
EBS is a virtual hard disk that is connected to an EC2 instance (virtual machine).
EFS is Elastic File System used for Linux.
https://aws.amazon.com/s3/
If full hierarchical file system is not needed and you need to store plain object which seems to be the case according to use case given by you then Amazon s3 is the way to go.
Amazon EBS is the block storage service that can only be attached to one machine it resembles the physical hardisk attached to your home computer for more information read this.
EBS
Amazon EFS stands for elastic file system its also a block storage service, its different from EBS in terms that it can be shared accross multiple machines, it resembles the NAS in a datacenter. for more information on EFS read this.
EFS
I have a couple of windows server instances running on Amazon EC2 and would like to make them a bit more fault tolerant by running a duplicate instance with load balancers.
The problem is the specific data, as an example it does no good to fail over from one web server to another web server if the contents of the document root i.e. C:/htdocs/ (Apache) or C:/Repositories (VisualSvn Server) are not identical.
Is there a way to share a volume across two or more instances?
My idea is share folder between EC2 istances:
I read it's not possible to attach the same EBS volume to multiple instances. I believe also AWS is not NFS friendly either in case I want to mount them across NFS.
And finally, I've also checked S3 bucket mounted with s3fs but I found out it's not a good option too.
Can anyone help point me in the right direction?
You are right, at the moment it is not possible to add an EBS volume to multiple instances. To create a common storage for all instances, there are options like NFS, mounting S3 buckets or using a distributed cluster filesystem like GlusterFS.
However in most cases you can simplify your setup. Try to offload static assets to another (static) domain or even host it on an website-enabled S3 bucket. This way you only have to care about the dynamic application logic or scripts on your app servers.
Also try to use some automated deployment and/or configuration management tools. With these you can for example create new machines easily, or you can use them to deploy the latest code on your machines.
Recently My Website shifted on Amazon.
I create EC2 Instance. Install lamp and Setup CodeIgnitor in /var/www/http folder.
the structure
Codeignitor folder I have folder name 'UPLOAD'.this folder is used for uploaded images and files.
I make AMI image from EC2 Instance.
I have setup Auto scaling of Ec2 instances.
When my old ec2 instance is failed then automatically new instance is created. But My all data from "UPLOAD" of folder on old ec2 instance has lost.
I want to separate "UPLOAD" folder in codeignitor from ec2 instance.
So whenever new instance is create it will get UPLOAD folder and its contents without loss.
I want to separate this upload folder. so when new instance is create then it will get this data.
how to do this.
Thanks for Advance.
Note . I have used MYSQL on Amazon RDS.
You can use a shared Elastic Block Storage mounted directory.
If you manually configure your stack using the AWS Console, go to the EC2 Service in the console, then go to Elastic Block Storage -> Volumes -> Create Volume. And in your launch configuration you can bind to this storage device.
If you are using the command line tool as-create-launch-config to create your launch config, you need the argument
--block-device-mapping "key1=value1,key2=value2..."
If you are using Cloudformation to provision your stack, refer to this template for guidance.
This assumes Codeignitor can be configured to state where its UPLOAD directory is.
As said by Mike, you can use EBS, but you can also use Amazon Simple Storage Service (S3) to store your images.
This way, whenever an instance starts, it can access all the previously uploaded images from S3. Of course, this means that you must change your code for the upload, to use the AWS API and not the filesystem to store your images to S3.