I have an application deployed on EB that need to download a file from a remote server then serve to visitors
As I understand, Its recommended to save files to S3 instead then grant users access to these files. However, I believe there is no option for S3 to initiate the download of a file on a remote server therefore the process would be :
EB application get the files => EB application upload the files to S3.
That would double the wait time for users.
Should I save files directly to the application directory instead as I will only use 200-300MB max then clean it daily.
Is there any risk or better approach to this problem?
Why would it double the time? The upload to S3 would be extremely quick. You could even stream the file to S3 as it is being downloaded.
Saving the files to the server will prevent you from scaling your application beyond a single server.
Related
I have a Nextflow pipeline executed in AWS Batch. Recently, I tried to add a process that uploads files from local machine to S3 bucket so I don't have to upload files manually before each run. I wrote a python script that handles the upload and I wrapped it into a Nextflow process. Since I am uploading from a local machine, I want the upload process with
executor 'local'
This requires a Fusion filesystem enabled in order to have a Work Dir in S3. But when I enable the Fusion filesystem I don't have access to my local filesystem. In my understanding, when Fusion filesystem is enabled, the task runs in Wave container without access to host filesystem. Does anyone have experience with running Nextflow with FusionFS enabled and how to access host filesystem? Thanks!
I don't think you need to manage a hybrid workload here. Pipeline inputs can be stored either locally or in an S3 bucket. If your files are stored locally and you specify a working directory in S3, Nextflow will already try to upload them into the staging area for you. For example, if you specify your working directory in S3 using -work-dir 's3://mybucket/work', Nextflow will try to stage the input files under s3://mybucket/work/stage-<session-uuid>. Once the files are in the staging area, Nextflow can then begin to submit jobs that require them.
Note that a Fusion file system is not strictly required to have your working directory in S3. Nextflow includes support for S3. Either include your AWS access and secret keys in your pipeline configuration or use an IAM role to allow your EC2 instances full access to S3 storage.
I'm currently running an icecast server for streaming audio on an EC2 Instance.
Currently all my .mp3 files are stored on the EC2 instance and I want to move them to AWS S3 for storage. So far I've been able to find scripts that will update the playlist but will not make the server request external sources.
Is it possible to setup this architecture? Any help would be appreciated.
How about mounting the S3 bucket as a directory and just using that?
https://github.com/s3fs-fuse/s3fs-fuse / https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
https://github.com/russross/s3fslite
As you only read the files, this should be without major issues.
I have angular application with nodejs backend(REST API). I am confused with S3 and EC2. which one is better and what are the pros and cons deploying to each.Considering average load. Help will be highly appreciate.
I figured it out by myself.
S3 is used to store static assets like videos, photos, text file and
any other format file. It is highly scalable, reliable, fast, inexpensive
data storage infrastructure.
EC2 is a like your own server. And it is on Cloud so computing
capacity can be decreased or increased instantly as per server need.
So here my confusion is clear as ...
When we build Angular2 Application it generates .js files which called
bundle in term of angular2. Now, these files can be hosted on S3 Bucket. And can be accessed through CloudFront in front of it. Which is very fast cache enabled. And pricing model is pay per request.
But EC2 is like running own server. And we have to configure server it
self so for angular application it is not good. It is good for node
Applicationas it can do computation.
You can setup a popular ubuntu server in EC2 with
Nginx to serve your angular frontend and proxy request for your
NodeJs Api
S3 is a file storage mainly for serving static content and media files (jpg, fonts, mp4 etc)
Theoretically you can host everything in your EC2 instance, but with S3 it is easier to scale, backup, migrate your static asset.
You can probably start with one simple EC2 instance to run everything, when everything's working fine you can try move the static asset to S3
I need to transfer all our files (With folders structure) to AWS S3. I have researched lot about how this done.
Most of the places mentioned s3fs. But looks like this is bit old. And I have tried to install s3fs to my exsisting CentOS 6 web server. But its stuck on $ make command. (Yes there is Makefile.in)
And as per this answer AWS S3 Transfer Acceleration is the next better option. But still I have to write a PHP script (My application is PHP) to transfer all folders and files to S3. It is working same as how file save in S3 (API putObject), but faster. Please correct me if I am wrong.
Is there any other better solution (I prefer FTP) to transfer 1TB files with folders from CentOS 6 server to AWS S3? Is there any way to use FTP client in EC2 to transfer files from outside CentOS 6 to AWS S3?
Use the aws s3 sync command of the AWS Command-Line Interface (CLI).
This will preserve your directory structure and can be restarted in case of disconnection. Each execution will only copy new, changed or missing files.
Be aware that 1TB is a lot of data and can take significant time to copy.
An alternative is to use AWS Snowball, which is a device that AWS can send to you. It can hold 50TB or 80TB of data. Simply copy your data to the device, then ship it back to AWS and they will copy the data to Amazon S3.
I'm doing a project and have to access AWS in order to maintain files and other stuffs.
I've connected to my AWS instance and now want to upload some files from my own computer system. There I can use only terminal.
My question is that how can I upload those files or directories to my AWS instance from my computer or any other method like first I upload those files to any third party server like github and then download or clone it from there?
Please suggest me an easy way of uploading from my own computer or present me with an alternative.
Pushing your code to Github and pulling from Github when you ssh into an AWS instance is a good way to deploy.
https://gist.github.com/oodavid/1809044